Caching and tags when using ZF + memcached

Foreword

In the development process using the Zend Framework + Memcached bundle, we sometimes have to deal with the (excessive) abundance of the existing functionality of the framework, and with certain limitations. I will try to tell about one of such cases and the found solution in this article.

Description of the problem

As you know, Memcached is a relatively simple to use Key / Value storage with simple, necessary and sufficient functionality. The ZF interfaces provided for interacting with Memcached are included in the general cache library (also includes adapters for Sqlite, Xcache, ZendServer, etc.). Some of these caching systems support the use of tags for caching objects, however Memcached does not have such a function, therefore attempts to use standard ZF class interfaces for caching objects with tags when working with Memcached will only lead to errors (in logs) up to exceptions. (You can read more in the documentation ).
')


One of the tasks in the development was the "smart" use of the cache in the following sense:

• if necessary, all objects and lists of objects for any model can be put in the cache;
• if the object is changed (edited in the admin):
  
  1. it should be removed from the cache, no matter how it was found (i.e., whatever the search parameters of this object would be, even by the primary key, at least by any other set of parameters);
  2. All lists that have this object (or all lists for this model (table)) should be removed from the cache.
  
  
• such functionality should be transparent for controllers, final models, and, in part, for mappers (except for possible indication of using or not using the cache when retrieving data);
• The cache interface should not change.

Thus, it turns out that by saving the results of a database query in the cache, you need to add a certain tag for it, by which you can determine which object / list / model the value stored in the cache belongs to. Yes, and store this tag is also somehow required.

Some ideas for solving the problem in due time came from an article on the Habré about clever deletion from the cache using the memcached + MongoDB bundle . However, there were no possibilities to add to the MongoDB server. Now and in the near future, other caching systems (Redis etc.) on the server are not provided.

Solution and Code

The basic idea is simple: storing all caching keys for a specific tag in an array — in the cache itself, using the tag as the caching key for that array. In this case, the tag itself is built automatically based on the type of request and the caching key itself.
Without spreading further along the tree, we will immediately consider the first part of the base mapper code, from which all mappers used in the project are inherited (except for the solution code for the task itself, the example will have a certain amount and accompanying code that is necessary for the system to work and is partially related to the task) . Immediately I apologize if the code seems a lot, but inside it left enough comments.

class System_ModelMapper { protected $_dbTable; //         protected $_modelName = ''; //  ,       -      public function __construct() { parent::__construct(); if (empty($this->_modelName)) { $parts = explode('_', get_class($this)); $this->_modelName = str_replace('Mapper', '', $parts[2]); } } //      dbtable    /** * @return /Zend_Db_Table_Abstract */ public function getDbTable() { if (null === $this->_dbTable) { $this->setDbTable('Application_Model_DbTable_' . $this->_modelName); } return $this->_dbTable; } public function setDbTable($dbTable) { if (is_string($dbTable)) { $dbTable = new $dbTable(); } if (!$dbTable instanceof Zend_Db_Table_Abstract) { throw new Exception('Invalid table data gateway provided'); } $this->_dbTable = $dbTable; return $this; } /** *       * @param array $params * @return System_Model */ public function getModel($params = array()) { $getInstance = 'Application_Model_' . $this->_modelName; return new $getInstance($params); } … } 

Then there are several methods for searching for a single object:

  /** *             * @param array $data   * @return string */ protected function objectCacheId($data) { $fields = array_keys($data); $values = md5(json_encode(array_values($data))); return 'find_' . $this->_modelName . '_' . join('_', $fields) . '_' . $values; } /** *           * @param $object System_Model * @return string */ public function getObjectCacheTag($object) { return 'object_' . $this->_modelName . '_' .$object->get_id(); } /** *       * @param numeric $id  ID * @param mixed $obj ,      * @param bool $cache         * @return bool|System_Model */ public function find($id, $obj = false, $cache = false) { return $this->findByFields(array('id' => $id), $obj, $cache); } /** *      * @param array $data     * @param mixed $obj ,      * @param bool $cache         * @return bool|System_Model */ public function findByFields($data, $obj = false, $cache = false) { //     -    ,        (        ,         (   :) ) if ($cache) { $cacheId = $this->objectCacheId($data); if (Zend_Registry::isRegistered(CACHE_NAME) { /** @var $cache System_Cache_Core */ $cache =& Zend_Registry::get(CACHE_NAME); //      -    if ($cache->test($cacheId)) { return $cache->load($cacheId); } } else { $cache = false; } } //   Zend_Db_Table         $select = $this->getDbTable()->select(); foreach ($data as $field => $value) { $select->where($select->getAdapter()->quoteIdentifier($field) . ' = ?', $value); } $row = $this->getDbTable()->fetchRow($select); if ($row) { if ($obj === false) { $obj = $this->getModel(); } $obj->setOptions($row->toArray()); } else { $obj = false; } //    -     if ($cache) { $cache->save($obj, $cacheId); } return $obj; } 

By the above code - as you can see, the object search methods have no idea about the tag system. It will work at the $ cache object level.
Further, there are several methods for searching a set of objects or even all objects in a table, taking into account pagination and sorting:

  /** *          ,    * @param array $data   * @param bool|string|array $order   * @param bool|System_Paginator $paginator     * @return string */ protected function listCacheId($data = array(), $order = false, $paginator = false) { $fields = array_keys($data); $values = md5(json_encode(array_values($data))); return sprintf('%s_%s_%s_%s_%s', $this->getListCacheTag(), join('_', $fields), $values, empty($order) ? '' : md5(json_encode($order)), is_object($paginator) ? $paginator->page . '_' . $paginator->limit : '' ); } /** *         * @return string */ public function getListCacheTag() { return 'list_' . $this->_modelName; } /** *        * @param array $data   * @param bool|string|array $order   * @param bool|System_Paginator $paginator     * @param bool|string $cache         */ public function fetchByFields($data = array(), $order = false, $paginator = false, $cache = false) { if ($cache) { $cacheId = $this->listCacheId($data, $order, $paginator); $cache .= 'Cache'; if (Zend_Registry::isRegistered(CACHE_NAME)) { /** @var $cache System_Cache_Core */ $cache =& Zend_Registry::get(CACHE_NAME); if ($cache->test($cacheId)) { return $cache->load($cacheId); } } else { $cache = false; } } //   ,           ,    $select = $this->getDbTable()->select(); $select_paginator = $this->getDbTable()->select(true); foreach ($data as $field => $value) { $s = '='; // value      ('=', 2)  ('<=', 10) if (is_array($value)) { $s = $value[0]; $value = $value[1]; } $select->where($select->getAdapter()->quoteIdentifier($field) . " $s ?", $value); $select_paginator->where($select->getAdapter()->quoteIdentifier($field) . " $s ?", $value); } //    if (!empty($order)) { $select->order($order); } else { $select->order('id ASC'); } //   ,     if (is_object($paginator)) { //        ,    $fetch_count = $this->getDbTable()->fetchRow($select_paginator->columns('count(id) as _c'))->toArray(); $paginator->total = $fetch_count['_c']; //    ,     ,     if ($paginator->page > $paginator->getLastPage()) $paginator->page = $paginator->getLastPage(); //       $select->limitPage($paginator->page, $paginator->limit); } $resultSet = $this->getDbTable()->fetchAll($select); $result = $this->rowsToObj($resultSet); //      -         (        limit) if (is_object($paginator)) { $paginator->inlist = count($result); } if ($cache) { $cache->save($result, $cacheId); } return $result; } /** *         * @param bool|string|array $order   * @param bool|System_Paginator $paginator     * @param bool|string $cache         * @return array|bool */ public function fetchAll($order = false, $paginator = false, $cache = false) { return $this->fetchByFields(array(), $order, $paginator, $cache); } /** *         * @param Zend_Db_Table_Rowset_Abstract $rowset     * @return array|bool */ protected function rowsToObj($rowset) { if (!empty($rowset)) { $entries = array(); foreach ($rowset as $row) { /** @var $entry System_Model */ $entry = $this->getModel($row->toArray()); $entries[$entry->get_id()] = $entry; } return $entries; } return false; } 

By the code given here, it is also clear that the sampling methods themselves are not directly related to the tag system.

Below is the class code that inherits from the standard Zend_Cache_Core and is used when bootstrap objects are initialized to work with Memcached. And after that we will return again to the mapper and methods for saving, updating and deleting objects in the database.

 class System_Cache_Core extends Zend_Cache_Core { /** *    save   */ public function save($data, $id = null, $tags = array(), $specificLifetime = false, $priority = 8) { //      $ida = explode('_', $id); //        ,          switch ($ida[0]) { case 'list': //       2   (    ) $tag = join('_', array_splice($ida, 0, 2)); $this->updateTagList($tag, $id); break; case 'find': //   -       if ($data instanceof System_Model) { $tag = $data->get_mapper()->getObjectCacheTag($data); $this->updateTagList($tag, $id); } break; } //           return parent::save($data, $id, $tags, $specificLifetime, $priority); } /** *        * @param string $tag * @param string $cacheId */ public function updateTagList($tag, $cacheId) { //       $list = $this->getListByTag($tag); $list[] = $cacheId; //         $this->saveListByTag($tag, $list); } /** *      * @param string $tag */ protected function getListByTag($tag) { $tagcacheId = '_taglist_' . $tag; $list = array(); if ($this->test($tagcacheId)) { $list = $this->load($tagcacheId); } return $list; } /** *         * @param string $tag * @param array $list */ protected function saveListByTag($tag, $list) { $tagcacheId = '_taglist_' . $tag; $this->save($list, $tagcacheId); } /** *         * @param System_Model $object */ public function removeByObject($object = null) { if ($object instanceof System_Model) { //       $this->removeByTag($object->get_mapper()->getListCacheTag()); //             if ($object->get_id()) { $this->removeByTag($object->get_mapper()->getObjectCacheTag($object)); } } } /** *         * @param string $tag */ public function removeByTag($tag) { //      $list = $this->getListByTag($tag); //      foreach ((array)$list as $cacheId) { $this->remove($cacheId); } //      , ,    $this->saveListByTag($tag, array()); } } 

Well, it remains to mention the methods of the mapper for saving and deleting objects:

  /** *     * @param System_Model $object   * @param boolean $isInsert    * @return array|bool|mixed */ public function save($object, $isInsert = false) { $data = $object->toArray(); $find = array('id = ?' => $object->get_id()); if (null === ($id_value = $object->get_id())) { $isInsert = true; unset($data['id']); } if ($isInsert) { $pk = $this->getDbTable()->insert($data); if ($pk) { $object->set_id($pk); } $this->resetCache(); return $pk; } else { //    -       return $this->getDbTable()->update($data, $find) && $this->resetCache($object); } } /** *     * @param $object System_Model * @return array|bool|mixed */ public function insert($object) { return $this->save($object, true); } /** *     * @param $object System_Model   * @return bool */ public function remove($object) { $primary = $this->getDbTable()->get_primary(); $where = array('id = ?' => $object->get_id()); //   -      return ($this->getDbTable()->delete($where) && $this->resetCache($object)); } /** *         * @param System_Model $object * @param array $cacheIds * @return bool */ public function resetCache($object = null, $cacheIds = array()) { //       if (Zend_Registry::isRegistered(CACHE_NAME)) { /** @var $cache System_Cache_Core */ $cache = Zend_Registry::get(CACHE_NAME); if (!empty($object)) { //       $cache->removeByObject($object); } else { //     $cache->removeByTag($this->getListCacheTag()); } foreach ($cacheIds as $cacheId) { $cache->remove($cacheId); } } return true; } } 

As a result, we got a certain basic tag system for objects and models, which is transparent when used and does not require an explicit indication of the tag name either in the models or in the mappers themselves - the tag names are automatically generated based on the data of the model itself and the objects.

For other problems and details (which are planned to be made and solved):

• put all prefixes ('list', 'object', '_taglist_' etc.) in constants;
• The tag for lists inside System_Cache_Core is defined as $ tag = join ('_', array_splice ($ ida, 0, 2)); which is not very good and does not directly correspond to the definition of the tag in the mapper. Localization of the tag definition for the list in one place (in the mapper method) will allow the end mappers to override this method and add parameters to the tag generation algorithm and the tag itself;
• the tag for an object is built on the basis of the primary key of this object, which can lead to the following problems: if the object is searched taking into account, for example, the parameter is_published = 1 and is not (it is, but not published), then the caching key for the request is understandable reasons will not be included in the tag list for this object, and after we have “published” the object in the admin panel of the project - the cache will not clear the record by this key and the object will not be found again (and will take false from the cache) until TTL end for this cache entry.

One way or another, we will gradually solve all these problems.
However, I want to listen to reviews and experienced habravchan and get a fair share of criticism of the proposed method.

PS Happy New Year!