Geekly Articles each Day
📜 ⬆️ ⬇️

PHP OPCache is to blame?


When I started my career as a developer, I was very surprised to read the phrase attributed to Phil Carlton: “There are only two difficulties in computer science: cache invalidation and naming . ” I was incredulous of this because I did not understand the essence of the phrase. But a little later, I began to understand.


I want to talk about the problem that we encountered not so long ago in our production-infrastructure. Immediately after successful deployment, when updating pages modified by a new release, for a while the new code was not displayed. In fact, this is not uncommon for web applications written in PHP. We faced the same before, and after the transition to a new production-environment, the problem became more noticeable. Therefore, we decided to investigate.


Our Deploy Procedure


Our technology is mostly written in PHP , and also uses the symfony and Zend frameworks. To send the code in production, we use the internal project shark-do , its author is the leader of the Luca team.


Shark-do philosophy


"If you can do this, then you can do it in bash."

The project is a bash script that can identify a task and execute it according to an algorithm. Each project has its own algorithm for managing different stages, for example, deleting unnecessary files, generating configuration files, etc.


For example, more than five times a day, I use the shark-do deploy collaboratori command to launch deployment tasks for the collaboratori project I'm working on. Typically, the deployment consists of the following steps:


  1. The last commit is extracted from the master branch.
  2. Folders are configured, unnecessary files are deleted, release creation begins.
  3. Parameters are set, installation of the linker is launched, resources are downloaded and added.
  4. A release archive is created, then it is moved to the bastion machine and unpacked.
  5. To launch a rollback release using the REST API of our infrastructure, the Ansible procedure is called.
  6. The system switches to the new release, the old releases are cleared and removed from the bastion vehicle.
  7. The new release is marked in New Relic, and in our Slack-channel there is a notification about the end of the deployment task.

Consider the fifth step. Ansible script responds:



Each deployment procedure consists of many necessary operations, but the turning point is a change in the current project folder: this is done using the symlink transfer from the previous release folder to the new one. The current project folder is the root location of the documents for a specific web application.


For example:


 ln -sf /var/www/{APP_NAME}/releases/@YYYYMMDDHHIISS /var/www/{APP_NAME}/current

The option -s used to create a symbolic link, and -f - to force the creation of such a link, if the target object already exists. {APP_NAME} - the name of the project.


We use the standard PHP deployment strategy. The releases of one application are stored on production servers, and the current version is accessed by a symbolic link. This allows you to deploy atomic and secure , without affecting the working traffic.


Finally, behind the carousel (round-robin) policy we have 15 front-end servers (more than two times more than before). Question: what happens after the release switch?


PHP OPCache (?) Is to blame


Some reservations: we will not delve into the flow of PHP scripts, and discuss the main things to make it easier for you to understand my reasoning about the problem. We will also consider only PHP 7.


Sometimes it is useful to remember how PHP code is executed. When you run the script, our source code goes through four phases:



The first phase is controlled by the PHP lexical analyzer . It is responsible for matching the keywords of a language like function , return and static with individual parts, which are usually called tokens . Each token is often complemented by the metadata needed for the next phase.


The second phase is controlled by the PHP parser . He is responsible for analyzing one or more tokens, as well as for comparing them with patterns of language structures. For example, $foo + 5 recognized as a binary "addition" operation, and the variable $foo and the number 5 recognized as operands. The parser recursively builds an abstract syntax tree (AST) . Usually the operation of the lexical analyzer and parser is considered one task.


The third phase is compiling . AST is converted to an ordered sequence of opcode instructions. Each opcode can be considered a low-level operation of the Zend virtual machine . A full list of supported opcodes can be found here .


Finally, the last phase is execution . Zend VM performs every task described in opcodes and generates a result.


The first three phases (lexical analyzer, parser and compiler) are combined into a “pipeline”. And the third phase takes much more time and consumes more resources (memory and processor). To reduce the weight of the compilation phase, in PHP 5.5 they introduced the Zend OPCache extension . It caches the compile-phase output (opcodes) in shared memory (shm, mmap, etc.), so that each PHP script is compiled only once, and different queries can be executed without a compile phase. If in an environment not intended for development, the code rarely changes, then the execution speed of PHP increases at least twice.


The OPCache extension is also responsible for optimizing opcodes, but this is already beyond the scope of the article.


In connection with the foregoing, it is logical to assume that OPCache is to blame for the strange behavior that we encountered in our production-environment. To test this assumption, I made a simple demo environment from the Docker container, PHP 7.0 and Apache 2.4. The full code can be downloaded from here .


To simplify the work, I wrote several scripts:



You can simply clone the GitHub repository, and everything is ready to check, if you already have Docker installed.


 git clone https://github.com/salvatorecordiano/facile-it-realpath_cache cd facile-it-realpath_cache docker pull salvatorecordiano/realpath_cache

To reproduce the problem with the cache, you need to run these commands in parallel in three different command lines:


 # start the container with production configuration ./start.sh production # start switching the current release ./release-switcher.sh # start watching the current web server response ./release-watcher.sh

Result of performance:



Execution with production configuration.


The problem with the cache repeated: after switching the release, we do not see the correct code after executing the HTTP request.


Now turn off OPCache and repeat the test.


 # start the container with production configuration and opcache disabled ./start.sh production-no-opcache # start switching the current release ./release-switcher.sh # start watching the current web server response ./release-watcher.sh


Execution with production-no-opcache configuration.


Surprisingly, the problem remained, so the assumption was wrong: OPCache was not at fault.


realpath_cache: the real culprit


Perhaps, when using the include/require function or PHP autoload, you need to remember about realpath_cache . The real path cache allows you to cache path permissions for files and folders in order to spend less time searching the disk and improve performance. This is very useful when working with many third-party libraries or frameworks like Symfony, Zend and Laravel, because they use a huge number of files.


The caching mechanism appeared in PHP 5.1.0. Today, this feature is not mentioned in official documents, except for the functions realpath_cache_get() , realpath_cache_size() , clearstatcache() and php.ini parameters realpath_cache_size and realpath_cache_ttl . From external sources, I was able to find only the old post written by Julien Paulie in 2014. Poly, a well-known PHP developer, explains how the path resolution mechanism works.


When we access a file, PHP tries to resolve its path using stat() , the Unix system call: it returns the file attributes (permissions, extension, and other metadata) for the inode . In the Unix world, an inode is a data structure used to describe a file system object, such as a file or directory. PHP puts the result of a system call into a data structure called realpath_cache_bucket , with the exception of such things as permissions and owners. So if you try to access the same file a second time, then when searching in a bucket in memory (bucket lookup), we will be saved from another slow system call. If you want to learn more, check out the PHP source code .


The realpath_cache_get function appeared in PHP 5.3.2. It allows you to get an array consisting of the cache of real paths. In each element of the array, the key is the resolved path, and the value is another array with data like key , is_dir , realpath , expires .


Next comes the output print_r(realpath_cache_get()) ; in our test Docker environment:


 Array ( [/var/www/html] => Array ( [key] => 1438560323331296433 [is_dir] => 1 [realpath] => /var/www/html [expires] => 1504549899 ) [/var/www] => Array ( [key] => 1.5408950988325E+19 [is_dir] => 1 [realpath] => /var/www [expires] => 1504549899 ) [/var] => Array ( [key] => 1.6710127960665E+19 [is_dir] => 1 [realpath] => /var [expires] => 1504549899 ) [/var/www/html/release1] => Array ( [key] => 7631224517412515240 [is_dir] => 1 [realpath] => /var/www/html/release1 [expires] => 1504549899 ) [/var/www/current] => Array ( [key] => 1.7062595747834E+19 [is_dir] => 1 [realpath] => /var/www/html/release1 [expires] => 1504549899 ) [/var/www/current/index.php] => Array ( [key] => 6899135167081162414 [is_dir] => 0 [realpath] => /var/www/html/release1/index.php [expires] => 1504549899 ) )

Here:



In the previous example, we had six paths, but all of them are related to the resolution of the path /var/www/current/index.php . PHP created six cache keys to allow only one path. So the path is divided into parts, each of which is alternately resolved. In our case, the “real” path is /var/www/html/release1/index.php , because /var/www/current is a symbolic link to the /var/www/html/release1 .


The post by Julien Pauli also states:


"The cache of this path is tied to the process and does not fit into the common memory."

This means that the cache must expire for each process . If we use PHP-FPM to clean the entire web server, then we will have to wait for the cache to become obsolete for each worker in the pool. This helps to understand what happens during testing using the production-no-opcache . Even if you disable OPCache after receiving a symbolic link, PHP will leisurely notify all processes of obsolescent paths.


In our real production-environment, we had to take into account that we have 15 front-end servers on which many web applications are hosted. On each server there is one PHP-FPM pool, each of which consists of 35 workers and one master process. This explains why “strange behavior” has become more noticeable in the new environment. You can adjust the effect of the present path cache on our web application by using the parameters realpath_cache_size and realpath_cache_ttl : the first determines the size of the bucket that PHP will use. This is an integer, and increasing it is useful for web applications that work with a huge number of files. The second parameter realpath_cache_ttl , as already mentioned, is the duration of caching information about the real path (in seconds).


Now everything is clear, you can re-enable OPCache and disable the cache of this path by adjusting its size and lifetime:


 realpath_cache_size=0k realpath_cache_ttl=-1

Run the test again:


 # start the container with production configuration, opcache enabled and realpath_cache disabled ./start.sh production-no-realpath-cache # start switching the current release ./release-switcher.sh # start watching the current web server response ./release-watcher.sh


Execution with the configuration production-no-realpath-cache.


I want to note that our latest configuration is strongly discouraged from being used in a production-based environment, because PHP is forced to allow each path encountered, which is bad for performance.


Conclusion


I wanted to talk about solving a mysterious cache problem, learning about OPCache and the cache of the current path, as well as about their differences. The script described at the beginning of the article was invented, but, for example, if the request starts with one version of the code, then tries to access other files during execution, and they were updated, moved or deleted in subsequent versions of the code, then real problems may arise. In the worst case, it is necessary to ensure compatibility of two consecutive releases, but under the conditions described this is very difficult to achieve.


It is necessary to implement an atomic deployment strategy (in the strict sense). For example, you can use containers or a new isolated PHP-FPM memory pool for each deployed release. In the latter case, you need to at least double the amount of memory so that you can keep more simultaneously running FPM-pools.


You can also use an apache module called mod_realdoc to support atomic deployments. It was written by Rasmus Lerdorf ( Rasmus Lerdorf ). The trick is implemented in this module: at the beginning of the request, the real path is called via the symbolic link DOCUMENT_ROOT , while the absolute path for the entire request is set as the real document root. Therefore, queries that begin before a symbolic link change will be executed in relation to the previous symbolic target object. The main drawback of the module is that you need to use the Apache Multi-Processing Module (MPM). This prefork implements a non-threaded server using forking based. The server spawns new processes and keeps them to service requests. This is the best MPM to isolate each request, so if there is a problem with one request, other requests will not be affected. But when the server is under heavy load, MPM is more likely to hurt, because it uses one process per request, and as a result, there will not be enough resources for simultaneous requests, they will have to wait until the server process becomes free. The same results as with mod_realdoc can be achieved at the PHP level in the front controller of the application, if you define the main root folder in realpath(__FILE__) .


If before PHP you use nginx , then you are lucky! To avoid updating symbolic links when executing queries, you need to force nginx to resolve symbolic links and assign them to DOCUMENT_ROOT . It is enough to change a few lines of code in the server blocks:


 # default configuration fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param DOCUMENT_ROOT $document_root; # configuration with real path resolution fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; fastcgi_param DOCUMENT_ROOT $realpath_root;

As a result, nginx will resolve symbolic links, hiding them from PHP.


These are just some of the ways to deal with the cache problems of the present path. There is no universal, "correct" way. You will have to find your ideal solution depending on your requirements and infrastructure.


Links



')

Source: https://habr.com/ru/post/341912/

More articles:


All Articles