How to choose the PHP-framework. Comparative Testing

When developing any software product, the development team is primarily faced with the task of competently choosing a software platform that defines the structure of a software system.



To do this, you need to take into account a sufficiently large number of characteristics, from “how fast everything will work” to “do we really need this feature?”. And so every time. It is during moments of brainstorming that the team compares the convenience of the framework, speed, a set of features that are implemented in it or in compatible modules.

')

But which is still better, faster and more productive?



Developers are constantly comparing frameworks to clarify this issue for themselves. For example, the article Lukasz Kujawa compares PHP frameworks. One “but” is an article for 2013. But time is ticking ... Therefore, we decided to conduct our own, actual comparison of frameworks.



To evaluate the performance was used PHP Framework Benchmark . He offers many frameworks for comparison (not just the ones mentioned above), but the author is not in a hurry to add new versions of projects to the repository, which, of course, is sad, though not fatal. If you want to add a new version is not difficult.



One of the main goals of this article is also an attempt to determine in practical terms improvements in the performance and efficiency of new versions of PHP. Therefore, testing was conducted on PHP 5.6 / 7.0 / 7.1

What will we compare?

For comparison, the following frameworks were selected:

• slim-3.0
• ci-3.0
• lumen-5.1
• yii-2.0
• silex-1.3
• fuel-1.8
• phpixie-3.2
• zf-2.5
• zf-3.0
• symfony-2.7
• symfony-3.0
• laravel-5.3
• laravel-5.4
• bluz (version 7.0.0 - for PHP5.6 and version 7.4 for PHP7.0 and above)
• ze-1.0
• phalcon-3.0

Testing is conditionally divided into 4 types:

• throughput,
• occupied memory (memory)
• runtime (exec time)
• number of included files.

Test methods and test bench

The machine on which the testing was carried out has the following characteristics:



Operation system: Linux Mint 17 Cinnamon 64-bit

Cinnamin Version 2.2.16

Linux Kernel: 3.13.0-24-generic

Processor: Intel Core i3-4160 CPU 3.60 Ghz X 2

Memory: 8 GB



Server version: Apache/2.4.7 (ubuntu)

Server build: Jul 15 2016

php 7.1 / php7.0 / php5.6



Enter the git clone https://github.com/kenjis/php-framework-benchmark command git clone https://github.com/kenjis/php-framework-benchmark - and the frame is already on our machine. Since we used Mint, you need to configure:



# Added net.netfilter.nf_conntrack_max = 100000 net.nf_conntrack_max = 100000 net.ipv4.tcp_max_tw_buckets = 180000 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 10



sudo sysctl -p



A bit about the structure of the php-framework-benchmark itself:



/ benchmarks - contains bash scripts that are responsible for collecting information about the number of requests per second (using the ab utility), the amount of information, how much time was spent and how many files were called from the "starting point" file.



/ lib - the directory where the files are located that are responsible for processing the information received after the output of the “Hello world” page, the output of tables with the results and the construction of diagrams.



/ output - directory in which logs are added after testing. Here there are two files for each tested file: .ab.log - the log after running the ab utility, and .output - contains information that was displayed on the screen (usually this is hello world and data on memory, runtime, used files).



The rest of the folders are frames stubs, in which one controller has already been added, which will return the string “hello world” when accessing by the URI compiled according to the rules for accessing this framework.



To run the test, you first need to set up frameworks. Consider two approaches.



The bash setup.sh command bash setup.sh set up those frameworks that are described in the list.sh file. You can edit it: add and delete folders for testing. That is, configure as you need.



bash setup.sh fatfree-3.5/ slim-3.0/ lumen-5.1/ silex-1.3/ command, you can set up some separate frameworks by setting them with parameters to the command. In some cases this is convenient, but we used the first approach.



After setting up the frameworks, we started testing with the help of bash benchmark.sh .



At the end of the work in the terminal, a table appeared with a list of tested frameworks, the number of requests per second, the relative value occupied by the memory, and the relative values ​​of these indicators.



To display the graphs, we used the link http: // localhost / php-framework-benchmark / .



As you understand, it was necessary to configure Apache and make it look in the folder with the frame. All this is described in the readme, so there are no questions.

Framework Test Results

Each section has a structure consisting of two forms of presentation of results.



The first form is the visual type of presentation. Each characteristic contains 4 diagrams. Each diagram displays a comparison of the frameworks among themselves, plus a cumulative diagram. It was built using a specific version of PHP. Thus, the evolution of improvements in PHP and frameworks can be traced.



The second form is the result of testing in the form of a table (suffice clarity, let's talk seriously - give me more numbers!).

Performance (throughput)

In relation to our situation, the throughput characteristic is measured in the number of requests that our framework can process in a second. Consequently, the higher this number, the more productive our application, since it will be able to correctly process requests from a large number of users.



We got the following results (queries per second):

	php 5.6	php 7.0	php 7.1
phalcon-3.1.2	5058.00	5130.00	7535.00
ci-3.0	2943.55	4116.31	4998.05
slim-3.0	2074.59	3143.94	3681.00
yii-2.0	1256.31	2276.37	2664.61
silex-1.3	1401.92	2263.90	2576.22
lumen-5.1	1316.46	2384.24	2741.81
ze-1.0	1181.14	1989.99	1741.81
phpixie-3.2	898.63	1677.15	1896.23
fuel-1.8	1044.77	1646.67	1770.13
bluz-7.3.1	- *	1774.00	1890.00
zf-2.5	198.66	623.71	739.12
zf-3.0	447.88	1012.57	1197.26
symfony-2.7	360.03	873.40	989.57
symfony-3.0	372.19	853.51	1022.28
laravel-5.3	258.62	346.25	625.99
laravel-5.4	219.82	413.49	600.42

* - bluz-7.3.1 does not support php 5.6



For clarity, built graphics for each version of PHP:



PHP5.6:





PHP7.0:





PHP7.1:





Summary Cumulative Chart (by framework):

Occupied memory (peak memory)

This characteristic (in megabytes) is responsible for the amount of memory used by the framework when performing the task assigned to it. The smaller this number, the better for us and for the server:

	php 5.6	php 7.0	php 7.1
phalcon-3.1.2	0.27	0.38	0.37
ci-3.0	0.42	0.38	0.38
slim-3.0	0.61	0.55	0.55
yii-2.0	1.31	0.91	0.91
silex-1.3	0.74	0.65	0.65
lumen-5.1	0.80	0.63	0.63
ze-1.0	0.79	0.56	0.56
phpixie-3.2	1.22	0.82	0.82
fuel-1.8	0.7	0.6	0.6
bluz-7.3.1	- *	0.69	0.69
zf-2.5	3.06	1.34	1.34
zf-3.0	2.12	1.09	1.08
symfony-2.7	3.11	1.41	1.42
symfony-3.0	2.86	1.30	1.32
laravel-5.3	2.91	2.04	2.04
laravel-5.4	3.04	1.45	1.49

* - bluz-7.3.1 does not support php 5.6



PHP 5.6:





PHP 7.0:





PHP 7.1:





Summary Cumulative Chart (by framework):

lead time

The lead time is the time spent by the system to complete the task. It is measured from the beginning of the task to the issue of the result by the system.

We looked at how many requests per second the framework can handle, how much memory it takes at the same time. Now consider how much we need to expect to get a response from the server. The lower this value, the better for us , and for the nervous system of the client of our application.



The time is given in milliseconds (ms):

	php 5.6	php 7.0	php 7.1
phalcon-3.1.2	1,300	1.470	1.080
ci-3.0	0.996	0.818	1.007
slim-3.0	1.530	1.228	0.662
yii-2.0	1.478	1.410	1.639
silex-1.3	4.657	1.625	2.681
lumen-5.1	2.121	1.829	1.228
ze-1.0	2.629	2.069	1.528
phpixie-3.2	9.329	4.757	1.911
fuel-1.8	3.283	2.684	1.425
bluz-7.3.1	- *	1.619	1.921
zf-2.5	22.042	5.011	3.998
zf-3.0	12.680	2.506	2.989
symfony-2.7	6.529	3.902	2.384
symfony-3.0	9.335	3.987	2.820
laravel-5.3	19.885	4.840	2.622
laravel-5.4	19.561	4.758	3.940

PHP 5.6:





PHP 7.0:





PHP 7.1:





Summary Cumulative Chart (by framework):

Include files

The characteristic that is responsible for the number of included files, which are described in the framework's entry point file. It is clear that the system spends some time searching and connecting. Consequently, the smaller the files, the faster the first launch of the application will be, as the framework will usually work with the cache in subsequent times, which speeds up the work:

phalcon-3.1.2	five
ci-3.0	26
slim-3.0	53
yii-2.0	46
silex-1.3	63
lumen-5.1	37
ze-1.0	68
phpixie-3.2	163
fuel-1.8	53
bluz-7.3.1	95
zf-2.5	222
zf-3.0	188
symfony-2.7	110
symfony-3.0	192
laravel-5.3	38
laravel-5.4	176

The difference in the number of connected files between Laravel 5.3 and Laravel 5.4 may seem strange and give rise to discussions, disputes, etc. We hasten to clarify the situation. As you know, using the command



php artisan optimize --force



in Laravel 5.3, you can generate the compiled.php file, and thereby reduce the number of include files by collecting them into one. But there is one “but”: the commands for generating this file are no longer in Laravel 5.4. The developer decided to remove this feature, since he considered ( https://github.com/laravel/framework/pull/17003 ) that it is better to use opcache to tune performance.

Is it worth updating?

The summary data on the versions more than clearly shows how the productivity and efficiency of resource utilization will increase during the transition (or initial choice) to the new version of PHP.



When switching from PHP 5.6 to PHP 7.0, the average performance increase was almost + 90%, while the minimum performance increase was + 33% for Laravel 5.3, and the maximum -> 200% for Zend Framework 2.5.



The transition from version 7.0 to 7.1 is no longer so shocking, but still, on average, it gives an almost 20% performance boost.



Summarizing all the data on the performance of different versions of PHP, we get these "mattresses":



Fun fact : Laravel 5.3 showed the smallest performance gains when migrating from PHP 5.6 to PHP 7.0, but the highest gains from migrating from version 7.0 to version 7.1, and as a result, the performance of Laravel 5.3 and 5.4 to PHP 7.1 is almost the same.



Memory consumption has also been optimized, so switching from PHP 5.6 to PHP 7.0 will allow your application to consume 30% less memory.



Upgrading from version 7.0 to version 7.1 practically does not give an increase, and in the latest Symfony and Laravel, we’ve left the “minus” altogether, because they are starting to “eat” a little more.



It remains to look at the execution time, and yes, here, too, everything is fine:

• Moving from PHP 5.6 to PHP 7.0 will give you an average acceleration of 44%.
• Moving from PHP 7.0 to PHP 7.1 will give you an acceleration of another 14%.

Note. Testing with ab - what we encountered

“What about slim and phpixie” - this question prompted an investigation into the behavior of the ab utility when interacting with these frameworks.



Perform a test separately for Slim-3.0:



ab -c 10 -t 3 http://localhost/php-framework-benchmark/slim-3.0/index.php/hello/index



Concurrency Level: 10

Time taken for tests: 5.005 seconds

Complete requests: 2

Failed requests: 0

Total transferred: 1800 bytes

HTML transferred: 330 bytes

Requests per second: 0.40 [#/sec] (mean)

Time per request: 25024.485 [ms] (mean)

Time per request: 2502.448 [ms] (mean, across all concurrent requests)

Transfer rate: 0.35 [Kbytes/sec] received



Something is wrong - the number of requests per second is only 0.4 (!)



ab -c 10 -t 3 http://localhost/php-framework-benchmark/laravel-5.4/public/index.php/hello/index



Concurrency Level: 10

Time taken for tests: 3.004 seconds

Complete requests: 1961

Failed requests: 0

Total transferred: 1995682 bytes

HTML transferred: 66708 bytes

Requests per second: 652.86 [#/sec] (mean)

Time per request: 15.317 [ms] (mean)

Time per request: 1.532 [ms] (mean, across all concurrent requests)

Transfer rate: 648.83 [Kbytes/sec] received



It was a Keep Alive connection, more details can be found here.

“When you make requests with“ Connection: keep-alive ”, it will be the same TCP connection. This is called HTTP persistent connection. It helps to reduce the time of the response.



If the request is made with "Connection: close", it means that you need to make it. Connection will be established.



By default HTTP 1.1 client / server doesn’t support keep-alive by default 1.0 client / server

Thus, the test for Slim should look like this:



ab -H 'Connection: close' -c 10 -t 3 http://localhost/php-framework-benchmark/slim-3.0/index.php/hello/index



Concurrency Level: 10

Time taken for tests: 3.000 seconds

Complete requests: 10709

Failed requests: 0

Total transferred: 2131091 bytes

HTML transferred: 353397 bytes

Requests per second: 3569.53 [#/sec] (mean)

Time per request: 2.801 [ms] (mean)

Time per request: 0.280 [ms] (mean, across all concurrent requests)

Transfer rate: 693.69 [Kbytes/sec] received

Conclusion

As expected, the undisputed leader in performance (but not development speed) is Phalcon. The second place - and in fact the first among PHP frameworks (and not C, on which the Phalcon source code is written) - is taken by CodeIgniter 3!



Of course, you should not forget that each instrument has its own purpose. If you choose a small and lightweight framework and are going to write something different from the simplest applications or the REST API on it, then you will most likely encounter problems when extending the functionality. And vice versa - the redundancy of full-featured, large frameworks will entail the financial costs of hosting maintenance, even for elementary applications under heavy load.



This testing was carried out in order to convince / tell / strengthen the position of PHP language versions 7.0 and 7.1 in your mind and in future projects, to convey information that productivity has indeed increased.

Refactoring internal data structures and adding an additional step before compiling the code as an abstract syntax tree - Abstract Syntax Tree (AST) - led to superior performance and more efficient memory allocation. The results themselves look promising: tests performed on real-world applications show that PHP 7 is on average twice as fast as PHP 5.6, and also uses 50% less memory during query processing, which makes PHP 7 a strong contender for the HHVM JIT compiler from Facebook

The tests fully confirm both the twice accelerated request processing in PHP7, and the reduced amount of memory used.