Pinba - monitor php in real time

As stated on the official website -

 Pinba for PHP using MySQL as a read-only interface.

And indeed it is. It will allow you to receive real-time statistics on the running application, while not slowing down the application itself.
What to do with statistics is your business, for example, we output it to Zabbiks and use it to monitor stability (there are no errors, few are long requests), and for analytics.
Surprisingly, about this truly remarkable invention there has not been a single article on Habré yet.



I wanted to do something good for this opensource project, and what could be better than mentioning in Habré;)

Installation

Pinba itself consists of two parts - this is php extention and engine (engine for mysql).
Both are supplied in source code. The installation itself is quite trivial and well described in the reference manual .

The principle of operation and configuration

Client (module)
On each server where php is running, you need to install the module and add the following lines to php.ini:
; configuration for php pinba module
extension=pinba.so
pinba.enabled=1
pinba.server=192.168.1.42:3300 ;

')
After that, each php script, including cli, will send a udp report to the specified address and port before shutting down.
The installation is complete, the client is configured.

Server (mysql engine).
When I first read the documentation, I could not understand why pinbe needed mysql. It turned out that everything is simple - mysql is the interface / entry point to the analytics system.
This is very convenient, because under this DBMS there are already a lot of solutions and everyone knows how to work with it.

The pinba server is configured as a mysql engine. There are not many settings and their names are obvious:

 [mysqld]
 pinba_port = 3300 # port
 pinba_address = 192.168.1.205
 pinba_stats_gathering_period = 10000 # (microseconds)
 pinba_stats_history = 900 # (seconds)
 pinba_temp_pool_size = 10000
 pinba_request_pool_size = 1000000 # <max expected requests per second> * <stats_history> = <request pool size>
 pinba_tag_report_timeout = -1 #Default value is -1, ie keep the data updated forever. 

See the documentation for details.

After you install the server and enable the module on the clients, the pinbu (mysql) will receive data on all scripts that have been executed with this module.
Data on all unique requests will go to the request tab. Also available to you groupings:
report_by_hostname, report_by_hostname_and_script, report_by_hostname_and_server, report_by_hostname_server_and_script, report_by_script_name, report_by_server_and_script, report_by_server_name.

And in general, it's mysql, so:
show tables;


I summarize - without tuning practically nothing, we got the analytics system of all php in production.
For example, we know the total number of hits (rps):

 mysql> select req_per_sec from report_by_server_name where server_name = 'ro.plus1.wapstart.ru';
 + ------------- +
 |  req_per_sec |
 + ------------- +
 |  547.161 |
 + ------------- +
 1 row in set (0.00 sec)

The same, but for each host (server for sale):

 mysql> select hostname, req_per_sec from report_by_hostname_and_server where server_name = 'ro.plus1.wapstart.ru';
 + ---------- + ------------- +
 |  hostname |  req_per_sec |
 + ---------- + ------------- +
 |  a .... |  81.7561 |
 |  b .. |  59.0298 |
 |  c ... |  90.8049 |
 |  f .... |  54.5014 |
 |  f .... |  54.5122 |
 |  h ...... |  63.5664 |
 |  k ... |  54.5068 |
 |  s .... |  90.8211 |
 + ---------- + ------------- +
 8 rows in set (0.00 sec)

And a large set of different reports. All reports will be updated promptly and contain the latest data.
This is not an analysis of apache logs, it is profiling and monitoring right on the production!

Timers and tags

Now about the most delicious.
You can measure the duration of an event in scripts and tag it. This data will also be available for reports.

For example, if you have a long sample of users from the database, you can do this:
$t = pinba_timer_start(array("group"=>"mysql", "server"=>"dbs2", "operation"=>"select"));
//
pinba_timer_stop($t);


I added timers directly to the framework that we use, and now, to some extent, I measure all database requests, memcached, etc.

Separately, I note that this data is easily accessible in the server interface.
They can be selected either from the raw data - see the tables report, timer, timertag, or create tables in which the data will be already in the aggregated state.

Example (from wiki ):
CREATE TABLE `tag_info_group_server` (
`group_value` varchar(32) DEFAULT NULL,
`server_value` varchar(32) DEFAULT NULL,
`req_count` int(11) DEFAULT NULL,
`req_per_sec` float DEFAULT NULL,
`hit_count` int(11) DEFAULT NULL,
`hit_per_sec` float DEFAULT NULL,
`timer_value` float DEFAULT NULL
) ENGINE=PINBA DEFAULT CHARSET=latin1 COMMENT='tag2_info:group,server'

NOTE : Comment to the table is important.
After creation, data on the group tag and server will be automatically (!) Aggregated into this table.

Stability

We use it in production. She works ;)

---

I do not aim to make some complete description of pinba, especially since I do not want to copy the manual.
If you use php in production, try pinba. She is really good!

Literature:
* wiki
* google group

upd: Meet the author of the current implementation - tony2001 !