mod_performance - 0.4

The final series of articles on the apache module is mod_performance.

History tour

This is the usual Apache 2.x module for Linux:

1. the module is designed to collect and accumulate statistics on the use of resources (CPU and memory, script execution time) by the Apache 2.4 / 2.2 / 2.0 web server;
2. The module allows to analyze the collected data.

It allows you to keep track of how many resources consumed the request received by the web server. Each time saving the following information:

1. virtual host that received the request;
2. the file that is requested;
3. Request URI;
4. CPU load in%;
5. memory usage in%;
6. request processing time.

And the accumulated statistics allows you to analyze. SQLite (MySQL, PostgreSQL, error_log) is used as a database for saving and analysis.
The module allows you to track absolutely all requests, as well as specific ones, filtered by the rule using regular expressions. More precisely, it will be said that the module ALWAYS processes only those requests that match the filter containing the regular expression.
')
An earlier description of the module can be read on the links:

1. 0.2-0.3 version - http://habrahabr.ru/post/120660/
2. 0.1 and earlier - http://habrahabr.ru/post/119011/

The functionality of the module during its existence has been expanded, new features have been added, and the mechanism of saving in the database has been optimized.
I decided to put together all the innovations and talk about them.

A small digression ...

There are many alternative methods of collecting various statistics, there are wonderful modules mod_status, mod_log_config, which can also perform similar (and not so) with the described module actions. The article only tells about another tool that can be useful for monitoring and debugging issues on the server.

What is the difference between module version 0.4 and 0.3?

First difference

I think a very important difference. Previously, for each request, the module ran two threads that monitored the request; they ended with a stream, writing data on the stream to the database (log). There was a limit on the number of threads monitored and the stack size for the thread being created — PerformanceMaxThreads and PerformanceStackSize. These parameters are no longer used by the module. Also removed from it is a pool of threads that monitored memory allocated to threads.
Now all requests are monitored by three threads. I will not specifically describe what they do, but there are only three of them. This means that there is no limit on the number of requests that will be monitored.

Second difference

The data in the database (log) is written by one stream, and not many as previously. Thus, the number of locks to the database decreases - this is especially true for SQLite. Buffering of data recording in the database. Ie now the data in the log does not fall instantly, but when there is an opportunity and time to write it there.

Third difference

It is no secret that the statistics on the process is taken by the module through procfs. Previously, data was taken only by Process ID, now it is possible to read data and by Thread ID - the so-called TID. This should be relevant for apache worker (event).

Fourth difference

Added the ability to create your own reports (a report is a method for analyzing data collected by a module). The module still carries a certain set of reports, the so-called “out of box”. But in the source code of the module there is a file custom.tpl, which expands the number of available reports of the module. And also allows you to create your own database queries and display the results. How to do this, I will describe below.

Fifth difference

A number of bugs fixed that caused apache server restarts (hopefully, no new bugs were added). Added display of which method GET, POST, etc. The script was called.

Sixth difference

Now you can write data to a centralized remote database (for MySQL, PostgreSQL mode). Those. Now data from different machines can be accumulated on the same server.

More details

A little bit about the interface

Since version 0.3, the interface has been redesigned for analyzing the collected data. I'll tell you a little about its features (Figure 1).
Figure 1.



1. Hide / Show - the button "Hide / Show filter" - designed to minimize and maximize the filter window.
2. Report - drop-down list of available reports (reports of analysis of accumulated data)
3. Sort Field - the number of the displayed field, which will be sorted (does not affect custom reports)
4. Sort Type - sorting type of the selected field - in ascending or descending order (does not affect custom reports)
5. Period (days) - the interval in days from the current day for which the data will be analyzed
6. Period begin (Period End) - the exact date and time of the beginning and end of the analyzed period (if these values ​​are specified, then the Period (days) field is ignored)
7-8-9. - Additional filter parameters of the analyzed data. Perhaps as a strict match, and not strict - for example, the value ending in test -% test.
10. Link to download the latest version
11. Reference to module documentation
12. Icon for sorting data on the page (v - descending, ^ - ascending)
13. Column Header Row
14. Fixed row column numbers. Does not move even when scrolling the page.

The module interface uses javascript.

Custom reports

An example of setting up custom reports (for CentOS / Redhat / Fedora) supplied with the module source code:

mkdir ~/tmp cd ~/tmp wget http://www.lexvit.dn.ua/modperf/getlast -O mod_performance.tar.gz tar -xvf mod_performance.tar.gz –strip 1 mkdir -p /opt/performance/ chown apache:apache /opt/performance/ chmod 755 /opt/performance/ cp ~/tmp/custom.tpl /opt/performance/   mod_performance.conf  PerformanceCustomReports /opt/performance/custom.tpl service httpd restart 

Report example:
Figure 2.



List of custom reports:

1. Show the average load generated by the site in different time periods
2. Show most popular script by site - show the most frequently called script for each site.
3. Show CPU load range by site - show CPU load distribution for each site
4. Show Memory Usage range by site - show the memory allocation for each site
5. Show Max CPU load in different time periods by site - show the maximum CPU load in different time periods for each site
6. Show Max Memory Usage - show the maximum memory usage in different time periods for each site.
7. Show Max IO Usage - show the maximum IO load in different time periods for each site

To create a custom report, use the following syntax in the custom.tpl file:

 [ ] header=Fld 1|Fld 2|Fld 3; ssql=   SQLite; msql=   MySQL; psql=   PostgerSQL; sort=1,2; 

The name of the report is written in square brackets; it will be displayed in field 2 of fig.1.
Header - the name and list of displayed fields in order in select separated by the “|” sign. In the example, the first picked up field from select will be displayed in column Fld 1, the second in column Fld 2, and so on. List of fields number 13 in Figure 1.
Ssql, msql, psql - database queries. I recommend not using the direct names of the table fields, but their pseudonyms. The report is displayed if there is a query for the current database configuration (SSQL, MSQL, PSQL)
Match table field and alias:
id: ITEMNUMBER: - record identifier
dateadd: DATEADD: - the date the record was added
host: FORHOST: - the host to which the request was made
uri: REQUESTURI: - Request URI
script: CALLEDSCRIPT: - called script
cpu: CPUUSAGE: - CPU in%
memory: MEMUSAGEINPRCT: - memory in%
exc_time: EXECUTIONTIME: - request execution time
cpu_sec: CPUUSAGEINSEC:
memory_mb: MEMUSAGEINMB: - memory in Mb
bytes_read: BYTESREAD: - byte read
bytes_write: BYTESWRITE: -, fqn pfgbcfyj
hostnm: HOSTNAME: - the request was made on the server ...
Filter fields:
: FILTER: - consider filter fields
: PERIOD: - take into account the entered period
performance -: TBL: - table alias

Example:

 select count(*) as t1, sum(:CPUUSAGE:)/count(*) as t2, sum(:MEMUSAGEINPRC T:)/count(*) as t3, sum(:BYTESREAD:+:BYTESWRITE:)/count(*) as t4, :CALLEDSCRIPT:, :FORHOST: from :TBL: where :FILTER: and :PERIOD: group by :CALLEDSCRIPT:,: FORHOST: 

Saving data in one centralized repository

Configuring the module for working with centralized storage:
1) in a centralized repository (MySQL, PostgreSQL), you must create a user, database and table.
MySQL: CREATE TABLE IF NOT EXISTS performance (id INT NOT NULL AUTO_INCREMENT, dateadd DATETIME, host VARCHAR (255), uri VARCHAR (512), script VARCHAR (512), cpu FLOAT (15.5), memory FLOAT (15.5 ), exc_time FLOAT (15.5), cpu_sec FLOAT (15.5), memory_mb FLOAT (15.5), bytes_read FLOAT (15.5), bytes_write FLOAT (15.5), hostnm CHAR (32), PRIMARY KEY (id))
PostgreSQL: create table performance (id SERIAL, dateadd timestamp, host varchar (255), uri varchar (512), script varchar (512), cpu float (4), memory float (4), exc_time float (4), cpu_sec float (4), memory_mb float (4), bytes_read float (4), bytes_write float (4), hostnm char (32), PRIMARY KEY (id))
2) on servers from which data will be collected to register in mod_performance.conf
PerformanceLogType MySQL or Postgres
PerformanceHostId HostName (enter a unique name for each host, for example its IP or other)
PerformanceDbUserName username
PerformanceDBPassword userpassword
PerformanceDBName dbname
PerformanceDBHost hostname - IP address or server name with centralized data storage

For those who have installed the module version 0.3

In version 0.4, the data table format was changed. For the 0.4 version to work correctly, the old table needs to be modified:
SQLITE: ALTER TABLE performance ADD hostnm CHAR (32)
MySQL: ALTER TABLE performance ADD hostnm CHAR (32)
PostreSQL: ALTER TABLE performance ADD hostnm CHAR (32)

Links and documentation

Documentation section mod_performance 0.4:
http://wiki.bayrepo.net/ru/docs
http://wiki.bayrepo.net/en/docs

Download module:

 wget http://git.bayrepo.net/modperformance/snapshot/modperformance-master.tar.gz