A simple way to monitor the response time of Sphinx indexes using Zabbix

Task

For example, you have the Zabbix monitoring service already configured and distributed throughout the company, and you also use the Sphinx search engine. It searches quickly, but has no built-in tools for live monitoring of its performance in the context of the index. For example, you have a lot of search servers and you want to correlate the consumption of system resources with each specific index in order to understand how to distribute them among servers — and also to see which collection starts to respond longer than we would like — and whether it correlates with increase user load or something else.

Solution design

Sphinx provides the ability to display more detailed information on resource consumption per request with a small loss of performance and a slight increase in the size of the log. To do this, you can start the searchd process with the searchd --iostats --cpustats parameters, while starting with version 2.0.1-beta, you must run searchd with the configuration query_log_format = sphinxql to display the full log. In this case, the log line looks like this:

/* Sun Jun 8 16:05:00.098 2014 conn 531 real 0.01 wall 0.06 found 1 */ SELECT tmstmp FROM index ORDER BY tmstmp desc LIMIT 0,1; /* ios=0 kb=0.0 ioms=0.0 cpums=0.6 */ 

Where
conn - the sequence number of the request from the start
real - the total time spent on a request from all cores
wall - total response time for the client
found - the number of records found
ios - the number of I / O operations
kb - the volume of read index files
ioms - iowait time per request
cpums - user CPU time on request
')
1.) Removal of parameters with Sphinx:
The most transparent and safest option for the main process seems to be launching a simple tail -F above the text log - and processing it line by line.
2.) Taking parameters in Zabbix:
Among the possible options you can consider jmx, snmp, zabbix trap, - for jmx, you need to additionally raise jvm on each search server, for snmp you need to work out your own tree of MIB elements, zabbix trap remains - www.zabbix.com/documentation/2.0/manual/config / items / itemtypes / trapper - a piece that allows you to send data to zabbix directly using the zabbix_sender program - requires the installation of an agent, but if you already have Zabbix monitoring, it is probably already installed.
3.) There are no performance restrictions imposed on the implementation tool - Python is pre-installed on the linux server at the minimum - we’ll select it as the implementation

Decision

The source code of the solution can be downloaded here: github.com/kuptservol/SphinxLogMonitor - the solution is a python process that starts a tail subprocess, receives line by line from query.log, parses it at a given regexp <log_parse_pattern> and sends data to Zabbix , can aggregate data by calculating the arithmetic average of the indicator for a given period of <time_aggregation_period_sec>.

1.) First you need to install zabbix-agent, if it is not installed:

- tar -xzvf zabbix-xxxtar.gz
- cd zabbix-xxx
- ./configure --enable-agent --prefix = / usr / local / zabbix
- make install

2.) Then start the Host with the search server in Zabbix, create Items of the Zabbix trapper type by specifying key for each observed log parameter

3.) Configure in zabbix_conf.ini the address of the zabbix server [ZabbixServer] mapping pulled out from the string of log parameters to those created in the zabbix key [ZabbixKeyLogFieldMapping]

4.) startSphinxLogMonitor.sh, stopSphinxLogMonitor.sh are made to be able to start the daemon in the background and correctly stop the python process and the tail subprocess

- make sh executable chmod + x startSphinxLogMonitor.sh
- chmod + x stopSphinxLogMonitor.sh

5.) To start: ./startSphinxLogMonitor.sh <path to query.log> <path to zabbix_conf.ini> <path to log>
To stop: ./stopSphinxLogMonitor.sh