Taming py-rrdtool

As they say in one ancient wisdom: one can forever observe three things: how fire burns, how water flows and how rrdtool draws graphics. This post is just about the last.
And so, what is rrdtool? rddtool is a great utility that allows you to manage the data stored in the form rrd (round-robin database). This format is remarkable in that it was originally intended for storing periodic information, and so that the size of the database itself is fixed (rather small) regardless of the amount of data. At the same time, it is possible to receive reports for long periods of time with arbitrary accuracy. The most widespread this system found in the monitoring systems, where, as they say, it is better to see the picture once than to read the logs 7 times. At the end of the article you can get something similar to





')
This will require: python , rrdtool , py-rrdtool .
After installing the necessary components, we get ready-made python-bindings to rrdtool, with which we can continue to fantasize. Of course, the number of rrdtool features may scare off at first, but I will try to look at a few basic actions + I’ll give you some custom code that you can directly launch and enjoy.

They say that real geeks draw graphs of everything they can, up to the daily number of walks of their dogs. For example, I consider two types of stored-analyzed information:

1. information automatically calculated by approximating the derivative of the input values ​​(for example, the speed at the network interface based on the number of bytes transferred)
   
2. simple information that is recorded and stored as separate, independent values ​​(temperature hdd, number of tape subscribers, etc.)
   

Working with rrd, as with any other database, begins with defining the structure of the stored data.
From python, it looks like this: first of all, import the required module and specify the name of the database file:

from rrdtool import *
fname = 'database.rrd'
rrd = RoundRobinDatabase ( fname )
further, actually, we create a DB, setting the structure.


rrd . create (
# The database will contain two independent data sources (`rrdtool.DataSource`),
# called 'in' and 'out', of type DeriveDST is a derivative. Limits are immediately set.
# values ​​of these indicators:
DataSource ( "in" , type = DeriveDST , heartbeat = 600 , min = 0 , max = 12,500,000 ) ,
DataSource ( "out" , type = DeriveDST , heartbeat = 600 , min = 0 , max = 12,500,000 ) ,
# describe which reports we want to store in the database.
# last 48 hours, every 5 min
RoundRobinArchive ( cf = AverageCF , xff = 0.5 , steps = 1 , rows = 576 ) ,
# last 2 weeks, average dig. over 30
RoundRobinArchive ( cf = AverageCF , xff = 0.5 , steps = 6 , rows = 672 ) ,
# last 2 months, average every 2 hours
RoundRobinArchive ( cf = AverageCF , xff = 0.5 , steps = 24 , rows = 732 ) ,
# last 2 years, average for 12 hours
RoundRobinArchive ( cf = AverageCF , xff = 0.5 , steps = 144 , rows = 1460 ) ,
# step 300c - the data stored in the database will be tied to the "grid" in increments of five minutes
step = 300
)


Example 2 will differ only in the rows with the DataSource:

DataSource ( "value" , type = GaugeDST , heartbeat = 600 , min = _min , max = _max )

Now we have a database in which at intervals of 5 minutes we need to enter data from two sources - in and out .

To do this, use the rrd.update() method, which takes as its argument a value (of the rrd.Val class) and an optional template necessary to set the data in a different order than the default.

For example:


values = ( 323132312 , 128539593 )
template = ( "in" , "out" )
rrd . update ( Val ( * values ) , template = template )
It is logical to arrange this action in the form of a separate script and run it with a crown with a five-minute periodicity for this example.

The next step is actually creating the very picture with information for the period period (day, week, month, year) from the database in the file fname


graph = RoundRobinGraph ( 'eth0-% s.png' % ( period ) )


graph . graph (

Def ( "in" , fname , data_source = "in" , cf = AverageCF ) ,


Def ( "out" , fname , data_source = "out" , cf = AverageCF ) ,


Cdef ( "out_neg" , "out, -1, *" ) ,

AREA ( "in" , rrggbb = "32CD32" , legend = "Incoming" ) ,


LINE1 ( "in" , rrggbb = "336600" ) ,

GPRINT ( "in" , cf = MaxCF , format = r "Max \:% 5.1lf% S" ) ,


GPRINT ( "in" , cf = AverageCF , format = r "Avg \:% 5.1lf% S" ) ,


GPRINT ( "in" , cf = LastCF , format = r "Current \:% 5.1lf% Sbytes / sec \ n" ) ,


AREA ( "out_neg" , rrggbb = "4169E1" , legend = "Outgoing" ) ,


LINE1 ( "out_neg" , rrggbb = "0033CC" ) ,

GPRINT ( "out" , cf = MaxCF , format = r "Max \:% 5.1lf% S" ) ,


GPRINT ( "out" , cf = AverageCF , format = r "Avg \:% 5.1lf% S" ) ,


GPRINT ( "out" , cf = LastCF , format = r "Current \:% 5.1lf% Sbytes / sec \ n" ) ,


HRULE ( "0" , rrggbb = "000000" ) ,


vertical_label = "bytes / sec" ,


lower_limit = 0 ,

start = "-1% s" % ( period , ) ,


title = title ,

width = 600 ,

height = 80 ,


lazy = None ,

)


In a nutshell - here you can find the data from the database that you need to display and the details of the design, such as: lines, filled areas, their colors, etc.

As a result of the work of this piece, we can get something resembling



or for example 2 - the graph of the number of subscribers to the tape from google reader:



I tried to arrange these actions into a more comfortable to use (although, it seems, bad for expansion) library , which includes several ready-to-use counter-drawers (for traffic, temperature hdd, number of clients on the wifi interface, number of subscribers) and suitable to extend the simplest class GaugeRRD , with which you can independently create your own classes to work with py-rrdtool.

Examples of use can be found at http://undefined.org.ua/static/rrdtool/src .
My example of the work of scripts, designed in the “analytical center” lies on http://xa4a.org.ua/rrdtool .

In conclusion, I want to note that the purpose of the article was to show the possibility of using a tool like rrdtool for analyzing non-trivial data, or in my python applications. If you need a system monitoring tool that is ready to work out of the box, there are already many ready-made projects, for example: munin , mrtg , cacti and others.


The original article is in the personal blog at: http://undefined.org.ua/blog/2008/08/05/py-rrdtool/