Weather Monitoring or Cacti HowTo
This post could start in different ways. It is possible on business: how the monitoring system is needed to search for system errors and how it helps to detect bottlenecks. But, now it's summer, time for outdoor recreation, and I'll start with how I decided to find out how often the weather forecast changes using the popular monitoring tool Cacti. Under the cut, about a curious reader!, You are waiting for stories about how to set up monitoring of arbitrary data in Cacti , and not just, but with pictures.
So, a week before June 19th (Friday, you know ...) I started watching, what will the weather be like? I watched from Yandex - I really love these web meteorological sites. The weather changed day by day and I was curious - what is the frequency of these changes? The feeling of the weather is a subjective thing, and the frequency of changes in the forecast on the site is quite measurable. For these purposes, I decided to use the long-known Cacti monitoring tool. The article is intended mainly for programmers and system administrators, so I don’t consider how to install Cacti - these people have either already been installed or can handle in 5 minutes.
If you are wondering how often the weather changes in forecasts, but Cacti is not interesting, skip right to the end.
Cacti is based on RRDTool - a tool for working with circular databases (RRD - round-robin database). Such databases consider the time coordinate consisting of the intervals and record the values ​​of the stored parameter in the nodes of such intervals, interpolating, if necessary, the measured value. In a single database, there can be several RRA (round-robin archive) —this is achieved the ability to store a value with different time intervals (that is, with different detail). All these operations Cacti hides from the user - all we need to do is to set up monitoring of the value - to provide a means to get this value and tell Cacti to start the poller (data collector - pulls data from all the sources and saves them to that very RRD ).
')
First, a few words about the “entities” that exist in Cacti. They are listed in the left menu in the admin panel:
So:
To find out the forecast at a given time, you need to upload a page with Yandex weather and parse. Since we don’t know in advance what day we are interested in, we’ll make a script that will accept a date for an input, give a temperature at the output for it in the afternoon. You don’t need to contact Yandex every 5 minutes - the weather is unlikely to change so often, so let's make the script cache the data. I will not go deep here, just give the script code:
Now that the script is ready, you need to put it somewhere (it’s up to you). I, for example, put in / usr / share / cacti /. But do not forget to take into account the rights to read and run (just to create a file) - poller will run this script under the user www-data . Next, create a Data Input Method :
Using the <day> construct, we make the template parameter for the input method. It must be declared in the Input Fields list - so that the Data Template knows about it. You also need to declare the Output Field with the name res (the name is not important, since our script returns one value). Making the Data Input Method is a pleasure, the most interesting thing to come)
We create Data Template :
By default, we are prompted to enter all the parameters, but some of them can be delegated to Data Source . In our case, we delegate the name and parameter to the Data Source script.
Making Data Source is also pretty simple:
Select the newly created Data Template , the host name can be left empty - in our case everything happens on the local machine. We write a unique name for Data Source (uniqueness is not required - it is necessary so that it would be easier to insert it into the schedule) and the day we are monitoring, in our case, the 19th number. The file name is calculated based on the id of the given Data Source - but you can choose your own one (the main thing is that there are no collisions).
Most of all, perhaps, the settings for graphs. Create a new Graph Template and see:
First, the elements of the chart are listed - lines, legends, other signatures. Further, the template parameters (where without them, this is a template). And then there is a bunch of graph parameters (I did not give them in the figure, they are pretty obvious - limits, logarithmic scale and other things that are not necessary in our case).
To begin with, we create visual elements (in the settings for a chart, the captions can be made by analogy):
We leave the name of the Data Source and the color empty - we’ll move them to the template parameters. The main parameters of the visual element:
When visual elements are described, you can create template parameters in the Graph Item Item Inputs list:
In this window, you need to specify that we want to template (data source, color, transparency, or something else) and to which elements this will apply. In our case, you need to submit a template data source for all elements.
Now that we have a data source and a graph template, we create the graph itself in the Graph Managment menu:
If everything is done correctly, the schedule should appear immediately. But there is one BUT - at least two pollings must pass in order to draw the data on the chart - that is, about 10 minutes.
In the process of setting up new data sources, it is convenient to use remote information, which is sufficient in Cacti:
So, for the sake of which there were all the dances with a tambourine, the dynamics of the weather forecast change for June 19 by Yandex. Weather:
And, as a bonus, the dynamics of changes in the dollar exchange rate by the Central Bank of the Russian Federation (there is an empty area - this is not a symbol of the crisis, but I just cut down the server):
What you want to monitor you - the mistakes of the Apache, the number of users, or the number of songs in your playlist - a question of fantasy. Good luck to all who are interested in experimenting with Cacti!
PS While I was writing this post, Yandex decided that the 19th would be even colder by two degrees ... But, nevertheless, the weather from Yandex is very cool, even it was nice to parse.
PPS Cacti is far from the only tool for working with RRD - there is Munin, Ganglia, Nagios and a whole lot more.
So, a week before June 19th (Friday, you know ...) I started watching, what will the weather be like? I watched from Yandex - I really love these web meteorological sites. The weather changed day by day and I was curious - what is the frequency of these changes? The feeling of the weather is a subjective thing, and the frequency of changes in the forecast on the site is quite measurable. For these purposes, I decided to use the long-known Cacti monitoring tool. The article is intended mainly for programmers and system administrators, so I don’t consider how to install Cacti - these people have either already been installed or can handle in 5 minutes.
If you are wondering how often the weather changes in forecasts, but Cacti is not interesting, skip right to the end.
Cacti Philosophy
Cacti is based on RRDTool - a tool for working with circular databases (RRD - round-robin database). Such databases consider the time coordinate consisting of the intervals and record the values ​​of the stored parameter in the nodes of such intervals, interpolating, if necessary, the measured value. In a single database, there can be several RRA (round-robin archive) —this is achieved the ability to store a value with different time intervals (that is, with different detail). All these operations Cacti hides from the user - all we need to do is to set up monitoring of the value - to provide a means to get this value and tell Cacti to start the poller (data collector - pulls data from all the sources and saves them to that very RRD ).
')
First, a few words about the “entities” that exist in Cacti. They are listed in the left menu in the admin panel:
So:
- Data Input Method - describes how the data will be obtained. The main methods are SNMP (an interesting topic, but we will not go deep) and the external script is our case.
- Data Template - describes how often and with what parameters the Data Input Method will be used.
- Data Source - RRD itself - stores data in a file, it is created based on the Data Template
- Graph Template . You can describe in it all the details of the display, except for the data source, and you can create template parameters for color, transparency and other things
- Graph - the actual graph. You can prepare by mixing Data Source, Graph Template and specifying all template parameters from the Graph Template
- Graph Tree - a tree of graphs - if you have a lot of graphs (for example, monitor 10 machines), then it is convenient to create a hierarchy
- There is much more, but we will not need it yet.
Script for mining weather
To find out the forecast at a given time, you need to upload a page with Yandex weather and parse. Since we don’t know in advance what day we are interested in, we’ll make a script that will accept a date for an input, give a temperature at the output for it in the afternoon. You don’t need to contact Yandex every 5 minutes - the weather is unlikely to change so often, so let's make the script cache the data. I will not go deep here, just give the script code:
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
#!/bin/bash cd / usr / share / cacti flagdate =$ ( date "+%Y-%m-%d %H:00" ) flagfile =yandex.flag touch $flagfile -d " $flagdate " if [ ! -e index.html -o index.html -ot $flagfile ] ; then wget http: // weather.yandex.ru / 27612 / -O index.html > / dev / null 2 >& 1 fi day =$ 1 pos =$ ( cat index.html | grep "<th class= \" weekday \" >" -A1 | grep -o "<th class= \" week[^0-9]*[0-9]*[^0-9]*</th>" | grep -o "[0-9]*" | grep $day -n | grep -o "^[0-9]" ) let pos =pos+ 1 res =$ ( cat index.html | grep "<tr class= \" data day \" >" -A1 | grep -o "+[0-9]*" | sed -n " $pos " "p" ) echo $res | grep -o "[0-9]*"
Data Input Method
Now that the script is ready, you need to put it somewhere (it’s up to you). I, for example, put in / usr / share / cacti /. But do not forget to take into account the rights to read and run (just to create a file) - poller will run this script under the user www-data . Next, create a Data Input Method :
Using the <day> construct, we make the template parameter for the input method. It must be declared in the Input Fields list - so that the Data Template knows about it. You also need to declare the Output Field with the name res (the name is not important, since our script returns one value). Making the Data Input Method is a pleasure, the most interesting thing to come)
Data template
We create Data Template :
By default, we are prompted to enter all the parameters, but some of them can be delegated to Data Source . In our case, we delegate the name and parameter to the Data Source script.
Data source
Making Data Source is also pretty simple:
Select the newly created Data Template , the host name can be left empty - in our case everything happens on the local machine. We write a unique name for Data Source (uniqueness is not required - it is necessary so that it would be easier to insert it into the schedule) and the day we are monitoring, in our case, the 19th number. The file name is calculated based on the id of the given Data Source - but you can choose your own one (the main thing is that there are no collisions).
Graph Template
Most of all, perhaps, the settings for graphs. Create a new Graph Template and see:
First, the elements of the chart are listed - lines, legends, other signatures. Further, the template parameters (where without them, this is a template). And then there is a bunch of graph parameters (I did not give them in the figure, they are pretty obvious - limits, logarithmic scale and other things that are not necessary in our case).
To begin with, we create visual elements (in the settings for a chart, the captions can be made by analogy):
We leave the name of the Data Source and the color empty - we’ll move them to the template parameters. The main parameters of the visual element:
- Graph Item Type is an item type. In our case there will be AREA (the actual graph) or GPRINT (the caption is at the bottom of the graph - we will write the average and maximum values)
- Consolidation Function - for the schedule should be AVERAGE , when creating signatures, you can also choose LAST / MAX / MIN
- GPRINT Type - for signatures, you can specify an extended output format. You can create your own format in the Graph Managment -> GPRINT Presets menu . There's the usual printf format, nothing interesting)
When visual elements are described, you can create template parameters in the Graph Item Item Inputs list:
In this window, you need to specify that we want to template (data source, color, transparency, or something else) and to which elements this will apply. In our case, you need to submit a template data source for all elements.
Almost Final
Now that we have a data source and a graph template, we create the graph itself in the Graph Managment menu:
If everything is done correctly, the schedule should appear immediately. But there is one BUT - at least two pollings must pass in order to draw the data on the chart - that is, about 10 minutes.
In the process of setting up new data sources, it is convenient to use remote information, which is sufficient in Cacti:
- There are many logs in / var / log / cacti / , in particular poller errors poller-error.log
- In the System Utilities menu -> View Cacti Log File
The final
So, for the sake of which there were all the dances with a tambourine, the dynamics of the weather forecast change for June 19 by Yandex. Weather:
And, as a bonus, the dynamics of changes in the dollar exchange rate by the Central Bank of the Russian Federation (there is an empty area - this is not a symbol of the crisis, but I just cut down the server):
What you want to monitor you - the mistakes of the Apache, the number of users, or the number of songs in your playlist - a question of fantasy. Good luck to all who are interested in experimenting with Cacti!
PS While I was writing this post, Yandex decided that the 19th would be even colder by two degrees ... But, nevertheless, the weather from Yandex is very cool, even it was nice to parse.
PPS Cacti is far from the only tool for working with RRD - there is Munin, Ganglia, Nagios and a whole lot more.
Source: https://habr.com/ru/post/62312/
All Articles