We monitor everything: the expansion of Windows and Linux agents using scripts

If we need to monitor the status of servers and other computerized workstations using Zabbix, then this can be done in two ways.

The first way is with the help of SNMP requests, which Zabbix does wonderfully with sending. So you can pull out and load network interfaces, and CPU, memory. On top of this, server manufacturers can give us a lot of information about the state of iron via SNMP.

The second is to use Zabbix agent , which we will run on the monitored system. The list of monitored parameters includes such simple things as CPU usage, memory usage, and more cunning ones, such as reading text log files with rotation support or tracking the fact that any file has changed. You can even use the output of any arbitrary command on the system as a parameter. Zabbix agent capabilities are growing from version to version.
')


What to do if what we want to control via Zabbix is ​​not in the list of features of Zabbix agent? Wait until it is implemented by the developers in the next release? Not necessary.

We were left with a few standard interfaces in order to expand the capabilities of Zabbiks for monitoring servers as much as our imagination and the availability of free time to write scripts will allow us. The interfaces are these UserParameter and zabbix_sender. We will talk about the first one, and as examples we will show how you can collect the status of SMART hard drives and monitor when someone uninstalls or installs new programs on their Windows machine.

Little materiel

If you have already configured Zabbix agent on the server at least once, then it will not be difficult to start using UserParameter . To add a new parameter you need to do several things:

• Add a line at the end of the zabbix_agentd.conf configuration file

UserParameter=<>,<> 

Where:

<key> is a unique name that we invent ourselves. We will use it when setting up a data item in Zabbix.
<command> is the command to be executed on the monitored host .

But at once a very simple example that lies in every standard Linux config:

 UserParameter=system.test,who|wc -l 

So, the key is system.test , and we execute the command who | wc -l , which returns us the number of open sessions in the system. Add (or uncomment this line if you already have), go ahead.

• In Zabbix Web Console, create a new data item with the key that we used; if we take the example above, it is system.test.

To do this, click "Create a data item"


and then set the key the same as indicated in the config file, and the type of Zabbix agent:

• Reboot the Zabbix agent to make changes in the config file take effect.

Observe the result in the latest data :

SMART monitoring via UserParameter

The example above has little practical use, given that the standard system.users.num key already exists, which does exactly the same thing.

So now we will consider an example which already will look more like a realistic one.

If we are interested in monitoring the moment when it is time to change hard drives according to plan, then there are two options:

1. If the disks are behind a hardware RAID controller, then, as a rule, the operating system does not see the disks themselves. Therefore, we are looking for ways to get information about the status of hard drives through utilities or SNMP-subagent, which we kindly provided (or did not provide) the manufacturer of the RAID controller. For each separate series of controllers, your path to this information.
2. If we are talking about just workstations, servers with software-based RAID, etc., then the disks can be accessed from the operating system, and we are free to use various utilities to read their status. In the case of Zabbix, the smartctl utility comes from the SMARTMONTOOLS package .

In Debian, installing SMARTMONTOOLS comes down to:

 sudo apt-get install smartmontools 

and the utility is ready to use.

For each disk that is in the system, first verify that SMART is enabled:

 sudo smartctl -i /dev/sda | grep SMART SMART support is: Available - device has SMART capability. SMART support is: Enabled 

if suddenly SMART is supported by a disk, but turned off, then activate it:

 sudo smartctl -s on -S on -o on /dev/sda smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF ENABLE/DISABLE COMMANDS SECTION === SMART Enabled. SMART Attribute Autosave Enabled. SMART Automatic Offline Testing Enabled every four hours. 

Now we can check the SMART status with the command:

 sudo smartctl -H /dev/sda |grep "test"| cut -f2 -d: |tr -d " " 

This is the command that we will write to our zabbix_agentd.conf:

 UserParameter=uHDD.health,sudo smartctl -H /dev/sda |grep "test"| cut -f2 -d: |tr -d " " 

where uHDD.health is the key.

SMART monitoring via Flexible UserParameter

This raises a reasonable question, how to be if there are two disks. The easiest way to solve this problem is the ability of the UserParameter to pass parameters to the agent, about which we have not yet mentioned. But everything is done very simply, just an example:

 UserParameter=uHDD.health.[*],sudo smartctl -H /dev/$1 |grep "test"| cut -f2 -d: |tr -d " " 

In the Zabbix web interface in the key, we will substitute the parameters in square brackets instead of *. For example, for one data element we write sda , and for another one sdb . In the command, this parameter will be reflected where the $ 1 variable is .



Let's create an item for the second disk:



And after some time we will be able to observe the result in the latest data:

SMART monitoring via Flexible UserParameter with Low-level Discovery

Everything worked out. But then there is a reasonable question, how to be if the disks are not two but twenty two. And here we have a great low-level detection (LLD) feature that we already talked about .

Low-level detection allows the monitoring system to detect how many elements of the same type are present on a network node and dynamically create the necessary data elements, triggers and graphs for these elements using a template. Out of the box, the system is able to find file systems, network interfaces, and SNMP OIDs. However, here too, the developers left the opportunity to supplement the standard features; you just need to transfer to the system information about which elements were found in the JSON format. We will use this.

Create a small perl script, smartctl-disks-discovery.pl . It will find all disks in the system and display this information in JSON, passing also information whether the SMART disk is enabled or not, and also tries to enable SMART itself if it is turned off:

 #!/usr/bin/perl #must be run as root $first = 1; print "{\n"; print "\t\"data\":[\n\n"; for (`ls -l /dev/disk/by-id/ | cut -d"/" -f3 | sort -n | uniq -w 3`) { #DISK LOOP $smart_avail=0; $smart_enabled=0; $smart_enable_tried=0; #next when total 0 at output if ($_ eq "total 0\n") { next; } print "\t,\n" if not $first; $first = 0; $disk =$_; chomp($disk); #SMART STATUS LOOP foreach(`smartctl -i /dev/$disk | grep SMART`) { $line=$_; # if SMART available -> continue if ($line = /Available/){ $smart_avail=1; next; } #if SMART is disabled then try to enable it (also offline tests etc) if ($line = /Disabled/ & $smart_enable_tried == 0){ foreach(`smartctl -i /dev/$disk -s on -o on -S on | grep SMART`) { if (/SMART Enabled/){ $smart_enabled=1; next; } } $smart_enable_tried=1; } if ($line = /Enabled/){ $smart_enabled=1; } } print "\t{\n"; print "\t\t\"{#DISKNAME}\":\"$disk\",\n"; print "\t\t\"{#SMART_ENABLED}\":\"$smart_enabled\"\n"; print "\t}\n"; } print "\n\t]\n"; print "}\n"; 

When you run the script gives:

 $ /usr/local/bin/smartctl-disks-discovery.pl 

 { "data":[ { "{#DISKNAME}":"md0", "{#SMART_ENABLED}":"0" }, { "{#DISKNAME}":"md1", "{#SMART_ENABLED}":"0" }, { "{#DISKNAME}":"sda", "{#SMART_ENABLED}":"1" }, { "{#DISKNAME}":"sdb", "{#SMART_ENABLED}":"1" }] } 

Now, in order for the script to be automatically launched by Zabbix, simply add another UserParameter to the zabbix_agentd.conf :

 UserParameter=uHDD.discovery,sudo /usr/local/bin/smartctl-disks-discovery.pl 

Having finished configuring the configuration, go to the web interface, where we create a new detection rule for smartctl:



Pay attention to the key and filter , ({#SMART_ENABLED} = 1) thanks to the latter, only those detected disks that support SMART will be added. Now we can rewrite our two data elements for sda and sdb disks into one prototype of data elements, simply replacing the disk name with the macro {#DISKNAME}:



Last, before Zabbix can run the commands that we have registered in zabbix_agentd.conf as root and monitor SMART, you need to add permissions for its user to run this command without entering a password, for this we add the following line to / etc / sudoers:

 zabbix ALL= (ALL) NOPASSWD: /usr/sbin/smartctl,/usr/local/bin/smartctl-disks-discovery.pl 

A ready-made template for monitoring SMART with the rest of the data elements, I attach triggers, as well as the config file configured for it.

Monitoring the installation of new programs on Windows

Zabbix agent installed on Windows can also be extended via UserParameter , but the commands will be different. Although, for example, smartctl is a cross-platform utility, and you can also use it to control hard drives in Windows.

Briefly consider another example. The task is to receive a notification each time a user uninstalls or installs programs.
For this we will use our vbs-script:



To integrate it with Zabbix, add a UserParameter to the config file:

 UserParameter=uDiffPrograms, cscript.exe /nologo "C:\zabbix\uDiffPrograms.vbs" & type C:\zabbix\uDiffPrograms_temp.txt 

Add a data item to the template for Windows:



Add a trigger :



and the action that will send an email notification:



The whole monitoring process looks like this: every hour the Zabbix script is started by the agent that compares two lists of programs: the current and the previous one. Then the script writes all the changes to a separate file. If there are no changes, then 0x0 is written to the file.

The content of the file goes to the Zabbix server, where the trigger is raised if the value of the uDiffProgramms data element is different from 0x0. Then a separate action sends a notification by mail with a list of what was installed or deleted on this computer:

Eventually

UserParameter is a great and easy way to expand the functionality of the system yourself. It is worth mentioning the alternatives: zabbix_sender, which, for example, is suitable for those cases when you need to send data to Zabbix not on a schedule, (as the UserParameter does), but on some event; and system.run [] , which is similar to UserParameter , but it is more convenient because you do not need to make changes to all configs of agents, it is enough just to add this data element to the template. Moreover, in the next major release of Zabbix 2.2, another new way of expanding agent capabilities awaits us — plug-ins . Look forward to!

So, consider that if you can learn something about the system with a script or a command, it means that you can always transfer it to Zabbix.