Automatic monitoring of newly installed software in ZABBIX
ZABBIX has an excellent mechanism that allows you to automatically detect and monitor file systems, network interfaces, CPUs, CPU cores and other objects. But unfortunately, he can't do the same with the software out of the box.
With just a couple of scripts, one of which must be put on the server, and the second scattered across clients, you can do low-level auto-detection of nginx, mongod, rabbitmq, mysql, postgresql and any other service.
Of course, my version of this implementation is not without flaws and most likely the guru will shower me with tomatoes! I would be extremely grateful to constructive criticism and advice.
Hanging on the host template auto detection "Template service auto discovery" in the template or already on the host in the macro "{$ SERVICES}" add a list of services (separated by spaces) that need to be detected and put on monitoring.
')
If the required service is installed and started, for example, “nginx”, then the data element “SERVICE_AUTO_DISCOVERY: detected_and_run_nginx” appears on the host
and the SERVICE_AUTO_DISCOVERY trigger: trigger_detected_and_run_nginx,
after 30 seconds, the trigger is triggered, and on the triggered trigger, the action is started,
the action is performed by the “auto-discovery.py” script (do not forget to change the login “zabbixapi_API_user” and the password “password” to your own) on the zabbix server.
The script in turn through the API-zabbix disables the corresponding trigger and hangs the template required for this service.
It would be possible to directly contact the API directly from the client, but then you have to keep this script on the client. It is not very safe. the API password will have to be kept on the host i. in the script. By the way, we have zabbix server for NAT.
It was decided that we will discover the running software, because there is no point in putting on monitoring what is not working (a lot of alerts, panic, calls, etc.).
You can use absolutely any template to monitor any service. To do this, you need to rename the template by giving it the name of the form:
"Template_ <service name from macro>".
That is, to monitor the mongo database, you must add the mongod to the macro "{$ SERVICES}", and rename the template for the mongo monitoring to Template_mongod.
The scripts and templates themselves are on a githaba in the autodiscovery folder.
With just a couple of scripts, one of which must be put on the server, and the second scattered across clients, you can do low-level auto-detection of nginx, mongod, rabbitmq, mysql, postgresql and any other service.
Of course, my version of this implementation is not without flaws and most likely the guru will shower me with tomatoes! I would be extremely grateful to constructive criticism and advice.
The course of action, the description of the functional and principle of operation
Hanging on the host template auto detection "Template service auto discovery" in the template or already on the host in the macro "{$ SERVICES}" add a list of services (separated by spaces) that need to be detected and put on monitoring.
')
picture with a list in a macro
If the required service is installed and started, for example, “nginx”, then the data element “SERVICE_AUTO_DISCOVERY: detected_and_run_nginx” appears on the host
and the SERVICE_AUTO_DISCOVERY trigger: trigger_detected_and_run_nginx,
after 30 seconds, the trigger is triggered, and on the triggered trigger, the action is started,
action picture
the action is performed by the “auto-discovery.py” script (do not forget to change the login “zabbixapi_API_user” and the password “password” to your own) on the zabbix server.
picture describing the operation when the action is triggered
The script in turn through the API-zabbix disables the corresponding trigger and hangs the template required for this service.
Triggered triggers
It would be possible to directly contact the API directly from the client, but then you have to keep this script on the client. It is not very safe. the API password will have to be kept on the host i. in the script. By the way, we have zabbix server for NAT.
It was decided that we will discover the running software, because there is no point in putting on monitoring what is not working (a lot of alerts, panic, calls, etc.).
Summing up:
You can use absolutely any template to monitor any service. To do this, you need to rename the template by giving it the name of the form:
"Template_ <service name from macro>".
That is, to monitor the mongo database, you must add the mongod to the macro "{$ SERVICES}", and rename the template for the mongo monitoring to Template_mongod.
PS
Please pay attention to the underscore in the template name, it is with it that the script searches for a suitable template. The name of the template and the name of the trigger must be "absolute" the same after "_" otherwise either the template is not thrown or the trigger does not work.
Pattern Examples
The scripts and templates themselves are on a githaba in the autodiscovery folder.
Advanced Host Script Functionality
The script on the host was trained to detect systemctl sevises. Developers in our company often write their services that need to be monitored off / on. Auto-discovery of these services occurs if the service is "is-enabled".
The macro is called {$ SERVICE}, you can omit the ".service" in the service names, and also list them separated by spaces.
Please note that there are no templates here, but a data element and a trigger necessary for monitoring the service are being created.
The macro is called {$ SERVICE}, you can omit the ".service" in the service names, and also list them separated by spaces.
Please note that there are no templates here, but a data element and a trigger necessary for monitoring the service are being created.
A little about debag:
On the host, check how the auto-detection script handles:
Ie, if the service is not found, the script will return an empty JSON.
Most likely the script cannot connect to the zabbix API. Checking what is happening:
From the side of the zabbix server, run the script and pass two parameters to it? the first is the name of the host, and the second is the name of the trigger. The host name as the name of the trigger is taken from the webmord zabbixa, the name of the trigger is taken from the name of the trigger in the host.
If the service is not detected, i.e. no triggers are created:
On the host, check how the auto-detection script handles:
/etc/zabbix/scripts/run_service.sh mongo {"data":[{"{#SERVICE}":"mongo"}]}root@bla_bla_bla:/tmp# _______________________________________________________________________ /etc/zabbix/scripts/run_service.sh mongo nginx supervisor {"data":[{"{#SERVICE}":"mongo"},{"{#SERVICE}":"nginx"},{"{#SERVICE}":"supervisor"}]}root@bla_bla_bla:/tmp# _______________________________________________________________________ /etc/zabbix/scripts/run_service.sh mongodb {"data":[]}root@bla_bla_bla:/tmp# Ie, if the service is not found, the script will return an empty JSON.
If the service is found, but the triggers are not disabled:
Most likely the script cannot connect to the zabbix API. Checking what is happening:
From the side of the zabbix server, run the script and pass two parameters to it? the first is the name of the host, and the second is the name of the trigger. The host name as the name of the trigger is taken from the webmord zabbixa, the name of the trigger is taken from the name of the trigger in the host.
/usr/bin/python3 /lib/zabbix/externalscripts/api/auto-discovery.py {HOST.NAME} {TRIGGER.NAME} Chicken or egg?
The next stage I think is to finish the integration with Ansible:
But then the question arises: What should be the first zabbix or ansible?
If zabbix, then erroneous actions in the monitoring system will lead to unnecessary installation of unnecessary software.
If the first is ansible, then its integration with zabbix is ​​superfluous, because zabbix will detect and monitor everything, and throw the scripts and configs necessary for the zabbix-agent during the expansion of the playbook.
There is a third option, when throwing a template with auto-detection (in which standard services are listed in the macro) to the host, zabbiz will simultaneously launch a playbook for deploying scripts and configs for zabbix-agent. But again, if the services are standard then on the host, at least when deploying these standard services, it is necessary to deploy the role for monitoring.
But then the question arises: What should be the first zabbix or ansible?
If zabbix, then erroneous actions in the monitoring system will lead to unnecessary installation of unnecessary software.
If the first is ansible, then its integration with zabbix is ​​superfluous, because zabbix will detect and monitor everything, and throw the scripts and configs necessary for the zabbix-agent during the expansion of the playbook.
There is a third option, when throwing a template with auto-detection (in which standard services are listed in the macro) to the host, zabbiz will simultaneously launch a playbook for deploying scripts and configs for zabbix-agent. But again, if the services are standard then on the host, at least when deploying these standard services, it is necessary to deploy the role for monitoring.
Source: https://habr.com/ru/post/347734/
All Articles