For some reason, they write about icinga2 surprisingly little, and what they write somehow does not create a common concept. In one place they write how to write a script, in another how to establish the whole thing, and what to do with it later is not clear.

I myself have been using icing somewhere since the beginning of 2013, then there was only the first version and it was very far from Nagios, it did not go away. With the release of the second version, a lot has changed and for me the choice is obvious.

Then I want to tell you how to quickly install all this and, more interestingly, what to do next.

Little description

Icinga2 is needed to monitor the status of servers, services, printers, routers and everything else where there is Linux or Windows or even where not.
')
If the first version worked on the kernel from Nagios, then in the second version everything was done in a new way so that it was fast and beautiful. To do this, the icinga team had to rewrite the kernel and now, without much effort, you can do either a distributed system or the same thing, but add a cluster to this as well.

But this is done for large networks (over 10k in one segment). When the second version was just coming out, I read that one server could handle up to 10k hits per second on their tests, I don’t keep so many servers, so I can’t confirm.

Beginning and slightly theory

In 2013, we at our company decided to make a system for monitoring and install from our clients, taking money from them for services in case of any problems with their systems, the question arose how to actually get on their systems.

Icinga has two ways to check the status of systems - either actively accessing the servers by knocking on any port, or open a port and wait for something to come there. Accordingly, it is called active and passive methods. In most cases, the active method is used, although I don’t understand why. Also on the Internet is found mainly description of the active survey.

Active mode

There are several options, you can use icinga2 director and client for windows or Linux. After installing them, you need to write configs or templates for icinga2. In this case, icinga2 accesses the server and performs a check through the client or somehow an arbitrary script. Previously, NRPE was used for this, as it happens now I do not know.

The active method has two big drawbacks, which actually decided everything:

1. It is necessary on the server side to open ports, and if it is a different network, then configure port forwarding, and for each server its own.

2. Setting up for each system is more complicated than in the passive version, since you have to prescribe commands for each case, well, or use ready-made templates, which of course must have been done before.

Passive mode

In this case, we open port 5667 for NSCA on the side of icinga2 server or turn on Icinga2 APi (port 5665) and just wait until we get the status of a system, while icinga is watching when the status has been updated the last time and if the status is not updated certain time, then icinga marks it as unknown. And of course, somewhere else without a spoon of tar - if you need to check a router or a printer, or somewhere to poll SNMP, then with passive polls this cannot be done. But here you can write a crutch, one of the servers in the same network can poll via SNMP and send status to icinga, but for this you need to write a script.

NSCA is already in some measure an outdated solution, but for Windows systems, there is not yet a good alternative (so that you don’t have to do a lot with your hands). In Linux, I have already written a number of scripts and they turn to Icinga2 Api. Most recently, I also wrote the library for node.js (link below).

Installation ()

As was already clear above, I want to install Icinga2 and configure it in passive mode.
For Icinga2 I made a container (docker).

What is in the container:

1. Icinga Web 2 - a web interface where you can view status
2. Graphite - to display beautiful graphics
3. Icinga2 Classic UI - also gui, but from the first version, so that you can use old programs like Nagstamon (for Windows) or adagios (for android)
4. = Integration with AD, but it turned out that we do not need it, so I have not checked it for a long time, it may turn out that it does not work.
5. NSCA Server - needed to receive metrics from Windows or from those who use nsca
6. Icinga2 API - api from icinga. You can do everything - set status, add / delete hosts and services, etc.

For those who have never used a docker, I’m showing how to install all this on ubuntu (16.04). First, we install the docker, and for convenience, you can also docker-compose.

sudo apt install docker.io docker-compose -y 

Then we start the container with Icinga2 without Active Directory:

 sudo docker run -i -p 80:80 -p 5667:5667 -p 5665:5665 -p 8080:8080 -h monitoring.example.com \ -v /storage/icingaweb2:/icingaweb2 -v /storage/icinga2:/icinga2conf -v /storage/mysql:/mysql \ -v /storage/graphite:/var/lib/graphite/whisper \ -e NOTIFICATION_INTERVAL=0 -e GRAPHITE_HOST=192.168.42.64:8080 \ -e APIUSER=root -e APIPASS=PASS -e ICINGA_PASS="icinga" \ -e MAILSERVER="mail.example.com" -e EMAILADDR="user@example.com" -e NSCAPASS="pass" -e NSCAPORT="5667" \ --name icinga2 -t adito/icinga2 

In the description on hub.docker.com there is a launch option with AD, you can see it here .

A bit of description for variables and the rest.

Ports:
80 - this is understandable, then you can watch everything
5667 - NSCA port, maybe not everyone needs it
5665 - Icinga2 API
8080 - The graphite page must be open, as icinga2 refers to it to draw graphs.

Variables:

“-H” is the host name, then displayed on the page
“-V / storage ...” - configs and db are stored
“GRAPHITE_HOST” is the ip of the server where the container works, through this address icinga2 refers to the graphite
“APIUSER” is clear and true. It is advisable to change to something like “0ilkasjdf09123malskdf”
“APIPASS” is clear too
“ICINGA_PASS” - password for the user “icingaadmin”
“MAILSERVER” is the host where you have a mail server, I have exchange on the same network, so it just works as a relay
“EMAILADDR” is an address that uses icinga2.
“NSCAPASS” and “NSCAPORT” I think is understandable.

After that you have a working monitoring server that monitors itself. You can go to see what is there and how.

Available at GRAPHITE_HOST: 80

There is also an old version by GRAPHITE_HOST / icinga2-classicui. This version can be used to connect to the anag example and watch the status through the phone. Or there is still a nagstamon for the computer, it seems to be able to handle the Icinga2 API, there is an option, but I didn’t watch much.

I say at once that it is better to use them than to look at emails, because there will be a lot of them, or all this should be very finely tuned.

If someone does not want to work with the container, then you can take the script (need icinga2.sh) install from the git repository and run it, also should work, just need to correct the passwords, for something more reliable.

Now you need to add two templates to the container, one for the host and one for the services. They should be put in the / storage / icinga2 folder that we set at the start of the container.

Host - passive-host.conf - can be called whatever you want, but should end in .conf

 template Host "passive-host" { max_check_attempts = 2 check_interval = 300s retry_interval = 200s enable_active_checks = true enable_passive_checks = true check_command = "passive" vars.notification["mail"] = { groups = [ "icingaadmins" ] } } 

Service:

 template Service "passive-service" { max_check_attempts = 2 check_interval = 300s retry_interval = 200s enable_active_checks = true check_command = "passive" vars.notification["mail"] = { groups = [ "icingaadmins" ] } } 

A small description of the patterns

Waiting for the status to come two times, if both show an error, then the status changes to an error.

enable_active_checks = true is needed so that if the status is not received from the server, change it to “unknown”.

If two times the wrong result, then sends an email. By the way, from experience, I can say that letters flow without interruption and very much, especially at the beginning and it is also not always clear from them which service has already started working and which one does not. Therefore, I advise you to use nagstamon or always look at the village.

I used to remember some letters sent immediately to OTRS, which created the ticket, was also convenient and the client saw everything at once.

These two files should be placed in the folder “/ storage / icinga2” and restart the container, or the icinga service in it.

 sudo docker exec -it icinga2 service icinga2 restart 

Everything, templates are added, now it is possible to add systems and all that is necessary.

What's next?

In principle, before that, there was nothing new, now I will show why the passive version is better.

Showing

For windows

For Windows, there is a ready client - nsclient ++ . It can work in both versions (passive and active) while passive it sends data via NSCA, this service is active in the Icinga2 container and can be used, you need to use it through port 5667. For Windows, this is a good option, since there is already everything ready You can check the status of services, watch eventlog and, well, things like disk, processor or memory. Also, if you wish, you can execute arbitrary scripts (you must first write them. For example, I wrote for lsi raid) whose output can be sent via nsca.

I already wrote above that we wanted to install icinga with clients, but writing configs for nsclient ++ is quite tedious and they need to be written for each server and, moreover, on the side of icinga you also need to write configs.

I simplified all this and wrote a small program (then I could only AutoIt), which can generate these two files for the host and icinga. The program is HERE (you can download it in releases), called “Agen” (Alerts Generator).

Once unpacked, you need to fix the config.xml

 <?xml version="1.0" encoding="UTF-16"?> <CONFIG> <monserver> <interval>2</interval> <adresse>monitoring.server.local</adresse> <password>NSCAPASS</password> </monserver> <konfiguration> <htemplate>passive-host</htemplate> <stemplate>passive-service</stemplate> <hgruppe>HOSTGROUP</hgruppe> <sgruppe>SERVICEGROUP</sgruppe> </konfiguration> </CONFIG> 

interval - check time in minutes
address - dns or ip icinga server
password - NSCA password (which was written above when creating the container)
htemplate - the name of the template for the host (it was already higher)
stemplate - the name of the template for services
hgroup is a group of hosts, if for example you want to share them. For example, Hosting, DataCenter2, etc., or for different clients, as was the case in my case. You can by the way then bind the user to one group
sgroup - same as above, only for services

Now you can run Agen and select the folder where source.txt and config.xml are located. It looks like this:



Some of the information is taken from congig.xml, you just need to add “Host Alias” and “Host Display Name” (will be displayed on the page).

The information on the left is taken from source.txt. How to use:

On the left, choose what works for you on this server. For example, choose “winExchange2013” ​​and “eveExchange2013”, write Host Alias ​​(without spaces) and Host Display Name (with spaces), if you need the configuration for icinga for this, then tick. After these manipulations, two files will appear:

hostalias.conf - it must be copied to the container (/ storage / icinga2) and restart icinga
nsclient.ini - copy to the folder where nsclient ++ is installed

What is in source.txt. Actually the whole Agen revolves around this file, everything is divided into two parts:
“Win” - windows services
“Eve” - events from eventlog

That is, if you select “winExchange2013”, then all services related to Exchange 2013 will be added to the configs and will be monitored. If you also choose “EveID Exchange 2013” ​​to this, then events from the eventlog will also be added to this.

That is, after you have done this and clicked OK, you will have two files “nsclient.ini” and “SERVER.conf”. After that, you need to put nsclient.ini in the folder where nsclient ++ is installed, and SERVER.conf add to the container (as when creating passive-host.conf and passive-service.conf). And after that you need to restart everything - restart the “nsca” service, so you also need to restart the icinga2 service.

The only point is that services in source.txt are called in German, so they will have to be renamed.

Example:

MSExchangeADTopology: Windows_Dienst_Ex_AD_Topology
MSExchangeADTopology is the same everywhere, regardless of the system language, but the second part is different everywhere.

Actually about this video all.

Linux

It's all a bit more complicated, since there is no client like nsclient ++. There are three ways, you can set “send_nsca”, then write a script and send the status through “send_nsca”, the scripts will be executed every two minutes by krone.

The second way is to send the same data via icinga2 api. I use the second option, for this I use node.js and for it I made a module, it lies on npmjs.com.

The third way is to use ready-made scripts and still set up active polls. But here I will not tell you, because I try to avoid it all the same.

Despite the fact that I practically use only Linux to write scripts for Linux, I practically don’t have to, since everything works with us through the docker.

For the docker, I also made a container that can see (through docker.sock) how many containers there are on the host. It can also create a host for each container in icinga2 and then monitor it. That is, a dynamic monitoring is obtained. If the container is removed from the docker of the host, then it is also removed from the monitoring.

Conclusion

In principle, everything is already said above, I can only add that if someone decides what to use, then he can try. Docker container greatly simplifies the installation process and is very well suited to try.

By itself, icinga2 is quite easy to set up and after a while it becomes clear. For myself, I did a few a few things (which I described above) to make it easier to use. Also for those who can use nodejs can write scripts for themselves, the link below has a library for this.

I especially do not like to delve into the technical parts, everything is trivially googled and located.

References:

1. Icinga2 Docker - container with the image of icinga2 2.6
1a Git repository , if someone needs only icinga2 installation scripts.
2. Agen - configuration generator for nsclient ++
3. Nsclient ++ - client for windows
4. dockerhost-monitoring - a container to monitor the status of all containers on a separate docker host.
5. Docker Container NSCA monitoring - the old container monitoring option (via nsca)
6. Icinga2 nodejs API - module / library for nodejs
7. aditosnmp - snmp polling example. Here is the moment when you have to poll the status via snmp, and then send the status. You need to watch "app.js"

And a little off topic, container for backup containers and not only - nodebackup . It's a pity that I poured it onto the githab, and no one knows.