How to teach Zabbix to send problem notifications directly to the desktop

Picture: source

Hi, Habr! My name is Ilya Ableev, I work in the monitoring department of Badoo.

You can take a long time on the topic of which monitoring system is cooler. The main tool for reacting to problems in Badoo is Zabbix, and we have repeatedly shared our experience in customizing it.
')
"Out of the box" to learn about new problems can be either through the web interface, or by mail / SMS. It works well and until a certain moment we were satisfied with these methods.

Today, there are several server instances in our system, thousands of hosts, hundreds of thousands of checks, and about the same number of triggers that need to be addressed as quickly as possible. After a series of incidents, when we only spent up to ten minutes on finding a problem (which is completely unacceptable), we realized that we needed other ways to report problems. Then, perhaps, our reaction would be faster.

The best solution, in our opinion, would be to display important notifications on top of all windows (browsers, chats, consoles). In this article we will talk about how we finished Zabbix, teaching them to send them directly to the desktop.

To suit this solution:

duty administrators, who not only look at the monitor around the clock, but are also involved in other tasks;
all other administrators who do not like to look through the PROBLEM sheet and OK in mail or SMS, but who want to know about critical issues in a timely manner.

We'll have to spend time installing, but I think if you are using Zabbix and have already used custom scripts, it will not be difficult for you.

Attention! Receiving notifications in this way is addictive, its long-term use creates the impression that it is an integral part of Zabbix.

I'll tell you in steps what and how we did. The technical implementation is quite simple:

Zabbix sends an alert for an event.
The script sender on the server sends data to the client on the desktop via UDP.
The recipient script receives the notification and triggers the appearance of a pop-up message through an additional application.

The third item may differ depending on which operating system you are using and which application you like best.

Step One: Configure Zabbix

The official documentation details how to create your own alerts .

The project code is available on GitHub: https://github.com/ableev/ZbxDsktp .

Beforehand we will put zbxpush.py in AlertScriptsPath on the Zabbix server.
Next in the web interface: Administration → Media types → Create .

3. We prescribe the IP / FQDN in advance, to which we will send alerts.

This item can be upgraded.

For example, we have done this: there is a main monitoring server, with an SSH login, the script remembers the incoming IP and starts sending alerts to it. Thus, in order to activate data acquisition in your home, you do not need to change the action every time.

In the web interface: Administration → Users → % username% → Media .

4. Create an action on the triggers.

In the web interface: Configuration → Actions .

Send format

{TRIGGER.NSEVERITY}@@@{HOST.NAME}@@@{TRIGGER.NAME}

The first is the numeric format of the trigger (4 or 5 affects the displayed icon: High or Disaster), the second the notification header, and the third the message body.

@@@ - so that it is convenient to break into columns, because anything can be written in the names of the triggers (at least, we have :)), but not exactly this combination.

Step two: the server sends data to the client

The logic is simple: we receive an event, send it to the client.

zbxpush.py

Here you can change UDP to TCP. We chose UDP for one simple reason: Zabbix sends all notifications sequentially, which means that if your computer is not available, if you use TCP, notifications will be sent with a long delay.

But UDP is unreliable.

The reader, who comes to a joke about UDP, will notice that in the case of sending alerts via UDP, there is a probability of losing them somewhere between the server and the desktop. And he will be right. But the point is not to deliver the message guaranteed, but to simply and quickly draw attention to the dashboards with problems: “Hey, something happened there, go and see.”

Do not forget to make the script executable!

Step three: receive the event and display the notification

zbxlistenerd.py - a script that will spin in the background and launch notifications

settings.cfg - client settings.cfg file

icons/5.png , icons/4.png - icons for different criticalities of triggers
Further our paths diverge. The “client” part involves a bit of creativity, as each creates an environment for himself.

What is necessary:

icons corresponding to the critical points of the triggers;
sound of pop-up notifications;
performing an action by clicking on a notification (SSH login, opening a link, initiating a ticket about a problem, etc.).

In our company, Linux and Mac OS are mainly used as the desktop OS, so let's look at a few examples for them. If you implement the same for Windows, welcome to the comments! We are happy to complement the post.

Linux

I will give an example that happened to use: notify-send in Xfce (in the screenshot above). Why notify-send, but not the same Python using libnotify? Yes, because it is in all popular desktop distributions (Ubuntu, Fedora, SUSE) and works with all (at least popular) DE (Gnome, KDE, Xfce).

Advice: if you experience periods of mass unavailability of something (for example, the switch fell off - and you are overwhelmed with a mountain of triggers about server unavailability), immediately configure the hotkey to complete the notify-send process.

Mac OS

Since the solution was initially implemented on the basis of a free application, and then we tried a paid one, there will be two examples.

terminal-notifier

After a brief search, a free terminal-notifier was chosen.

Pros:

You can set different icons for different criticalities;
You can create links to graphics in Zabbix and open them by clicking on the notification;
You can implement the opening of SSH-links: saw the notification -> clicked -> immediately went to the host.

Disadvantages: none (especially after icon support was added).

Growl

To enable it, set growl_enabled = True in settings.cfg.

At the time when the terminal-notifier did not know how to use custom icons, Growl did quite well with this. Therefore, our curiosity won the greed - and we tried this quite popular application (which, I must say, can not only change icons - this is the whole center of notifications).

Pros:

you may already have the application installed;
You can select different icons for different criticalities;
as well as in the terminal-notifier, you can customize actions by clicking on notifications.

Minus: paid.

If you don’t have Growl, I recommend using terminal-notifier.

Conclusion

~~So, with the help of simple devices ...~~

What we got in the end:

fast delivery of notifications to admins;
a decrease in the reaction time to events (from the moment “learned” to “repaired” or “passed on”);
the ability to perform simple semi-automatic actions:

Further, it all depends on your creative potential.

Once upon a time, when I was working in a company provider, when the Internet channel fell from all the speakers and headphones in the workplace, the sound of a siren could be heard. This led the attendant (me) to a stupor, did not adequately report the problem by telephone, but he (I) tried to solve this problem as soon as possible.

Personally, I already know cases of screwing to a script the sound of a pig from a popular antivirus and a talker from a popular translator. :)

With free time, desire and modern technology can be done

Quick to you reactions to incidents!

Source: https://habr.com/ru/post/418907/

All Articles