Monitoring Java applications in Zabbix, customizing JavaGateway for JMX LLD

Introduction

In this article I will explain how you can customize Zabbix JavaGateway a little for the most convenient low-level detection of JMX metrics. Here, github.com/mfocuz/zabbix_plugins/tree/master/jmx_discovery can take a patch to version 2.0.11 and see examples of external scripts. But first things first.



Since version 2.0, Zabbix has added native support for monitoring Java applications via JMX . But maybe not everyone knows that, apart from collecting metrics, we can also discover them in Zabbix out of the box. In the documentation, either they missed this moment, or considered the feature not quite ready (although maybe I just didn’t find it in the dock?), But this feature is there, and, in my opinion, it is really not quite ready. Although I am not sure that it works at all, it did not come to testing.



In the zabbix_javagw source in the JMXItemChecker.java class for version 2.0.11, line 157 and on:

else if (item.getKeyId().equals("jmx.discovery”)) { … for (ObjectName name : mbsc.queryNames(null, null)) { logger.trace("discovered object '{}'", name); for (MBeanAttributeInfo attrInfo : mbsc.getMBeanInfo(name).getAttributes()) { logger.trace("discovered attribute '{}'", attrInfo.getName()); if (!attrInfo.isReadable()) { logger.trace("attribute not readable, skipping"); continue; } try { logger.trace("looking for attributes of primitive types"); String descr = (attrInfo.getName().equals(attrInfo.getDescription()) ? null : attrInfo.getDescription()); findPrimitiveAttributes(counters, name, descr, attrInfo.getName(), mbsc.getAttribute(name, attrInfo.getName())); } catch (Exception e) { Object[] logInfo = {name, attrInfo.getName(), e}; logger.trace("processing '{},{}' failed", logInfo); } } } } 

')

As can be seen from the code, if in the configuration of the discovery rules set the key with the name “jmx.discovery” , then the detection will work, which will return all the attributes for all found mbeans .

Here, first of all, I would like to note that there is no special need for such a request. Firstly, such a request does not work very quickly, because a request for the attributes of each mbina passes by a separate request, and there can be quite a lot of bins. Secondly, the discovery key of the rule is a unique value in Zabbix, which means that using jmx.discovery we can create one single discovery rule for the JMX metrics within the same host, which is completely inappropriate for most tasks. And thirdly, the attributes for each java mbean usually do not change, that is, their number and designation within the same bean is constant. This means that the attributes themselves are better to be set in item prototypes and a part of the code, which is actually the most brake:

  for (MBeanAttributeInfo attrInfo : mbsc.getMBeanInfo(name).getAttributes()) { ....... try { (attrInfo.getName().equals(attrInfo.getDescription()) ? null : attrInfo.getDescription()); findPrimitiveAttributes(counters, name, descr, attrInfo.getName(), mbsc.getAttribute(name, attrInfo.getName())); ...... 

becomes not needed. But the number of mbeans can be variable. For example, Oracle Coherence creates a certain number of bins for each node in the cluster. As a result, we have bins whose name differs only in nodeId . And for example, when restarting a cluster, all nodeId changes. That is, Java metrics have dynamic names mbeans . This is where we use LLD , but:



1. We need only mbeans themselves, without attributes (you can find them too, but IMHO is superfluous).

2. It is important for us to be able to create several discovery rules for one host.

3. We need the ability to encode a variety of logic for discovery rules.



Not so long ago, one article appeared on JMX Discovery www.zabbix.org/wiki/Docs/howto/jmx_discovery . Last year I used a similar solution. In short: the essence of this solution is to run the .jar file as an external script. Its great disadvantage is that Java is not designed to run multiple times, as is usually done for interpreted languages ​​like python or perl. This minus makes the solution practically non-working. For example, on a virtual machine with 2 cores to run about 20 of the rules discovery, the CPU load went off the shelf, as a result of external checks just fell off on time out. On the glands where it was at 8k seems to have worked properly. But vseravno, launching a pack of JVM once every N minutes is not a beautiful solution and can only be considered as a crutch.

How the new functionality will work

To begin, consider how Zabbix server and JavaGateway interact. To collect JMX metrics, Zabbix server connects to JavaGateway via TCP, and then makes a request for the necessary data via Zabbix protocol. On of the site there is a general description of the protocol, I will describe it a bit more closely. We need him for the LLD .

The message to the server and the response from it consists of 3 parts:



1. Title ZBXD \ 1

2. Message length

3. Message in JSON format.



It looks like this:







I wanted to make a minimum of changes in the Zabbix code, so we don’t touch the Zabbix server code, in the scheme above we replace the Zabbix server with an externalscript that will make the same connection. And in the JavaGateway code, we add the function we need. And now communication with JavaGateway for LLD looks like this:







That is, the regexp field and the new type request = “java jmx lld” was added.

Change JavaGateway code

What needs to be done to make JMX Discovery convenient and fast? As conceived by JavaGateway developers, there are two possible request requests, they can be found in the ItemChecker.java code, these are JSON_REQUEST_INTERNAL constants for collecting a pair of internal JavaGateway metrics, and JSON_REQUEST_JMX — which is the main request and serves for collecting JMX metrics. In the SocketProcessor.java code we see:

 if (request.getString(ItemChecker.JSON_TAG_REQUEST).equals(ItemChecker.JSON_REQUEST_INTERNAL)) checker = new InternalItemChecker(request); else if (request.getString(ItemChecker.JSON_TAG_REQUEST).equals(ItemChecker.JSON_REQUEST_JMX)) checker = new JMXItemChecker(request); ….. JSONArray values = checker.getValues(); 

That is, what type of checker will be determined here. We will add the 3rd type of query - JSON_JMX_LLD . And there is one more condition for our request in SocketProcessor.java:

 …... else if (request.getString(ItemChecker.JSON_TAG_REQUEST).equals(ItemChecker.JSON_JMX_LLD)) checker = new JMXItemDiscoverer(request); …… JSONArray values = checker.getValues(); 

Now, when the server receives a request for JSON_JMX_LLD , an instance of the JMXItemDiscoverer class will be created and the getValues method will be called . It remains to add the JMXItemDiscoverer class, which will make the discovery rule as it would be convenient for us, namely, it will make a request for all available bins and will return the list of bins at the output according to a specified regexp. The code for the new class can be seen in the patch.

Setting up a new Java gateway

There are two options, you can replace JavaGateway itself, and you can raise another one, which will only work for LLD. If I change the gateway to PRO, then:

1. On the Zabbix side, you only need to change JavaGateway. Under the link at the beginning of the article, you can find a patch on zabbix java gateway version 2.0.11. I used a little diff on code 2.2 and 2.0.11, so the java gateway is not much different there, so I think with basic knowledge of Java it will not be difficult to transfer the patch to the latest version.

2. Next we roll the patch and build JavaGateway for the installed version of Java.

3. We get the .jar file that we receive in place of the old one, all other java gateway files including the config file, or leave it as it is.

4. We start. We get the same Java gateway, but now it can handle another type of requests.

5. For the requests themselves, we write a script that will cling to the server via TCP and actually make the request itself. I sketched a simple JMXDiscovery.pm module for Perl to make it easier to write a discovery script plus an example discovery of the jmx_discovery.pl script and the use of the module. (You can also find it at the link in the beginning).

6. And finally, create discovery rules, type externalscript in the type. The parameters passed to the script will ensure the uniqueness of the key, which will allow you to create any number of rules.



If you want to raise a separate server for LLD , then skip step 3 and after the assembly, along with the patch, simply install JavaGateway as a separate service.

Conclusion

It works fast and stable.