Geekly Articles each Day
📜 ⬆️ ⬇️

On-the-Knee Monitoring - Using Cacti to Control Java Applications

The article describes a monitoring solution using Cacti using the example of analyzing and controlling the resource consumption of a large Java application.

I was faced with the task - in a short time to propose measures to stabilize a large three-tier Java application that has problems with memory consumption and performance. Time, as usual, is not enough: 1-2 weeks for everything. The company lacked a suitable application monitoring infrastructure, and it was not my task to create it. The JConsole option was not suitable because of the need to analyze consumption over a long time and watch it after possible sudden reloads of applications.

In one of the companies where I worked, an impressively convenient and simple solution was implemented for monitoring Java applications based on the RRD Tool . It consisted of a simple add-on on perl-scripts that ensure the collection and display of data through HTTP and a number of improvements to the data collection agents in the application itself. For me, this was the idea of ​​a solution, however, I did not have time to write the strapping over RRD.
')
After a careful search, I found a free tool that implements the add-on I need - Cacti. Cacti is an application written in the Apache-PHP-MySql framework that allows you to customize the collection and display of monitoring data based on the web interface. It turned out to be easy to deal with it, a couple of days for raising the infrastructure, then setting up and adding data collection agents and that's it.

The rest of the article describes in detail the solution that allowed me to solve my problem and, in the end, to successfully stabilize the application in the company.



What you need to work:

(the versions on which the monitoring was launched are indicated, any working Apache-PHP-MySql bundle will do)

I will describe the principle of the monitoring operation on Cacti: using the “assigned tasks” of Windows (or cron in unix), the polling of data collection agents is performed, which in my case were: the JVM itself (memory consumption) and specialized updates of the application. The collected information is placed in the RRD database - i.e. in circular buffers as files. Further, the accumulated data from the RRD can be viewed through the Cacti web interface at various scales in the context of minutes, hours, days, months, etc.

The work plan is to raise all the necessary infrastructure, adapt the analyzed applications to collect data and configure the collection and output of data in Cacti.

Infrastructure setup

Php


Add the path to the php.exe in the variable PATH, the same path is written in the variable PHPRC

Copy the php.ini-production file in php.ini and make the following changes to php.ini:

Uncomment the lines:
 extension_dir = c: \ php \ ext
 extension = php_mysql.dll
 extension = php_snmp.dll
 extension = php_sockets.dll
 cgi.force_redirect = 0
 date.timezone = "Europe / Moscow"


Apache

Add the following lines to conf \ httpd.conf:
 LoadModule php5_module c: \ php \ php5apache2_2.dll
 AddType application / x-httpd-php .php
 DirectoryIndex index.html index.htm index.php


MySql

Add the path to mysql.exe in the variable PATH

Creating a cacti schema:
 mysql --user = root --password create cacti


Import the cacti schema data structure
 mysql --user = root --password cacti <c: \ apache2 \ htdocs \ cacti \ cacti.sql


Create user cactiuser:
 mysql --user = root --password


Next, on the MySql command line:
 mysql> create user cactiuser @ localhost IDENTIFIED BY 'cactiuser'
 mysql> GRANT ALL ON cacti. * TO cactiuser @ localhost;
 mysql> flush privileges;


For a quick test of the php-mysql-apache bundle, I used the following php script:
 <? php
 mysql_connect ("localhost", "cactiuser", "cactiuser") or die ("Can't connect");
 mysql_query ("USE cacti") or die ("Can't select mysql database");
 echo "Success \ n";
 ?>


It should be put into a file with a name, for example, testphp.php, copied to the Apache htdocs / directory and load the localhost: 8080 / testphp.php page. The message “Success” should appear.

Cacti Setup


Unzip the cacti distribution in the Apache / htdocs directory.
Check that the file cacti /include/config.php contains the following lines:

 $ database_default = "cacti";
 $ database_hostname = "localhost";
 $ database_username = "cactiuser";
 $ database_password = "cactiuser";
 $ database_port = "3306";


Go to the address localhost: 8080 / cacti / under the login admin / admin.

In the Settings-> Paths settings, specify the paths to external utilities (it is recommended to use Unix-style paths, for example, c: /php/php.exe).

Configure the launch of the command "php cacti / poller.php" every 5 minutes (via Windows Scheduled Tasks). I have used the batch file for this:
 start / MIN php.exe cacti \ poller.php


Configure Cacti Spine (optional: this is a poller written in C ++, which is used to speed up polling, recommended by Cacti)
Unzip the spine archive into the cacti directory to make sure that spine.conf contains the following lines:

 DB_Host 127.0.0.1 # strictly not localhost !!!
 DB_Database cacti
 DB_User cactiuser
 DB_Password cactiuser
 DB_Port 3306


Select and configure data collection


I tried two ways to collect data - SNMP and JMX server polling as part of the JVM and applications. SNMP is supported by Cacti and it is reasonable to use it if you only need to watch JVM memory usage and, you need to do it very quickly. I started with SNMP, but after the first successes I switched to JMX. Cacti does not support JMX , so you have to pay for additional flexibility - it takes effort to write a Java receiving and receiving part.

Below is the corresponding code.

The code for polling the JVM according to the state of memory (the code for an arbitrary JMX server is written in the same way):

import java.io.Closeable; import java.io.IOException; import java.lang.management.MemoryMXBean; import java.lang.management.MemoryUsage; import javax.management.JMX; import javax.management.MBeanServerConnection; import javax.management.ObjectName; import javax.management.remote.JMXConnector; import javax.management.remote.JMXConnectorFactory; import javax.management.remote.JMXServiceURL; public class JvmHeapJmxClient { public static void main(String[] args) { JMXConnector jmxc = null; try { String jmxHost = args[0]; String jmxPort = args[1]; String jmxConnectionString = "service:jmx:rmi:///jndi/rmi://" + jmxHost + ":" + jmxPort + "/jmxrmi"; JMXServiceURL url = new JMXServiceURL(jmxConnectionString); jmxc = JMXConnectorFactory.connect(url, null); MBeanServerConnection mbsc = jmxc.getMBeanServerConnection(); ObjectName memoryMBeanName = new ObjectName("java.lang:type=Memory"); MemoryMXBean memoryMBeanProxy = JMX.newMXBeanProxy(mbsc, memoryMBeanName, MemoryMXBean.class, true); MemoryUsage memoryUsage = memoryMBeanProxy.getHeapMemoryUsage(); echo( "used:" + memoryUsage.getUsed() + " " + "committed:" + memoryUsage.getCommitted() + " " + "init:" + memoryUsage.getInit() + " " + "max:" + memoryUsage.getMax()); } catch(Exception e) { e.printStackTrace(); } finally { closeStream(jmxc); } } private static void echo(String msg) { System.out.println(msg); } private static void closeStream(Closeable stream) { try { if (stream != null) { stream.close(); } } catch (IOException e) { e.printStackTrace(); } } }


Java code to run JMX. All these difficulties are needed to fix the port and host name, which is required when there is a firewall. Each service requires 2 ports, as can be seen from the code - one for http access, the other for RMI. Distribution of ports can obviously be done differently (in this case, RMI-port = http-port + 1), including explicitly specify both ports. The following words will need to be added to the application launch line (jmxagent - jar-file with agent code):

 -Djmxagent.port = <SERVER_JMX_PORT> -Djmxagent.host = <SERVER_HOST> -javaagent: jmxagent.jar



 import java.io.IOException; import java.lang.management.ManagementFactory; import java.rmi.registry.LocateRegistry; import java.util.HashMap; import javax.management.MBeanServer; import javax.management.remote.JMXConnectorServer; import javax.management.remote.JMXConnectorServerFactory; import javax.management.remote.JMXServiceURL; public class JmxFirewallAgent { private JmxFirewallAgent() { } public static void premain(String agentArgs) throws IOException { try { final int rmiRegistryPort = Integer.parseInt(System.getProperty("jmxagent.port")); LocateRegistry.createRegistry(rmiRegistryPort); final int rmiServerPort = rmiRegistryPort + 1; MBeanServer mbs = ManagementFactory.getPlatformMBeanServer(); HashMap<String,Object> env = new HashMap<String,Object>(); final String hostname = System.getProperty("jmxagent.host"); JMXServiceURL url = new JMXServiceURL("service:jmx:rmi://" + hostname + ":" + rmiServerPort + "/jndi/rmi://" + hostname + ":" + rmiRegistryPort + "/jmxrmi"); JMXConnectorServer cs = JMXConnectorServerFactory.newJMXConnectorServer(url, env, mbs); cs.start(); } catch (Exception e) { e.printStackTrace(); } } }


Next come the screenshots of the most important Cacti settings.

Configuring a poll is an important point in the general settings of Cacti. I use Spine, the survey runs every 5 minutes, the data is removed every 10 seconds. See below for the settings RRD in the Cacti - they need to think in advance, because further change without overwriting the database RRD problematic. Here is a good selection of tips on these settings:



My settings RRD.



Example of the contents of one setting:



Device Settings is a server with an IP address from which data will be retrieved. We have several servers, each of them running several Java services.



Configure Data Input Method - set the method of data collection. In my case, this is a batch file, which launches the Java JMX poll program (its source codes were listed above.) Its input parameters are the address and port of the server, i.e. Java applications.



An example of the input parameter description - in this case the server address. The predefined keyword is used - hostname, which is automatically filled in by Cacti. The second parameter will be populated for each Data Source data collection setting, as will be seen later.



Output parameters that batch file returns. Cacti understands them in the format:
 key1: value1 key2: value2


My program returns 4 output values, here is an example of setting one of them (commited):



Next, create a Data Template - a template for specifying data sources. The most important setting in Cacti, in my opinion. The template is associated with the setting of the RRD database, it specifies the parameters for storing in the database (Data Source Item) and additional settings, in this case it is jmx host and jmx port, the first one, as already mentioned, is filled automatically, the second one will be filled in on each data source i.e. The data source will correspond to a single Java application.



Based on the template, a Data Source is created - a data source, which can then be displayed on Graph diagrams. In the source, indicates the Device to which to connect for data collection, the source template, the name and location of the RRD database. In my case, you also need to specify an additional parameter - the JMX port. The important point is that with significant updates to the data collection settings in the Data Source or Data Template (for example, deleting-adding Data Source Item parameters), the RRD database needs to be re-created. This can be done by hands outside Cacti (I haven't gotten to this before) or re-create the Data Source with the loss of all previous data. This is probably the most unpleasant feature of the Cacti-RRD bundle with which I had to face.



Proceed to display the data by defining Graph Template templates. Settings basically set the way data is visualized. You need to specify the parameters from the Data Template that you want to display on the chart and in what form. As far as I understand, all these settings are a direct shell over the RRD command interface.



An example of setting the Data Source parameter. I use to display the memory consumption of 5 elements in the diagram: Max, min, and the current consumption in the form of a solid fill (the order of the output is important!), Then two parameters - the maximum allocated and reserved consumption in the form of lines. Example, see below.



Graph Chart - set the template, server and assignment of correspondence between a specific Data Source and pametra of a chart template:



An example of a final diagram using all the specified settings: It can be seen that the consumption is at an average of 3 GB, but quite often reaches the maximum selected border of 6 GB (-Xmx). Data can be viewed with arbitrary detail (from 1 tick of collection to 2 years, as in an example and more). It all depends on the specified RRD base settings. With my settings I had enough information to solve consumption problems.



This is where the description of my decision ends. I emphasize that the main idea of ​​the solution is the speed of implementation with the high quality of the result obtained. Questions and ideas for improvement are welcome.

I did not give that part of the settings, which concerns my specific performance parameters, since they are completely similar to those already described.

Thanks for attention!

Findings:


Cacti allowed me to successfully solve the problem of quick setup of monitoring Java applications. If you need to quickly make monitoring from scratch, I recommend this as one of the working options. If your plans include building long-term monitoring, then, in my opinion, it makes sense to look at the following options: implement powerful systems like Nagios, or write your own specialized add-on above RRD. Pros and cons of Cacti:

Solution Minuses:



Advantages of the solution:

Source: https://habr.com/ru/post/179391/

More articles:


All Articles