Aggregation of logs from multiple servers with log4j tools
log4j is a well-known logging library that has found its application in many projects. Its capabilities are not limited to "adding lines to log files." Based on log4j, you can organize a complex system of aggregation of logs to a central server. In addition, the community has GUI utilities for analyzing logs, which are conveniently connected to a central server for analyzing logs.
When a company has several servers or a scattering of various applications logging data into different files, it becomes extremely inconvenient to track all events occurring in applications. Sometimes this becomes impossible due to the lack of access rights to a particular server. It is in such systems that it is necessary to aggregate data on one central server. Consider the easiest way to implement such a system using the log4j library.
Integrating log4j into Java is extremely simple. You need to connect the library itself and create the log4j.properties or log4j.xml configuration file . If we consider a maven project, its configuration will be approximately as follows.
')
The log4j.properties file must be in the classpath for log4j to detect it. It is possible to explicitly specify the location of the configuration file using the java -Dlog4j.configuration = pathToFile command line argument, for more details on the configuration of log4j .
In log4j , there is a concept appender , it defines event handlers, in the example we used the standard ConsoleAppender, which logs all events to the console. Fortunately, the standard set also includes SocketAppender and SocketHubAppender .
SocketAppender creates a connection to a remote log server and sends events to this server. And the serialized LoggingEvent is sent , i.e. all the information about the event and not the string is transmitted. If the remote server is not available, the messages will be discarded, and when the server starts working again, the connection will be restored automatically.
SocketHubAppender is similar to SocketAppender , but it works the other way around. SocketHubAppender creates a socket to which remote clients can connect and when events occur, they are sent to all connected clients.
To configure log4j to work with a remote server, add the following to the configuration:
With these settings, the application will try to connect to localhost: 4560 to send log messages. Now, it is time to configure the server that will aggregate the messages. Such a server can be started with a single command.
The first parameter is the server port, and the second is the server logger configuration. All messages that arrive at the server will be processed as if the server itself generated these events, i.e. since you configure log4j-server.properties , messages will still be displayed. So you can organize complex chains of message transfer from server to server.
If both the server and the client are started simultaneously, all messages that have occurred on the client will be printed in the server console (provided that you added ConsoleAppender ).
All logs are now collected on one server, we will configure the server so that it opens a socket and send events to connected clients. To do this, use SocketHubAppender .
With this configuration, you can connect to the server using log analysis software.
chainsaw - the program originated in the depths of log4j, allows you to connect to SocketHub and display information about events in real time. It seems that the chainsaw is long dead, because last build was already in 2006. Nevertheless, it copes with its task.
After starting the chainsaw, you need to add a data source by clicking on the appropriate icon in the right pane. In the context menu, select SocketHubReciver , then specify the host, port, and name. If successful, a tab with the same name will appear in the main panel in which the latest events that have occurred will be displayed. Chainsaw has quite primitive possibilities for filtering events by level and by application, all events are stored in memory, therefore, when a large number of them begin to slow down. I do not recommend to use.
otroslogviewer is a more modern program that is being actively developed. Connection to SocketHub is performed via the menu item File -> Connect to Log4J SocketHub . Similarly, specify the host and port. Events will be displayed in a new tab in the main panel.
Outwardly, of course it looks scary, Swing is not going away from him, but the functionality here is much richer. Flexible filters by level, time, line entry, streams, and so on.
lilith - the author states that this program can work with SocketHub . But I did not understand how to set it up.
SocketHubAppender does not provide for any authorization, i.e. theoretically any users can connect to it. Therefore, to open in global access this kind of logging server is imprudent. The easiest way to secure yourself is to forward ports through an ssh tunnel. Those. we start the server on a remote machine on port 4561, but we close access to this port using a firewall. When it becomes necessary to see the latest events, we forward ports with a command.
After that, you can connect log analyzers to localhost: 4561.
PS There are other technologies that solve this problem, such as scribe from facebook, but it seems to me that for simple projects its advantages are not significant, and it is more difficult to set it up.
When a company has several servers or a scattering of various applications logging data into different files, it becomes extremely inconvenient to track all events occurring in applications. Sometimes this becomes impossible due to the lack of access rights to a particular server. It is in such systems that it is necessary to aggregate data on one central server. Consider the easiest way to implement such a system using the log4j library.
log4j
Integrating log4j into Java is extremely simple. You need to connect the library itself and create the log4j.properties or log4j.xml configuration file . If we consider a maven project, its configuration will be approximately as follows.
')
./pom.xml
<project xmlns = "http://maven.apache.org/POM/4.0.0" xmlns: xsi = "http://www.w3.org/2001/XMLSchema-instance" xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion> 4.0.0 </ modelVersion> <groupId> com.github.caiiiycuk </ groupId> <artifactId> log4j-app </ artifactId> <version> 1.0 </ version> <name> log4j app </ name> <dependencies> <dependency> <groupId> log4j </ groupId> <artifactId> log4j </ artifactId> <version> 1.2.17 </ version> </ dependency> </ dependencies> </ project>
./src/main/java/Log4JApp.java
import org.apache.log4j.Logger; public class Log4JApp { private final static Logger LOGGER = Logger.getLogger(Log4JApp.class); public static void main(String[] args) throws InterruptedException { while (true) { LOGGER.debug("I'm doing science and I'm still alive."); LOGGER.info("I feel fantastic and I'm still alive."); LOGGER.warn("While you're dying I'll be still alive."); LOGGER.error("And when you're dead I will be, still alive."); LOGGER.fatal("Still alive, still alive."); Thread.sleep(1000); } } } ./src/main/resources/log4j.properties
log4j.rootLogger = debug, stdout log4j.appender.stdout = org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout = org.apache.log4j.PatternLayout # Log message template log4j.appender.stdout.layout.ConversionPattern =% 5p [% t] (% F:% L) -% m% n
The log4j.properties file must be in the classpath for log4j to detect it. It is possible to explicitly specify the location of the configuration file using the java -Dlog4j.configuration = pathToFile command line argument, for more details on the configuration of log4j .
In log4j , there is a concept appender , it defines event handlers, in the example we used the standard ConsoleAppender, which logs all events to the console. Fortunately, the standard set also includes SocketAppender and SocketHubAppender .
SocketAppender creates a connection to a remote log server and sends events to this server. And the serialized LoggingEvent is sent , i.e. all the information about the event and not the string is transmitted. If the remote server is not available, the messages will be discarded, and when the server starts working again, the connection will be restored automatically.
SocketHubAppender is similar to SocketAppender , but it works the other way around. SocketHubAppender creates a socket to which remote clients can connect and when events occur, they are sent to all connected clients.
To configure log4j to work with a remote server, add the following to the configuration:
log4j.rootLogger = DEBUG, stdout, server (...) log4j.appender.server = org.apache.log4j.net.SocketAppender log4j.appender.server.Port = 4560 log4j.appender.server.RemoteHost = localhost log4j.appender.server.ReconnectionDelay = 10000 log4j.appender.server.Application = Log4JApp log4j.appender.server.LocationInfo = true
- Port - the server port to which to send events, by default 4560
- RemoteHost - server host to which to send events
- ReconnectionDelay - timeout for reconnection
- Application - the name of the application generating the logs
- LocationInfo - whether to include location information
With these settings, the application will try to connect to localhost: 4560 to send log messages. Now, it is time to configure the server that will aggregate the messages. Such a server can be started with a single command.
java -classpath log4j.jar org.apache.log4j.net.SimpleSocketServer 4560 log4j-server.properties
The first parameter is the server port, and the second is the server logger configuration. All messages that arrive at the server will be processed as if the server itself generated these events, i.e. since you configure log4j-server.properties , messages will still be displayed. So you can organize complex chains of message transfer from server to server.
If both the server and the client are started simultaneously, all messages that have occurred on the client will be printed in the server console (provided that you added ConsoleAppender ).
Example of server operation
Configuration
Console output
log4j.rootLogger = debug, stdout log4j.appender.stdout = org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout = org.apache.log4j.PatternLayout # Log message template log4j.appender.stdout.layout.ConversionPattern =% 5p [% t] (% F:% L) -% m% n
Console output
INFO [main] (SimpleSocketServer.java:60) - Listening on port 4560 INFO [main] (SimpleSocketServer.java:63) - Waiting to accept a new client. INFO [main] (SimpleSocketServer.java:65) - Connected to client at /127.0.0.1 INFO [main] (SimpleSocketServer.java:66) - Starting new socket node. INFO [main] (SimpleSocketServer.java:63) - Waiting to accept a new client. DEBUG [main] (Log4JApp.java:9) - I'm doing science and I'm still alive. INFO [main] (Log4JApp.java:10) - I feel fantastic and I'm still alive. WARN [main] (Log4JApp.java:11) - While you're still dying. ERROR [main] (Log4JApp.java:12) - And when you're dead, I will be still alive. INFO [SimpleSocketServer-4560] (SocketNode.java:94) - Caught java.io.EOFException closing conneciton.
Error Caught java.io.EOFException closing conneciton
This error occurs when the connection between the server and the client is broken, it always occurs, there is nothing terrible in it.
Log Analysis
All logs are now collected on one server, we will configure the server so that it opens a socket and send events to connected clients. To do this, use SocketHubAppender .
log4j-server.properties
log4j.rootLogger = debug, stdout, server
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern =% 5p [% t] (% F:% L) -% m% n
log4j.appender.server = org.apache.log4j.net.SocketHubAppender
log4j.appender.server.Port = 4561
log4j.appender.server.LocationInfo = true
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern =% 5p [% t] (% F:% L) -% m% n
log4j.appender.server = org.apache.log4j.net.SocketHubAppender
log4j.appender.server.Port = 4561
log4j.appender.server.LocationInfo = true
With this configuration, you can connect to the server using log analysis software.
chainsaw v2
chainsaw - the program originated in the depths of log4j, allows you to connect to SocketHub and display information about events in real time. It seems that the chainsaw is long dead, because last build was already in 2006. Nevertheless, it copes with its task.
After starting the chainsaw, you need to add a data source by clicking on the appropriate icon in the right pane. In the context menu, select SocketHubReciver , then specify the host, port, and name. If successful, a tab with the same name will appear in the main panel in which the latest events that have occurred will be displayed. Chainsaw has quite primitive possibilities for filtering events by level and by application, all events are stored in memory, therefore, when a large number of them begin to slow down. I do not recommend to use.chainsaw v2
otroslogviewer
otroslogviewer is a more modern program that is being actively developed. Connection to SocketHub is performed via the menu item File -> Connect to Log4J SocketHub . Similarly, specify the host and port. Events will be displayed in a new tab in the main panel.
otroslogviewer
Outwardly, of course it looks scary, Swing is not going away from him, but the functionality here is much richer. Flexible filters by level, time, line entry, streams, and so on.
lilith
lilith - the author states that this program can work with SocketHub . But I did not understand how to set it up.
Security questions
SocketHubAppender does not provide for any authorization, i.e. theoretically any users can connect to it. Therefore, to open in global access this kind of logging server is imprudent. The easiest way to secure yourself is to forward ports through an ssh tunnel. Those. we start the server on a remote machine on port 4561, but we close access to this port using a firewall. When it becomes necessary to see the latest events, we forward ports with a command.
ssh -L 4561:localhost:4561 remote-hostAfter that, you can connect log analyzers to localhost: 4561.
PS There are other technologies that solve this problem, such as scribe from facebook, but it seems to me that for simple projects its advantages are not significant, and it is more difficult to set it up.
Source: https://habr.com/ru/post/172075/
All Articles