Taming Graylog2 - a visualized and functional server log files
With a large enough park of servers, with thousands of services spinning on them, demons, scripts, it is rather difficult to keep track of the numerous errors inside. Somewhere out of memory, somewhere stuck a demon, somewhere the database behaves inadequately. The centralized log storage servers have been discussed more than once, I want to talk about one more convenient and powerful tool - Graylog2 .
Immediately I apologize for the abundance of English words, I do not know how to translate them and not lose meaning.
Graylog2 is a free open source system for the centralized collection, storage and analysis of information, which is written in syslog, graylog2 is made according to the DevOps concept.
')
Graylog2 consists of three components:
- graylog2-webui - Rails web interface,
- graylog2-server - Java TCP / UDP log collector,
- mongodb to store the actual logs and settings of the entire system as a whole.
Graylog2-server allows using TCP / UDP protocols, like regular syslog, to receive logs from everywhere, mongodb stores, rails draws everything nicely.
In addition to the usual functions for syslog-server, I would also like to note the following interesting things:
1. Aggregation of messages in streams. For a keyword, we combine the stream of logs from several hosts, you can hang alerts on stream and make these alerts come to someone in the mail.
2. Aggregation of hosts into groups. You can combine streams from different hosts into one group.
3. Samples from the entire array by regexp, by time, by importance, by facility. You can find anything and anytime.
4. Bleklisty for logs. You can filter logs by regexp. All that we ban, the database will not fall.
5. Autorotation logs. No need to worry about cleaning up old records, mongodb itself will do all the work using the capped collections mechanism.
6. Ability to use GELF - graylog extended log format, thus extending the standard syslog message length to 1024 bytes. With GELF, you can monitor not only system messages, but already the logic of the code, sending expanded messages directly from the application.
7. The project is free (GPLv3), is actively developing.
The bundle is generally stable and very smart. I was watching myself as the system processed up to 17,000 messages per second. At the same time, the mongodb scaling mechanisms were not involved, all our components revolve on one small virtual machine.
Unfortunately, there are downsides.
1. mongodb - the main disadvantage, forgive me fans. Maybe we do not know how to cook it, maybe the load is too large. It happens that falls, it happens that eats all the memory. Versions are suitable> 2.0, otherwise sampling by regulars will not work.
2. Java is also quite demanding on resources.
3. Sampling by regular, sampling by time, paging through pages does not always work correctly. To see the pattern is quite difficult, sometimes it works, sometimes not. Webui need to finish.
4. Autorotation is a good thing, but old messages will not survive anywhere and die.
I did not tie critical data to graylog2. All this is still damp. Therefore, on the neighboring port, the log also accepts the good old syslog-ng and adds this data to disk as well. But for monitoring and analyzing the total flow of logs - the tool is quite sane.
The installation is trivial, we download the binaries, we prepare the environment for the rails, we put the last Mongu and after 10 minutes everything works. We have not written init scripts yet, but we will definitely do it.
References:
- project page
- GELF , AMQP
- DevOps
- mongodb
UPD:
Well, while writing the article, the developer announced what will happen in the next version 0.9.6, before the release of the beta of which there are a few days left.
- mongodb was an error, the project goes to ElasticSearch as a repository of information. Honestly, I don’t know what it is)) A huge increase in productivity is promised,
- change of logic in analytics, faster drawing of graphs, more performance increase due to this,
- revised analytics console,
- new rules for filtering, file name, string, etc.
- host groups were removed, considering that this functionality is redundant and is already present in streams,
- bugfixing, improvements.
In general, the author promises even more performance, more stability, more convenience, which, for example, makes me extremely happy. To be honest, it was only after writing the article that I finally realized that the scope of this tool is very, very wide.
Immediately I apologize for the abundance of English words, I do not know how to translate them and not lose meaning.
What it is?
Graylog2 is a free open source system for the centralized collection, storage and analysis of information, which is written in syslog, graylog2 is made according to the DevOps concept.
')
Graylog2 consists of three components:
- graylog2-webui - Rails web interface,
- graylog2-server - Java TCP / UDP log collector,
- mongodb to store the actual logs and settings of the entire system as a whole.
Graylog2-server allows using TCP / UDP protocols, like regular syslog, to receive logs from everywhere, mongodb stores, rails draws everything nicely.
Pros and features.
In addition to the usual functions for syslog-server, I would also like to note the following interesting things:
1. Aggregation of messages in streams. For a keyword, we combine the stream of logs from several hosts, you can hang alerts on stream and make these alerts come to someone in the mail.
2. Aggregation of hosts into groups. You can combine streams from different hosts into one group.
3. Samples from the entire array by regexp, by time, by importance, by facility. You can find anything and anytime.
4. Bleklisty for logs. You can filter logs by regexp. All that we ban, the database will not fall.
5. Autorotation logs. No need to worry about cleaning up old records, mongodb itself will do all the work using the capped collections mechanism.
6. Ability to use GELF - graylog extended log format, thus extending the standard syslog message length to 1024 bytes. With GELF, you can monitor not only system messages, but already the logic of the code, sending expanded messages directly from the application.
7. The project is free (GPLv3), is actively developing.
Performance.
The bundle is generally stable and very smart. I was watching myself as the system processed up to 17,000 messages per second. At the same time, the mongodb scaling mechanisms were not involved, all our components revolve on one small virtual machine.
Minuses
Unfortunately, there are downsides.
1. mongodb - the main disadvantage, forgive me fans. Maybe we do not know how to cook it, maybe the load is too large. It happens that falls, it happens that eats all the memory. Versions are suitable> 2.0, otherwise sampling by regulars will not work.
2. Java is also quite demanding on resources.
3. Sampling by regular, sampling by time, paging through pages does not always work correctly. To see the pattern is quite difficult, sometimes it works, sometimes not. Webui need to finish.
4. Autorotation is a good thing, but old messages will not survive anywhere and die.
I did not tie critical data to graylog2. All this is still damp. Therefore, on the neighboring port, the log also accepts the good old syslog-ng and adds this data to disk as well. But for monitoring and analyzing the total flow of logs - the tool is quite sane.
The installation is trivial, we download the binaries, we prepare the environment for the rails, we put the last Mongu and after 10 minutes everything works. We have not written init scripts yet, but we will definitely do it.
References:
- project page
- GELF , AMQP
- DevOps
- mongodb
UPD:
Well, while writing the article, the developer announced what will happen in the next version 0.9.6, before the release of the beta of which there are a few days left.
- mongodb was an error, the project goes to ElasticSearch as a repository of information. Honestly, I don’t know what it is)) A huge increase in productivity is promised,
- change of logic in analytics, faster drawing of graphs, more performance increase due to this,
- revised analytics console,
- new rules for filtering, file name, string, etc.
- host groups were removed, considering that this functionality is redundant and is already present in streams,
- bugfixing, improvements.
In general, the author promises even more performance, more stability, more convenience, which, for example, makes me extremely happy. To be honest, it was only after writing the article that I finally realized that the scope of this tool is very, very wide.
Source: https://habr.com/ru/post/132116/
All Articles