slit - a new word in the world of PAGERS, or how to spend less time viewing logs
It just so happens that I regularly have to look through a lot of logs. One thing pleases, not so much as the people working with me who sometimes have this main job. These logs do not lie in any centralized system, but are stored in s3 and we watch them downloading with redirection of the output to less.
less is installed for everyone, everyone is used to working with it, they know about basic things, like searching back and forth, filtering by &, go to the end (G) of a file, go to the beginning (g) and so on.
And also, everyone has already come to terms with the fact that at any moment, when adding a filter, less can hang for an indefinite period, display along a line of 5 seconds, and so on. Ultimately, especially when reading logs from stdin, you have to be careful. The filter may or may not work.
')
Actually, at that moment, that I had a chance to walk through a couple of hundred log files for several days - it became obvious that the world needed to be changed for the better ...
Under the cut demo (gif 2.2mb) and a bit of history.
The main claims for less were 3:
1) The filter can fail at any time;
2) The filter basically works rather slowly, especially with stdin, especially with logs> 5MB;
3) Cannot add multiple filters.
Searching in the open spaces of the network - I realized that there is nothing. There is one "log navigator" - but, not intuitive, a lot of excess, but what you need is very inconvenient.
Starting to write his PAGER-a on Go - first of all he began to compare different approaches.
The first was with a swoop, the idea that there is a lot of memory on the machines, you can simply load the entire file into memory, prepare it - and then quickly search for everything there.
Everything turned out to be not so trivial, on a 50MB file, with its line-by-line splitting, recognition of control characters to separate text from color, memory allocation for the resulting data (separate color information, separate text for quick search in text) - it took longer than if just read everything, break, search, but not allocate memory for storage.
When streaming, data moves from function to function in the stack, which is much faster.
Ultimately, it came to the conclusion that memory is allocated only for storing the offset of each line, and the separation of text from colors and the transfer of ready-made suitable lines to the buffer occurs in streaming mode. The buffer is always stored 2-3 windows in each direction, for fast navigation back and forth without additional readings. If possible, the search is performed in this buffer, if this is not enough - even then the file is re-read from the desired position.
It goes without saying that such an approach also means that you don’t have to wait at the beginning to upload the entire file.
In principle, in the future I am thinking of adding a flag that will in any case keep the entire file in memory, but with stream processing. But not sure what it needs :)
In general, search, filtering, including multi-level, turned out much faster than in less and much more stable. About 4 times a simple search, the difference in filtering reaches infinity, since less can just hang. But even if not, then there is already an order of magnitude.
Multi-level filters
The sequence of these filters will help to quickly leave on the screen what you need.
Also, for more detailed study will actively help:
Another useful feature:
Shift + K - Keep. Allows you to specify the number of characters from the beginning of the line that will be displayed when the screen is horizontally shifted. Thus, you can fix the time, commit hash, etc.
You can either enter the number, or use the up-down arrows to adjust the number of fixed characters.
- Mostly hotkeys follow less
- Search-filters save stories between sessions. There is no search for ctrl + r yet, this is in the plans.
- Files (so far) can be viewed only one at a time or with stdin-a
- If slit is started with output redirection - then the incoming stream will be simply redirected, without any logic, like cat and less
- Actively tested on linux, osx, a bit on windows, not tested at all for freebsd.
Collected under it collected, but there is nothing to check. According to the idea should work
The project at a very early stage, I personally have already replaced the PAGER in the system. But perhaps someone lacks something that would seem obvious. Do not be silent :)
There are a number of plans:
- Add switching case sensitivity. Currently, only case-sensitive search;
- Add search type switch (regex / plain / extended glob);
- Add a menu of filters to review what is at the moment with selective deletion-disconnection;
- Add support for multiple files. To start with switching between them as in less;
- CTRL + R history search;
- Delete a substring from each line, to remove noise, without deleting the lines themselves;
- Direct output to the terminal if the text is smaller than the screen (analog of the -F flag in less-e), perhaps with support for -F from the environment variable LESS.
What is in question:
- Read the entire stdin or not? Personally, it helps me a lot, that he reads everything before I decided that I needed it. But usually PAGERs do not.
- not Sly opportunities to work with time, i.e. recognize timestamps and enable filtering by them. I do not know if anyone in the logs
What is required to remain despite the addition of features:
- Collect no more than one line from the terminal
- No more time should be spent on opening a file than in the case of less
This is my second project on Go, the first was here .
The absence of various functionalities, such as the notorious generics, sometimes the need to implement seemingly trivial things (atomic boolean) sometimes frustrates. In python, I write a lot more, but every time I touch the go, I really enjoy the restrictions that have been imposed on me :)
But most of all I am pleased with the ecosystem. For some reason, it is often forgotten about her properly to mention, comparing or cursing the language. Perhaps the language slows down the spelling, until the “run” button is pressed for the first time. After it, the go quickly rushes forward.
1) go build / run / install --race
The application will run in race condition search mode, access data from different streams without synchronization. It is actually very cool to help catch bugs that are quite difficult to catch. And sometimes to understand that in vain I decided to go the easy way and pass the pointer, and not to add an additional level of communication between parts of the application through threads
2) net / http / pprof - that's great. Simple import, registration of the local port and when a deadlock occurs, you can go through the browser to the selected port and see all the information about goroutine-ah that are running now
3) Cross compilation . This is said of course more, but every time it never ceases to please. That you do not need to dance with a tambourine, put additional dependencies - it just works
4) Goglang , thank you, Jetbrains, for putting everything together, again, no need to dance with a tambourine, as well as in the case of Pycharm, I continue to work in a similar environment, with vim mode, auto import, auto go get, declaring methods and etc.
Before the next choice of language, I compared how some other ascending languages ​​are doing and it became obvious that they still have room to grow ...
And the main thing, where to take:
If you have the Go environment set up, then the best thing to do is to build:
If you prefer ready-made binaries, then from the releases page .
And the question to the studio, I would like to notify users about the fact that there is a newer version. It is unpleasant if I make a release, I suddenly notice a mistake, after 5 minutes I release a new release, but he has already downloaded. What are the options for an unobtrusive but understandable notification?
So far, I think to climb into the network no earlier than once per hour and at startup, until some action is taken to display a message in the navigation bar. Maybe there will be more elegant ideas.
Also happy with the ideas that it is worth improving-add. But I do not promise everything :)
less is installed for everyone, everyone is used to working with it, they know about basic things, like searching back and forth, filtering by &, go to the end (G) of a file, go to the beginning (g) and so on.
And also, everyone has already come to terms with the fact that at any moment, when adding a filter, less can hang for an indefinite period, display along a line of 5 seconds, and so on. Ultimately, especially when reading logs from stdin, you have to be careful. The filter may or may not work.
')
Actually, at that moment, that I had a chance to walk through a couple of hundred log files for several days - it became obvious that the world needed to be changed for the better ...
Under the cut demo (gif 2.2mb) and a bit of history.
Demo (2.2mb):
The main claims for less were 3:
1) The filter can fail at any time;
2) The filter basically works rather slowly, especially with stdin, especially with logs> 5MB;
3) Cannot add multiple filters.
Searching in the open spaces of the network - I realized that there is nothing. There is one "log navigator" - but, not intuitive, a lot of excess, but what you need is very inconvenient.
Starting to write his PAGER-a on Go - first of all he began to compare different approaches.
The first was with a swoop, the idea that there is a lot of memory on the machines, you can simply load the entire file into memory, prepare it - and then quickly search for everything there.
Everything turned out to be not so trivial, on a 50MB file, with its line-by-line splitting, recognition of control characters to separate text from color, memory allocation for the resulting data (separate color information, separate text for quick search in text) - it took longer than if just read everything, break, search, but not allocate memory for storage.
When streaming, data moves from function to function in the stack, which is much faster.
Ultimately, it came to the conclusion that memory is allocated only for storing the offset of each line, and the separation of text from colors and the transfer of ready-made suitable lines to the buffer occurs in streaming mode. The buffer is always stored 2-3 windows in each direction, for fast navigation back and forth without additional readings. If possible, the search is performed in this buffer, if this is not enough - even then the file is re-read from the desired position.
It goes without saying that such an approach also means that you don’t have to wait at the beginning to upload the entire file.
In principle, in the future I am thinking of adding a flag that will in any case keep the entire file in memory, but with stream processing. But not sure what it needs :)
In general, search, filtering, including multi-level, turned out much faster than in less and much more stable. About 4 times a simple search, the difference in filtering reaches infinity, since less can just hang. But even if not, then there is already an order of magnitude.
Speaking of killer features
Multi-level filters
- & - intersection with the previous sample. Ie the same as in less, but with all the previous filters
- + Add matches to the selection
- - Exclude matches from the selection
The sequence of these filters will help to quickly leave on the screen what you need.
Also, for more detailed study will actively help:
- U (shift + u) - remove the last filter
- = - remove all filters
- C (shift + c, Context) - temporarily disable all filters, allows you to see everything that happened after this line, this line is considered the first on the screen
Another useful feature:
Shift + K - Keep. Allows you to specify the number of characters from the beginning of the line that will be displayed when the screen is horizontally shifted. Thus, you can fix the time, commit hash, etc.
You can either enter the number, or use the up-down arrows to adjust the number of fixed characters.
- Mostly hotkeys follow less
- Search-filters save stories between sessions. There is no search for ctrl + r yet, this is in the plans.
- Files (so far) can be viewed only one at a time or with stdin-a
- If slit is started with output redirection - then the incoming stream will be simply redirected, without any logic, like cat and less
- Actively tested on linux, osx, a bit on windows, not tested at all for freebsd.
Collected under it collected, but there is nothing to check. According to the idea should work
The project at a very early stage, I personally have already replaced the PAGER in the system. But perhaps someone lacks something that would seem obvious. Do not be silent :)
There are a number of plans:
- Add switching case sensitivity. Currently, only case-sensitive search;
- Add search type switch (regex / plain / extended glob);
- Add a menu of filters to review what is at the moment with selective deletion-disconnection;
- Add support for multiple files. To start with switching between them as in less;
- CTRL + R history search;
- Delete a substring from each line, to remove noise, without deleting the lines themselves;
- Direct output to the terminal if the text is smaller than the screen (analog of the -F flag in less-e), perhaps with support for -F from the environment variable LESS.
What is in question:
- Read the entire stdin or not? Personally, it helps me a lot, that he reads everything before I decided that I needed it. But usually PAGERs do not.
- not Sly opportunities to work with time, i.e. recognize timestamps and enable filtering by them. I do not know if anyone in the logs
What is required to remain despite the addition of features:
- Collect no more than one line from the terminal
- No more time should be spent on opening a file than in the case of less
And a little about Go
This is my second project on Go, the first was here .
The absence of various functionalities, such as the notorious generics, sometimes the need to implement seemingly trivial things (atomic boolean) sometimes frustrates. In python, I write a lot more, but every time I touch the go, I really enjoy the restrictions that have been imposed on me :)
But most of all I am pleased with the ecosystem. For some reason, it is often forgotten about her properly to mention, comparing or cursing the language. Perhaps the language slows down the spelling, until the “run” button is pressed for the first time. After it, the go quickly rushes forward.
1) go build / run / install --race
The application will run in race condition search mode, access data from different streams without synchronization. It is actually very cool to help catch bugs that are quite difficult to catch. And sometimes to understand that in vain I decided to go the easy way and pass the pointer, and not to add an additional level of communication between parts of the application through threads
2) net / http / pprof - that's great. Simple import, registration of the local port and when a deadlock occurs, you can go through the browser to the selected port and see all the information about goroutine-ah that are running now
3) Cross compilation . This is said of course more, but every time it never ceases to please. That you do not need to dance with a tambourine, put additional dependencies - it just works
4) Goglang , thank you, Jetbrains, for putting everything together, again, no need to dance with a tambourine, as well as in the case of Pycharm, I continue to work in a similar environment, with vim mode, auto import, auto go get, declaring methods and etc.
Before the next choice of language, I compared how some other ascending languages ​​are doing and it became obvious that they still have room to grow ...
And the main thing, where to take:
If you have the Go environment set up, then the best thing to do is to build:
go get github.com/tigrawap/slit If you prefer ready-made binaries, then from the releases page .
And the question to the studio, I would like to notify users about the fact that there is a newer version. It is unpleasant if I make a release, I suddenly notice a mistake, after 5 minutes I release a new release, but he has already downloaded. What are the options for an unobtrusive but understandable notification?
So far, I think to climb into the network no earlier than once per hour and at startup, until some action is taken to display a message in the navigation bar. Maybe there will be more elegant ideas.
Also happy with the ideas that it is worth improving-add. But I do not promise everything :)
Source: https://habr.com/ru/post/327120/
All Articles