Detailed statistics of requests from the logs
Recently there was a post about collecting statistics in the Google App Engine. Here is one of the recipes that I use for such purposes:
This is how it turns out and the statistics for the last 10 days for the toptuby.appspot.com application is shown in the browser .
appcfg.py allows you to download server logs . In the example above, they are placed in the file today.txt. These logs are then analyzed by the visitors' minimalist log analyzer. It is indicated that all logs should be taken into account, the maximum number of fields in the tables, the time shift, and the fact that the typical transitions within the site need to be shown. The result is an html page that can be viewed in any browser (all charts and diagrams are made using html elements). There is a sample report on the program website. You can learn more about the parameters and appcfg.py, and the visitors simply by typing them in the command line with no parameters.
Thus, at one time, you can get up to date statistics, including how robots behave. appcfg.py does not have a limit of 1000 requests - it has been possible to download logs and more than 5000 entries, but this takes time. Visitors generate reports on the number of unique visitors per day, referrals from Google, external links, browsers, operating systems, robots, and about a dozen more.
')
The disadvantage of visitors is that under Windows it should be collected from source code in C (I never did it there). Under Linux is going to cheer, in the Ubuntu repository is also available. Surprisingly, the utility is not mentioned on the web analyzers Wikipedia page . With this approach, you can use all that are referred to there as "logfile-based".
If someone uses interesting log analyzers that are not in the lists in Wikipedia - beacon in the comments.
#!/bin/bash appcfg.py --num_days=10 request_logs code/ today.txt visitors -A -m 50 -T --time-delta 10 --trails --prefix http://toptuby.appspot.com/ today.txt > today.html firefox today.html This is how it turns out and the statistics for the last 10 days for the toptuby.appspot.com application is shown in the browser .
appcfg.py allows you to download server logs . In the example above, they are placed in the file today.txt. These logs are then analyzed by the visitors' minimalist log analyzer. It is indicated that all logs should be taken into account, the maximum number of fields in the tables, the time shift, and the fact that the typical transitions within the site need to be shown. The result is an html page that can be viewed in any browser (all charts and diagrams are made using html elements). There is a sample report on the program website. You can learn more about the parameters and appcfg.py, and the visitors simply by typing them in the command line with no parameters.
Thus, at one time, you can get up to date statistics, including how robots behave. appcfg.py does not have a limit of 1000 requests - it has been possible to download logs and more than 5000 entries, but this takes time. Visitors generate reports on the number of unique visitors per day, referrals from Google, external links, browsers, operating systems, robots, and about a dozen more.
')
The disadvantage of visitors is that under Windows it should be collected from source code in C (I never did it there). Under Linux is going to cheer, in the Ubuntu repository is also available. Surprisingly, the utility is not mentioned on the web analyzers Wikipedia page . With this approach, you can use all that are referred to there as "logfile-based".
If someone uses interesting log analyzers that are not in the lists in Wikipedia - beacon in the comments.
Source: https://habr.com/ru/post/78485/
All Articles