View requests to the apache web server in real time

The load on the server is an important part of web server administration, tracking requests helps to quickly search for erroneous requests and eliminate them before they return to you with webmaster statistics. By default, the host’s Apache configuration file has a commented definition of the host in dns, which in turn makes it even more difficult to understand what comes to the site. It was possible to simplify all this by writing a script that displays all requests to the web server in real time. Saving a few hundred requests it was possible to calculate the load time. This is a percentage of the actual elapsed time of the amount of time spent on the withdrawal of resources. Convenient display of time spent on the execution of each page of the site has become indispensable in finding weaknesses. A small array with a list of search engine templates made it possible to highlight them in a separate color in the feed of requests.

Output is started using the Linux console by forwarding over ssh logging. The plus was the minimum load on the server, since all the calculations occur on the local page. A small php script helps to calculate the input to which data is being output from /var/log/apache2/access.log, which at one time highlights important parts of the log with a bright backlight.

Unfortunately, I did not find the standard logging levels suitable for my conditions. In my case, the important parameter was the host name where the request arrived, but the% V parameter that is responsible for displaying the full host name in the standard settings was missing. Had to add the necessary parameters to common

LogFormat "%h %V %l %u %t \"%r\" %>s %O %D" common 

It is important to see in such a bots ribbon, this task was solved by setting the output in the DNS log of the client name. This setting is also disabled by default in the configuration file.
')

 HostnameLookups On 

The script itself is a small php file that parses all the incoming data on it in parts and highlights processes important for further output. It looks like this. For security reasons, all local hosts and addresses in the video are replaced. In addition, it may be convenient to add important pages to your list of interesting addresses. The script highlights these addresses in the request feed. The video in the second parameter displays the average number of requests per second, while the third is the time taken to issue resources, taking into account the last thousand requests.



Well, the script itself that displays this output.

 /usr/local/bin/log.php 

 <?php $colors = array( "200"=>"[0;32m", "404"=>"[1;31m", "410"=>"[0;31m", "403"=>"[0;91m", "301"=>"[1;33m", "302"=>"[1;36m", "304"=>"[1;36m", ); #   $bots = array("localhost", "googlebot", "yandex", "spider", "ahrefs", "mail.ru", "bot"); #   ,    $url = ["login", "admin"]; #       require "/home/mpak2/www/idna_convert.class.inc"; $idna = new idna_convert(); $in = fopen('php://stdin', 'r'); $tm = []; $t = microtime(true); $nn = 0; $microtime = microtime(true); while($str = fgets($in)){ if(!$ar = explode(" ", $str)){ print_r("   "); }elseif(!$n = number_format($nt = (100*(count($tm)-1)+($nn%100))/(microtime(true)-$t), 2)){ print_r("    "); }elseif(!($percent = number_format((array_sum(array_column($tm, 'msec'))/1e6)/(microtime(true)-$t)*100, 2)) & false){ print_r("   "); }elseif(!$mtime = $ar[11]/1e6){ print_r("    "); }elseif(!$mtm = (($_tm = number_format($mtime, 3)) > 1 ? "\e[1;31m{$_tm}\e[0m" : $_tm)){ print_r("  "); }elseif(!$uri = urldecode($ar[7])){ mpre("   "); }elseif(!$uri = (array_filter(array_map(function($u) use($uri){ return strpos($uri, $u); }, $url)) ? "\e[1;37m". urldecode($ar[7]). "\e[0m" : urldecode($ar[7]))){ print_r("   "); # }elseif(!$uri = "/-/"){ mpre(" "); }elseif(!$status = (array_key_exists($ar[9], $colors) ? "\e{$colors[$ar[9]]}{$ar[9]}\e[0m" : $ar[9])){ print_r("  "); }elseif(!$host = "\e[1;34m". $idna->decode($ar[1]). "\e[0m"){ print_r("   "); # }elseif(!$host = "-."){ print_r("    "); }elseif(!$bot = ((array_filter(array_map(function($b) use($ar){ return (strpos($ar[0], $b) !== false ? $b : false); }, $bots))) ? "\e[1;35m{$ar[0]}\e[0m" : "\e[1;32m{$ar[0]}\e[0m")){ print_r("  "); }elseif(!$size = number_format($ar[10]/1024, 2). ""){ print_r("   "); }else{// print_r($ar); if(($nn++ % 100) == 0){ array_unshift($tm, ['msec'=>0, 'microtime'=>microtime(true)]); if(count($tm) > 10){ $t = array_pop($tm)['microtime']; } } if(!($nn % 1000) && ($cmd = "echo '". strtr($n, ['.'=>' ']). "' | festival --tts --language russian &")){ passthru($cmd); } $tm[0]['msec'] += $ar[10]; passthru("echo '{$nn} {$n} {$percent}% {$mtm}c {$status} http://{$host}{$uri} ({$bot}) {$size}'"); } }; 

Run by the command: ssh root@khabrakhabr.rf 'tail -f /var/log/apache2/access.log' | php -f /usr/local/bin/log.php