Command Line Tools for Web Developer
The life of a web developer is overshadowed by difficulties. It is especially unpleasant when the source of these difficulties is unknown. Is this a problem with sending a request, or with a response, or with a third-party library, or is the external API buggy? There is a bunch of different prilad, able to simplify our lives. Here are some command line tools that I personally find invaluable.
cURL
cURL is a program for transferring data using various protocols, similar to wget. The main difference is that by default, wget saves to a file, and cURL prints to the command line. So you can very simply see the content of the website. For example, here’s how to quickly get your current external IP:
The -i (show headers) and -I (show only headers) options make cURL an excellent tool for debugging HTTP responses and analyzing exactly what the server sends you:
The -L parameter is also useful, it forces cURL to automatically follow redirects. cURL supports HTTP authentication, cookies, tunneling through HTTP proxies, manual settings in headers and much, much more.
')
Siege
Siege is a load testing tool. Plus, it has a convenient option -g , which is very similar to curl –iL , but in addition it also shows you http-request headers. Here's an example from google.com (some headers removed for short):
But for what Siege is really great, so it is for load testing. Like the Abache benchmark ab , it can send multiple parallel requests to the site and see how it handles traffic. The following example shows how we test Google using 20 queries for 30 seconds, after which the result is displayed:
One of the most useful features of Siege is that it can work not only with a single address, but also with a list of URLs from a file. This is great for load testing, because you can simulate real traffic on the site, rather than just pressing the same URL again and again. For example, here's how to use Siege to load the server using addresses from your Apache log:
Ngrep
For serious traffic analysis, there is Wireshark with thousands of settings, filters and configurations. There is also a tshark command line version . But for simple tasks, Wireshark functionality is considered redundant. So as long as I do not need a powerful weapon, I use ngrep . It allows you to do with network packets the same thing that grep does with files.
For web traffic, you almost always want to use the -W option to preserve string formatting, as well as the -q option, which hides redundant information about inappropriate packages. Here is an example of a command that intercepts all packets with a GET or POST command:
You can add an additional filter for packets, for example, by a given host, IP address or port. Here's a filter for all incoming and outgoing traffic on google.com, port 80, which contains the word “search”.
cURL
cURL is a program for transferring data using various protocols, similar to wget. The main difference is that by default, wget saves to a file, and cURL prints to the command line. So you can very simply see the content of the website. For example, here’s how to quickly get your current external IP:
$ curl ifconfig.me 93.96.141.93 The -i (show headers) and -I (show only headers) options make cURL an excellent tool for debugging HTTP responses and analyzing exactly what the server sends you:
$ curl -I habrahabr.ru HTTP/1.1 200 OK Server: nginx Date: Thu, 18 Aug 2011 14:15:36 GMT Content-Type: text/html; charset=utf-8 Connection: keep-alive Keep-alive: timeout=25 The -L parameter is also useful, it forces cURL to automatically follow redirects. cURL supports HTTP authentication, cookies, tunneling through HTTP proxies, manual settings in headers and much, much more.
')
Siege
Siege is a load testing tool. Plus, it has a convenient option -g , which is very similar to curl –iL , but in addition it also shows you http-request headers. Here's an example from google.com (some headers removed for short):
$ siege -g www.google.com GET / HTTP/1.1 Host: www.google.com User-Agent: JoeDog/1.00 [en] (X11; I; Siege 2.70) Connection: close HTTP/1.1 302 Found Location: http://www.google.co.uk/ Content-Type: text/html; charset=UTF-8 Server: gws Content-Length: 221 Connection: close GET / HTTP/1.1 Host: www.google.co.uk User-Agent: JoeDog/1.00 [en] (X11; I; Siege 2.70) Connection: close HTTP/1.1 200 OK Content-Type: text/html; charset=ISO-8859-1 X-XSS-Protection: 1; mode=block Connection: close But for what Siege is really great, so it is for load testing. Like the Abache benchmark ab , it can send multiple parallel requests to the site and see how it handles traffic. The following example shows how we test Google using 20 queries for 30 seconds, after which the result is displayed:
$ siege -c20 www.google.co.uk -b -t30s ... Lifting the server siege... done. Transactions: 1400 hits Availability: 100.00 % Elapsed time: 29.22 secs Data transferred: 13.32 MB Response time: 0.41 secs Transaction rate: 47.91 trans/sec Throughput: 0.46 MB/sec Concurrency: 19.53 Successful transactions: 1400 Failed transactions: 0 Longest transaction: 4.08 Shortest transaction: 0.08 One of the most useful features of Siege is that it can work not only with a single address, but also with a list of URLs from a file. This is great for load testing, because you can simulate real traffic on the site, rather than just pressing the same URL again and again. For example, here's how to use Siege to load the server using addresses from your Apache log:
$ cut -d ' ' -f7 /var/log/apache2/access.log > urls.txt $ siege -c<concurreny rate> -b -f urls.txt Ngrep
For serious traffic analysis, there is Wireshark with thousands of settings, filters and configurations. There is also a tshark command line version . But for simple tasks, Wireshark functionality is considered redundant. So as long as I do not need a powerful weapon, I use ngrep . It allows you to do with network packets the same thing that grep does with files.
For web traffic, you almost always want to use the -W option to preserve string formatting, as well as the -q option, which hides redundant information about inappropriate packages. Here is an example of a command that intercepts all packets with a GET or POST command:
ngrep -q -W byline "^(GET|POST) .*" You can add an additional filter for packets, for example, by a given host, IP address or port. Here's a filter for all incoming and outgoing traffic on google.com, port 80, which contains the word “search”.
ngrep -q -W byline "search" host www.google.com and port 80 Source: https://habr.com/ru/post/126605/
All Articles