Studying the wget command with 12 examples

All of us sometimes download files from the Internet. If you use programs with a graphical interface for this, then everything turns out to be extremely simple. However, when working in the Linux command line, the case is somewhat complicated. Especially - for those who are not familiar with suitable tools. One such tool is the extremely powerful wget utility, which is suitable for all kinds of downloads. We bring to your attention twelve examples, having disassembled which, you can master the main features of wget.

1. Download single file

If all you need is to download one file, the following construction will suit us:

$ wget https://downloads.sourceforge.net/project/nagios/nagios-4.x/nagios-4.3.1/nagios-4.3.1.tar.gz?r=&ts=1489637334&use_mirror=excellmedia 

After entering such a command, Nagios Core will start downloading. During this process, you will see data on the download, for example - information about how much data is already loaded, the current speed, and how much time is left until the end of the download.
')

2. Download the file and save it with a new name

If we want to save the downloaded file under a name different from its original name, we can use the wget command with the -O parameter:

 $ wget -O nagios_latest https://downloads.sourceforge.net/project/nagios/nagios-4.x/nagios-4.3.1/nagios-4.3.1.tar.gz?r=&ts=1489637334&use_mirror=excellmedia 

With this approach, the downloaded file will be saved under the name nagios_latest .

3. Limiting the speed of downloading files

If necessary, the speed of downloading files using wget can be limited. As a result, this operation will not occupy the entire available data transfer channel and will not affect other processes associated with the network. This can be done using the --limit-rate parameter and specifying the speed limit expressed in bytes (as a normal number), kilobytes (adding after the K number) or megabytes ( M ) per second:

 $ wget ––limit-rate=500K https://downloads.sourceforge.net/project/nagios/nagios-4.x/nagios-4.3.1/nagios-4.3.1.tar.gz?r=&ts=1489637334&use_mirror=excellmedia 

Here the download speed limit is set to 500 Kb / s.

4. Completion of interrupted download

If during the file download this operation was interrupted, you can resume the download using the -c parameter of the wget command:

 $ wget –c https://downloads.sourceforge.net/project/nagios/nagios-4.x/nagios-4.3.1/nagios-4.3.1.tar.gz?r=&ts=1489637334&use_mirror=excellmedia 

If this parameter is not used, then the download of the file that was not downloaded will start over from the beginning.

5. Background file download

If you are uploading a huge file and want to perform this operation in the background, you can do it using the -b parameter:

 $ wget –b https://downloads.sourceforge.net/project/nagios/nagios-4.x/nagios-4.3.1/nagios-4.3.1.tar.gz?r=&ts=1489637334&use_mirror=excellmedia 

6. Upload multiple files

If there is a list of URLs of files that need to be downloaded, but you don’t want to manually start downloading these files, you can use the -I option. However, before starting the download, you need to create a file containing all the addresses. For example, you can do this with the following command:

 $ vi url.txt 

Add addresses to this file, one per line. Further, it remains only to run wget , passing the newly created file with the list of downloads to this utility:

 $ wget –I url.txt 

Execution of this command will lead to the sequential loading of all files from the list.

7. Increasing the total number of file download attempts

In order to configure the number of file download retries, you can use the --tries parameter:

 wget ––tries=100 https://downloads.sourceforge.net/project/nagios/nagios-4.x/nagios-4.3.1/nagios-4.3.1.tar.gz?r=&ts=1489637334&use_mirror=excellmedia 

8. Uploading Files from an FTP Server

The command to download a file from an anonymous FTP server using wget looks like this:

 $ wget FTP-URL 

If a username and password are required to access the file, the command will look like this:

 $ wget –-ftp-user=dan ––ftp-password=********* FTP-URL 

9. Creating a local copy of the website

If you need to load the contents of the entire website, you can do this by using the --mirror :

 $ wget --mirror -p --convert-links -P /home/dan xyz.com 

Note the additional command line options:

• -p : download all the files required for the correct display of HTML-pages.
  
• --convert-links : the links in the documents will be converted for local viewing purposes.
  
• -P /home/dan : the materials will be saved in the /home/dan folder.
  

10. Download from the site only files of a certain type.

In order to download only certain types of files from the site, you can use the -r -A options:

 $ wget -r -A.txt Website_url 

11. Skipping files of a certain type

If you want to copy an entire website, but you don’t need files of a certain type, you can disable their download using the --reject parameter:

 $ wget --reject=png Website_url 

12. Download using your own .log file

To upload a file and use your own .log file, use the -o option and specify the name of the log file:

 $ wget -o wgetfile.log https://downloads.sourceforge.net/project/nagios/nagios-4.x/nagios-4.3.1/nagios-4.3.1.tar.gz?r=&ts=1489637334&use_mirror=excellmedia 

Results

Wget is quite easy to use, but a very useful Linux utility. And, in fact, what we said is only a small part of what she can do. We hope that this review will help those who were not familiar with wget, evaluate this program, and, possibly, include it in their everyday arsenal of command line tools.

Dear readers! Do you use Linux command line tools to download files? If so, please talk about them.