Xargs: a variety of use cases

A lot has been written about the xargs utility - what else can I write? But if, as they say, to dig deeper, then it turns out that in many publications only the very basics are laid out, but there is no main thing: it is not explained how xargs can be applied in actual practice. Unfortunately, there are very few articles with the analysis of complex and non-trivial application options for this very useful tool for the system administrator. That is why we wrote our article and tried to include as many examples as possible of using xargs to solve various problems.

First, we look at the principle of xargs and analyze examples more simply, and then proceed to the analysis of complex and interesting cases.

')

Remember the basics

The principle of operation of xargs can be described as follows: the program takes data from standard input or from a file, splits it in accordance with the specified parameters, and then passes it to another program as an argument.

In general, the syntax of the xargs command can be represented as follows:

 [command_generator_list] |  xargs [xargs_options] [command]

Consider how it all works, on the material of simple and textbook examples.

Deleting files

One of the most common situations in which xargs is used is deleting files found using the find command.

Imagine the following situation: there is a directory in which a large number of files are stored. It is necessary to remove files of a certain type from it (in our example, files with the * .sh extension). To perform this operation, you need to pass xargs the output of the find command, and the -rm command will be applied to files with the specified extension:

 $ ls
 one.sh one.py two.sh two.py

 $ find.  -name "* .sh" |  xargs rm -rf

 $ ls
 one.py two.py

Note that the operation of deleting files can be done without xargs, and using the command

 $ find.  -name "* .sh" -exec rm -rf '{}' \

The described method will not work if the name of one of the files to be deleted contains a space. A name consisting of two words separated by a space will not be perceived as a whole.

We illustrate this with the following example:

 $ ls
 new file.sh one.sh one.py two.sh two.py

 $ find.  -name "* .sh" |  xargs rm -rf

 $ ls
 new file.sh one.py two.py

As you can see, the file with a space in its name was not deleted.

To solve this problem, use the print0 option for the find command and the -0 option for the xargs command. It replaces the standard delimiter (a line break to a null character (\ x0), which means the end of the stored string:

 $ find.  -name "* .sh" -print0 |  xargs -0 rm -rf

Xargs can also help, for example, quickly remove all temporary files that have the tmp extension:

 $ find / tmp -name "* .tmp" |  xargs rm

File compression

Compress all files in the current directory using gzip by typing the following command:

 $ ls |  xargs -p -l gzip

Consider another example: using tar to compress all files with the * .pl extension:

 $ find.  -name "* .pl" |  xargs tar -zcf pl.tar.gz

Rename files

With xargs, you can rename files in bulk. Imagine that we have a group of files with the extension * .txt, and we need to replace this extension with * .sql. This can be done with xargs and the sed text flow editor:

 $ ls |  sed -e "p; s / .txt $ /. sql /" |  xargs -n2 fmv

As a result of its execution, the list of renamed files will be displayed on the console.

With xargs, you can also add additional elements to file names (for example, a date):

 $ ls |  xargs -I FILE mv {} <...> - {}

Instead of <..> you can substitute anything.
The curly braces {} in this example mean the “current argument” (i.e. the current file name).

Change permissions for folders and files

With xargs, you can also speed up the process of changing permissions on files and folders for a specific user or group. Suppose we need to find all the folders of the root user and replace their owner with temp. This operation is performed using the command:

 $ find.  -group root -print |  xargs chown temp

To find all the folders of the root group and replace the group with temp, use the command:

 $ find.  -group root -print |  xargs chgrp temp

Xargs and find: complex operations

Using the find and xargs commands, you can perform more complex operations. So, for example, you can delete temporary files created more than 7 days ago:

 $ find / tmp -type f -name '*' -mtime +7 -print0 |  xargs -0 rm -f

And this is how to forcefully stop processes that are already running for more than 7 days:

 $ find / proc -user myuser -maxdepth 1 -type d -mtime +7 -exec basename {} \;  |  xargs kill -9

Xargs and Sut

Xargs are often used in conjunction with the cut command, which allows you to cut lines from text files. Consider some practical examples. Using the command below, a list of all users of the system will be displayed on the console:

 $ cut -d: -f1 </ etc / passwd |  sort |  xargs echo

And the view team

 file * |  grep ASCII |  cut -d ":" -f1 |  xargs -p vim

will consistently open files for editing in vim.
Note the -p option. Thanks to it, the command will be executed interactively: before opening each file, confirmation (y / n) will be requested.

In conclusion, we present another complex and interesting example - the recursive search for the largest files in a directory:

 $ find.  -type f -printf '% 20s% p \ n' |  sort -n |  cut -b22- |  tr '\ n' '\ 000' |  xargs -0 ls -laSr

Parallel running processes

Xargs is often used to run multiple processes in parallel. So, for example, you can simultaneously compress several directories in tar.gz:

 $ echo dir1 dir2 dir3 |  xargs -P 3 -I NAME tar czf NAME.tar.gz NAME 

In the example above, the –P key is used. It indicates the maximum number of processes that will be executed simultaneously. Suppose we have 10 arguments in our input. If we enter the xargs command with the -P 3 key, 3 instances of the command following the xargs will be launched with each of these arguments.

With xargs, you can also download multiple files from the Internet in parallel:

 $ wget -nv <link> |  egrep -o "http: // [^ [: space:]] * .jpg" |  xargs -P 10 -n 1 wget -nv 

In the above example, all graphic files with the extension jpg will be downloaded from the specified address; the -P switch indicates that you need to download 10 files at a time.

Preliminary results

Let's summarize the preliminary results and formulate several rules for working with xargs.

1. Xargs does not work with files whose name contains a space. To solve this problem with the xargs command, the −0 option is used. You can also bypass the space in the file name as follows:
   

    $ xargs -I FILE my_command “FILE”
   

   
   
2. The xargs command accepts commands from standard input, separated by a space or a line break. To group these commands, you can use double quotes or single quotes. You can also specify a delimiter using the -d option;
3. If no arguments are passed to the xargs command at all, then the default command is / bin / echo;
4. In many cases, the xargs command can be replaced with a for loop. For example, the command
   

    $ find.  -type f -and -iname "* .deb" |  xargs -n 1 dpkg -I
   

   
   completely equivalent to a cycle
   

    $ for file in `find.  -type f -and -iname "* .deb" `;  do dpkg -I "$ file";  done
   

   
   

Nontrivial examples

We recalled the basics, we considered typical use cases ... Now we turn to more complex and non-trivial examples. We came up with some of them on our own, working on everyday tasks, and some learned from http://www.commandlinefu.com (for those who want to learn the subtleties of working with the command line, we highly recommend visiting it from time to time - sometimes you can find useful tips).

Ban IPs from the list

To ban IP-addresses from the list, you need to add them to IP tables with the DROP rule. This operation is performed using the command:

 $ cat bad_ip_list |  xargs -I IP iptables -A INPUT -s IP -j DROP

You can do a more complicated operation and ban all the addresses on AS:

 $ / usr / bin / whois -H -h whois.ripe.net -T route -i origin AS <number> | egrep "^ route" | awk '{print $ 2}' | xargs -I NET iptables -A INPUT - s NET -j DROP 

Change URL format

To convert URLs like “http% 3A% 2F% 2Fwww.google.com” to “ www , google.com”, use the command:

 echo "http% 3A% 2F% 2Fwww.google.com" |  sed -e's /% \ ([0-9A-F] [0-9A-F] \) / \\\\\ x \ 1 / g '|  xargs echo -e

We generate a password of 10 characters

You can generate a strong password using the following command:

 $ tr -dc A-Za-z0-9_ </ dev / urandom |  head -c 10 |  xargs

You can generate passwords without the help of xargs: there is a specialized utility pwgen for this. Some other ways to generate passwords are also described here .

We are looking for binary files installed without using dpkg

Such an operation may be required if, for example, the machine has become a victim of a hacker attack and malicious software has been installed on it. The following command will help to identify what the attackers have put in the programs (she is looking for running binary packages installed without using the dpkg package manager):

 $ at /var/lib/dpkg/info/*.list> / tmp / listin;  ls / proc / * / exe | xargs -l readlink |  grep -xvFf / tmp / listin;  rm / tmp / listin

Remove obsolete kernel packages

 $ dpkg -l linux- * |  awk '/ ^ ii / {print $ 2}' |  grep -v -e `uname -r |  cut -f1,2 -d "-" `|  grep -e [0-9] |  xargs sudo apt-get -y purge

The problem of removing old kernels has already been discussed at Habré - see here (you can find interesting examples of commands at this link).

Convert the script to a string

Sometimes it is necessary to convert a large script into one line. You can do it like this:

 $ (sed 's /#.*// g' | sed '/ ^ * $ / d' | tr '\ n' ';' | xargs echo) <script.sh

Conclusion

As can be seen from the review, the possibilities of xargs are much wider than it might seem at first glance. We hope that the examples given in the article will be useful for you. If you know of other interesting uses for xargs, welcome to comments. Readers who for one reason or another can not leave comments here are invited to our blog .