Frequent Bash programming errors

The quality of the scripts used to automate and optimize the system is the key to its stability and longevity, and also saves the time and nerves of the administrator of this system. Despite the seeming primitiveness of bash as a programming language, it is full of pitfalls and cunning currents that can significantly spoil the mood of both the developer and the administrator.

Most of the manuals available are about how to write. I will tell you how to write it is NOT necessary :-)

This text is a free translation of the bash pitfalls wiki page as of December 13, 2008. Due to the wiki form of the source, this translation may differ from the original. Since the volume of the text is too large to be published in its entirety, it will be published in parts.
')

1. for i in `ls * .mp3`

One of the most common errors in bash scripts are loops like this:

  for i in `ls * .mp3`;  do # Wrong!
     some command $ i # Wrong!
 done 

It will not work if the name of one of the files contains spaces, since The result of the ls *.mp3 substitution is word breaking. Suppose we have the file 01 - Don't Eat the Yellow Snow.mp3 in the current directory 01 - Don't Eat the Yellow Snow.mp3 . The for loop will go through each word from the file name and $i will take the values: "01" , "-" , "Don't" , "Eat" , "the" , "Yellow" , "Snow.mp3" .

Enclose the whole team in quotes too:

  for i in "` ls * .mp3` ";  do # Wrong!
     ... 

All output is now treated as one word, and instead of going through each of the files in the list, the loop will run only once, and i will take the value, which is the concatenation of all file names separated by spaces.

In fact, the use of ls completely unnecessary: ​​it is an external command that is simply not needed in this case. How, then, is it right? That's how:

  for i in * .mp3;  do # Much better, but ...
     some command "$ i" # ... see trick number 2
 done 

Provide bash yourself with file names. Such a substitution will not lead to the division of the line into words. Each file name that satisfies the *.mp3 pattern will be considered as one word, and the loop will go through each file name once.

Additional information can be found in paragraph 20 of the Bash FAQ .

The attentive reader should have noticed the quotes in the second line of the above example. This smoothly brings us to the trick number two.

2. cp $ file $ target

What is wrong with this team? It seems to be nothing special if you absolutely know for sure that the variables $file and $target do not contain spaces or wildcard characters.

But if you don’t know what files you’ll get, or you’re paranoid, or just try to follow a good bash programming style, you’ll enclose the names of your variables in quotes so as not to break them into words.

  cp "$ file" "$ target" 

Without double quotes, the script will execute the command cp 01 - Don't Eat the Yellow Snow.mp3 /mnt/usb , and you will get a lot of errors like cp: cannot stat `01': No such file or directory . If the values ​​of the variables $file or $target contain the characters *,?, [..] or (..) used in wildcard naming patterns, then in the case of the existence of files that match the pattern, the values ​​of the variables will be converted in the names of these files. Double quotes solve this problem, unless "$file" starts with a dash - in this case, cp thinks that you are trying to give it another command line option.

One way to work around this is to insert a double hyphen ( -- ) between the cp command and its arguments. A double hyphen tells cp to stop searching for options:

  cp - "$ file" "$ target" 

However, you may get caught by one of the systems in which such a trick does not work. Or the command you are trying to execute does not support the -- option. In this case, read on.

Another way is to make sure that file names always begin with the name of the directory (including ./ for the current one). For example:

  for i in ./*.mp3;  do
     cp "$ i" / target
     ... 

Even if we have a file whose name starts with "-", the template substitution mechanism ensures that the variable will contain something like ./-foo.mp3 , which is absolutely safe to use with cp .

3. [$ foo = "bar"]

In this example, the quotes are placed incorrectly: in bash, there is no need to enclose a string literal in quotes; but you should definitely quote the variable if you are not sure that it does not contain spaces or wildcards.

This code is erroneous for two reasons:

1. If the variable used in the condition [ does not exist or is empty, the string

  [$ foo = "bar"] 

will be perceived as

  [= "bar"] 

which will cause an “unary operator expected” error. (The operator "=" is binary, not unary, so the command [ will be shocked by this syntax)
2. If a variable contains a space inside it, it will be split into different words before being processed with the command [ :

  [multiple words here = "bar"] 

Even if you personally feel that this is normal, this syntax is erroneous.

It will be correct like this:

  ["$ foo" = bar] # is near! 

But this option will not work if $ foo starts with - .

In bash, the keyword [[ , which includes and significantly extends the old test command (also known as [ )

  [[$ foo = bar]] # correct! 

Inside [[ and ]] no longer need to quote variable names, since variables are no longer broken down into words and even empty variables are handled correctly. On the other hand, even if once again to put them in quotes, it does not hurt anything.

You may have seen a code like this:

  [x "$ foo" = xbar] # is correct too! 

The hack x"$foo" is required in code that should work in shells that do not support [[ , because if $foo starts with - , the command [ will be disoriented.

If one of the parts of the expression is a constant, you can do this:

  [bar = "$ foo"] # that's right too! 

The command [ does not care that the expression to the right of the "=" sign begins with - . It simply uses this expression as a string. Only the left side requires such close attention.

4. cd `dirname" $ ​​f "`

So far, we are mostly talking about the same thing. In the same way as with the disclosure of variable values, the result of the command substitution is subjected to word splitting and file name expansion (pathname expansion). Therefore, we must enclose the command in quotes:

  cd "` dirname "$ f" `" 

What is not completely obvious here is a sequence of quotes. A C programmer could assume that the first and second quotes are grouped, as well as the third and fourth quotes. However, in this case it is not. Bash treats double quotes within a command as the first pair, and outer quotes as the second.

In other words, the parser treats backquotes ( ` ) as a level of nesting, and the quotes inside it are separated from the outside.

The same effect can be achieved using the more preferred $() syntax:

  cd "$ (dirname" $ ​​f ")" 

Quotes inside $() are grouped.

5. ["$ foo" = bar && "$ bar" = foo]

You cannot use && inside the “old” test command or its equivalent [ . The bash parser sees && outside the brackets and splits your command into two, before and after && . Better use one of the options:

  [bar = "$ foo" -a foo = "$ bar"] # That's right!
 [bar = "$ foo"] && [foo = "$ bar"] # That's right too!
 [[$ foo = bar && $ bar = foo]] # That's right too! 

Please note that we swapped the constant and variable inside [ - for the reasons discussed in the previous paragraph.

The same applies to || . Use [[ , or -o , or two commands [ .

6. [[$ foo> 7]]

If the > operator is used inside [[ ]] , it is treated as a string comparison operator, not a number. In some cases, this may or may not work (and this will happen just when you least expect it). If > is inside [ ] , it is still worse: in this case, it is a redirection of the output from the file descriptor with the specified number. An empty file with the name 7 appears in the current directory, and the test command completes successfully, unless the $foo variable is empty.

Therefore, the> and <operators cannot be used to compare numbers inside [ .. ] or [[ .. ]] .

If you want to compare two numbers, use (( )) :

  ((foo> 7)) # That's right! 

If you are writing for Bourne Shell (sh) and not for bash, the correct way is this:

  [$ foo -gt 7] # That's right too! 

Note that the command test ... -gt ... will give an error if at least one of its arguments is not an integer. Therefore, it does not matter if the quotes are correctly placed: if the variable is empty, or contains spaces, or its value is not an integer, an error will occur in any case. Just carefully check the value of a variable before using it in the test command.

Double brackets also support this syntax:

  [[$ foo -gt 7]] # That's right too! 

7. count = 0; grep foo bar | while read line; do ((count ++)); done; echo "number of lines: $ count"

At first glance, this code looks fine. But in fact, the $count variable will remain unchanged after exiting the loop, much to the surprise of the bash developer. Why it happens?

Each command in the pipeline is executed in a separate subshell (subshell), and changes to the variable inside the subshell do not affect the value of this variable in the parent shell instance (that is, in the script that caused this code).

In this case, the for loop is part of the pipeline and runs in a separate subshell with its copy of the variable $count , and the unified value of the variable $count from the parent shell: "0". When the loop ends, the copy of $count used in the loop is discarded and the echo command shows the unchanged initial value of $count (“0”).

There are several ways to get around this.

You can perform a cycle in your subshell (slightly crooked, but it is simpler and clearer and works in sh):

  # POSIX compatible
 count = 0
 cat / etc / passwd |  (
     while read line;  do
         count = $ ((count + 1))
     done
     echo "total number of lines: $ count"
 ) 

To completely avoid creating a subshell, use redirection (in Bourne shell (sh), a subshell is also created for redirection, so be careful, this trick will only work in bash):

  # bash only!
 count = 0
 while read line;  do
     count = $ (($ count + 1))
 done </ etc / passwd
 echo "total number of lines: $ count" 

The previous method works only for files, but what if you need to process the output of a command line by line? Use process substitution:

  while read LINE;  do
     echo "-> $ LINE"
 done <<(grep PATH / etc / profile) 

Another couple of interesting ways to solve the problem with sub-shells are discussed in the Bash FAQ # 24 .

8. if [grep foo myfile]

Many people are embarrassed by the practice of putting square brackets after if and newbies often get the false impression that [ is part of a conditional syntax, just like brackets in conditional C language constructs.

However, such an opinion is a mistake! The opening square bracket ( [ ) is not part of the syntax, but a command that is equivalent to the test command, except that the last argument of this command must be a closing bracket ] .

if syntax

  if COMMANDS
 then
     COMMANDS
 elif COMMANDS # optional
 then
     COMMANDS
 else # optional
     COMMANDS
 fi 

As you can see, there are no [ or [[ !

Once again, [ is a command that accepts arguments and issues a return code; like all normal commands, it can display error messages, but, as a rule, it does not produce anything in STDOUT.

if executes the first set of commands, and depending on the return code of the last command from this set, determines whether a block of commands from the “then” section is executed or the script will continue.

If you need to make a decision depending on the output of the grep , you do not need to enclose it in round, square or curly braces, backward quotes, or any other syntax element. Just write grep as a command after the if :

  if grep foo myfile> / dev / null;  then
     ...
 fi 

Notice that we discard the standard output of grep : we don’t need a search result, we just want to know if the line is in the file. If grep finds a string, it returns 0, and the condition is met; otherwise (no line in the file), grep returns a value other than 0. In GNU grep, the redirection >/dev/null can be replaced with the -q option, which tells grep 'u that you don't need to output anything.

9. if [bar = "$ foo"]

As explained in the previous paragraph, [ is a command. As with any other command, bash assumes that the command is followed by a space, then the first argument, then a space, and so on. Therefore, you can not write anything without spaces! Right like this:

  if [bar = "$ foo"] 

bar , = , "$foo" (after substitution, but without word division) and ] are arguments to the command [ , so there must be a space between each pair of arguments so that the shell can determine where which argument starts and ends.

10. if [[a = b] && [c = d]]

Again the same mistake. [ Is a command, not a syntax element between an if and a condition, and certainly not a means of grouping. You cannot take the C syntax and convert it to the bash syntax by simply replacing the round brackets with square brackets.

If you want to implement a complex condition, here is the right way:

  if [a = b] && [c = d] 

Notice that here we have two commands after if, combined with the && operator. This code is equivalent to the following command:

  if test a = b && test c = d 

If the first test command returns false (any non-zero number), the condition body is skipped. If it returns true , the second condition is satisfied; if it returns true , the condition body is executed.

To be continued.

The first publication of this translation took place on the pages of my blog .