Interesting bash programming techniques
These techniques were described in Google’s internal project “Testing on the Toilet” (Testing in the toilet - distributing leaflets in the toilets to remind developers of the tests).
In this article, they were revised and supplemented.
I start each script with the following lines.
It protects against two common mistakes.
1) Attempts to use non-declared variables
2) Ignoring the abnormal completion of commands
If the command can complete abnormally, and it suits us, you can use the following code:
It must be remembered that some commands do not return a crash code such as “mkdir -p” and “rm -f”.
There are also difficulties with calling subroutine chains (command1 | command 2 | command3) from a script, to bypass this restriction, you can use the following construction:
In this case, the '&&' operator will not allow the next command to execute, for more details see the 'http://fvue.nl/wiki/Bash:_Error_handling'
Bash allows you to use functions as normal commands, it greatly increases the readability of your code:
Example 1:
Example 2:
Example 3:
Try transferring all your code to functions leaving only global variables / constants and calling the “main” function in which there will be all high-level logic.
')
Bash allows you to declare variables of several types, the most important:
local (For variables used only inside functions)
readonly (Variables attempt to reassign which causes an error)
It is possible to make a variable of type readonly from the already declared:
Strive to ensure that all your variables are either local or readonly, this will improve readability and reduce the number of errors.
Back quotes are poorly readable and in some fonts can be easily confused with single quotes.
The $ () construction also allows the use of nested calls without a headache with screening:
Double square brackets avoid unintended use of paths instead of variables:
In some cases, simplify the syntax:
As well as provide additional functionality:
New operators:
Extended / modified operators:
Examples:
Starting with bash 3.2, regular expressions or wildcard expressions should not be quoted, if your expression contains spaces, you can put it in the variable:
Comparing string variables with substitution is also available in the case statement:
Bash has several (underestimated) capabilities for working with string variables:
Basic:
Replacement with substitution:
Separation of variables:
Deletion with substitution:
# Delete from the beginning of the line, before the first match
# Delete from the beginning of the line, until the last match
# Delete from the end of the line, before the first match
# Delete from the end of the line, until the last match
Some commands expect the file name to be input, the operator '<()' will help us with it, it accepts a command as input and converts it into something that can be used as a file name:
# download two URLs and transfer them to diff
Use a marker to pass multiline variables:
# MARKER - any word.
If you need to avoid substitution, then you can put the marker in quotes:
# construct returns '$ var' and not the variable value
Example:
conclusion:
Syntax check (saves time if the script runs longer than 15 seconds):
Trace:
Tracing with the disclosure of complex commands:
The -v and -x parameters can be set in the code, this can be useful if your script runs on one machine and logging is done on another:
If your project matches the items in this list, consider Python or Ruby languages for it.
References:
Advanced Bash-Scripting Guide: tldp.org/LDP/abs/html
Bash Reference Manual: www.gnu.org/software/bash/manual/bashref.html
Original article: robertmuth.blogspot.ru/2012/08/better-bash-scripting-in-15-minutes.html
In this article, they were revised and supplemented.
Security
I start each script with the following lines.
#!/bin/bash set -o nounset set -o errexit It protects against two common mistakes.
1) Attempts to use non-declared variables
2) Ignoring the abnormal completion of commands
If the command can complete abnormally, and it suits us, you can use the following code:
if ! <possible failing command> ; then echo "failure ignored" fi It must be remembered that some commands do not return a crash code such as “mkdir -p” and “rm -f”.
There are also difficulties with calling subroutine chains (command1 | command 2 | command3) from a script, to bypass this restriction, you can use the following construction:
(./failing_command && echo A) In this case, the '&&' operator will not allow the next command to execute, for more details see the 'http://fvue.nl/wiki/Bash:_Error_handling'
Functions
Bash allows you to use functions as normal commands, it greatly increases the readability of your code:
Example 1:
ExtractBashComments() { egrep "^#" } cat myscript.sh | ExtractBashComments | wc comments=$(ExtractBashComments < myscript.sh) Example 2:
SumLines() { # iterating over stdin - similar to awk local sum=0 local line=”” while read line ; do sum=$((${sum} + ${line})) done echo ${sum} } SumLines < data_one_number_per_line.txt Example 3:
log() { # classic logger local prefix="[$(date +%Y/%m/%d\ %H:%M:%S)]: " echo "${prefix} $@" >&2 } log "INFO" "a message" Try transferring all your code to functions leaving only global variables / constants and calling the “main” function in which there will be all high-level logic.
')
Variable declaration
Bash allows you to declare variables of several types, the most important:
local (For variables used only inside functions)
readonly (Variables attempt to reassign which causes an error)
## DEFAULT_VAL , , '-7' readonly DEFAULT_VAL=${DEFAULT_VAL:-7} myfunc() { # local some_var=${DEFAULT_VAL} ... } It is possible to make a variable of type readonly from the already declared:
x=5 x=6 readonly x x=7 # failure Strive to ensure that all your variables are either local or readonly, this will improve readability and reduce the number of errors.
Use $ () instead of backquotes ``
Back quotes are poorly readable and in some fonts can be easily confused with single quotes.
The $ () construction also allows the use of nested calls without a headache with screening:
# : ABCD echo "A-`echo B-\`echo C-\\\`echo D\\\`\``" echo "A-$(echo B-$(echo C-$(echo D)))" Use double brackets [[]] instead of single []
Double square brackets avoid unintended use of paths instead of variables:
$ [ a < b ] -bash: b: No such file or directory $ [[ a < b ]] In some cases, simplify the syntax:
[ "${name}" \> "a" -o ${name} \< "m" ] [[ "${name}" > "a" && "${name}" < "m" ]] As well as provide additional functionality:
New operators:
- || Logical OR (logical or) - only with double brackets.
- && Logical AND (logical and) - only with double brackets.
- < Comparison of string variables (string comparison) - no escaping is required with double brackets.
- == Comparing string variables with globbing (string matching with globbing) - only with double brackets.
- = ~ Comparing string variables using regular expressions (string matching with regular expressions) - only with double brackets.
Extended / modified operators:
- -lt Digital comparison (numerical comparison)
- -n String variable is not empty (string is non-empty)
- -z String variable is empty (string is empty)
- -eq Digital Equality (numerical equality)
- -ne Digital no equality (numerical inequality)
Examples:
t="abc123" [[ "$t" == abc* ]] # true (globbing) [[ "$t" == "abc*" ]] # false (literal matching) [[ "$t" =~ [abc]+[123]+ ]] # true (regular expression) [[ "$t" =~ "abc*" ]] # false (literal matching) Starting with bash 3.2, regular expressions or wildcard expressions should not be quoted, if your expression contains spaces, you can put it in the variable:
r="a b+" [[ "a bbb" =~ $r ]] # true Comparing string variables with substitution is also available in the case statement:
case $t in abc*) <action> ;; esac Work with string variables:
Bash has several (underestimated) capabilities for working with string variables:
Basic:
f="path1/path2/file.ext" len="${#f}" # = 20 ( ) # : ${<>:<_>} ${<>:<_>:<_>} slice1="${f:6}" # = "path2/file.ext" slice2="${f:6:5}" # = "path2" slice3="${f: -8}" # = "file.ext" ( '-') pos=6 len=5 slice4="${f:${pos}:${len}}" # = "path2" Replacement with substitution:
f="path1/path2/file.ext" single_subst="${f/path?/x}" # = "x/path2/file.ext" ( ) global_subst="${f//path?/x}" # = "x/x/file.ext" ( ) Separation of variables:
f="path1/path2/file.ext" readonly DIR_SEP="/" array=(${f//${DIR_SEP}/ }) second_dir="${array[1]}" # = path2 Deletion with substitution:
# Delete from the beginning of the line, before the first match
f="path1/path2/file.ext" extension="${f#*.}" # = "ext" # Delete from the beginning of the line, until the last match
f="path1/path2/file.ext" filename="${f##*/}" # = "file.ext" # Delete from the end of the line, before the first match
f="path1/path2/file.ext" dirname="${f%/*}" # = "path1/path2" # Delete from the end of the line, until the last match
f="path1/path2/file.ext" root="${f%%/*}" # = "path1" Getting rid of temporary files
Some commands expect the file name to be input, the operator '<()' will help us with it, it accepts a command as input and converts it into something that can be used as a file name:
# download two URLs and transfer them to diff
diff <(wget -O - url1) <(wget -O - url2) Use a marker to pass multiline variables:
# MARKER - any word.
command << MARKER ... ${var} $(cmd) ... MARKER If you need to avoid substitution, then you can put the marker in quotes:
# construct returns '$ var' and not the variable value
var="text" cat << 'MARKER' ... $var ... MARKER Embedded variables
- $ 0 name of the script
- $ 1 $ 2 ... $ n Parameters passed to the script / function (positional parameters to script / function)
- $$ PID Script (PID of the script)
- $! PID of the last command executed in the background (PID of the last command executed (and run in the background))
- $? Status returned by the last command (exit status of the last command ($ {PIPESTATUS} for pipelined commands))
- $ # Number of parameters passed to script / function (number of parameters to script / function)
- $ @ All parameters passed to the script / functions presented as words (separate arguments)
- $ * All parameters passed to the script / functions, represented as a single word (sees arguments as single word)
- Usually:
- $ * Rarely helpful
- $ @ Correctly handles empty parameters and parameters with spaces.
- $ @ When used, usually enclosed in double quotes - "$ @"
Example:
for i in "$@"; do echo '$@ param:' $i; done for i in "$*"; do echo '$! param:' $i; done conclusion:
bash ./parameters.sh arg1 arg2 $@ param: arg1 $@ param: arg2 $! param: arg1 arg2 Debugging
Syntax check (saves time if the script runs longer than 15 seconds):
bash -n myscript.sh Trace:
bash -v myscripts.sh Tracing with the disclosure of complex commands:
bash -x myscript.sh The -v and -x parameters can be set in the code, this can be useful if your script runs on one machine and logging is done on another:
set -o verbose set -o xtrace Signs that you should not use shell scripts:
- Your script contains more than a few hundred lines.
- You need data structures harder than regular arrays.
- You zadolbalo engage lewdness with quotes and shielding.
- You need to process / change many string variables.
- You do not need to call third-party programs and there is no need for pipes.
- Speed / performance is important to you.
If your project matches the items in this list, consider Python or Ruby languages for it.
References:
Advanced Bash-Scripting Guide: tldp.org/LDP/abs/html
Bash Reference Manual: www.gnu.org/software/bash/manual/bashref.html
Original article: robertmuth.blogspot.ru/2012/08/better-bash-scripting-in-15-minutes.html
Source: https://habr.com/ru/post/221273/
All Articles