findNclean: find and clean

No, no, this post is not about pain and suffering! Even a little bit the other way around. Just a picture reminds of the significance of the first line of the future code.
At first, I just wanted to describe a fresh utility , piping hot from under the keyboard. By itself, it is completely nothing (even to the brilliance of the badge of the newly-made demob, to polish and polish it), but to describe it only turned out to be somehow boring. Then I decided that using her example I could share bashism, which I had not used before. Then I thought that you can kill two birds with one stone and describe it together with life examples. But in the end I realized that it was not worthwhile to torture the already pretty shabby little animals, and decided to simply comb them a little. The third hare (which is not a hare at all, but a boa constrictor) wisely chose to refrain from participating in this mess.

So if all of a sudden, you regularly have to look for files and can even then move them somewhere; or don't you care what layer of dust lies on the file system of your server with uptime of several hundred years and you are just interested in bash; or if you just passed by {, a, o} , then do not pass by!
')
findNclean is created on the basis of a great and mighty find , no less a great mv , and all this is in the soup from bash .

Of course, you could write everything in python and not bother with the “subtleties” of bash , but I didn’t have such a choice, plus bash has better portability between different environments — just an executable file and no libraries, virtual environments, and related problems with launching on vinaigrette from platforms. Who does not agree - throw in me ... something, just to get it :). So I had to tinker, including adaptation for old CentOS 5 environments. But the dependencies turned out to be at least for new axes (such as Ubuntu 16.04): find mv wc , for old ones plus date touch . The rest is all clean bash.

Main usage scenario

• You create a configuration file where you describe the rules for searching for a specific category of files / directories / symlinks. Each rule is usually a separate find process that writes the found files to its own log.
• If you need not only a list of files in itself, but for further movement, you also describe the options for moving there.
• Run the search process. He will create a csv-log with the list of items found.
• You check this list, if suddenly something is wrong there - delete these lines.
• If the global parameter APPROVED_SUFFIX is specified in the config (in order to avoid moving files from an APPROVED_SUFFIX list), add this suffix to the log names. By default, APPROVED_SUFFIX empty, i.e. no renaming is required.
• You start the process of moving. It will move all the files for which the logged parameters match the real, and for which it does not match - will not move with the message to the movement error log. Of course, it will create a log when, what, where it moved.

All documentation can be reached through a call with no parameters at all: ./findNclean

The utility consists of executable and configuration files. All ./findNclean help -c options available can be found in the output ./findNclean help -c
You can run multiple instances of findNclean at the same time with different configs.

If suddenly something is going to go wrong, you can set the NOTIFICATION_SCRIPT variable with the path to the script, then parameters will be passed to it with which it can do something, for example, send alerts.

All characters are supported in file names except for line breaks, therefore files with hyphenations are ignored by default, although if you only need to compile a list of files (without further movement), you can also turn them on with the option IGNORE_FILES_WITH_NEWLINES = false .

In the code, all documentation is crammed into variables (which are then displayed on the screen if necessary):

 read -r -d '' VARIABLE <<'EOF' bla-bla line EOF 

this is better than a periodically occurring construct:

 VARIABLE=$(cat <<'EOF' The bla-bla line EOF ) 

for it uses the built-in function without creating a subshell and calling an external program.

Creating a process is a rather slow operation compared to running a built-in command, so if you don’t pull system binaries without a special need, the speed of execution of a slowpoke bash script can increase by an order of magnitude.

When documentation variables are no longer needed, they are reset, empty, but nice:

 unset DESCRIPTION USAGE CONFIG_USAGE TRICKS # No need help variables yet 

Closer to the header, the versions of the programs used are checked to see in which environment the utility is now located. For the bash version, it looks something like this:

 printf -v BV '%d%03d%03d' "${BASH_VERSINFO[0]}" "${BASH_VERSINFO[1]}" "${BASH_VERSINFO[2]}" # Bash version in numbers like 4003046, where 4 is major version, 003 is minor, 046 is subminor. ((BV < 3002025)) && echo "WARNING: bash version ${BASH_VERSION} is too old (below 3.2.25)! This app was not tested with too ancient versions!" >&2 

printf in this case is better than BV=$(bla-bla) , because again the built-in utility will be executed without launching the subshell. In addition, it allows you to format the source data.

Perhaps there is a more elegant way of checking the version than stuffing it into an int with further comparison as with a decimal number, if you have such a thing lying around in your bins, lay it out!
In the second line of the example ((BV < 3002025)) also better than [[ "${BV}" -lt 3002025 ]] , since faster, shorter and more human readable.


For boolean variables, it is also convenient to use double parentheses:

 RULES_JOIN_ALLOW=1 # Allow optimize performance by join Rules with identical search_paths collection if ((RULES_JOIN_ALLOW)); then : # Do something here fi 

Many people know about the notation of the type $'\r'$'\n' , it is better to use them with the echo built-in command instead of the often recommended escape sequences for the external utility /bin/echo , called with echo -e (the built-in does not support them) . But in quoted expressions, this is somewhat inconvenient, you often have to close and open quotes. Fortunately, the same line break can be assigned to a variable and then no openings-closures

 readonly EOL=$'\n' # Newline ... approve|confirm) quit 255 "This action is not implemented yet.${EOL}You have to manually overview and approve list of files by appending${EOL}corresponding APPROVED_SUFFIX to the result_list value.${EOL}By default APPROVED_SUFFIX is none, so all result_list will be treated as approved." 

The same example appeals to the eternal holivar "tabs vs spaces". So as not to spoil the formatting when using tabs, you could implement it via a here-doc with indents:

  read -r -d '' tmp <<-'EOF' This action is not implemented yet. You have to manually overview and approve list of files by appending corresponding APPROVED_SUFFIX to the result_list value. By default APPROVED_SUFFIX is none, so all result_list will be treated as approved. EOF quit 255 "${tmp}" unset tmp 

When using spaces, you can also get out:

  tmp='This action is not implemented yet.'$'\n' tmp+='You have to manually overview and approve list of files by appending'$'\n' tmp+='corresponding APPROVED_SUFFIX to the result_list value.'$'\n' tmp+='By default APPROVED_SUFFIX is none, so all result_list will be treated as approved.' quit 255 "${tmp}" unset tmp 

But I like my solution more, and it doesn’t matter whether there are tabs or spaces, although the string becomes indecently long.

By the way, at one time, tabs were preferable for me precisely because of the possibility to use here-doc with the destruction of indents when parsing. But the code with tabs is very problematic to insert into an interactive terminal for tests - all sorts of autocompletions spoil everything. So now I do not even know how to live in this imperfect world. Google Inc. is on the side of the gaps and I am gradually leaning in the same direction .

The insertion of a timestamp from time immemorial has been done using date , although in new versions of bash you can use printf %(datefmt)T , but without the support of fractions of a second:

 readonly TIMESTAMP_FORMAT='%Y-%m-%dT%H:%M:%S' ... if ((BV > 4002000)); then # Modern bash versions log() { ## Fast (builtin) but sec is min sample for most implementations printf "%(${TIMESTAMP_FORMAT})T %5d %s\n" '-1' $$ "$*" # %b convert escapes, %s print as is } else # Legacy bash versions log() { ## Slow (subshell, date) but support nanoseconds echo "$(exec -c date +"${TIMESTAMP_FORMAT}") $$ $*" } fi 

Here you can see the use of $(exec -c some command) to run an external utility in a subshell. This allows you to get rid of the additional layer process between the current shell and the utility, well, just faster (by 0-2% depending on the situation, but with the world on a thread on the cap will be typed). This is what the $(some command) tree clipping will look like:

 ├── /bin/bash #   └── /bin/bash #   └── some command 

Together with exec there will be no extra layer, and with the -c option the command will be run with an empty environment, which will also speed up the work slightly.

The following construction is used to calculate the execution time of a part of the code (up to a second):

 if ((BV > 4002000)); then # Modern bash versions ## Set global variable with the name $1 and time format $2 set_timestamp() { printf -v "$1" "%($2)T" '-1' } else # Legacy bash versions set_timestamp() { printf -v "$1" '%s' "$(exec -c date +"$2")" } fi ... set_timestamp ts '%s' #      set_timestamp TS '%s' log "Completed in $((TS-ts)) sec" 

As an alternative, the special variable SECONDS could be used. It is equal to 0 at the start of the shell and is increased by one every second. If you nullify it, and then refer to it, you will get the execution time of the code segment between these events in seconds. In order not to lose the current value of the script execution time from the very beginning, you can also use intermediate variables:

 ts=${SECONDS} #      TS=${SECONDS} log "Completed in $((TS-ts)) sec" 

This, I think, is even more preferable than using the set_timestamp function.

Reading a file line by line is also not entirely trivial:

 while IFS= read -r line || [[ -n ${line} ]]; do ... done </path/to/some/text/file 

Cleaning IFS for read necessary so that the initial and final tabs or spaces are not removed;
-r - so that the string is read as is, without interpreting escape sequences of the \t\r ;
|| [[ -n ${line} ]] || [[ -n ${line} ]] - so that the code inside the loop is executed even if the last line of the file does not contain a line break, without it read returns a non-zero code and the cycle ends immediately.

Interesting things happen with the time of creating a file in Linux (see the born option): even if the file system (for example ext4) supports it and diligently writes it down every time a new file is created, stat and, accordingly, find cannot read it:

 user@host:~$stat / : '/' : 4096 : 8  /: 4096  : fc00h/64512d Inode: 2 : 25 : (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) : 2017-03-16 15:28:06.121675004 +0700 : 2017-02-27 17:09:09.937545795 +0700 : 2017-02-27 17:09:09.937545795 +0700 : - 

but it is!

 user@host:~$sudo debugfs -R "stat <$(stat -c %i /)>" /dev/ROOTFSDRIVE Inode: 2 Type: directory Mode: 0755 Flags: 0x80000 Generation: 0 Version: 0x00000000:00000086 User: 0 Group: 0 Size: 4096 File ACL: 0 Directory ACL: 0 Links: 25 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x58b3fac5:df87410c -- Mon Feb 27 17:09:09 2017 atime: 0x58ca4c96:1d0273f0 -- Thu Mar 16 15:28:06 2017 mtime: 0x58b3fac5:df87410c -- Mon Feb 27 17:09:09 2017 crtime: 0x5801149b:00000000 -- Sat Oct 15 00:23:39 2016 Size of extra inode fields: 32 EXTENTS: (0):9249 

Such is the flaw .

To move, mv used in a slightly unusual way (for an interactive terminal):

 #  ,  dest_name -  mv --backup=t -vT -- "${name}" "${dest_name}" #  ,  dest - ,  ,   '--', -     mv --backup=t -vt "${dest}" -- "${name}" #      , #       ,     ! mv --backup=off -vft "${dest}" -- "${name}" 

--backup[=CONTROL] is a useful thing, in case of some kind of malfunction or unforeseen situation the probability of turning everything backwards greatly increases.
-- also useful in scripts (this applies not only to mv , but also to others), because to some extent decouple from the use of a limited set of characters in the values ​​of variables (for example, if the value starts with a hyphen). In this case, it is not so necessary, because all variables for mv will first have either / or ./ to avoid, but it is better to use this feature anyway.

To check the existence of a binary in the system instead of the frequently used which , it is better to use the built-in hash :

 for util in ${DEPENDENCIES}; do hash "${util}" &>/dev/null || quit 1 "ERROR: '${util}' not found on this system" done; unset util 

To parse the command line options (who don't know yet), it is convenient to use the getopts builtin command:

 OPTIND=2 # Ignore first argument ACTION even if it has leading hyphen while getopts ':dvc:' OPT; do [[ "${OPTARG:0:1}" = '-' ]] && quit 1 "ERROR: Option argument cannot start with hyphen, got: ${OPTARG}" case "${OPT}" in d|v) DEBUG=1; v=v; ;; c) CONFIG=${OPTARG} ;; :) quit 1 "ERROR: Option -${OPTARG} requires an argument"; ;; *) quit 1 "ERROR: Unrecognized option: -${OPTARG}" esac done 

True, the scope of its application is somewhat limited by the fact that it does not support long options (starting with -- ).

To convert the values ​​of variables (not themselves, but only output) or arrays, there is a whole set of functions following the variable name in curly brackets:

 ${var#word} #     ${var##word} #     ${var%word} #     ${var%%word} #     ${var/pattern/string} #    ${var^pattern} #       ${var^^pattern} #       ${var,pattern} #       ${var,,pattern} #       

There is even an undocumented possibility:

 ${var~} #      ${var~~} #      

In the code there are many where you can meet. Here is one of the most entertaining for an array of commands for parallel execution:

 parallel_run "${Commands[@]/#/exec -c }" # Prepending of 'exec -c ' need to avoid additional subshell creation 

It inserts ' exec -c ' at the beginning of each cell in the array.

In the same way, all the values ​​inside the array can be summed (only integers of course): $((${MOVE_OK_COUNT[@]/#/+}))


In the m [io] re bash still a lot of pitfalls and you can dive after them until you drown even if you have mastered man bash and we sew random passersby regularly take off their hat in front of you.

I will end with a quote from the above epic:

 BUGS It's too big and too slow. 

Therefore, if possible, choose the #!/bin/bash shebang #!/bin/bash !


PS For a snack, since we are talking about search and bash

 ## Find files by shebang shebang='#!/bin/bash'$'\n' search_path=./ while IFS= read -rd $'\0' filename; do IFS= read -rd '' -N ${#shebang} firstbytes <"${filename}" [[ "${firstbytes}" == "${shebang}" ]] && echo "${filename}" done < <(exec -c find "${search_path}" -type f -size +${#shebang}c -readable -print0) 

You already understand the functionality of some of the “excesses” in this set of beeches;)?