Command line art

For a week now, the English version of the art of command line has been hanging in the trending section on Github. For myself, I found this material incredibly useful and decided to help the community with its translation into Russian. There are probably several flaws in the translation, so you are welcome to send pull-requests to me here or to the author of the original work Joshua Levy here . (If PR is sent to me, then after reviewing the changes I will send them to Josh's master brunch). Special thanks to jtraub for help and typos.

')
( tss, in the read more readable )

Description

Main:

This publication is intended for both beginners and experienced people. Objectives: volume (to collect all the important aspects of using the command line), practicality (to give concrete examples for the most frequent use cases) and brevity (you should not delve into non-obvious things that you can read about elsewhere).
This document is written for Linux users, with the only exception being the “ MacOS only “ section. Everything else is suitable and can be installed under all UNIX / MacOS systems (and even Cygwin).
Focusing on the interactive Bash, but many things can also be used with other shells; and generally applicable to bash scripting
This instruction includes standard Unix commands and those for which you need to install third-party packages - they are so useful that they are worth installing.

Notes:

In order for the manual to remain one-page, all the information is inserted right here. You are smart enough to explore the issue yourself in more detail elsewhere. Use apt-get / yum / dnf / pacman / pip / brew (depending on your package management system) to install new programs.
On Explainshell, you can find a simple and detailed explanation of what commands, flags, pipes, etc. are.

The basics

Learn the basics of Bash. Just take and type man bash in the terminal and at least view it; it's pretty easy to read and it's not very big. Other shells can also be good, but Bash is a powerful program and Bash is always at hand (using only zsh, fish, etc., which certainly looks cool on your laptop, it limits you a lot, for example, you cannot use the capabilities of these shells on already existing server).
Learn at least one console text editor. Ideal Vim ( vi ), because he has no competitors, when you need to correct something quickly (even if you are constantly sitting on Emacs / some heavy IDE or on some fashionable hipster editor)
Know how to read the documentation through man (for the curious man, man ; man in the corners of the document in brackets adds a number, for example, 1 for ordinary commands, 5 for files, conventions, 8 for administrative commands). Look for manuals through apropos , and remember that some commands are not binaries, but Bash's built-in commands, and help on them can be obtained through help and help -d .
Learn how to redirect input and output via > and < and pipe | . Remember that > - overwrites the output file, and >> adds to it. Learn more about stdout and stderr.
Learn more about file glob expansion with * (and perhaps ? And { ... } ), quotes as well as the difference between double " and single " quotes. (For more on expanding variables, see below)
Be familiar with working with processes in Bash: & , ctrl-z , ctrl-c , jobs , fg , bg , kill , etc.
Know ssh , and the basics of passwordless authentication through ssh-agent , ssh-add , etc.
File basics: ls and ls -l (in particular, find out what each column in ls -l means), less , head , tail, and tail -f (or even better, less + F ), ln and ln -s ( find out the difference between symbolic links and hard links and why hard links are better), chown , chmod , du (for a quick summary of disk usage: du -hk * ). For file system management, df , mount , fdisk , mkfs , lsblk .
Networking basics: ip or ifconfig , dig .
Well know regular expressions and different flags to grep / egrep . Flags like -i , -o , -A , and -B are worth knowing.
Learn how to use apt-get , yum , dnf or pacman package management systems (depending on the distribution). Know how to search and install packages and be sure to have a pip installed to install command-line utilities written in Python (some of the ones that you find below are easiest to install via pip )

Daily use

Use the tab in Bash for autocomputing arguments to commands and ctrl-r for searching the command line history
Use ctrl-w in Bash to remove the last word in a command; ctrl-u to remove the command completely. Use alt-b and alt-f to run between the words of the command, ctrl-k to jump to the end of the line, ctrl-l to clear the screen. Look at the man readline to find out about all of Bash's shortcuts. A lot of them! For example, alt-. runs on the previous arguments of the command, and alt- * expands the glob. ??
If you like Wim shortcuts make set -o vi .
To view history enter history . There are also many abbreviations, for example ! $ - the last argument !! - the last command, although these abbreviations are often replaced by shortcuts ctrl-r and alt-. .
To jump to the last working directory - cd -
If you wrote a command in half and suddenly changed your mind, press alt- # to add # to the beginning and send the command as a comment. Then you can return to it through the story.
Do not forget to use xargs (or parallel ). This is a very powerful thing. Please note that you can control the number of commands per line, as well as parallelism. If you are not sure that you are doing something right, start with xargs echo . Another -I {} is a useful thing. Examples:

find . -name '*.py' | xargs grep some_function cat hosts | xargs -I{} ssh root@{} hostname

pstree -p is a useful type of process tree output.
Use pgrep or pkill to find / send signals to processes by name ( -f helps).
Know the different signals that can be sent to processes. For example, to pause a process, use kill -STOP [pid] . For a complete list, see man 7 signal .
Use nohup or disown to make the background process run indefinitely.
Find out what processes are listening to ports via netstat -lntp or ss -plat (for TCP; add -u for UDP).
Also lsof to view open sockets and files.
Use alias to alias frequently used commands. For example, alias ll = 'ls -latr' will create a new alias ll .
In Bash Scrites, use set -x to debug an output. Use strict modes wherever possible. Use set -e to stop execution on errors. Use set -o pipefail to be strictly error-prone (this is a bit of a deep topic). For more complex scripts, also use trap .
In Bash scripts, subshells (subshells) are a convenient way to group commands. One of the most common examples is to temporarily move to another working directory, like this:

  # do something in current dir (cd /some/other/dir && other-command) # continue in original dir

Bash has many types of variable space. Check if the variable exists - $ {name:? Error message} . For example, if a bash script needs just one argument, just type input_file = $ {1:? Usage: $ 0 input_file} . Arithmetic scope: i = $ (((i + 1)% 5)) . Sequences: {1..10} . Crop lines: $ {var% suffix} and $ {var # prefix} . For example, if var = foo.pdf then echo $ {var% .pdf} .txt will print foo.txt .
The output of any command can be converted to a file via <(some command) . For example, comparing the local file `/ etc / hosts with the remote:

 diff /etc/hosts <(ssh somehost cat /etc/hosts)

Know about the heredoc-syntax in Bash, it works like this cat << EOF ....
In Bash, redirect standard output as well as standard errors like this: some-command> logfile 2> & 1 . Often, in order to make sure that the team does not leave the file open, linking it to the open terminal is considered good practice to add </ dev / null .
Use man ascii for a good ASCII table, with hex and decimal values. For information on the basic encodings useful: man unicode , man utf-8 , and man latin1 .
Use screen or tmux to have multiple screens in one terminal, this is especially useful when you are working with a remote server, then you can connect / disconnect from sessions. A more minimalistic approach to this is using dtach .
In SSH, it is useful to know how to port tunnel with the -L and -D (and sometimes -R ) flags, for example, to access the site from a remote server.
It may also be useful to optimize your SSH configuration, for example, this ~ / .ssh / config file contains settings that help avoid lost connections in some network environments, use compression (which is useful with scp over slow connections) and increase the number of channels to one server through This config is like this:

 TCPKeepAlive=yes ServerAliveInterval=15 ServerAliveCountMax=6 Compression=yes ControlMaster auto ControlPath /tmp/%r@%h:%p ControlPersist yes

Some other SSH settings can greatly affect security and must be changed carefully, for example for a specific subnet or specific machine or in home networks: StrictHostKeyChecking = no , ForwardAgent = yes
In order to get file permissions in octal, which is useful for system configuration, but cannot be obtained from ls , you can use something like:

  stat -c '%A %a %n' /etc/timezone

To interactively highlight the results of other commands, use percol or fzf .
To work with files listed by another command (for example Git) use fpp ( PathPicker ).
To quickly set up a web server in the current directory (and sub-directories) that is accessible to everyone on your network, use:
python -m SimpleHTTPServer 7777 (for port 7777 and Python 2) and python -m http.server 7777 (for port 7777 and Python 3).
To execute a particular command with privileges, use sudo (for root) and sudo -u (for another user). Use su or sudo bash to run a shell on behalf of this user. Use su - in order to simulate a fresh login from the root or another user.

File and Information Processing

To find a file in the current directory, make a find. -iname '* something *' . To search for a file throughout the system, use locate something (but do not forget that updatedb could not yet index newly created files).
For a basic search through the contents of files (more complex than grep -r ), use ag .
To convert HTML to text: lynx -dump -stdin
To convert different types of markup (HTML, Markdown, etc.) try pandoc .
If you need to work with XML, there is an old but good utility - xmlstarlet .
To work with JSON, use jq .
To work with Excel and CSV files, use csvkit (the program provides the commands in2csv , csvcut , csvjoin , csvgrep , etc.)
For working with Amazon S3, it is convenient to work with s3cmd and s4cmd (the latter is faster). For other Amazon services, use standard aws .
Know about sort and uniq , including the -u and -d flags, see the examples below. Take another look at comm .
Know about cut , paste , and join for working with text files. Many people use cut forgetting to join .
Be aware of wc : for counting line breaks ( -l ), for characters - ( -m ), for words - words ( -w ), for byte counting - ( -c ).
Know about tee , to copy to a file from stdin and stdout, something like ls -al | tee file.txt .
Keep in mind that your locale affects many commands, including sorting orders, comparisons, and performance. Many Linux distributions automatically set LANG or any other variable to suit your region. Because of this, the results of sorting functions can work unpredictably. Routines i18n can significantly reduce the performance of sorts. In some cases, you can completely avoid this (except in rare cases), sorting traditionally byte-by-byte, for this export LC_ALL = C
Know the basics of awk and sed for simple data manipulation. For example, to get the sum of all the numbers that are in the third column of a text file, you can use awk '{x + = $ 3} END {print x}' . Most likely, it is 3 times faster and 3 times easier than doing it in Python.
To replace all occurrences of a substring in one or more files:

  perl -pi.bak -e 's/old-string/new-string/g' my-files-*.txt

To rename many files at once by a template, use rename . For complex renames, repren can help:

 #    foo.bak  foo: rename 's/\.bak$//' *.bak #    ,      foo  bar. repren --full --preserve-case --from foo --to bar .

Use shuf to shuffle lines or select a random line from a file.
Know the sort flags. For numbers, use -n , to work with human-readable numbers, use -h (for example, du -h ). Know how the keys work ( -t and -k ). In particular, do not forget that you need to write -k1,1 in order to sort only the first field; -k1 means sorting taking into account the entire line. Also stable sorting can be useful ( sort -s ). For example, in order to sort the most important by the second field, and the second by the first, you can use sort -k1,1 | sort -s -k2,2`.
If you ever have to write a tab code in a terminal, for example, to sort by taboo with the -t flag, use the shortcut ctrl-v [Tab] or type $ '\ t' . The latter is better because it can be copied.
The standard tools for patching source codes are diff and patch . Just look at diffstat to see diff statistics. diff -r works for the entire directory. Use diff -r tree1 tree2 | diffstat for a full summary of changes.
For binaries, use hd for simple hex-dump and bvi for binary changing binaries.
strings (in grep or something similar) helps to find strings in binaries.
To see the difference in binary (delta encoding), use xdelta3 .
To convert encodings use iconv . For more complex tasks, uconv , it supports some complex Unicode features. For example, this command translates lines from a file into lower case and removes accents (which are, for example, in Spanish)

  uconv -f utf-8 -t utf-8 -x '::Any-Lower; ::Any-NFD; [:Nonspacing Mark:] >; ::Any-NFC; ' < input.txt > output.txt

To split a file, use split (splits into pieces by size), or csplit (by pattern or regular expression)
Use zless , zmore , zcat , and zgrep to work with compressed files.

System debugging

For web debugging, use curl and curl -I , or their alternative to wget . There are also more modern alternatives, such as httpie .
For information on disk / CPU / network, use iostat , netstat , top (or the best alternative to htop ) and especially dstat . A good start to understand what is happening in the system.
For more details use glances . This program shows several different statistics in one terminal window. Useful when following multiple systems at once.
In order to monitor the memory, learn to understand free and vmstat . In particular, do not forget that cached values (“cached” value) are the memory that the kernel holds and these values are part of free .
Debagging Java is a completely different fish, but some manipulations of the Orakla or any other virtual machine will allow you to do kill -3 <pid> and trace stack and hip reports (including the details of the garbage collector’s work that are very useful) and can be dumped in stderr or logs.
Use mtr for better tracing to find network problems.
To find out why the disk is completely clogged, use ncdu , this saves time compared to the same du -sh * .
To find out which socket or process uses the Internet, use iftop or nethogs .
ab , which comes together with Apache, is useful for quick, inaccurate testing of web server performance. For more serious load-testing, use siege .
For more serious network debugging, use wireshark , tshark , and ngrep .
Know about strace and ltrace . These commands can be useful if the program crashes or hangs and you do not know why, or if you want to test the performance of the program. Do not forget about the possibility of debugging ( -c ) and the opportunity to cling to the process ( -p ).
Do not forget about ldd to check the system libraries.
Know how to cling to the running process via gdb and get a stack trace.
Use / proc . Sometimes it is incredibly useful for debugging running programs. Examples: / proc / cpuinfo , / proc / xxx / cwd , / proc / xxx / exe , / proc / xxx / fd / , / proc / xxx / smaps .
When debugging something that has broken in the past, use sar - it can be very useful. Shows the history of CPU, memory, network, etc.
For an analysis of deeper systems and performance, look at stap ( SystemTap ), perf , and sysdig .
Find out what operating system you have through uname or uname -a (basic Unix-info / information about the kernel) or lsb_release -a (information about the distribution).
Use dmesg when something behaves very strange (for example, hardware or drivers).

One line

Let's put it all together and write a few commands:

It's pretty cool that you can find multiple file intersections, merge sorted files and see the difference in several files through sort / uniq . This is a quick approach and works on files of any size (including multi-gigabyte files). (The sorting is not limited by memory, but you may have to add -T if / tmp is on a small logical drive). Another look at what was said above about LC_ALL . The -u option flag is not used below to make it clearer:

 cat ab | sort | uniq > c # c is a union b cat ab | sort | uniq -d > c # c is a intersect b cat abb | sort | uniq -u > c # c is set difference a - b

Use grep. * in order to view the contents of all files in the directory, especially when you have many configs such as / sys , / proc , / etc.
Get the sum of all the numbers that are in the third column of the text file. (Most likely, it is 3 times faster and 3 times easier than doing it in Python):

  awk '{ x += $3 } END { print x }' myfile

If you need to see the sizes and dates of file tree creation, use:

 find . -type f -ls

This is almost like recursive ls -l , but more readable than ls -lR :

Use xargs (or parallel ). This is a very powerful thing. Please note that you can control the number of commands per line, as well as parallelism. If you are not sure that you are doing something right, start with xargs echo . Another -I {} is a useful thing. Examples:

 find . -name '*.py' | xargs grep some_function cat hosts | xargs -I{} ssh root@{} hostname

Let's imagine that we have some kind of text file, for example, the log of some server and a value appears on some lines, the lines with which we are interested, for example acct_id . Let's calculate how many such queries in our log:

  cat access.log | egrep -o 'acct_id=[0-9]+' | cut -d= -f2 | sort | uniq -c | sort -rn

Run this script to get random advice from this instruction:

  function taocl() { curl -s https://raw.githubusercontent.com/jlevy/the-art-of-command-line/master/README-ru.md | pandoc -f markdown -t html | xmlstarlet fo --html --dropdtd | xmlstarlet sel -t -v "(html/body/ul/li[count(p)>0])[$RANDOM mod last()+1]" | xmlstarlet unesc | fmt -80 }

Difficult but useful

expr : to perform arithmetic and boolean operations, as well as regular expressions
m4 : a simple macro processor
yes : output a string in an infinite loop
cal : cool calendar
env : to execute a command (useful in bash scripts)
printenv : print out environment variables (debugging and scripts)
look : find english words (or strings) in a file
cut :, paste and join : data manipulation
fmt : paragraph formatting in text
pr : format text into pages / columns
fold : (wrap) limit the length of lines in the file
column : format text in columns or tables
expand : and unexpand : conversion between tabs and spaces
nl : add line numbers
seq : display numbers
bc : calculator
factor : raise numbers to a power
gpg : encrypt and sign files
toe : terminfo terminal table with description
nc : network debugging and data transfer
socat : socket switch and tcp port forwarding (similar to netcat )
slurm : network traffic visualization
dd : transfer information between files and devices
file : get file type
tree : show directories and subdirectories in the form of a tree; like ls but recursively
stat : file information
tac : output file in reverse (lasipan)
shuf : random selection of lines from a file
comm : compare sorted files line by line
pv : monitoring the progress of the passage of information through the pipe
hd : and bvi : dump and edit binaries
strings : find text in binaries
tr : char manipulation (character type)
iconv : and uconv : convert encodings
split : and csplit : split files
sponge : read the whole input before writing it is useful when reading from the same file where you write it, for example like this: grep -v something some-file | sponge some-file
units : converter, meters to kelometers, versts per span (see /usr/share/units/definitions.units )
7z : High Compression Archiver
ldd : shows program dependencies on system libraries
nm : we get the names of all functions that are defined in .o or .a
ab : web server benchmarking
strace : system call debug
mtr : better tracing for debugging network
cssh : several terminals in one UI
rsync : synchronize files and folders via SSH
wireshark : and tshark : packet capture and network debug
ngrep : grep for network layer
host : and dig : learn DNS
lsof : descriptor processing and socket information
dstat : useful system statistics
glances : high-level, multi-system statistics
iostat : CPU and hard disk usage statistics
htop : improved top version
last : login history to the system
w : under what user are you sitting
id : user / group information
sar : history of system statistics
iftop : or nethogs : use of a network by a specific socket or process
ss : socket statistics
dmesg : buta errors and system errors
hdparm : manipulation with SATA / ATA
lsb_release : Linux distribution information
lsblk : list of computer block devices: tree of your disks and logical disks
lshw :, lscpu , lspci , lsusb , dmidecode : information about hardware including, CPU, BIOS, RAID, graphics, devices, etc.
fortune :, ddate , and sl : hm, I don’t know if funny quotes and trains crossing your terminal will be “useful” to you :)

MacOS only

Some things are relevant only for Mac.

Package management systems - brew (Homebrew) and port (MacPorts). They can be used to install most of the programs mentioned in this document.
Copy output of console programs to desktop via pbcopy , and paste input via pbpaste .
To open a file or desktop program like Finder, use open , like this open -a /Applications/Whatever.app .
Spotlight: Search for files in the console via mdfind and see the metadata (for example, EXIF photo information) via mdls .
Do not forget that MacOS is based on BSD Uni and many commands (for example, ps , ls , tail , awk , sed ) have some slight differences with Linux. This is due to the effect of UNIX System V and GNU Tools . The difference can be noticed by seeing the “BSD General Commands Manual.” Header for the programs. In some cases, you can install GNU versions of programs on Mac, for example gawk and gsed . When writing cross-platform Bash scripts, try to avoid commands that may differ (for example, use Python or perl better), or test everything thoroughly.

Disclaimer

With a few exceptions, all the code is written so that others can read it.

To whom much is given, much is asked. The fact that something can be written in Bash does not mean that it should be written there. ;)

License

Translated with the help of friends from Canada .

The original work and translation into Russian is distributed under the Creative Commons Attribution-ShareAlike 4.0 International License .

️ bootstrap.build

Source: https://habr.com/ru/post/262127/

All Articles