Console to the masses. Go to the bright side. Automate routine tasks

Introduction

Machines will always be faster, no matter how productive we are and how quickly we recruit teams. The harsh truth of life. On the other hand, if we perform the same action many times, then why not make the machines suffer. Write a script in bash (your favorite programming language) and call this script each time, rather than typing monotonous commands that take so much time, effort and energy. And we, while the script will do its work, we can dream about how the spacecraft surf the expanses of our Universe.

In the last article, we looked at the basics of bash programming. Today we will apply this knowledge in practice.

Automation plan

1. Fast diff
2. Quick diff + Jira API
3. Cleaning _dist
4. Up a large number of repositories
5. Cloning a large number of repositories
6. Useful aliases

The list includes tasks that I perform daily several times a day, or even an hour. In general, the automation of routine processes is a creative and highly personal process.
You can automate anything you can think of. I hope that at the end of the article you will have your own plan for automation and you will make the machines suffer. Make yourself a cup of aromatic coffee and sit back. Waiting for you a fascinating journey into the world of automation with bash .
')

Fast diff

In our work on the project, we use Git . Creating a diff is a fairly common task. To create a diff for a specific branch, run the following command:

 git diff origin/master origin/<branch-name> > "${HOME}/diff/diff-<branch-name>.diff" 

<branch-name> - the name of the branch for which you want to create a diff

Disadvantages of this approach

1. Each time the team must be typed manually
2. High probability of error in the recruitment process
3. Hard to remember

These flaws can be easily resolved with bash . Ideally, everything should work like this:

1. Typed team
2. Give her the name of the branch
3. Received diff

Final Team View

 gdd <branch-name> 

Automating

Now instead of a long line of commands, just type on the keyboard ./fast_diff.sh <branch-name> . If you forgot to specify the name of the branch, this script will kindly tell us about it.

Final touch

Stop, you say. But what about the final appearance of the team. Indeed, in the current form, this script is not very convenient, we are tied to the directory in which it is located.

Let us consider in more detail how to make a separate command for the executable file, rather than writing a relative / absolute path to it each time.

Each user has a bin subdirectory in his home directory ( ~ ). If not, then it can be created. It can store executable files. Convenience lies in the fact that such files are accessible by name and they do not need to indicate the relative / absolute path. I put the gdd file in this directory, which is responsible for creating the diff :

 #!/bin/bash "${HOME}/htdocs/rybka/tools/fast_diff.sh" "$@" 

A few important points:

1. The file is not taken to specify the extension.
2. For the file, you must explicitly specify the x (chmod +x <filename>) attribute x (chmod +x <filename>) .
   
3. If the bin directory is not in the $PATH variable, add it explicitly to PATH="${PATH}:${HOME}/bin" .

To make this file available, restart the terminal. Now, to create a diff , just run the following command:

 gdd <branch-name> 

It is not very convenient all the time to create a separate file in the bin directory for each command. We can optimize this by using symbolic links:

 ln -s "${HOME}/htdocs/rybka/tools/fast_diff.sh" gdd 

Quick diff + Jira API

If you use the Jira task manager or any other task manager that provides an API in your work, then you can go even further. For example, you can use the Jira API to attach diff to a specific task. To do this, we also need cURL .

Algorithm of the decision

1. Call the script
2. Pass the task id
3. If the task id not transmitted, we display a message to the user.
4. If everything is correct, we generate diff and attach it to the task.

Final Team View

 gdd_jira <issue_id> 

Automating

As you can see, the name of the branch does not need to be specified. We get it with the help of simple manipulations with git commands:

 branch=$(git rev-parse --abbrev-ref HEAD) 

Cleaning _dist

To begin with, let's see what the _dist directory is responsible _dist . This is the place where CSS , JavaScript files, various templates ( Jade / Pug , Handlebars , etc.) and other files get after the build system starts ( Grunt , Gulp , etc.). This directory does not have to be called _dist . Variations are possible.


For one of the projects we use Grunt. Quite often, our team faces the problem that Grunt does not always notice changes in some files (the problem is mostly with Less files). To correct this situation, you need to clear the _dist directory for one of the topics or for all topics at once. Yes, this problem can be solved with the help of Grunt. Even you can always delete this directory manually. But it will not be as efficient and convenient as in the case of bash . The number of these directories ( _dist ) is not one and not two, and not even ten or twenty. A lot of them. The main requirement for the script is not to apply extra wrappers and / or dependencies unless necessary.

Consider the option without bash . We use all the power of the shell to solve this problem:

 find <path-to-themes> -type d -name "_dist" | xargs rm -rfv 

<path-to-themes> - path to the directory where all themes are located

The disadvantages of this approach are approximately the same as I mentioned in the case of the task of creating a diff . Plus, in this case, it is not possible to specify for which particular topic you want to delete the _dist directory.

Algorithm of the decision

1. Call the script
2. If the topic name was not transferred, delete the _dist directory in all topics
3. If the topic name was transferred, we delete the _dist directory in a specific topic.

Final Team View

 clean_dist [<theme_name>] 

Automating

Up a large number of repositories

Imagine you are working with a large project. In this project there is a directory reserved for third-party repositories that you do not develop, but must keep them up-to-date. Of course, if these repositories are two or three, then this is not such a big problem. Although here I would argue. And if you have such repositories 10-15, and this figure is constantly growing. As a result, you do not have time to keep track of them or spend a disproportionately long time on support. Why not automate this task.

Algorithm of the decision

1. Go to the repository directory
2. Check that the repository is on the master branch.
3. If the repository is not on the master branch, do git checkout
4. Do git pull

An important point . Even if the repository is on the master branch, we cannot be sure that it is up to date. Based on this, we do git pull anyway.

Final Team View

 up_repo 

Automating

Cloning a large number of repositories

This task is closely related to the previous point of automation. In order for the end user to have the opportunity to use the previous command in practice, it is necessary to provide a set of third-party repositories that will be located in the bash/core/vendors , and about which the user, by and large, does not need to know anything. By analogy with the npm modules, this set of repositories should not be supplied with the main repository. All the user needs to do is execute the command and wait for the repository to clone.

Algorithm of the decision

1. The list of repositories is given as an array.
2. Run a loop on this array
3. Pay special attention if one vendor has more than one repository
4. We perform additional checks
5. We execute git clone

Final Team View

 clone_repo 

Automating

Useful aliases

I have a few questions for readers. You must answer honestly to yourself. How often do you use this command?

 git branch 

And this team?

 git status 

But this team?

 git push origin <branch-name> 

How about this command?

 ps aux | grep <user-name> 

That's right, this list is endless and he has his own. And then, unexpectedly, insight comes to us:



Right. For commands that you use often - create aliases. Here is just a small list of those aliases that I use:


To check which aliases are set for you, just run the alias command without parameters.

Where to put aliases

In most cases, for such purposes use the .bashrc , which is located in the user's home directory. There is also a file called .gitconfig , to which you can add aliases to work with git .

Do not change aliases late at night

Aliases are a powerful tool. Only here as with passwords. You should not change aliases late at night. One fine night, I changed one of the aliases and forgot. The next day, I spent half a day figuring out why nothing worked.

Instead of conclusion

As soon as I began to understand the basics of bash programming, the first thought that came to me was: “Stop, this is needed more for system administrators ...”. But at the same time, I understood that I needed this knowledge in order to at least somehow save myself from the daily routine tasks. Now I can say with confidence that this knowledge is needed not only by system administrators. They will be useful to anyone who at least interacts with a remote server or works on OS *nix like systems. For users who work on Windows OS, this knowledge will also be useful ( Bash on Ubuntu on Windows , Windows and Ubuntu Interoperability ). In the simplest case, a script is nothing more than a simple list of system commands written to a file. Such a file can facilitate your everyday work and eliminate the need to perform routine tasks manually.

Useful links for some of the bash features that were used in the examples:

1. I / O Redirection
2. Functions
3. Arrays
4. Double parentheses
5. Combining commands into chains (pipeline)
6. Completion and Completion Code
7. Aliases
8. How to add paths to the $ PATH variable correctly

That's all. Thanks for attention. Who read to the end, special thanks.