Makefile - smart shell scripts for Django
Make is one of the basic Unix utilities, is on every Unix machine, and, unfortunately, unknown to many web developers. In this article I will show how it can simplify the work, and first - the formulation of the problem.
To deploy a project on Dzhanga or Rails on a new server, you need to execute commands five to ten. If the developer has corrected the database scheme, he needs to execute a couple of commands and restart the server. If you change the list of third-party packages, you need to rebuild and restart, 2 more commands. Changed the translation - two more. And if I changed 2 things at once, DB and translation? Or 3? It is necessary to remember, check, errors are possible. But we use a computer! He is smart enough to figure out what to update. Imagine a factory of drills, where you could not make a screwdriver and work with screwdrivers. Stupid right
Whether Jeff Atwood, or Paul Graham once wrote that programmers should spend more time developing tools for themselves. Since we are writing code abstractions (functions, modules) to reduce its repetition, then we also need to deal with tools in order to have less repetitive work.
Let's see this: what do we need in the cases described above? (Installation, updates.) We set the server so that he did? Earned. We update packages and want what? To change something, and the server started working again. We do not care how, most importantly - quickly, that is, so that the excess is not updated.
')
Of course, for all these cases, you can write scripts, but again you have to remember which one. Run the wrong one - and time is wasted.
On the diagram at the beginning of the article are the actions that need to be performed in my fastdev-django project. Until recently, I had, in fact, had to keep an analogue of this scheme in my head and every time I remembered what to do. However, the final goals in the scheme are only those that are highlighted in yellow. The remaining actions are intermediate stops, and a good system should ensure that I do not remember them.
The Make utility is quite flexible and can track exactly what needs to be rebuilt, so that everything is updated, and not rebuild too much. This is how the rule in the Makefile looks like that the Django server needs:
( full text file )
When there is such a file, the work becomes easier:
1. Changed the translation - and if you need to look at it, go to the console and write make run. And if you do not need to look right away, you just correct the translation. Another time will update.
2. Added packages depending - just restart the server via make run or console in make shell_plus.
Here I will not describe how the Makefiles are written; there is enough material for this on the Web: a book from O'Reilly (in English), a manual in Russian , a book, Effective Use of GNU Make .
It was possible to customize the file so that any translations were compiled (not specific files), but only updated ones.
Unable to automate database migration. Of course, you can write a command that will delete the database and recreate it, but it is dangerous for the working server. There is a more subtle tool, South , but for now (version 0.7.3) it is better not to call it automatically. If we write every change in the database to the migration (they contain both the delts and the full state of the tables), then this is what happens: I added a field, a migration appeared. Then I worked, did not commit anything yet, but deleted this field. Another migration appeared on the disk. I saved the files in WOS , sent to other developers. They will have to run unnecessary migrations, and the repository will swell out of unnecessary files.
Listing: files in the migration folder. 0002 creates a field, 0003 deletes it. The size of each - 17 kilobytes!
If you disable the automatic launch of South, you will have to remember what changes are happening in the database. And here we return to the beginning of the article: we must remember what the computer can remember.
Once I heard this argument: “Let the developer remember. Remembering and rechecking everything is a matter of work culture. ”While I myself was programming in C and PHP, I also thought that I had to be careful and, for example, diligently go to the project wiki to maintain my library’s documentation there. In Python, there is a docstring, and the description of the module, class, or method lies next to the code. The same result is achieved with less effort, and mistakes happen less often: looking at the code you can check if the documentation is not outdated.
Even in other industries where technical solutions are more complex and expensive, they do the same. No matter how careful the pilot of the aircraft, he may be wrong, so, for example, the mechanism of raising the chassis is locked automatically, in order not to accidentally release the wheels at high speed, and not remove, standing on the ground.
Chassis An-124, Airliners.net
Therefore, I believe that the culture of work - on the contrary, reduce the load on memory and simplify work, save time for people (if the savings are worth it, of course), as well as educate workers so that they can repair their tools if necessary.
1. Restarting the server when updating a translation is also a waste of time, especially if there is an auto-restart. I want to patch django.utils.autoreload, so that it monitors not only the loaded modules, but also text and compiled translations.
2. In the Make syntax, there are variables, functions, and lazy calculations. This is another markup language to remember. It would be nice to write the same, but on the Python? Worth a try scons . However, it is not on all systems, and Make is on Linux and on Macs.
Addendum: in the comments, artifex suggests the Fabric system, scripts similar to Make on Python. In my opinion, the most convenient of the proposed.
To deploy a project on Dzhanga or Rails on a new server, you need to execute commands five to ten. If the developer has corrected the database scheme, he needs to execute a couple of commands and restart the server. If you change the list of third-party packages, you need to rebuild and restart, 2 more commands. Changed the translation - two more. And if I changed 2 things at once, DB and translation? Or 3? It is necessary to remember, check, errors are possible. But we use a computer! He is smart enough to figure out what to update. Imagine a factory of drills, where you could not make a screwdriver and work with screwdrivers. Stupid right
Whether Jeff Atwood, or Paul Graham once wrote that programmers should spend more time developing tools for themselves. Since we are writing code abstractions (functions, modules) to reduce its repetition, then we also need to deal with tools in order to have less repetitive work.
Let's see this: what do we need in the cases described above? (Installation, updates.) We set the server so that he did? Earned. We update packages and want what? To change something, and the server started working again. We do not care how, most importantly - quickly, that is, so that the excess is not updated.
')
Of course, for all these cases, you can write scripts, but again you have to remember which one. Run the wrong one - and time is wasted.
On the diagram at the beginning of the article are the actions that need to be performed in my fastdev-django project. Until recently, I had, in fact, had to keep an analogue of this scheme in my head and every time I remembered what to do. However, the final goals in the scheme are only those that are highlighted in yellow. The remaining actions are intermediate stops, and a good system should ensure that I do not remember them.
Decision
The Make utility is quite flexible and can track exactly what needs to be rebuilt, so that everything is updated, and not rebuild too much. This is how the rule in the Makefile looks like that the Django server needs:
.PHONY: run run: bin / django syncdb bin / sass compile_trans ./bin/django runserver 0.0.0.0:8000 .PHONY: shell_plus shell_plus: bin / django syncdb bin / sass ./bin/django shell_plus bin / django: bin / buildout buildout.cfg ./bin/buildout bin / buildout: python bootstrap.py --distribute syncdb: bin / django ./bin/django syncdb # touch a timestamp file to avoid re-syncing touch syncdb make_trans: bin / django ./bin/django makemessages -a -e 'py, html, haml' bin / sass: gem install sass --bindir bin # watch buildout.cfg buildout.cfg: compile_trans: make locale / * / LC_MESSAGES / django.mo locale /% / LC_MESSAGES / django.mo: locale /% / LC_MESSAGES / django.po ./bin/django compilemessages -l $ (word 2, $ (subst /, $ @)) locale / * / LC_MESSAGES / django.po:
( full text file )
When there is such a file, the work becomes easier:
1. Changed the translation - and if you need to look at it, go to the console and write make run. And if you do not need to look right away, you just correct the translation. Another time will update.
2. Added packages depending - just restart the server via make run or console in make shell_plus.
Here I will not describe how the Makefiles are written; there is enough material for this on the Web: a book from O'Reilly (in English), a manual in Russian , a book, Effective Use of GNU Make .
It was possible to customize the file so that any translations were compiled (not specific files), but only updated ones.
Unable to automate database migration. Of course, you can write a command that will delete the database and recreate it, but it is dangerous for the working server. There is a more subtle tool, South , but for now (version 0.7.3) it is better not to call it automatically. If we write every change in the database to the migration (they contain both the delts and the full state of the tables), then this is what happens: I added a field, a migration appeared. Then I worked, did not commit anything yet, but deleted this field. Another migration appeared on the disk. I saved the files in WOS , sent to other developers. They will have to run unnecessary migrations, and the repository will swell out of unnecessary files.
Listing: files in the migration folder. 0002 creates a field, 0003 deletes it. The size of each - 17 kilobytes!
29K 2011-10-13 12:33 0001_initial.py 17K 2011-11-13 02:07 0002_auto__add_field_address_test_field.py 17K 2011-11-13 02:07 0003_auto__del_field_address_test_field.py
If you disable the automatic launch of South, you will have to remember what changes are happening in the database. And here we return to the beginning of the article: we must remember what the computer can remember.
Work culture
Once I heard this argument: “Let the developer remember. Remembering and rechecking everything is a matter of work culture. ”While I myself was programming in C and PHP, I also thought that I had to be careful and, for example, diligently go to the project wiki to maintain my library’s documentation there. In Python, there is a docstring, and the description of the module, class, or method lies next to the code. The same result is achieved with less effort, and mistakes happen less often: looking at the code you can check if the documentation is not outdated.
Even in other industries where technical solutions are more complex and expensive, they do the same. No matter how careful the pilot of the aircraft, he may be wrong, so, for example, the mechanism of raising the chassis is locked automatically, in order not to accidentally release the wheels at high speed, and not remove, standing on the ground.
Chassis An-124, Airliners.net
Therefore, I believe that the culture of work - on the contrary, reduce the load on memory and simplify work, save time for people (if the savings are worth it, of course), as well as educate workers so that they can repair their tools if necessary.
What's next
1. Restarting the server when updating a translation is also a waste of time, especially if there is an auto-restart. I want to patch django.utils.autoreload, so that it monitors not only the loaded modules, but also text and compiled translations.
2. In the Make syntax, there are variables, functions, and lazy calculations. This is another markup language to remember. It would be nice to write the same, but on the Python? Worth a try scons . However, it is not on all systems, and Make is on Linux and on Macs.
Literature
- GNU Make , O'Reilly, 2003
- GNU Make Guide
- Effective use of GNU Make , Vladimir Ignatov, 2000
- Python's Makefile , Ian Bicking
- Setuptools documentation
Addendum: in the comments, artifex suggests the Fabric system, scripts similar to Make on Python. In my opinion, the most convenient of the proposed.
Source: https://habr.com/ru/post/132524/
All Articles