Automating code checking or some more about pre-commit hooks
I think there is no need to tell the habrew user what Git / GitHub is, pre-commit and how to put a hook to it on the right. Let's get right to the point.
There are many examples of hooks in the network, most of them are on shells, but no author has paid attention to one important point - you have to drag a hook from project to project. At first glance - do not worry. But suddenly there is a need to make changes to the hook that already lives in 20 projects ... Or suddenly you need to transfer the development from Windows to Linux, and the hook to PowerShell ... What to do???????? PROFIT ...
Examples, of course, are greatly exaggerated, but with their help, inconveniences have been revealed that I would like to avoid. It would be desirable, that the hook was not required to be dragged on all projects, it was not necessary to “finish” often, but so that he could:
And it looked like this:
It is clear that the hook itself is just a starter, and the script that it launches performs all thespecial street magic. Let's try to write such a script. Interested - welcome under cat.
But beforedownloading a ready-made example from the network to start developing, consider the parameters that accept them. At the same time on their example I will tell how everything works.
')
These parameters will set the basic behavior of the script:
Both parameters will be optional (to be able to leave only one type of check), but if you do not specify any of them, pre-commit.py will exit with the code “1” (error).
And add auxiliary parameters (all optional):
The logic is clear, now you can start writing the script itself.
To begin, configure the command line parser. Here we will use the argparse module (or “on the fingers” is well explained here and here ), since it is included in the basic Python package.
Run the script with the following parameters:
And display the contents of params on the screen:
Now the values ​​of all parameters are in the params dictionary and can be easily obtained by the key of the same name.
Add a check for the presence of the main parameters:
Everything is good, but you can simplify your life a little, without compromising flexibility. We know that in 99% of cases, the validation script is one and it is called, for example, 'pep8.py', and the power unit script is the same for us every time (and often it will also be one). Similarly, with the display of the log - we will always use the same program (let it be a “Notepad”). Let's make changes to the configuration of the parser:
And add the installation of default values:
After making changes, the parser configuration code should look like this:
Now the script startup line is shorter:
Parameters won, go further.
Configure the log object. The 'pre-commit.log' log file will be created in the root of the current project. For Git, the working directory is the project root, so the file path is not specified. Also, we specify the mode for creating a new file for each operation (we do not need to store the previous logs) and set the log format - only the message:
The last line of code will simplify your life a little more - we create an alias, which we will use further by code instead of logging.info .
We will need to repeatedly run child processes and read their output to the console. To fulfill this need, we write the shell_command function. Her responsibilities will include:
The function will take arguments:
The current commit's file list is easily obtained using the Git diff command . In our case, we will need modified or new files:
As a result, targets will contain something like this:
The most painful stage is completed - it will be easier further.
Everything is simple here - let's go through all the scripts specified in --check and run each with a list of targets :
An example of the contents of the log on the code that did not pass the validation check:
We do the same with unit tests, but without targets :
Depending on the global result code and the --openlog and --forcelog parameters , we decide to display the log or not:
Note. Works in versions of Python 2.6 (and above) and 3.x. On versions lower than 2.6 - no tests were performed.
And at the end of the script, don't forget to return the result code to the Git shell:
Everything. The script is ready to use.
A hook is a file called “pre-commit” (without an extension) that you need to create in the directory: <project_dir> /. Git / hooks /
To run correctly on Windows, there are a couple of important points:
1. The first line of the file should be: #! / Bin / sh
Otherwise, we will see such an error:
2. Using the standard delimiter when specifying the path leads to a similar error:
It is treated in three ways: use double backslash, or take all the way in double quotes, or use "/". For example, Windows eats it and does not choke:
Of course, it is not recommended to do this :) Use any method that you like, but one .
We will train "on cats":
Test commit has new, renamed \ modified and deleted files. Also included are files with no code; The code itself contains design errors and does not pass one of the unit tests. Create a hook with validation, tests and the opening of a detailed log:
And try to commit. After thinking for a couple of seconds, the Git desktop will signal an error:
And in the next window a notebook will display the following:
Repeat the same commit, only without a detailed log:
Result:
Correct the errors, repeat commit, and here it is, the long-awaited result: Git desktop does not swear, and the notebook shows an empty pre-commit.log . PROFIT.
You can see the finished example here .
Of course, this script is not a panacea. It is useful when all necessary checks are limited to running test scripts locally. In complex projects, the concept of Continuous Integration (or CI) is usually used, and here come to the aid Travis (for Linux and OS X) and its counterpart AppVeyor (for Windows).
[UPD] Another alternative is overcommit . A pretty functional tool for managing Git hooks. But there are nuances - to work overcommit, you need to locally deploy the Ruby interpreter.
All nice coding and correct commits.
There are many examples of hooks in the network, most of them are on shells, but no author has paid attention to one important point - you have to drag a hook from project to project. At first glance - do not worry. But suddenly there is a need to make changes to the hook that already lives in 20 projects ... Or suddenly you need to transfer the development from Windows to Linux, and the hook to PowerShell ... What to do?
“Better like this: 8 pies and one candle!”
Examples, of course, are greatly exaggerated, but with their help, inconveniences have been revealed that I would like to avoid. It would be desirable, that the hook was not required to be dragged on all projects, it was not necessary to “finish” often, but so that he could:
- check the code sent to the repository for validity (for example: compliance with the PEP8 requirements, availability of documentation, etc.);
- perform a comprehensive check of the project (unit tests, etc.);
- interrupt the commit operation in case of errors and display a detailed log for debriefing.
And it looked like this:
python pre-commit.py --check pep8.py --test tests.py It is clear that the hook itself is just a starter, and the script that it launches performs all the
pre-commit.py
But before
')
These parameters will set the basic behavior of the script:
- -c or --check [script1 ... scriptN] - launching validation scripts. The script should be located in the same directory as pre-commit.py . Otherwise, you need to specify the full path. Each script will be “fed” files from the current commit.
- -t or --test [test1 ... testN] - run unit tests and other scripts that do not require the current commit file. The test should be located in the current project directory. Otherwise, you need to specify the full path.
Both parameters will be optional (to be able to leave only one type of check), but if you do not specify any of them, pre-commit.py will exit with the code “1” (error).
And add auxiliary parameters (all optional):
- -e or --exec_interpreter_path - the full path (with the file name) to the interpreter that will execute the scripts from --check and --test . If the parameter is not specified, the interpreter used by pre-commit.py will be used.
- -v or --verbose - enables verbose logging. If not specified, the console output of those scripts that have completed with an error code is written to the log.
- -o or --openlog path_to_viewer - the full path (with the file name) to the program with which we will view the log.
- -f or --forcelog - forced opening of the log. If not specified, the log is opened only if errors are detected. The parameter is applicable if --openlog is specified .
The logic is clear, now you can start writing the script itself.
Command line parameters
To begin, configure the command line parser. Here we will use the argparse module (or “on the fingers” is well explained here and here ), since it is included in the basic Python package.
# -*- coding: utf-8 -*- import sys import argparse # parser = argparse.ArgumentParser() # . , # : 1-N parser.add_argument('-c', '--check', nargs='+') # --check parser.add_argument('-t', '--test', nargs='+') # -. , # True. - False parser.add_argument('-v', '--verbose', action='store_true') # . # - =default parser.add_argument('-e', '--exec', default=sys.executable) # . # - =None parser.add_argument('-o', '--openlog') # --verbose parser.add_argument('-f', '--forcelog', action='store_true') # 1- ( ), # dict params = vars(parser.parse_args(sys.argv[1:])) Run the script with the following parameters:
c:\python34\python c:\dev\projects\pre-commit-tool\pre-commit.py --check c:\dev\projects\pre-commit-tool\pep8.py --test tests.py And display the contents of params on the screen:
{'exec': 'c:\\python34\\python.exe', 'forcelog': False, 'test': ['tests.py'], 'check': ['c:\\dev\\projects\\pre-commit-tool\\pep8.py'], 'openlog': None, 'verbose': False} Now the values ​​of all parameters are in the params dictionary and can be easily obtained by the key of the same name.
Add a check for the presence of the main parameters:
# if params.get('check') is None and params.get('test') is None: print(' ') exit(1) Everything is good, but you can simplify your life a little, without compromising flexibility. We know that in 99% of cases, the validation script is one and it is called, for example, 'pep8.py', and the power unit script is the same for us every time (and often it will also be one). Similarly, with the display of the log - we will always use the same program (let it be a “Notepad”). Let's make changes to the configuration of the parser:
# 0-N parser.add_argument('-c', '--check', nargs='*') parser.add_argument('-t', '--test', nargs='*') # , const parser.add_argument('-o', '--openlog', nargs='?', const='notepad') And add the installation of default values:
if params.get('check') is not None and len(params.get('check')) == 0: # , pre-commit.py params['check'] = [join(dirname(abspath(__file__)), 'pep8.py')] if params.get('test') is not None and len(params.get('test')) == 0: params['test'] = ['tests.py'] After making changes, the parser configuration code should look like this:
# -*- coding: utf-8 -*- import sys import argparse from os.path import abspath, dirname, join parser = argparse.ArgumentParser() parser.add_argument('-c', '--check', nargs='*') parser.add_argument('-t', '--test', nargs='*') parser.add_argument('-v', '--verbose', action='store_true') parser.add_argument('-e', '--exec', default=sys.executable) parser.add_argument('-o', '--openlog', nargs='?', const='notepad') parser.add_argument('-f', '--forcelog', action='store_true') params = vars(parser.parse_args(sys.argv[1:])) if params.get('check') is None and params.get('test') is None: print(' ') exit(1) if params.get('check') is not None and len(params.get('check')) == 0: params['check'] = [join(dirname(abspath(__file__)), 'pep8.py')] if params.get('test') is not None and len(params.get('test')) == 0: params['test'] = ['tests.py'] Now the script startup line is shorter:
c:\python34\python c:\dev\projects\pre-commit-tool\pre-commit.py --check --test --openlog params content: {'check': ['c:\\dev\\projects\\pre-commit-tool\\pep8.py'], 'openlog': 'notepad', 'test': ['tests.py'], 'verbose': False, 'exec': 'c:\\python34\\python.exe', 'forcelog': False} Parameters won, go further.
Log
Configure the log object. The 'pre-commit.log' log file will be created in the root of the current project. For Git, the working directory is the project root, so the file path is not specified. Also, we specify the mode for creating a new file for each operation (we do not need to store the previous logs) and set the log format - only the message:
import logging log_filename = 'pre-commit.log' logging.basicConfig( filename=log_filename, filemode='w', format='%(message)s', level=logging.INFO) to_log = logging.info The last line of code will simplify your life a little more - we create an alias, which we will use further by code instead of logging.info .
Shell
We will need to repeatedly run child processes and read their output to the console. To fulfill this need, we write the shell_command function. Her responsibilities will include:
- running a subprocess (using Popen );
- read data from the subprocess console and convert them;
- write the read data to the log if the subprocess is terminated with an error code.
The function will take arguments:
- command is an argument for Popen . Actually what will run in Shell. But instead of a solid line (“python main.py”), it is recommended to specify with a list (['python', 'main.py']);
- force_report - control output to the log. Possible values: True - forced output to the log, False - output if an error code was received, None - disable output to the log.
from subprocess import Popen, PIPE def shell_command(command, force_report=None): # proc = Popen(command, stdout=PIPE, stderr=PIPE) # proc.wait() # # ( , "\r\n") transform = lambda x: ' '.join(x.decode('utf-8').split()) # ( ) stdout report = [transform(x) for x in proc.stdout] # stderr report.extend([transform(x) for x in proc.stderr]) # force_report if force_report is True or (force_report is not None and proc.returncode > 0): to_log('[ SHELL ] %s (code: %d):\n%s\n' % (' '.join(command), proc.returncode, '\n'.join(report))) # return proc.returncode, report Head revision
The current commit's file list is easily obtained using the Git diff command . In our case, we will need modified or new files:
from os.path import basename # result_code = 0 # commit' code, report = shell_command( ['git', 'diff', '--cached', '--name-only', '--diff-filter=ACM'], params.get('verbose')) if code != 0: result_code = code # "py" targets = filter(lambda x: x.split('.')[-1] == "py", report) # ( ) targets = [join(dirname(abspath(x)), basename(x)) for x in targets] As a result, targets will contain something like this:
['C:\\dev\\projects\\example\\demo\\daemon_example.py', 'C:\\dev\\projects\\example\\main.py', 'C:\\dev\\projects\\example\\test.py', 'C:\\dev\\projects\\example\\test2.py'] The most painful stage is completed - it will be easier further.
Validation check
Everything is simple here - let's go through all the scripts specified in --check and run each with a list of targets :
if params.get('check') is not None: for script in params.get('check'): code, report = shell_command( [params.get('exec'), script] + targets, params.get('verbose')) if code != 0: result_code = code An example of the contents of the log on the code that did not pass the validation check:
[ SHELL ] C:\python34\python.exe c:\dev\projects\pre-commit-tool\pep8.py C:\dev\projects\example\demo\daemon_example.py (code: 1):
C:\dev\projects\example\demo\daemon_example.py:8:80: E501 line too long (80 > 79 characters)
Running tests
We do the same with unit tests, but without targets :
if params.get('test') is not None: for script in params.get('test'): code, report = shell_command( [params.get('exec'), script], params.get('verbose')) if code != 0: result_code = code [UPD] Display the log
Depending on the global result code and the --openlog and --forcelog parameters , we decide to display the log or not:
if params.get('openlog') and (result_code > 0 or params.get('forcelog')): # Popen([params.get('openlog'), log_filename], close_fds=True) Note. Works in versions of Python 2.6 (and above) and 3.x. On versions lower than 2.6 - no tests were performed.
And at the end of the script, don't forget to return the result code to the Git shell:
exit(result_code) Everything. The script is ready to use.
Root of evil
A hook is a file called “pre-commit” (without an extension) that you need to create in the directory: <project_dir> /. Git / hooks /
To run correctly on Windows, there are a couple of important points:
1. The first line of the file should be: #! / Bin / sh
Otherwise, we will see such an error:
GitHub.IO.ProcessException: error: cannot spawn .git/hooks/pre-commit: No such file or directory
2. Using the standard delimiter when specifying the path leads to a similar error:
GitHub.IO.ProcessException: C:\python34\python.exe: can't open file 'c:devprojectspre-commit-toolpre-commit.py': [Errno 2] No such file or directory
It is treated in three ways: use double backslash, or take all the way in double quotes, or use "/". For example, Windows eats it and does not choke:
#!/bin/sh c:/python34/python "c:\dev\projects\pre-commit-tool\pre-commit.py" -c -tc:\\dev\\projects\\example\\test.py Of course, it is not recommended to do this :) Use any method that you like, but one .
Acceptance Testing
We will train "on cats":
Test commit has new, renamed \ modified and deleted files. Also included are files with no code; The code itself contains design errors and does not pass one of the unit tests. Create a hook with validation, tests and the opening of a detailed log:
c:/python34/python c:/dev/projects/pre-commit-tool/pre-commit.py -c -t test.py test2.py -vfo And try to commit. After thinking for a couple of seconds, the Git desktop will signal an error:
And in the next window a notebook will display the following:
[ SHELL ] git diff --cached --name-only --diff-filter=ACM (code: 0):
.gitattributes1
demo/daemon_example.py
main.py
test.py
test2.py
[ SHELL ] C:\python34\python.exe c:\dev\projects\pre-commit-tool\pep8.py C:\dev\projects\example\demo\daemon_example.py C:\dev\projects\example\main.py C:\dev\projects\example\test.py C:\dev\projects\example\test2.py (code: 1):
C:\dev\projects\example\demo\daemon_example.py:8:80: E501 line too long (80 > 79 characters)
C:\dev\projects\example\demo\daemon_example.py:16:5: E303 too many blank lines (2)
C:\dev\projects\example\demo\daemon_example.py:37:5: E303 too many blank lines (2)
C:\dev\projects\example\demo\daemon_example.py:47:5: E303 too many blank lines (2)
C:\dev\projects\example\main.py:46:80: E501 line too long (90 > 79 characters)
C:\dev\projects\example\main.py:59:80: E501 line too long (100 > 79 characters)
C:\dev\projects\example\main.py:63:80: E501 line too long (115 > 79 characters)
C:\dev\projects\example\main.py:69:80: E501 line too long (105 > 79 characters)
C:\dev\projects\example\main.py:98:80: E501 line too long (99 > 79 characters)
C:\dev\projects\example\main.py:115:80: E501 line too long (109 > 79 characters)
C:\dev\projects\example\main.py:120:80: E501 line too long (102 > 79 characters)
C:\dev\projects\example\main.py:123:80: E501 line too long (100 > 79 characters)
[ SHELL ] C:\python34\python.exe test.py (code: 1):
Test 1 - passed
Test 2 - passed
[!] Test 3 FAILED
[ SHELL ] C:\python34\python.exe test2.py (code: 0):
Test 1 - passed
Test 2 - passed
Repeat the same commit, only without a detailed log:
c:/python34/python c:/dev/projects/pre-commit-tool/pre-commit.py -c -t test.py test2.py -fo Result:
[ SHELL ] C:\python34\python.exe c:\dev\projects\pre-commit-tool\pep8.py C:\dev\projects\example\demo\daemon_example.py C:\dev\projects\example\main.py C:\dev\projects\example\test.py C:\dev\projects\example\test2.py (code: 1):
C:\dev\projects\example\demo\daemon_example.py:8:80: E501 line too long (80 > 79 characters)
C:\dev\projects\example\demo\daemon_example.py:16:5: E303 too many blank lines (2)
C:\dev\projects\example\demo\daemon_example.py:37:5: E303 too many blank lines (2)
C:\dev\projects\example\demo\daemon_example.py:47:5: E303 too many blank lines (2)
C:\dev\projects\example\main.py:46:80: E501 line too long (90 > 79 characters)
C:\dev\projects\example\main.py:59:80: E501 line too long (100 > 79 characters)
C:\dev\projects\example\main.py:63:80: E501 line too long (115 > 79 characters)
C:\dev\projects\example\main.py:69:80: E501 line too long (105 > 79 characters)
C:\dev\projects\example\main.py:98:80: E501 line too long (99 > 79 characters)
C:\dev\projects\example\main.py:115:80: E501 line too long (109 > 79 characters)
C:\dev\projects\example\main.py:120:80: E501 line too long (102 > 79 characters)
C:\dev\projects\example\main.py:123:80: E501 line too long (100 > 79 characters)
[ SHELL ] C:\python34\python.exe test.py (code: 1):
Test 1 - passed
Test 2 - passed
[!] Test 3 FAILED
Correct the errors, repeat commit, and here it is, the long-awaited result: Git desktop does not swear, and the notebook shows an empty pre-commit.log . PROFIT.
You can see the finished example here .
[UPD] Instead of conclusion
Of course, this script is not a panacea. It is useful when all necessary checks are limited to running test scripts locally. In complex projects, the concept of Continuous Integration (or CI) is usually used, and here come to the aid Travis (for Linux and OS X) and its counterpart AppVeyor (for Windows).
[UPD] Another alternative is overcommit . A pretty functional tool for managing Git hooks. But there are nuances - to work overcommit, you need to locally deploy the Ruby interpreter.
All nice coding and correct commits.
Source: https://habr.com/ru/post/302932/
All Articles