HowTo: continuous integration of the project on Django using TeamCity

Introduction

In the development process, creating a new functionality, more and more often, the old code began to touch the old code with wide strokes than to destroy the logic of its work. This made all the same to write a unit and integration tests for the old code and automate their launch, because chasing all the tests with my hands is somehow sad. I just remembered the recent CI Django guide in Jenkins and quite old on the Webtest in Django . As a result, an attempt was made to raise Jenkins, but he somehow did not take off on my ubunt and I remembered the sinful thing about TeamCity. “Since I’m writing to PyCharm and found an approach to it, I’m probably going to master TeamCity , because there’s only one desk!” (and megapodrobny), unlike my previous, manual :)

Total: who needs guidance on raising the TeamCity integration server, and testing Django projects in it with nose and webtest tests in a virtual python environment with its automatic (environment) update - welcome under cat.

Caution! TeamCity requires (according to the documentation) a sun / oracle version of the JVM ...

Theory

Virtual python environment using virtualenv

The system on my PC for development meets all the requirements of my project, but this is far from being on a server for testing (even python is 2.6 there, and I eventually needed 2.7), so I had to take care of virtualization of the environment in order to calmly put what I needed, in addition, it eliminates the need to have the power of root or sudo er.
')
To create a virtual environment, you need to install virtualenv or download and extract virtualenv.py from the archive. We will use the second option, because this completely discourages us from having to use the power of the superuser. Yes, the system still needs to have python installed.

 python virtualenv.py -p python2.7 .env 

Here, -p python2.7 sets the required version of python (the subtlety is that the necessary versions of python must be installed on the system itself, but not being the default python, the script will not be able to pull them out and install them) as well .env directory in which the environment will be placed. Further, to use python and modules from the environment, you need to call them through .env/bin/python but if you write this is laziness, you can do

 source .env/bin/activate 

which replaces the environment variable PATH .

Fabric automation

In addition to the python itself installed on the target system, we also need the fabric module. That is, on the testing server, you will have to execute the sudo pip install fabric or sudo easy_install fabric , during installation, pycrypto will pull to build python-dev python samples, so you need to install it first. Fabric allows you to execute scripts in python that perform actions on a local or remote (via SSH) system. After installation, the fab application will appear in the system, which searches for the file fabfile.py in the call folder and executes the specified script.

For example:

 fab test 

will execute the test function from the fabfile.py file.

Nose and webtest in django

Now a little about the tests themselves. Django has its own unit tests and functional tests (with the help of a test client), but there are flaws in them about which it was written more than once. In general, I began to use nose for unit testing and webtest for functional. In the introduction, I gave a link to an article about webtest, it shows a good idea how it works. We will need the following modules for testing (then we will add this list to the file with the list of modules for automatic installation via pip -r ):

 coverage>=3.0 nose webtest django-nose django-webtest 

In addition, the django_nose should be added to the INSTALLED_APPS file of the settings.py your django project, or in the same file specify TEST_RUNNER = 'django_nose.NoseTestSuiteRunner' .

Attention! If south used, then django_nose should be added after it.

Actually, if there are unit tests already written in the Djang test system - you can not touch them, everything will work and so. How to write functional tests through webtest can be read in the above article .

Testing starts as usual:

 python manage.py test 

This command will perform both nose tests and webtest tests, if the names of files, classes and functions with tests have the word "test" in the name - this is a bonus from nose . In addition, nose can use coverage to calculate code coverage by tests (it does not understand functional webtest tests, because tests work with the code not directly, but through the generated pages). Django-nose allows us to use the nose test launch options . To do this, they can be written in the project settings.py file. For example:

 NOSE_ARGS = [ '--nocapture', '--with-coverage', '--cover-html', '--cover-html-dir=%s' % os.path.join(PROJECT_PATH, 'cover', 'unit'), '--cover-package=django_dir' ] 

This spell will allow you to generate a test report in HTML format and place it in the cover/unit subdirectory of your project.

Practice

Project structure

My project has the following folder structure:

 . .. project_dir project_dir/django_dir project_dir/build project_dir/build/pipreq.txt project_dir/fabfile.py project_dir/virtualenv.py 

Where project_dir contains all sorts of useful things such as a directory with a virtual environment, a factory file and other useful things, and already in the django_dir subdirectory django_dir located the actual project on django . The version control system (in my case, this is GIT ) gets everything from project_dir with a few exceptions, like a directory with a virtual environment.

Test execution

Actually, our goal is to teach TeamCity after detecting changes in the repository (which he knows himself) to test our updated django project (which he doesn’t know) and inform us about the results (here he also needs some help).

The fact is that for the purity of the experiment, it is desirable that, in addition to the tests themselves, the whole environment be checked / updated, for this we will write a small factory which, after receiving the changes, will launch TeamCity.

project_dir/fabfile.py :

 # -*- coding: utf-8 -*- from fabric.api import local def test(): """  . """ local('python virtualenv.py -p python2.7 .env') #    local('./.env/bin/pip install -q -E .env -r build/pipreq.txt') #    #   build/pipreq.txt local('./.env/bin/python django_dir/manage.py test --noinput') #   

Please note that virtualenv.py lies at the root of the project so that it is not required to be installed on the system.

Required project_dir/build/pipreq.txt :

 django coverage>=3.0 nose teamcity-nose webtest django-nose django-webtest ...      /... 

Notice that the previously listed teamcity-nose is not listed in the file, this module also helps to pass the results of our testing to TeamCity. It does not require any configuration and connection - it starts automatically when it finds the environment variable TEAMCITY_PROJECT_NAME that the TeamCity creates during operation.

Actually everything is ready for automatic testing. To start it, just run the command:

 fab test 

in the general project root.

Install and configure TeamCity

I did not bother much, which I recommend to you - just downloaded the tar.gz option from the download page . All this happiness weighs indecent 333MB. But the bundle carries Tomcat itself apart from the integration server itself, which will save us from hemorrhoids with server settings and other crap.

TeamCity has a property (mentioned at the beginning) that someone may not like - it requires a Sanov / Oracle version of the JVM to work.

How to get it on Ubuntu can be read for example here .

Actually, after downloading, we unpack this good in the place where it will work (the data itself will by default be stored in the .BuildServer subdirectory of the .BuildServer home directory under which the server is running). The server is bin/runAll.dh start command bin/runAll.dh start , and bin/runAll.dh stop is killed (if we are in the folder with the unpacked archive). The default server is running on port 8111 , but there is another build agent that starts on port 9090 . The first can be changed in the conf/server.xml file, the second in the buildAgent/conf/buildAgent.dist.properties (sample) and buildAgent/conf/buildAgent.properties (specific agent).

All other settings (well, about which I will tell) will be done through the web interface. In general, we start teamcity and go to the browser (http: // localhost: 8111 if it was launched locally). Depending on the capacity of the system, you will either see the license agreement or a message:



and you will need to wait until it is replaced by a license agreement.

Further it will be offered to create an administrator account. After that we will see the TeamCity system itself.

Here we are interested in creating a project. Everything is simple - the name and description. After we get to the project settings page.



On the VCS Roots tab (VCS - version control system) you need to set a new repository. More precisely indicate where the system will take the code for testing. There is a long footcloth that needs to be filled with your data. All fields are signed, there are links to help pages, so there is nothing superhuman there. Here you can check the performance of the connection.

Next, on the General tab, you need to create a new Buid Configuration - the cornerstone of our testing.



Build counter - the number with which the numbering of the builds will begin, or rather the build number that will be assigned to the next build. You can safely put 1 , then just a screenshot is not fresh :)

Of the important fields on the first page here is the Artifact paths which tells the system how to find the files resulting from the build / test. In our case, these are test coverage reports. In my case, here it is enough to enter django_dir/cover/**/* . On the second page, select our previously defined repository.

Clicking on the Add Build Step will need to create the first (and, in our case, the only) build step (in our case, testing).



Actually the required set of settings you see on the screen. We save.

Next, on the right in the vertical menu, select Build Triggering and add the launch of our build when changing the repository:



And finally, we return to the settings of the project itself, go to the Report Tabs tab and create a new tab:



This will allow TemCity to display our reports in its interface.



The tab will appear when the specified file is in the specified location, so if it is not there, check the settings and see if the reports really lie where they should be.

That's all, it remains in the top menu in Projects to go to our project and click Run at the build for the first run.

Beautiful

All managers love graphics, so that it was necessary to show a few pictures. Alas, these pictures are added by the true programmer-admin method - making changes to the XML config. The manual is available here . One feels that in this place the developers have fairly rested (they could have made the forms) . Because There is no obvious way to find out one of the most important parameters for buildTypeId . The simplest thing is to look in the address bar of the browser: localhost:8111/viewType.html?tab=buildTypeStatusDiv& buildTypeId=bt2 localhost:8111/viewType.html?tab=buildTypeStatusDiv& buildTypeId=bt2 .

I will cite the config of my two graphs:

 <?xml version="1.0" encoding="UTF-8"?> <settings> <report-tabs> <report-tab title="Coverage" basePath="/unit/" buildTypeId="bt2"> <revisionRule name="lastFinished" revision="latest.lastFinished" /> </report-tab> </report-tabs> <custom-graphs> <graph title="Build Duration" defaultFilters="showFailed" seriesTitle="key"> <!--     --> <valueType key="BuildDuration" title="main test" buildTypeId="bt2"/> </graph> <graph title="Test Passes" defaultFilters="showFailed" seriesTitle="key"> <!--   --> <valueType key="FailedTestCount" title="fail" buildTypeId="bt2"/> <valueType key="PassedTestCount" title="pass" buildTypeId="bt2"/> </graph> </custom-graphs> </settings> 

This is the file ~/.BuildServer/config/django_dir/plugin-settings.xml . Changes to the file are picked up on the fly.

It looks something like this:



This is how the coverage report looks like:



List of assemblies:



Details on the assembly:

Conclusion

I apologize for the multi-bookmark, but it just so happened. I wanted to describe available. As a result, we have automated testing (modular and integration), in addition, we also check the availability of the necessary modules and the project's “liftability”. Showdowns with TemaCity's response to various build results I leave on your conscience. In this part, there is something to be happy about - standard soap notification, notification via jabber , and creation of release push.

Successful development!