Accelerate Testing Django-projects
The question of testing Django applications has received much attention in various articles, including on Habré. Almost in each of them, at least a couple of sentences are devoted to ways and hacks to speed up the passing of tests, and therefore it is not easy to say something fundamentally new here.
In the project of the hosting control panel, the development of which I do a significant part of my work with NetAngels , there are 120 tables and about 500 objects from fixtures are loaded during testing. It’s not that scary, but creating all the tables, adding indexes and loading objects each time you run a test is pretty annoying, especially if you run just one or a couple of tests.
Under the cat quite briefly listed several ways to accelerate testing, proposed earlier, and at the end provides a detailed description of another useful recipe, which for me now, I hope, has finally solved the problem of the speed of test execution.
In average applications, as a rule, most of the brakes will be associated with working with a database. It is not surprising that almost all proposals are aimed at optimizing this particular component.
')
In turn, work with the database is mainly hampered by the disk subsystem, and it is logical to assume that if we somehow reduce this load, we will get a noticeable performance gain.
Nobody cares if during the testing process the electricity suddenly disappears, and the data will not be completely written to disk. Usually, the database servers are set up in such a way as to minimize the sad consequences of such events. If you use a separate MySQL or PostgreSQL server for tests, you can safely change the settings to unsafe.
For MySQL, in particular, offer the following:
In PostgreSQL, for the same purpose, it is recommended to add the
A more radical approach is not to use a disk at all, but instead to work with data in memory. The general principle is that a new partition is created with the tmpfs file system, mounted in a separate directory, and then all database files are created in that directory. Extra bonus - simply unmounting a partition deletes all data.
In fact, the Django developers have already done a lot to ensure that tests run as quickly as possible. In particular, if you use SQLite as the database engine, then during testing, the database is created in memory (the string ": memory:" is passed to the driver as the file name), and this method alone is enough to most likely solve most problems with speed.
Sometimes they complain that ORM Django does not thoroughly hide the details of the database, and therefore in some cases it may turn out that the code that worked in SQLite (i.e., the tests passed) suddenly breaks when you roll out to the system where it lives and running mysql. Indeed, this sometimes happens, but as a rule, it is due to the fact that you did something “unusual”, for example, you began to manually create queries using the QuerySet.extra method. However, most likely, if you do such things, then you know what it may threaten.
As you know, when running tests, Django performs the following sequence of actions:
1. clears the test database with the user's permission if there is something in it
2. creates all the tables and indexes in the test database
3. download the fixtures set with the name initial_data
4. performs tests, one by one
5. deletes everything that was created during the execution of tests
The first and last stage may not be performed if the data lives in memory, but steps 2 and 3 take a significant amount of time and bring down the high rate of iterations “corrected code - launched the test”. Obviously, the database schema and set of fixtures change much less frequently than the code and tests, so it makes sense to somehow save on the constant creation of the database. By the way, steps 2 and 3 are the usual execution of the syncdb management command.
The general approach to speeding up the launch of tests is to run syncdb manually, and only when it is really necessary, and when running the tests, simply copy the previously prepared database. Using SQLite, you could get by copying files, but you didn’t want to lose the benefits of working with tests in ": memory:".
A short search showed that a solution to this case exists. It turns out that SQLite has an interface for “hot” backup (just like adults), and if we run such a copy from a previously prepared database into the database named: memory: before running the tests, we will get exactly what we need: an initialized database data in memory.
The first difficulty in implementation is that the standard Python module sqlite3 does not support, and may never support this API, therefore, to perform such copying with Python tools they suggest using a third-party module named APSW (Another Python SQLite Wrapper).
The second difficulty is that each new connection to the database in: memory: creates its own copy of the database (obviously empty), and therefore you must somehow teach the cursor that the ORM uses to use the connection initialized by APSW. Fortunately, a hack is provided for this case: instead of the line with the file name, when creating a connection using sqlite3, you can send an apsw.Connection object that will proxy all requests on its own.
Thus, the solution looks very simple:
1. Create two ASPW Connection objects, one of which refers to a previously prepared database, and the second to a base in memory.
2. Copy data from file to memory.
3. As the NAME parameter for an alias with the name “default”, we pass ASPW Connection, which refers to the memory.
4. Initialize the cursor, and run the tests.
The database is very simple to prepare: just add another alias with the name “quickstart” to the DATABASES settings.py variable, and then execute
All code that can perform these actions takes a little more than 20 lines and is shown below. It’s enough to make it work
1. Install APSW
2. Copy the code into a separate file and place it in the project
3. Add a database alias to settings.DABABASES named quickstart
4. Create a database by executing
5. Set the TEST_RUNNER variable so that it refers to the class of the newly saved object
6. Try to run some simple test.
As a result, the execution time of one single test was reduced from 18 to 2 seconds; it is quite comfortable to run the test as often as I would like.
This same code, but with comments and a fat warning “your tests can eat your data!” (Use only in the test environment) is posted on gist.github.com/1044215 .
I hope these simple recommendations will allow you to write code faster, more efficiently and more reliably.
For details and other useful information I recommend to refer to the documentation for your database server, as well as to the following sources:
- Speeding up Django unit test runs with MySQL
- Innodb Performance Optimization Basics
- Using the SQLite Online Backup API
- How to use SQLite's backup in Python
In the project of the hosting control panel, the development of which I do a significant part of my work with NetAngels , there are 120 tables and about 500 objects from fixtures are loaded during testing. It’s not that scary, but creating all the tables, adding indexes and loading objects each time you run a test is pretty annoying, especially if you run just one or a couple of tests.
Under the cat quite briefly listed several ways to accelerate testing, proposed earlier, and at the end provides a detailed description of another useful recipe, which for me now, I hope, has finally solved the problem of the speed of test execution.
In average applications, as a rule, most of the brakes will be associated with working with a database. It is not surprising that almost all proposals are aimed at optimizing this particular component.
')
In turn, work with the database is mainly hampered by the disk subsystem, and it is logical to assume that if we somehow reduce this load, we will get a noticeable performance gain.
Insecure transaction settings
Nobody cares if during the testing process the electricity suddenly disappears, and the data will not be completely written to disk. Usually, the database servers are set up in such a way as to minimize the sad consequences of such events. If you use a separate MySQL or PostgreSQL server for tests, you can safely change the settings to unsafe.
For MySQL, in particular, offer the following:
[mysqld]
default-table-type=innodb
transaction-isolation=READ-COMMITTED
innodb_flush_log_at_trx_commit = 0
skip-sync-frm=OFF
In PostgreSQL, for the same purpose, it is recommended to add the
fsync = off option to postgresql.conf.Using ramdisk
A more radical approach is not to use a disk at all, but instead to work with data in memory. The general principle is that a new partition is created with the tmpfs file system, mounted in a separate directory, and then all database files are created in that directory. Extra bonus - simply unmounting a partition deletes all data.
SQLite for tests
In fact, the Django developers have already done a lot to ensure that tests run as quickly as possible. In particular, if you use SQLite as the database engine, then during testing, the database is created in memory (the string ": memory:" is passed to the driver as the file name), and this method alone is enough to most likely solve most problems with speed.
Sometimes they complain that ORM Django does not thoroughly hide the details of the database, and therefore in some cases it may turn out that the code that worked in SQLite (i.e., the tests passed) suddenly breaks when you roll out to the system where it lives and running mysql. Indeed, this sometimes happens, but as a rule, it is due to the fact that you did something “unusual”, for example, you began to manually create queries using the QuerySet.extra method. However, most likely, if you do such things, then you know what it may threaten.
Preliminary creation of SQLite test database
As you know, when running tests, Django performs the following sequence of actions:
1. clears the test database with the user's permission if there is something in it
2. creates all the tables and indexes in the test database
3. download the fixtures set with the name initial_data
4. performs tests, one by one
5. deletes everything that was created during the execution of tests
The first and last stage may not be performed if the data lives in memory, but steps 2 and 3 take a significant amount of time and bring down the high rate of iterations “corrected code - launched the test”. Obviously, the database schema and set of fixtures change much less frequently than the code and tests, so it makes sense to somehow save on the constant creation of the database. By the way, steps 2 and 3 are the usual execution of the syncdb management command.
The general approach to speeding up the launch of tests is to run syncdb manually, and only when it is really necessary, and when running the tests, simply copy the previously prepared database. Using SQLite, you could get by copying files, but you didn’t want to lose the benefits of working with tests in ": memory:".
A short search showed that a solution to this case exists. It turns out that SQLite has an interface for “hot” backup (just like adults), and if we run such a copy from a previously prepared database into the database named: memory: before running the tests, we will get exactly what we need: an initialized database data in memory.
The first difficulty in implementation is that the standard Python module sqlite3 does not support, and may never support this API, therefore, to perform such copying with Python tools they suggest using a third-party module named APSW (Another Python SQLite Wrapper).
The second difficulty is that each new connection to the database in: memory: creates its own copy of the database (obviously empty), and therefore you must somehow teach the cursor that the ORM uses to use the connection initialized by APSW. Fortunately, a hack is provided for this case: instead of the line with the file name, when creating a connection using sqlite3, you can send an apsw.Connection object that will proxy all requests on its own.
Thus, the solution looks very simple:
1. Create two ASPW Connection objects, one of which refers to a previously prepared database, and the second to a base in memory.
2. Copy data from file to memory.
3. As the NAME parameter for an alias with the name “default”, we pass ASPW Connection, which refers to the memory.
4. Initialize the cursor, and run the tests.
The database is very simple to prepare: just add another alias with the name “quickstart” to the DATABASES settings.py variable, and then execute
./manage.py syncdb --database quickstart .All code that can perform these actions takes a little more than 20 lines and is shown below. It’s enough to make it work
1. Install APSW
2. Copy the code into a separate file and place it in the project
3. Add a database alias to settings.DABABASES named quickstart
4. Create a database by executing
./manage.py syncdb --database quickstart5. Set the TEST_RUNNER variable so that it refers to the class of the newly saved object
6. Try to run some simple test.
Copy Source | Copy HTML
- import apsw
- from django. test .simple import DjangoTestSuiteRunner
- from django.db import connections
- class TestSuiteRunner (DjangoTestSuiteRunner):
- def setup_databases (self, ** kwargs):
- quickstart_connection = connections [ 'quickstart' ]
- quickstart_dbname = quickstart_connection.settings_dict [ 'NAME' ]
- memory_connection = apsw.Connection ( ': memory:' )
- quickstart_connection = apsw.Connection (quickstart_dbname)
- with memory_connection.backup ( 'main' , quickstart_connection, 'main' ) as backup:
- while not backup.done:
- backup.step ( 100 )
- connection = connections [ 'default' ]
- connection.settings_dict [ 'NAME' ] = memory_connection
- cursor = connection.cursor ()
- def teardown_databases (self, old_config, ** kwargs):
- pass
As a result, the execution time of one single test was reduced from 18 to 2 seconds; it is quite comfortable to run the test as often as I would like.
This same code, but with comments and a fat warning “your tests can eat your data!” (Use only in the test environment) is posted on gist.github.com/1044215 .
I hope these simple recommendations will allow you to write code faster, more efficiently and more reliably.
Used sources
For details and other useful information I recommend to refer to the documentation for your database server, as well as to the following sources:
- Speeding up Django unit test runs with MySQL
- Innodb Performance Optimization Basics
- Using the SQLite Online Backup API
- How to use SQLite's backup in Python
Source: https://habr.com/ru/post/122574/
All Articles