Fighting Tastypie API performance

As I dragged the Tastypie package onto the Pony ORM and what came of it.

This article continues the story of my struggle for application performance in python and django.


')
Summary of the previous series:
- ORMs slow down , and very much - 3-5 times or more
- But the slapstick for Django ORM
- Wow, this ORM almost does not slow down and caches all that is possible.
- This is how you can fasten a high-performance Pony ORM to the Django project and use it in key places

Introduction

Having obtained quite satisfactory (compared to Django ORM) results on the speed of accessing the DBMS using the Pony ORM, I began to think about how to use them in practice. To begin with, I did a mapping of the Django data model to the Pony data model . Now, the finished data model can be used in the project.

Where code is written from scratch, using Pony instead of Django was not easy, but very simple: the `p` attribute of all Django models now points to the Pony model. However - what to do with those ready-made packages that already use Django ORM?

Unfortunately, the answer is one: rewrite or append, since the syntax and semantics of accessing models and collections in Pony ORM is completely different from Django ORM - so much so that to mask the appeal to Pony ORM under appeal to Django ORM is not yet possible. In addition, I strongly suspect that such a disguise, when performed, will kill all the advantages of using the Pony ORM due to losses on the intermediate layer. Still, python is an interpreter, and for example, losses on a function call are large enough to be noticeable even against the background of accessing the DBMS. These losses, among others, are likely to kill the performance of Django ORM and SQLAlchemy.

We plan to transfer the entire interface (or most of it) of our application to the client side, using the server as a data provider via the HTTP API. Thus, the key element is the API that we implement using the Tastypie package , so that increasing its performance should directly affect the performance of the application interface and reduce the overall load on the site.

And I took up the Tastypie .

How did the Tastypie structure help me?

The Tastypie library, although it relies on Django, has a data source abstraction layer (Resource), independent of Django ORM. Only the next heir, ModelResource, uses Django ORM-specific data calls. Thus, as I assumed at the beginning, I only needed to inherit from the Resource and implement a functional similar to the ModelResource.

In practice, I also had to make a class parallel to ToManyField, because, to my disappointment, the latter turned out to be several lines oriented specifically to Django ORM. I called my class SetField, since it is the Set class that is used in Pony ORM to denote a one-to-many relationship.

What happened

Everything turned out just fine. As a sample, I used the data model from django.contrib.auth, filling it in approximately 1000 users. I assigned one of the users 159 rights to various types of data in the system (we have accumulated a lot of them - data types) and requested this user along with his rights through the API, using this call as a model.

Here's what the data source definition for the old API v2, which used Django ORM, looked like (note that authentication and authorization are excluded - for correct measurement of performance).

class PermissionResource(ModelResource): class Meta: queryset = auth_models.Permission.objects.all() object_model = queryset.model filtering = dict([(n,ALL_WITH_RELATIONS) for n in object_model._meta.get_all_field_names()]) resource_name = 'auth/permission' class UserResource(ModelResource): user_permissions = ToManyField(PermissionResource,'user_permissions',related_name='user_set',null=True) class Meta: queryset = auth_models.User.objects.all() object_model = queryset.model filtering = dict([(n,ALL_WITH_RELATIONS) for n in object_model._meta.get_all_field_names()]) resource_name = 'auth/user' 

Well, this is how the definition for the new v3 API looks, based on the use of the Pony ORM together with the Djony package:

 class PermissionResource(DjonyResource): class Meta: object_model = auth_models.Permission filtering = dict([(n,ALL_WITH_RELATIONS) for n in object_model._meta.get_all_field_names()]) resource_name = 'auth/permission' class UserResource(DjonyResource): user_permissions = SetField(PermissionResource,'user_permissions',related_name='user_set',null=True) class Meta: object_model = auth_models.User filtering = dict([(n,ALL_WITH_RELATIONS) for n in object_model._meta.get_all_field_names()]) resource_name = 'auth/user' 

Please note that the metadata definition uses the Django model and not the Pony. This is due to the fact that the Pony model does not contain many of the attributes specific to Django ORM, but not needed to define the actual data structure, such as help_text, for example. The question of the placement of such metadata is actually important, but its discussion is probably beyond the scope of this article.

The server was deployed on my modest work computer with 2 2.2 GHz cores and 3 GB of RAM. The test client started from the same computer. As the last, the ab (Apache Benchmark) package was launched:

 seva@SEVA (2):~/djony$ ab -n 100 -c 4 "http://localhost:8080/api/v2/auth/user/3/?format=json" This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient).....done Server Software: nginx/1.1.19 Server Hostname: localhost Server Port: 8080 Document Path: /api/v2/auth/user/3/?format=json Document Length: 5467 bytes Concurrency Level: 4 Time taken for tests: 17.331 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 582900 bytes HTML transferred: 546700 bytes Requests per second: 5.77 [#/sec] (mean) Time per request: 693.256 [ms] (mean) Time per request: 173.314 [ms] (mean, across all concurrent requests) Transfer rate: 32.84 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 1.5 0 9 Processing: 313 685 486.1 575 3357 Waiting: 312 684 485.8 574 3355 Total: 313 685 486.7 575 3357 Percentage of the requests served within a certain time (ms) 50% 575 66% 618 75% 647 80% 670 90% 819 95% 1320 98% 2797 99% 3357 100% 3357 (longest request) seva@SEVA (2):~/djony$ ab -n 100 -c 4 "http://localhost:8080/api/v3/auth/user/3/?format=json" This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient).....done Server Software: nginx/1.1.19 Server Hostname: localhost Server Port: 8080 Document Path: /api/v3/auth/user/3/?format=json Document Length: 5467 bytes Concurrency Level: 4 Time taken for tests: 8.339 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 582900 bytes HTML transferred: 546700 bytes Requests per second: 11.99 [#/sec] (mean) Time per request: 333.557 [ms] (mean) Time per request: 83.389 [ms] (mean, across all concurrent requests) Transfer rate: 68.26 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 137 317 375.9 243 2753 Waiting: 137 316 375.7 243 2751 Total: 137 317 375.9 243 2753 Percentage of the requests served within a certain time (ms) 50% 243 66% 264 75% 282 80% 299 90% 351 95% 433 98% 2670 99% 2753 100% 2753 (longest request) 

As you can see, the use of Pony ORM gave here a 2-fold increase in API performance.

In the second example, I request all the objects from the table. Here, productivity increase is even more significant:

 seva@SEVA (2):~/djony$ ab -n 20 -c 4 "http://localhost:8080/api/v2/auth/user/?format=json&limit=0" This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient).....done Server Software: nginx/1.1.19 Server Hostname: localhost Server Port: 8080 Document Path: /api/v2/auth/user/?format=json&limit=0 Document Length: 306326 bytes Concurrency Level: 4 Time taken for tests: 40.891 seconds Complete requests: 20 Failed requests: 0 Write errors: 0 Total transferred: 6133760 bytes HTML transferred: 6126520 bytes Requests per second: 0.49 [#/sec] (mean) Time per request: 8178.157 [ms] (mean) Time per request: 2044.539 [ms] (mean, across all concurrent requests) Transfer rate: 146.49 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.1 0 0 Processing: 6235 7976 1035.4 7980 10671 Waiting: 6225 7959 1033.0 7958 10654 Total: 6235 7976 1035.4 7980 10671 Percentage of the requests served within a certain time (ms) 50% 7980 66% 8177 75% 8287 80% 8390 90% 10001 95% 10671 98% 10671 99% 10671 100% 10671 (longest request) seva@SEVA (2):~/djony$ ab -n 20 -c 4 "http://localhost:8080/api/v3/auth/user/?format=json&limit=0" This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient).....done Server Software: nginx/1.1.19 Server Hostname: localhost Server Port: 8080 Document Path: /api/v3/auth/user/?format=json&limit=0 Document Length: 306326 bytes Concurrency Level: 4 Time taken for tests: 11.841 seconds Complete requests: 20 Failed requests: 0 Write errors: 0 Total transferred: 6133760 bytes HTML transferred: 6126520 bytes Requests per second: 1.69 [#/sec] (mean) Time per request: 2368.136 [ms] (mean) Time per request: 592.034 [ms] (mean, across all concurrent requests) Transfer rate: 505.88 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.1 0 0 Processing: 1024 2269 806.2 2227 4492 Waiting: 1017 2252 803.6 2211 4472 Total: 1024 2269 806.2 2227 4492 Percentage of the requests served within a certain time (ms) 50% 2227 66% 2336 75% 2395 80% 2406 90% 4140 95% 4492 98% 4492 99% 4492 100% 4492 (longest request) 

API performance increased more than 4 times!

What remains to be

Authorization and authentication have not yet been implemented, however, I do not see any special problems in their implementation.

There is a problem with the implementation of filters on the criteria of the regex, week_day and search types, since similar criteria are not yet available in Pony ORM.

Conclusion

Implementing the Pony ORM as a replacement for the Django ORM in ready-made Django application packages is quite possible and very useful in terms of productivity. A good abstraction of classes inside a package helps a lot, in particular the presence of a layer independent of Django ORM.

I invite everyone who is interested in improving the performance of their Django applications to join the open djony projects (crossing Pony and Django) and tastypie_djony (accelerating Tastypie using Pony ORM) to bring them to a sustainable industrial design as quickly as possible .

Project Pony ORM is an open source project.

Traditionally, I remind you that the Pony ORM developers are still readonly-users of Habrahabr:

AlexeyMalashkevich m.alexey@gmail.com

metaprogrammer alexander.kozlovsky@gmail.com