Named tuples from selections

In Django, to speed up queries that return a large amount of data, there are methods QuerySet values() and values_list() . The first instead of models returns dictionaries, the second tuples. Working with those and with others is not as convenient as with instances of models, they say, pay the guys for speed convenience. But I don’t want to, and thanks to the nominal tuples from the standard collections module, I won’t.

For those who have not encountered, and for those who have not yet convinced, I will give a piece of code illustrating the current state of affairs:

qs = Item . objects . filter( ... )
for item in qs:
print item . title, item . amount
total += item . amount * item . price

#
qs = Item . objects . filter( ... ) . values( 'title' , 'amount' , 'price' )
for item in qs:
print item[ 'title' ], item[ 'amount' ]
total += item[ 'amount' ] * item[ 'price' ]

#
qs = Item . objects . filter( ... ) . values_list( 'title' , 'amount' , 'price' )
for item in qs:
print item[ 0 ], item[ 1 ]
total += item[ 1 ] * item[ 2 ]


Versions with dictionaries and tuples are not as pretty as with models, but they work faster and require less memory. Named tuples allow to access their fields by attribute, i.e. with them, our code will hardly differ from the code for models. Hence the accompanying bonus - in many cases we will be able to switch from models to tuples and back without changing the code, and where you have to change it, the changes will be minimal. In addition, tuples store field names in the class, rather than each object as dictionaries, so they take up less memory, which is nice in itself and will be even more important if you think of adding the results of queries to the cache.

What are named tuples?

Named tuples appeared in Python 2.6 and are tuples in which access to elements is possible by specified attributes. A small piece of code for the demonstration:

>>> from collections import namedtuple
>>> Point = namedtuple( 'Point' , 'x y' . split())
>>> p = Point(x =11 , y =22 ) #
>>> p2 = Point( 11 , 22 ) # ...
>>> p1 == p2
True
>>> p[ 0 ] + p[ 1 ] #
33
>>> x, y = p # ...
>>> x, y
(11, 22)
>>> p . x + p . y #
33
>>> Point . _make([ 11 , 22 ]) #
Point(x=11, y=22)


NamedTuplesQuerySet

It remains to create a descendant of QuerySet , which will produce named tuples. Due to the fact that all the query logic with marked fields is implemented in ValuesQuerySet (returned from QuerySet.values() ), we just need to process the result before issuing:

from itertools import imap
from collections import namedtuple

from django.db.models.query import ValuesQuerySet

class NamedTuplesQuerySet (ValuesQuerySet):
def iterator ( self ):
#
extra_names = self . query . extra_select . keys()
field_names = self . field_names
aggregate_names = self . query . aggregate_select . keys()
names = extra_names + field_names + aggregate_names

#
tuple_cls = namedtuple( ' %s Tuple' % self . model . __name__, names)

results_iter = self . query . get_compiler( self . db) . results_iter()
#
return imap(tuple_cls . _make, results_iter)


And, for ease of use, we assign the method to the QuerySet :

from django.db.models.query import QuerySet

def namedtuples ( self , * fields):
return self . _clone(klass = NamedTuplesQuerySet, setup = True , _fields = fields)
QuerySet . namedtuples = namedtuples


After that, the code given in the beginning of the article for models, dictionaries and tuples can be written as:

qs = Item . objects . filter( ... ) . namedtuples( 'title' , 'amount' , 'price' )
for item in qs:
print item . title, item . amount
total += item . amount * item . price


It's funny that the resulting class performs the functions of both the existing values() and values_list() accelerators. And it does it either better, or almost also and thus generates a more beautiful code. Maybe someday it will lead to their replacement.

PS Class code with the patch can be pulled from here gist.github.com/876324
PPS The order of the fields in the tuple may not correspond to the order of the arguments nametuples() , scored on this for speed. Implementing the reordering of fields, if anyone needs it, can be taken from ValuesListQuerySet.iterator() .