Learning to write multi-threaded and multi-process applications in Python

This article is not for experienced Python tamers, for whom unraveling this tangle of snakes is child's play, but rather a superficial overview of multi-threaded features for those who recently sat down on a python.

Unfortunately, there are not so much material in Russian on the topic of multithreading in Python, and the pythoners who have not heard anything, for example, about GIL, began to come across to me with enviable regularity. In this article I will try to describe the most basic features of a multi-threaded python, tell you what GIL is and how to live with it (or without it) and much more.


Python is a charming programming language. It perfectly combines many programming paradigms. Most of the tasks that a programmer can meet are solved here easily, elegantly and concisely. But for all these tasks, a single-threaded solution is often enough, and single-threaded programs are usually predictable and easy to debug. What can not be said about multi-threaded and multiprocess programs.
')

Multithreaded applications

Python has a threading module, and it has everything you need for multi-threaded programming: there are different types of locks, and a semaphore, and an event mechanism. One word - everything you need for the vast majority of multi-threaded programs. And to use all these tools is quite simple. Consider an example of a program that runs 2 threads. One thread writes ten “0”, the other - ten “1”, and strictly in turn.

import threading def writer(x, event_for_wait, event_for_set): for i in xrange(10): event_for_wait.wait() # wait for event event_for_wait.clear() # clean event for future print x event_for_set.set() # set event for neighbor thread # init events e1 = threading.Event() e2 = threading.Event() # init threads t1 = threading.Thread(target=writer, args=(0, e1, e2)) t2 = threading.Thread(target=writer, args=(1, e2, e1)) # start threads t1.start() t2.start() e1.set() # initiate the first event # join threads to the main thread t1.join() t2.join() 

No magic and voodoo-code. The code is clear and consistent. And, as you can see, we created a stream from a function. For small tasks it is very convenient. This code is also quite flexible. Suppose we have a 3rd process that writes “2”, then the code will look like this:

 import threading def writer(x, event_for_wait, event_for_set): for i in xrange(10): event_for_wait.wait() # wait for event event_for_wait.clear() # clean event for future print x event_for_set.set() # set event for neighbor thread # init events e1 = threading.Event() e2 = threading.Event() e3 = threading.Event() # init threads t1 = threading.Thread(target=writer, args=(0, e1, e2)) t2 = threading.Thread(target=writer, args=(1, e2, e3)) t3 = threading.Thread(target=writer, args=(2, e3, e1)) # start threads t1.start() t2.start() t3.start() e1.set() # initiate the first event # join threads to the main thread t1.join() t2.join() t3.join() 

We added a new event, a new stream and slightly changed the parameters with which
threads start (of course, you can write a more general solution using, for example, MapReduce, but this is already beyond the scope of this article).
As we see, there is still no magic. Everything is simple and clear. Let's go further.

Global Interpreter Lock

There are two most common reasons for using threads: firstly, to increase the efficiency of using the multicore architecture of modern processors, and hence the performance of the program;
secondly, if we need to divide the logic of the program into parallel fully or partially asynchronous sections (for example, be able to ping several servers at the same time).

In the first case, we are confronted with such a restriction of Python (or rather its main CPython implementation), as Global Interpreter Lock (or abbreviated GIL). The concept of GIL is that only one thread at a time can be executed by a processor. This is done to ensure that between threads there is no struggle for individual variables. The executable thread accesses the entire environment. This feature of the implementation of threads in Python greatly simplifies work with threads and gives a certain thread safety.

But there is a subtle point: it may seem that a multi-threaded application will work exactly the same time as a single-threaded one, doing the same thing, or for the sum of the execution time of each thread on the CPU. But here we are waiting for one unpleasant effect. Consider the program:

 with open('test1.txt', 'w') as fout: for i in xrange(1000000): print >> fout, 1 

This program simply writes a million “1” lines to the file and does it in ~ 0.35 seconds on my computer.

Consider another program:

 from threading import Thread def writer(filename, n): with open(filename, 'w') as fout: for i in xrange(n): print >> fout, 1 t1 = Thread(target=writer, args=('test2.txt', 500000,)) t2 = Thread(target=writer, args=('test3.txt', 500000,)) t1.start() t2.start() t1.join() t2.join() 

This program creates 2 threads. In each stream, she writes in a separate file for half a million lines "1". In essence, the amount of work is the same as that of the previous program. But over time, there is an interesting effect. The program can work from 0.7 seconds to as much as 7 seconds. Why is this happening?

This is due to the fact that when a thread does not need a CPU resource, it frees GIL, and at this moment it can try to get it, and another thread, and also the main thread. At the same time, the operating system, knowing that there are many cores, can aggravate everything by trying to distribute the flows between the cores.

UPD: at the moment in Python 3.2 there is an improved implementation of GIL, in which this problem is partially solved, in particular, due to the fact that each thread after losing control waits for a short period of time before it can again capture GIL (there is good presentation in English)

“It’s impossible to write effective multi-threaded programs in Python?” You ask. No, of course, there is a way out and even a few.

Multi-process applications

In a sense, to solve the problem described in the previous paragraph, Python has a subprocess module. We can write a program that we want to execute in a parallel thread (in fact, already a process). And run it in one or more threads in another program. Such a way would really speed up the work of our program, because the threads created in the GIL launching program do not pick up, but just wait for the completion of the running process. However, in this way there are a lot of problems. The main problem is that it becomes difficult to transfer data between processes. It would be necessary to somehow serialize objects, establish communication through PIPE or a friend of tools, and all this inevitably carries overhead and the code becomes difficult to understand.

Here another approach can help us. Python has a multiprocessing module. In terms of functionality, this module resembles threading . For example, processes can be created in the same way from normal functions. The methods for working with processes are almost all the same as for threads from the threading module. But for the synchronization of processes and data exchange it is customary to use other tools. We are talking about queues (Queue) and channels (Pipe). However, there are analogs of locks, events and semaphores that were in threading, too.

In addition, the multiprocessing module has a mechanism for working with shared memory. For this, the module has classes for the variable (Value) and the array (Array), which can be “generalized” (share) between processes. For the convenience of working with common variables, you can use classes managers (Manager). They are more flexible and easy to handle, but slower. It should be noted a pleasant opportunity to make common types from the ctypes module using the multiprocessing.sharedctypes module.

Even in the multiprocessing module, there is a mechanism for creating process pools. This mechanism is very useful for implementing the Master-Worker template or for implementing a parallel Map (which in a sense is a special case of the Master-Worker).

Of the main problems with working with the multiprocessing module, it is worth noting the relative platform dependence of this module. Since working with processes is organized differently in different operating systems, some restrictions are imposed on the code. For example, in Windows OS there is no fork mechanism, therefore the process split point should be wrapped in:

 if __name__ =='__main__': 

However, this design and so is a good form.

What else ...

There are other libraries and approaches for writing parallel applications in Python. For example, you can use Hadoop + Python or various MPI implementations in Python (pyMPI, mpi4py). You can even use wrappers for existing C ++ or Fortran libraries. Here one could mention such frameworks / libraries as Pyro, Twisted, Tornado and many others. But this is all beyond the scope of this article.

If you like my style, in the next article I will try to tell you how to write simple interpreters in PLY and why they can be used.