PyGTK: threads and magic wrappers

GTK + is good for everyone, but there is a big problem when working with it in multi-threaded applications. GTK itself is thread-safe, but requiring forced blocking by the user. The second problem is that the blocking is implemented through the mutexes, and you must block it strictly one time, otherwise your code will “hang” on linux while working fine on windows.

Experience has shown that the access method “Any stream appeals to the GUI, the main thing to cause blocking” was a failure: after some time, you can catch the core dump in the depths of GTK for a variety of reasons, which are useless to catch.

UP : The code is uploaded to Github , and slightly updated with the latest version of the code for the “main” project. Cosmetic changes (Period instead of Seconds, and logging of the use of an erroneous flow for finding problem areas has been added.

In this connection, in my small project I came to the following method of organizing the most trouble-free work with threads:

1. Any GTK calls are produced by only one thread. All changes to the GUI are made by dedicated status update features.
2. All locks are maintained through a common reentrant blocker.
3. The lock also monitors the GUI from the "wrong" thread, logging it.

General structure of the GUI part

The main part consists of a module gui.py, exporting the following "good":

GtkLocker = CGtkLocker() #  def GtkLocked(f): #    gui,        def IdleUpdater(f): #    ,      gui def SecondUpdater(f): #    ,       def GUIstop(): #       GUI def GuiCall(Func): #    GUI ""  def GuiIdleCall(Func): #    GUI       def GuiSecondCall(Func): #       def GUI(): #     

')
And the work of the application looks like this:

 import gui, gtk def InitApp(): """       """ with GtkLocker: window = gtk.Window(gtk.WINDOW_TOPLEVEL) window.set_title(u"  ") window.connect("destroy", gui.GUIstop) window.realize() vbox = gtk.VBox() window.add(vbox) label = gtk.Label("Text label") vbox.add(label) window.show_all() def __main__(): gui.GuiCall( InitApp ) gui.GUI() if __name__ == '__main__': __main__() sys.exit(0) 

This application does nothing but show a text box with the text label. Let's do it humanly: we will move all the application code to the class and add a button.

 class CMyApp(object): def __init__(self): self.label = None self.times = 0 def CreateGui(self): with gui.GtkLocker: window = gtk.Window(gtk.WINDOW_TOPLEVEL) window.set_title(u"  ") window.connect("destroy", gui.GUIstop) window.realize() vbox = gtk.VBox() window.add(vbox) label = gtk.Label("Welcome to our coool app!") vbox.pack_start(label) label = gtk.Label("Here will be counter") self.label = label vbox.pack_start(label) button = gtk.Button("Press me!") button.connect("clicked", self.Click) vbox.pack_start(button) window.show_all() @gui.GtkLocked def Click(self, widget): self.times += 1 self.label.set_text("You pressed button %d times" % self.times) MyApp = CMyApp() def InitApp(): """       """ MyApp.CreateGui() 

So what have we done? They created an object whose __init__ method simply prepared the future fields of the method, and all the actual creation is done in the CreateGui function, which will be called already from the general processing cycle of the event events.

Now for the magic that the Click method is subject to: notice that it is marked with the gui.GtkLocked wrapper. This means that this method is a method for handling GUI events, and is called strictly via connect, which means that at the time the method is called, it already has a GTK lock. This wrapper implements the GUI state of the Locker Already Locked, so using a lock inside the function will not cause any problems.

Add a second button that also makes the sea useful:

 class CMyApp(object): def CreateGui(self): with gui.GtkLocker: .... button = gtk.Button("Or me!") button.connect("clicked", gui.GtkLocked(self.Count)) vbox.pack_start(button) .... @gui.GtkLocked def Click(self, widget): self.Count() gui.GuiSecondCall( self.Count ) def Count(self, *args, **kwargs): with gui.GtkLocker: self.times += 1 self.label.set_text("You pressed button %d times" % self.times) 

So, we changed the Click method of the previous button to call the general Count method, and to the deferred call it with a delay of up to a second, which counts and updates the code regardless of the weather on Mars, and hung up on the second button Count .
Since the Count method involves calling it not only via connect , we cannot hang @gui.GtkLocked on it - the method can be called from a context that is not yet blocked (for example, it is simply called in the idle event), therefore we mark gui.GtkLocked directly into moment of connect As a result, the Count method can be called from an unblocked context and it will take the lock itself, but it is also bound to an event and another event handler calls it. Due to the magic with GtkLocker and GtkLocked no deadlock happens, everything works.

Now let's add His Highness Progress Bar, and a complex background process that updates its contents in the process:

 class CMyApp(object): def CreateGui(self): with gui.GtkLocker: .... progress = gtk.ProgressBar() self.progress = progress vbox.pack_start(progress) T = threading.Thread(name="Background work", target=self.Generate) T.setDaemon(1) T.start() @gui.GtkLocked def UpdateProgress(self): self.progress.pulse() def Generate(self): while(True): time.sleep(0.3) gui.GuiIdleCall( self.UpdateProgress ) 

So, our Generate method works on the background, and every 0.3 seconds it wants to update progress, for which it adds UpdateProgress execution UpdateProgress . Since UpdateProgress runs in the context of a thread's GUI, everything works. That is just what will happen if we do not know the time required for implementation? Update on every sneeze? Replace 0.3 with 0.001 - and admire the result. No, this is not an option. Add time measurements and artificially slow down the update? Generally not an option. Maybe instead of GuiIdleCall do GuiSecondCall ? Let's try ... M-yes. Every second there is a sharp update of all events executed during this second. Horror.

Let's add one more background process, and to it a “smart” update method:

 class CMyApp(object): def CreateGui(self): with gui.GtkLocker: .... fastprogress = gtk.ProgressBar() self.fastprogress = fastprogress vbox.pack_start(fastprogress) T = threading.Thread(name="Heavy background work", target=self.GenerateFast) T.setDaemon(1) T.start() @gui.SecondUpdater def SingleUpdateProgress(self): self.fastprogress.pulse() def GenerateFast(self): while(True): time.sleep(0.001) self.SingleUpdateProgress() 

Magic, delight! We simply declare the state update function and hang the necessary wrapper on it: @gui.SecondUpdater or @gui.IdleUpdater , and the method will be automatically called in the context of the GUI stream no more than once per second or in free time. Due to the wrappers, the double launch of the method in a row is excluded, it does not require an extra accounting code whether it was called and there is no need to think about folding into the execution queue.

Under the hood

So let's take a close look at what's inside the gui.py.

The common code is nothing complicated, just initialization:

 from __future__ import with_statement import logging, traceback logging.basicConfig(level=logging.DEBUG, filename='debug.log', filemode='a', format='%(asctime)s %(levelname)-8s %(module)s %(funcName)s %(lineno)d %(threadName)s %(message)s') import pygtk pygtk.require('2.0') import gtk gtk.gdk.threads_init() import gobject, gtk.glade, Queue, sys, configobj, threading, thread from functools import wraps import time, os.path gtk.gdk.threads_enter() IGuiCaller = Queue.Queue() IGuiIdleCaller = Queue.Queue() IGuiSecondsCaller = Queue.Queue() IdleCaller = [ None ] IdleCallerLock = threading.Lock() gtk.gdk.threads_leave() class CGtkLocker: .... #    GtkLocker = CGtkLocker() #   GUI  --   main_quit,   GUI  @IdleUpdater def GUIstop(*args, **kwargs): gtk.main_quit() # ,     . #    (gobject)  , #  threading.Queue,     . def GuiCall(Func): IGuiCaller.put(Func) with IdleCallerLock: if IdleCaller[0] == False: gobject.idle_add(GUIrun) IdleCaller[0] = True def GuiIdleCall(Func): IGuiIdleCaller.put(Func) with IdleCallerLock: if IdleCaller[0] == False: gobject.idle_add(GUIrun) IdleCaller[0] = True def GuiSecondCall(Func): IGuiSecondsCaller.put(Func) #    GUI  def GUIrun(): #     GuiCaller try: Run = IGuiCaller.get(0) #   idle ,  GTK   # ,   ,    #      "with GtkLocker:" ,  @GtkLocked with GtkLocker: Run() except Queue.Empty: #    --  GuiIdleCaller try: Run = IGuiIdleCaller.get(0) with GtkLocker: Run() except Queue.Empty: #     -- ,   with GtkLocker: IdleCaller[0] = False return False return True #   :     #     def GUIrunSeconds(): try: with GtkLocker: while (True): Run = IGuiSecondsCaller.get(0) Run() except Queue.Empty: pass return True #    def GUI(): #   gobject.idle_add(GUIrun) IdleCaller[0] = True gobject.timeout_add(1000, GUIrunSeconds) #     gtk.main gtk.gdk.threads_enter() #    GtkLocker, #   gtk.main    GtkLocker.FREE() #     GUI  gtk.main() #   main_quit,   main    gtk.gdk.threads_leave() 

Now look at the most interesting: wrappers. So, the most important of them is the implementation of a reentrant lock:

 class CGtkLocker: def __init__(self): #       self.lock = threading.Lock() self.locked = 1 # ,      self.thread = thread.get_ident() #   ,     GUI    self.mainthread = self.thread self.warn = True #   ,   N  # ,     with  def __enter__(self): #       with self.lock: DoLock = (thread.get_ident()!=self.thread) #   ,   -       if self.warn and self.mainthread != thread.get_ident(): logging.error("GUI accessed from wrong thread! Traceback: "+"".join(traceback.format_stack())) #        with  ,      if DoLock: #        gtk.gdk.threads_enter() #             with self.lock: self.thread = thread.get_ident() #  __enter__    self.locked += 1 return None #     with  (  -- ,   ..) def __exit__(self, exc_type, exc_value, traceback): #      ,       with self.lock: self.locked -= 1 if self.thread!=thread.get_ident(): print "!ERROR! Thread free not locked lock!" logging.error("Thread free not locked lock!") sys.exit(0) else: if self.locked == 0: self.thread = None gtk.gdk.threads_leave() return None #    :      . def FREE(self): self.locked -= 1 self.thread = None if self.locked != 0: print "!ERROR! Main free not before MAIN!" logging.error("Main free not before MAIN!") sys.exit(0) GtkLocker = CGtkLocker() 

It should be understood that in " with GtkLocker " any piece that works with the GUI should be wrapped.
And in most cases, it will work, even if all calls go from different threads, for that threads_enter / threads_leave and created. But only sometimes everything works for the time being, and suddenly falls into the crust somewhere in the depths of GTK.

GtkLocker wrapper GtkLocker that allows you to mark event methods that are called by the GTK kernel already inside the lock. Being called at the zero level increases the level of the blocking level, thus ensuring that we do not call threads_enter / threads_leave .

 def GtkLocked(f): @wraps(f) def wrapper(*args, **kwds): with GtkLocker.lock: if GtkLocker.thread == None or GtkLocker.thread==thread.get_ident(): GtkLocker.thread = thread.get_ident() GtkLocker.locked += 1 WeHold = True else: print "!ERROR! GtkLocked for non-owned thread!" logging.error("GtkLocked for non-owned thread!") WeHold = False ret = None try: return f(*args, **kwds) finally: if WeHold: with GtkLocker.lock: GtkLocker.locked -= 1 if GtkLocker.locked == 0: GtkLocker.thread = None return wrapper 

Well, the last magical pass: IdleUpdater / SecondUpdater :

 def IdleUpdater(f): @wraps(f) def wrapper(*args, **kwds): self = len(args)>0 and isinstance(args[0], object) and args[0] or f if '_idle_wrapper' not in self.__dict__: self._idle_wrapper = {} def runner(): if self._idle_wrapper[f]: try: return f(*args, **kwds) finally: self._idle_wrapper[f] = False return None if f not in self._idle_wrapper or not self._idle_wrapper[f]: self._idle_wrapper[f] = True #  SecondUpdater   GuiSecondCall GuiIdleCall( runner ) return wrapper 

In the object whose method we are calling, a _idle_wrapper dictionary is _idle_wrapper , in which the tracking is carried out, whether this method has already been queued for execution or not, and if not, we remember that we have inserted and added a launch wrapper that will execute this method and reset the flag . As a result, the first method call will add its launch to the GuiIdleCall (or *Seconds* ) queue, and repeated calls until its execution will be simply ignored.

Download codes and useful information

All sources of the example are available: pygtk-demo.tar.bz2 .
For useful information on PyGTK known issues in its official FAQ .
For writing under PyGTK, I often refer to the tutorial and manual .