Spreading applications on twisted

My job is to create various Twisted applications to work with numerous web services. Unlike the projects on TurboGears , which I spread as eggs using easy_install (via setuptools) I did not have a convenient way to roll out projects on Twisted.

Up to this point.

Twisted overcomes a convenient system of plug-ins that allow applications to be built into the "twistd" starter.

Only one problem - the complete lack of decent documentation about this process.
')
Below I want to give a short story about how you can build a simple application in a twisted package. Actually - I will take the well-known tutorial - Twisted finger and I will add step number 12: “How to create a package for finger as a Twisted application for twistd” (aka “Skipped step”).

Step 12: Twisted application plug-in for twistd installer

Create a directory structure as shown below.
finger
finger/__init__.py
finger/finger.py
MANIFEST.in
setup.py
twisted
twisted/plugins
twisted/plugins/finger_plugin.py

finger / finger.py is the finger app from here .

twisted / plugins - directory structure, contains the file finger_plugin.py, which will be described below. Please note - no __init__.py files in the twisted and twisted / plugins directories are important!

finger_plugin.py implements the IServiceMaker and IPlugin interfaces.

  # ==== twisted / plugins / finger_plugin.py ====
 # - Zope modules -
 from zope.interface import implements

 # - Twisted modules -
 from twisted.python import usage
 from twisted.application.service import IServiceMaker
 from twisted.plugin import IPlugin

 # - Finger modules -
 from finger import finger

 class Options (usage.Options):
     synopsis = "[options]"
     longdesc = "Make a finger server."
     optParameters = [
         ['file', 'f', '/ etc / users'],
         ['templates', 't', '/ usr / share / finger / templates'],
         ['ircnick', 'n', 'fingerbot'],
         ['ircserver', None, 'irc.freenode.net'],
         ['pbport', 'p', 8889],
     ]
    
     optFlags = [['ssl', 's']]

 class MyServiceMaker (object):
     implements (IServiceMaker, IPlugin)
    
     tapname = "finger"
     description = "Finger server."
     options = Options
    
     def makeService (self, config):
         return finger.makeService (config)

 serviceMaker = MyServiceMaker () 

setup.py is a standard installer file. Pay attention to the “packages” and “package_data” arguments in the setup () function and the refresh_plugin_cache () function, which is called after it has fully completed setup () s. The latter will manually reset the Twisted plugins cache (twisted / plugins / dropin.cache).

  # ==== twisted / plugins / finger_plugin.py ====
 '' 'setup.py for finger.

 This is an extension of the twisted finger tutorial demonstrating how
 to package the Twisted application and installable Python package and
 twistd plugin (consider it "Step 12" if you like).

 Uses twisted.python.dist.setup () to make this package installable as
 a Twisted Application Plugin.

 After the installation should be manageable as a twistd
 command.

 For example, to enter it in the foreground:
 $ twistd -n finger

 To view the options for finger enter:
 $ twistd finger --help
 '' '

 __author__ = 'Chris Miles'


 import sys

 try:
     import twisted
 except ImportError:
     raise SystemExit ("twisted not found. Make sure you"
                      "have installed the Twisted core package.")

 from distutils.core import setup

 def refresh_plugin_cache ():
     from twisted.plugin import IPlugin, getPlugins
     list (getPlugins (IPlugin))

 if __name__ == '__main__':
    
     if sys.version_info [: 2]> = (2, 4):
         extraMeta = dict (
             classifiers = [
                 "Development Status :: 4 - Beta",
                 "Environment :: No Input / Output (Daemon)",
                 "Programming Language :: Python",
             ])
     else:
         extraMeta = {}

     setup (
         name = "finger",
         version = '0.1',
         description = "Finger server.",
         author = __ author__,
         author_email = "you@email.address",
         url = "http://twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html",
         packages = [
             "finger",
             "twisted.plugins",
         ],
         package_data = {
             'twisted': ['plugins / finger_plugin.py'],
         },
         ** extraMeta)
    
     refresh_plugin_cache () 

MANIFEST.in contains one script, which, I believe, tells distutils to modify an existing Twisted package (to install twisted / plugin / finger_plugin.py) and something like that.
graft twisted

This is all so that you can start installing the package in the usual way:
$ python setup.py install

After that, you need to run twistd - to make sure that the application is installed normally and accessible. Look at the twistd options:
$ twistd --help
Usage: twistd [options]
...
Commands:
athena-widget Create a service which starts a NevowSite with a single
page with a single widget.
ftp An FTP server.
telnet A simple, telnet-based remote debugging service.
socks A SOCKSv4 proxy service.
manhole-old An interactive remote debugger service.
portforward A simple port-forwarder.
web A general-purpose web server which can serve from a
filesystem or application resource.
inetd An inetd(8) replacement.
vencoderd Locayta Media Farm vencoderd video encoding server.
news A news server.
words A modern words server
toc An AIM TOC service.
finger Finger server.
dns A domain name server.
mail An email service
manhole An interactive remote debugger service accessible via
telnet and ssh and providing syntax coloring and basic
line editing functionality.
conch A Conch SSH service.

Let's look at the options of our finger server in the plugin:
$ twistd finger --help
Usage: twistd [options] finger [options]
Options:
-s, --ssl
-f, --file= [default: /etc/users]
-t, --templates= [default: /usr/share/finger/templates]
-n, --ircnick= [default: fingerbot]
--ircserver= [default: irc.freenode.net]
-p, --pbport= [default: 8889]
--version
--help Display this help and exit.

Make a finger server.

Run the finger server in the console:
$ sudo twistd -n finger --file=users
2007/12/23 22:12 +1100 [-] Log opened.
2007/12/23 22:12 +1100 [-] twistd 2.5.0 (/Library/Frameworks/Python.framework/
Versions/2.5/Resources/Python.app/Contents/MacOS/Python 2.5.0) starting up
2007/12/23 22:12 +1100 [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'>
2007/12/23 22:12 +1100 [-] finger.finger.FingerFactoryFromService starting on 79
2007/12/23 22:12 +1100 [-] Starting factory <finger.finger.FingerFactoryFromService instance at 0x1d0a4e0>
2007/12/23 22:12 +1100 [-] twisted.web.server.Site starting on 8000
2007/12/23 22:12 +1100 [-] Starting factory <twisted.web.server.Site instance at 0x1d0a558>
2007/12/23 22:12 +1100 [-] twisted.spread.pb.PBServerFactory starting on 8889
2007/12/23 22:12 +1100 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x1d0a670>
2007/12/23 22:12 +1100 [-] Starting factory <finger.finger.IRCClientFactoryFromService instance at 0x1d0a5f8>


Twistd provides numerous useful features - such as starting a daemon, specifying the location of the log and pid files, etc ...

Unfortunately, Twisted and setuptools do not work well together, so I could not pack my Twisted application as an egg, I have to tinker with the dependencies resolution system setuptools, or install using easy_install.

References:

• twistedmatrix.com/projects/core/documentation/howto/plugin.html
• twistedmatrix.com/projects/core/documentation/howto/tap.html
• twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html

- Source