We write the extension module for Python on C

OMFG! - can exclaim the reader. Why write something on C when there is Python, and will be largely right. However, fortunately, unfortunately our green friend is not omnipotent. So…

Task Description

In the framework of the current project (a virtual machine management system based on Libvirt ), it was necessary to programmatically control the loop device in Linux. The first version when based on a call to the command line command losetup via subprocess.Popen () worked fairly well on my Ubuntu 8.04, however, after the deployment, there was a bug report that RHEL and some other systems did not work. After some trials, it turned out that the losetup accepts slightly different arguments in them, and it will not be possible to implement our task.



Having picked the source code of losetup, I saw that all the operations I needed are done by sending IOCTL calls to the device. With Python's fcntl.ioctl (), something went wrong for me. It was decided to go down to a lower level, write a module in C.

')

Disclaimer

As it turned out, fcntl.ioctl () is quite sufficient to implement all that I needed. I do not remember what scared me at the beginning. Probably need to work less than 10 hours a day;)



On the other hand, if I used it right away - this topic would not exist.



So again, for those who read diagonally, there is an excellent module fcntl.ioctl () in Python. All that read below is just an example.

API Planning

All that you can do on Python is to do on Python. What does not work out is to carry it to a low-level in C.



What I can’t do on python is a bit of a mess: actually mounting / unmounting an image, and checking if the device is busy.



As part of the task, there were no requirements for supporting encryption, and other frills, therefore from the C side, the interface turned out to be quite simple:

• mount (device, imagepath) - mounts the imagepath in the device.
• unmount (device, imaepath) - frees the device.
• is_used (device) - 1 if the device is mounted, and 0 if free

Making a skeleton

The module, by analogy with the command line utility, will be called losetup. We start favorite Eclipse + PyDev and we create the project. In it we create losetup.py in which there will be all Python code of the module.



The module that implements the low-level interaction with the system is called _losetup. Our losetup will import _losetup and use it to implement a high-level API.



Create a folder src, in which we put two files losetupmodule.c and losetupmodule.h



losetupmodule.c

#include <Python.h&rt;



#include "losetupmodule.h"



// -

static PyObject * LosetupError;



//

static PyObject *

losetup_mount (PyObject * self, PyObject * args)

{

return Py_BuildValue( "" );

}



//

static PyObject *

losetup_unmount (PyObject * self, PyObject * args)

{

return Py_BuildValue( "" );

}



// , -

static PyObject *

losetup_is_used (PyObject * self, PyObject * args)

{

int fd, is_used;

const char * device;

struct loop_info64 li;



if ( ! PyArg_ParseTuple(args, "s" , & device)) {

return NULL ;

}



if ((fd = open (device, O_RDONLY)) < 0 ) {

return PyErr_SetFromErrno(LosetupError);

}



is_used = ioctl(fd, LOOP_GET_STATUS64, & li) == 0 ;



close(fd);

return Py_BuildValue( "i" , is_used);

}



//

// , , ,

static PyMethodDef LosetupMethods[] = {

{ "mount" , losetup_mount, METH_VARARGS, "Mount image to device. Usage _losetup.mount(loop_device, file)." },

{ "unmount" , losetup_unmount, METH_VARARGS, "Unmount image from device. Usage _losetup.unmount(loop_device)." },

{ "is_used" , losetup_is_used, METH_VARARGS, "Returns True is loopback device is in use." },

{ NULL , NULL , 0 , NULL } /* Sentinel */

};



//

PyMODINIT_FUNC

init_losetup ( void )

{

PyObject * m;



// _losetup

m = Py_InitModule( "_losetup" , LosetupMethods);

if (m == NULL )

return ;



//

LosetupError = PyErr_NewException( "_losetup.error" , NULL , NULL );

Py_INCREF(LosetupError);

PyModule_AddObject(m, "error" , LosetupError);

}



#include <Python.h&rt;



#include "losetupmodule.h"



// -

static PyObject * LosetupError;



//

static PyObject *

losetup_mount (PyObject * self, PyObject * args)

{

return Py_BuildValue( "" );

}



//

static PyObject *

losetup_unmount (PyObject * self, PyObject * args)

{

return Py_BuildValue( "" );

}



// , -

static PyObject *

losetup_is_used (PyObject * self, PyObject * args)

{

int fd, is_used;

const char * device;

struct loop_info64 li;



if ( ! PyArg_ParseTuple(args, "s" , & device)) {

return NULL ;

}



if ((fd = open (device, O_RDONLY)) < 0 ) {

return PyErr_SetFromErrno(LosetupError);

}



is_used = ioctl(fd, LOOP_GET_STATUS64, & li) == 0 ;



close(fd);

return Py_BuildValue( "i" , is_used);

}



//

// , , ,

static PyMethodDef LosetupMethods[] = {

{ "mount" , losetup_mount, METH_VARARGS, "Mount image to device. Usage _losetup.mount(loop_device, file)." },

{ "unmount" , losetup_unmount, METH_VARARGS, "Unmount image from device. Usage _losetup.unmount(loop_device)." },

{ "is_used" , losetup_is_used, METH_VARARGS, "Returns True is loopback device is in use." },

{ NULL , NULL , 0 , NULL } /* Sentinel */

};



//

PyMODINIT_FUNC

init_losetup ( void )

{

PyObject * m;



// _losetup

m = Py_InitModule( "_losetup" , LosetupMethods);

if (m == NULL )

return ;



//

LosetupError = PyErr_NewException( "_losetup.error" , NULL , NULL );

Py_INCREF(LosetupError);

PyModule_AddObject(m, "error" , LosetupError);

}



In losetupmodule.h, just a definition set ruthlessly torn from util-linux-ng

Customize the assembly

Modules can be assembled in different ways, but the easiest and most reliable is through setuptools (distutils).



Create setup.py

from setuptools import setup, Extension

setup(name = 'losetup' ,

version = '1.0.1' ,

description = 'Python API for "loop" Linux module' ,

author = 'Sergey Kirillov' ,

author_email = 'serg@rainboo.com' ,

ext_modules = [Extension( '_losetup' , [ 'src/losetupmodule.c' ], include_dirs = [ 'src' ])],

py_modules = [ 'losetup' ]

)



All white magic in the string "ext_modules = [Extension ('_ losetup', ['src / losetupmodule.c'], include_dirs = ['src'])]". It describes the extension named _losetup, whose code is in src / losetupmodule.c, including src. This is enough for the distutils to build an extension, install it, make all kinds of pekedges out of it (including the win32 installer, although it's not that simple).



We check that everything builds by calling "python setup.py build"

Build muscle

Implement the mount () method

static PyObject *

losetup_mount (PyObject * self, PyObject * args)

{

int ffd, fd;

int mode = O_RDWR;

struct loop_info64 loopinfo64;

const char * device, * filename;



// Check parameters

if ( ! PyArg_ParseTuple(args, "ss" , & device, & filename)) {

return NULL ;

}



// Initialize loopinfo64 struct, and set filename

memset( & loopinfo64, 0 , sizeof (loopinfo64));

strncpy(( char * )loopinfo64.lo_file_name, filename, LO_NAME_SIZE - 1 );

loopinfo64.lo_file_name[LO_NAME_SIZE - 1 ] = 0 ;



// Open image file

if ((ffd = open(filename, O_RDWR)) < 0 ) {

if (errno == EROFS) // Try to reopen as read-only on EROFS

ffd = open(filename, mode = O_RDONLY);

if (ffd < 0 ) {

return PyErr_SetFromErrno(LosetupError);

}

loopinfo64.lo_flags |= LO_FLAGS_READ_ONLY;

}



// Open loopback device

if ((fd = open(device, mode)) < 0 ) {

close(ffd);

return PyErr_SetFromErrno(LosetupError);

}



// Set image

if (ioctl(fd, LOOP_SET_FD, ffd) < 0 ) {

close(fd);

close(ffd);

return PyErr_SetFromErrno(LosetupError);

}

close (ffd);



// Set metadata

if (ioctl(fd, LOOP_SET_STATUS64, & loopinfo64)) {

ioctl (fd, LOOP_CLR_FD, 0 );

close (fd);

return PyErr_SetFromErrno(LosetupError);

}

close(fd);



return Py_BuildValue( "" );

}





static PyObject *

losetup_mount (PyObject * self, PyObject * args)

{

int ffd, fd;

int mode = O_RDWR;

struct loop_info64 loopinfo64;

const char * device, * filename;



// Check parameters

if ( ! PyArg_ParseTuple(args, "ss" , & device, & filename)) {

return NULL ;

}



// Initialize loopinfo64 struct, and set filename

memset( & loopinfo64, 0 , sizeof (loopinfo64));

strncpy(( char * )loopinfo64.lo_file_name, filename, LO_NAME_SIZE - 1 );

loopinfo64.lo_file_name[LO_NAME_SIZE - 1 ] = 0 ;



// Open image file

if ((ffd = open(filename, O_RDWR)) < 0 ) {

if (errno == EROFS) // Try to reopen as read-only on EROFS

ffd = open(filename, mode = O_RDONLY);

if (ffd < 0 ) {

return PyErr_SetFromErrno(LosetupError);

}

loopinfo64.lo_flags |= LO_FLAGS_READ_ONLY;

}



// Open loopback device

if ((fd = open(device, mode)) < 0 ) {

close(ffd);

return PyErr_SetFromErrno(LosetupError);

}



// Set image

if (ioctl(fd, LOOP_SET_FD, ffd) < 0 ) {

close(fd);

close(ffd);

return PyErr_SetFromErrno(LosetupError);

}

close (ffd);



// Set metadata

if (ioctl(fd, LOOP_SET_STATUS64, & loopinfo64)) {

ioctl (fd, LOOP_CLR_FD, 0 );

close (fd);

return PyErr_SetFromErrno(LosetupError);

}

close(fd);



return Py_BuildValue( "" );

}







It seems to be easy, but perhaps not quite clear what is happening here. Let's take a look at the main elements.

if ( ! PyArg_ParseTuple(args, "ss" , & device, & filename)) {

return NULL ;

}



if ( ! PyArg_ParseTuple(args, "ss" , & device, & filename)) {

return NULL ;

}





Functions declared as METH_VARARGS get arguments in the form of a tuple. PyArg_ParseTuple () checks that the arguments match the specified pattern (in this case, “ss” is two strings), and receives data, or, if the argument does not match the pattern, sets an error, and returns false. Details on how this works can be found in Extracting Parameters in Extension Functions.



From the point of view of python, it looks like this:

 >>> import _losetup
 >>> _losetup.mount ("aaa")
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 TypeError: function takes exactly 2 arguments (1 given)
 >>> _losetup.mount (1,2)
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 TypeError: argument 1 must be string, not int
 >>> 

Go ahead

return PyErr_SetFromErrno(LosetupError);

PyErr_SetFromErrno creates an exception with the specified type, receives an error code from the global variable errno, and returns NULL - which means that the exception has occurred. Documentation Links: Intermezzo: Errors and Exceptions , Exception Handling



For python, it looks like this:

 >>> _losetup.mount ('/ dev / loop0', '/ tmp / somefile')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 _losetup.error: (2, 'No such file or directory')
 >>> 

return Py_BuildValue( "" );



return Py_BuildValue( "" );



Our function does not need to return any specific data, so we return None. Read more in Building Arbitrary Values



The remaining functions are implemented similarly.

PyPI Publication

So the module is written. We need to give humanity a chance to use it. The easiest way to do this is to publish a module in the Python Package Index .



Register on PyPI.



After registration, write to the console

python setup.py register

Enter your account information, and setuptools will create a PyPI package.

python setup.py sdist upload

does source destribution (tgz archive with code and metadata), and uploads it to PyPI.



The result can be seen here http://pypi.python.org/pypi/losetup/



We go to the hated RHEL, write easy_install -U losetup, and while we speak the magic words “cryble-craft-bumc”, setuptools will download our package, bring it down and install it into the system.



Add losetup as a dependency in the setup.py of the main application. Now when it is installed, setuptools will also be installed by our module.

Completion

So, it was unexpectedly easy to fall from Python to the abstraction level below, and write a module for low-level interaction with the system.



We also got a good example of what you need to think more and do less. Our Green Friend is powerful, and even such exotic tasks can be solved without parting with it.



What you want.

Used literature

• Extending and Embedding the Python Interpreter Guido van Rossum