Embedding PyPy Code in C Applications

Note translator:
As correctly suggested in the comments, in spite of the title, this article will focus not on directly embedding code, but on creating shared libraries in Python. But since this is just a translation, I decided to leave the name closer to the original.

---

At the PyGrunn 2016 conference, I gave a talk on the Python cffi package and its use for embedding PyPy code in C applications.

With the release of cffi 1.5.0 and its subsequent inclusion in PyPy 5, it becomes possible to embed PyPy code. This is done by compiling Python code into a dynamic library, which can then be used in any other language. In this article, I will show you how to do this.

Embedded API

The first step is to define an interface that defines how the C application should call our Python code. We need to point this out using prototypes of C functions. For our example, we consider a function that performs some kind of computation, but, of course, it can be anything.

float compute(float first, float second); 

Now we need to implement these calculations in Python:

 from my_library import ffi, lib @ffi.def_extern() def compute(first, second): """      . """ return abs(first - second) 

This fragment contains several special things for its proper embedding. The first line imports ffi and lib objects from a dynamic library. By doing this, the implementation accesses the functions provided by cffi and can be used for more complex tasks, such as allocating memory. The name my_library defined below and corresponds to the name of our dynamic library.

The second thing we notice in this fragment is the decorator @ffi.def_extern . He tells cffi that the decorated functions should be represented in a public API created by the C library. Decorated functions will be matched with the prototypes specified in the API declaration and their arguments, and the returned values ​​will be converted automatically.
')

Script generating library

Now that we have an API and its implementation, we need to actually embed it somewhere. For this we use a script that generates a dynamic library. It requires that the two code snippets listed above are in the api.h and implementation.py files.

 import cffi ffi = cffi.FFI() ffi.embedding_api(open("api.h").read()) ffi.embedding_init_code(open("implementation.py").read()) ffi.set_source("my_library", "") ffi.compile(verbose=True) 

This script is very simple. We must specify our API and provide its implementation. Both are read from disk and correspond to the code snippets given in the previous section.

After specifying the source code, we must say cffi the name of our library. In this example, this is my_library . In addition, there is a place to add additional C-code that provides types for the header file of our API, for example, by including the appropriate header files (which is not allowed in embedding_api ). It remains only to compile the source code to create a file of our library.

Running the script displays some information and creates our library:

 $ pypy embed.py generating ./my_library.c running build_ext building 'my_library' extension ... $ ls my_library.dylib -rwxr-xr-x 1 djinn staff 9856 May 15 14:46 my_library.dylib 

All that's left is to use it somewhere!

Using application

Using embedded Python code is actually very simple. You can do this with the following code:

 #include <stdio.h> #include "api.h" int main(void) { float result = compute(12.34f, 10.0f); printf("The result: %f\n", result); return 0; } 

As you can see, almost nothing is needed to call our Python code. Using the CPython API, you would have to start the interpreter and perform many parameter and return value conversions. But not with cffi! The library created takes over, so you can focus on really useful work. The last thing I have to show you is how to compile and run this code.

 $ clang -o test test.c my_library.dylib $ ./test The result: 2.340000 

And here it is! With just a few lines of code, we wrote a C program that runs the PyPy interpreter and executes our Python code as if it were C code. Of course, I showed you only the basics, but this is really powerful technology. For more information, see the cffi documentation .