Immediately, in order zavlekalochki. Our goal will be to learn how to write polyglot programs that can be interpreted in several programming languages ​​at once (one of which is “basic” Python). In this case, in the case of interpretation on one of them, the program will generate another program, functionally similar (or even equivalent) to that which is performed in the case of interpretation in another language.

And the most interesting: the approaches used in writing this program will be interesting not so much academically as practically - when developing a program using these approaches, development will be easier and more convenient (at least a bit unusual at first), and the program will be more efficient than without them.

However, it sounds worse than it is.
')
Go?

__debug__

Let's start with a fairly simple - with code generation. Python programmers know that in Python there is no preprocessor (as, for example, in C) that would allow you to create macros that, depending on some parameters, generate different code at the compile-time stage. The “dynamism” of Python, of course, allows you to generate anything and anytime at runtime, but ... sometimes the ability to influence the compile time is also useful, and even necessary.

The preprocessor in Python, of course not. But all Python programmers surely know the directive assert, which allows you to add a check to the program code that in some place the value of an expression is invariant.

At the same time, they will surely remember that this directive is ignored when the Python interpreter runs “optimized” (i.e., the -O key) and is not executed by the interpreter.

Compare, for example,

#! / usr / bin / python def func (x): assert x> 5 return x import dis; dis.dis (func)

→

4 0 LOAD_FAST 0 (x) 3 LOAD_CONST 1 (5) 6 COMPARE_OP 4 (>) 9 JUMP_IF_TRUE 7 (to 19) 12 POP_TOP 13 LOAD_GLOBAL 0 (AssertionError) 16 RAISE_VARARGS 1 >> 19 POP_TOP 5 20 LOAD_FAST 0 (x) 23 RETURN_VALUE

and

#! / usr / bin / python -O def func (x): assert x> 5 return x import dis; dis.dis (func)

→

5 0 LOAD_FAST 0 (x) 3 RETURN_VALUE

.

What follows from this? That, by varying the value of optimization when starting the generation of byte code, you can get completely different byte code. Code that behaves differently depending on whether it was run in optimization mode or without it. What is this if not the germ of a preprocessor?
Why do we need this? In order to significantly separate the behavior of debug-and release-build-s. Everything is like “in big languages” - a bunch of checks, logs and informativeness in debug-build, fast and efficient code in release-build. But is it not enough for this one directive assert?

So, assert is different depending on the optimization setting that was at the time of generating the bytecode (I hope everyone remembers / knows that it is not necessary to regenerate the byte code every time? Moreover, Ie files .pyc / .pyo , can be distributed without source codes from which they were generated).
From this it follows quite logically that the information on whether the optimization was enabled at the time of its launch was available to the bytecode generator. And how, by the way, does he know how to optimize the code?

So, this very information is available to him in the form of a constant __debug__ . Pay attention - really constants. Not so often in an absolutely dynamic Python language, you can find constants :) But since this is a constant, the optimizer is able to generate code differently, depending on its value:

#! / usr / bin / python def func (): if __debug__: print import dis; dis.dis (func)

→

5 0 PRINT_NEWLINE 1 LOAD_CONST 0 (None) 4 RETURN_VALUE

#! / usr / bin / python -O def func (): if __debug__: print import dis; dis.dis (func)

→

4 0 LOAD_CONST 0 (None) 3 RETURN_VALUE

And this, and, as a special case, optimization of the assert directive (well, plus the removal of docstrings with -OO ) is actually the whole spectrum of skills of the optimizer.

This optimizer is so simple that, although it can optimize code generation for the or and and operators on variables (so that the second part of the test will not be performed if unnecessary), but optimize the code generation when the first test is a constant (so that the second part of the check will not be generated if it is not needed); it is no longer able:

#! / usr / bin / python -O def func (x): if __debug__ and x> 5: print import dis dis.dis (func)

→

4 0 LOAD_GLOBAL 0 (__debug__) 3 JUMP_IF_FALSE 18 (to 24) 6 POP_TOP 7 LOAD_FAST 0 (x) 10 LOAD_CONST 1 (5) 13 COMPARE_OP 4 (>) 16 JUMP_IF_FALSE 5 (to 24) 19 POP_TOP 5 20 PRINT_NEWLINE 21 JUMP_FORWARD 1 (to 25) >> 24 POP_TOP >> 25 LOAD_CONST 0 (None) 28 RETURN_VALUE

Please note: although the optimization has been enabled, even though the generated code can skip the check for x > 5 , if after checking it __debug__ out that the constant __debug__ set to True , it will do the check itself. Therefore, if you need to get rid of even generating some code in the release-build, you will have to optimize the code “in the old manner” - to do nested checks, first for the __debug__ value, and only inside it at x > 5 :

#! / usr / bin / python -O def func (x): if __debug__: if x> 5: print import dis dis.dis (func)

→

4 0 LOAD_CONST 0 (None) 3 RETURN_VALUE

By the way, the ternary operator is also not optimized to check for a constant:

#! / usr / bin / python -O def func (x): return 5 if __debug__ else 7 import dis dis.dis (func)

→

4 0 LOAD_GLOBAL 0 (__debug__) 3 JUMP_IF_FALSE 7 (to 13) 6 POP_TOP 7 LOAD_CONST 1 (5) 10 JUMP_FORWARD 4 (to 17) >> 13 POP_TOP 14 LOAD_CONST 2 (7) >> 17 RETURN_VALUE

Recall why we are doing this: in order to be able in debug-builds to have arbitrarily detailed debugging information (for example, displayed in the service logs) that would be omitted without affecting the performance of release-builds .

But now, when we know exactly what and how you can do with the variable __debug__ (no composite logical operations, no ternary operations — just a clean check of one variable __debug__ for validity), we can do, for example, the following:

if __debug__: def log (msg): __do_some_heavy_logging (msg) else: def log (msg): pass

This is already something. Now we can install a log(-) at least every second line of code - this will not affect the performance of the release-build.

Well, to be honest, almost not at all. Suppose now that we do not write every line of the log on the release-build by terribly slow calling the __do_some_heavy_logging() procedure to print a line on an ancient roll-on ADC - but the call to each log() function, albeit empty, will remain in the release-build. And the generation of a message for him (if it is also computationally nontrivial). But there's nothing you can do. In Python, there is no preprocessor, and we will not be able to completely omit the log() call on debug-build-ah. However…

GPP: GNU Python Preprocessor

... However, if the preprocessor is not in the Python itself, why not take it from the outside? What can you not do in an effort to optimize release-build.

Glory to the gods, there is a GPP in the world - the Generic Preprocessor, the General-Purpose Processor, and now the new backronym - the GNU Python Preprocessor. Why not?

Especially to describe it and re-read the man aloud reluctance. I can only say that its syntax actually repeats the syntax of the preprocessor in C: #define , #if , etc.

What is very convenient - in Python, comments begin with a sharp sign. Do you realize what it means? ..

Right. Now we will write a polyglot program.

Yes. This is quite a full polyglot. The program is in two languages ​​- the language of Python (not possessing a preprocessor), and the language of GPP.

At the same time, these programs are not of the print "Hello world" level print "Hello world" , equally behaving in 4242 different programming languages ​​and dialects. Not at all, our programs will even have different tasks. A Python program (1) will have to solve the problem for which we are writing it. A GPP (2) program should generate Python (3) / (4) programs that solve the problem for which we are writing it.

Moreover, pay attention to two funny facts. First, a GPP program will generate not one, but at least (in our case) two different Python programs. One (3) is the corresponding debug-build, the other (4) is the corresponding release-build. And secondly, (1) is definitely not at all equal to (4) (in order for the release build to differ from the middleware developer towards higher performance due to discarding the debugging code, we started everything), and it doesn’t even have to be equal ( 3)! But it is better, of course, that at least (1) be equal to (3). Otherwise we will get confused. Too many levels on which to think at the same time. A typical problem when writing a polyglot program.

Well, let's try?

Writing source code on GPP:

#ifdef debug #define log (msg) __do_some_heavy_logging (msg) #else #define log (msg) #endif

In general, not bad. If we called GPP with the -Ddebug parameter, then it will replace the function call __do_some_heavy_logging(msg) in each place of the source code where the function call log() __do_some_heavy_logging(msg) . Compared to code based on __debug__ , the direct benefit is already less than one intermediate function call. If we called GPP without such a parameter, then it will replace every call to the log() function with an empty string ...

Hooray! What we need ... or not?

And if we execute this code immediately in the Python interpreter without preprocessing? Then, for any call to the log() function, the interpreter simply does not find it — in our case, it is defined in the preprocessor.

So, it is necessary to define it and at the level of Python. And so as not to interfere with the preprocessor. The second pancake:

def log (msg): __do_some_heavy_logging (msg) #ifdef debug #define log (msg) __do_some_heavy_logging (msg) #else #define log (msg) #endif

Already better. Without preprocessing, we create the log() function and use it every time we call log() in the code. With preprocessing, without preprocessor variable debug - log() calls are replaced with an empty string. With preprocessing, with the debug preprocessor variable: replace log() calls with __do_some_heavy_logging() calls.

Good? Already better. But there is one caveat: we (1) turned out not equal to (3). Which, of course, is beautiful, but potentially dangerous during the development process. It is better to make sure that the debug code preprocessing in the mode is absolutely equivalent to the non-preprocessing code. And so put it all inside the #ifdef debug check.

#ifdef debug def log (msg): _do_some_heavy_logging (msg) #else #define log (msg) #endif

That's so much better. In the #ifdef debug section #ifdef debug is only a non-preprocessor code, and therefore the program will behave in the same way with both the preprocessor and the -Ddebug variable, and without the preprocessor. In the else section, only the preprocessor code (and therefore it will work only in the release-build). So that. And even the Python variable __debug__ no longer needed.

Cython

Now that we have learned how to generate Python code by the preprocessor and at the same time maintain start-up capability without passing the preprocessor, why not take the task more complicated?

Why don't we try to write a program that, after passing GPP in release mode (without the -Ddebug variable), generates code in a different programming language? Well, or almost another ?

What for?

And, for example, because this language can be Cython. Based on Python and close to it in syntax, but explicitly typed. And having a nice and useful feature in the household: a Cython program / module can be translated into C source code, and a module suitable for use from Python code can be created from this source code after compilation.

The only thing is that Cython programs are not backwards compatible with Python. But Python programs are usually correct Cython programs. On the one hand, this simplifies the task to almost triviality. On the other - we are not pioneers, in order to create difficulties for ourselves and solve them heroically, we just want to write in Python efficiently.

So, the goal: to write a module in Python (1 '), which has a function that performs some kind of calculations. This module should be safely imported and used by Python itself without any preprocessing - so that it is more convenient to develop. In addition, this module must be a GPP program (2 '). Which, in the case of appropriately specifying GPP variables, should generate a Cython program (3 '). Containing an algorithm that is completely similar to the program (1 '), but typed. Such that after translating the source code (3 ') into C (4') and compiling, a binary Python library will be created that performs the same as in (1 '), only due to the compiled code, much faster.

As a solvable problem, we take, for example, the second problem of the Project Euler : find the sum of all even Fibonacci numbers that do not exceed four million . Imagine that we need to solve it as part of a much larger problem, and we still want to solve the problem in Python. It's just that we want to optimize this function for performance by rewriting it in Cython. At the same time, we want to preserve the ability to run the same code without processing by Cython (and GPP), as usual Python code, for reasons of ease of development.

The desired process looks like the development: we make the Python module, from which we are in the process of development and debugging and we will import the function calculating our task solution. When generating a release-build, the same code will be interpreted by means of GPP, which will generate the equivalent Cython program (with the necessary data typing for this task); later in the release-build build process, the Cython program will be translated by Cython into a C program, which will be compiled into a library available for use by the Python code. And in the release-build, the calling Python code will import the function not from the Python module, but from the binary library, compiled and very fast.

What should the function that solves the second task look like? Spoiler, of course ... but this is the second task, and any reasonable programmer who has finished reading this line and has not escaped into the realm of Morpheus will surely calmly decide it himself. Therefore, it will not be a big loss for the educational process if I write this function:

def sum_even_fibo_le (upb): f1, f2 = 1, 2 res = 2 while f2 <upb: f2, f1 = f1 + f2, f2 if f2% 2 == 0: res + = f2 return res

Accordingly, we write the module fibo.py , which contains this function, in the main program we import the function from this module and call it, passing the upper bound for the task.

Although this function is even for the upper limit of four million and is performed fairly quickly even in interpreted Python, but in this case it simply demonstrates the capabilities that we can achieve. Therefore, we will try to optimize it, no matter what. Rewrite it in Cython.

In principle, the Python function we already have is a fully valid Cython function. Moreover, being compiled by Cython, it will even work faster. In two or three times. Because the bytecode processing stage will completely disappear, and instead of it, the machine code will immediately perform the necessary operations on Python objects.

But this is not the limit. If we decide to typify our data (while, unfortunately, having lost the pleasant unboundedness of the Python data type long), then our code will work even faster.

What will we (can) type in Cython? Input arguments to functions. Output results of functions. Intermediate variables.

So we typify everything. Let's calculate that 64 bit should be enough for everybody, and we will result in all variables to unsigned long long. At the same time we will get acquainted with Cython, who does not know him:

cdef unsigned long long sum_even_fibo_le (unsigned long long upb): cdef unsigned long long f1, f2, res f1, f2 = 1, 2 res = 2 while f2 <upb: f2, f1 = f1 + f2, f2 if f2% 2 == 0: res + = f2 return res

The code is good, quite readable for pythonists, and quite understandable. It has only two drawbacks:

1. It is no longer valid Python code. However, we will solve it. Preprocessor. But…
2. He does not work.

Well, not that it does not work at all. Just does not solve our problem. This code perfectly creates, in Cython terminology, the “C function”. Which can be called from other Cython code or even C, but which cannot be called from Python code. What does not suit us.
We'll have to modify this function so that it remains “Python function”. A bit slow call (however, the call from Cython code can be saved fast), which returns PyObject (if someone is disturbed by the Python's interpreter interpreter).

cpdef unsigned long long sum_even_fibo_le (unsigned long long upb): cdef unsigned long long f1, f2, res f1, f2 = 1, 2 res = 2 while f2 <upb: f2, f1 = f1 + f2, f2 if f2% 2 == 0: res + = f2 return res

So, we have a function in Python, which solves our problem, albeit slowly, but it is more convenient to debug the module with it — run it for interpretation without intermediate processing. And the function on Cython, which the usual Python interpreter no longer understands, but which is then the most productive:

def sum_even_fibo_le (upb): f1, f2 = 1, 2 res = 2 while f2 <upb: f2, f1 = f1 + f2, f2 if f2% 2 == 0: res + = f2 return res

∪

cpdef unsigned long long sum_even_fibo_le (unsigned long long upb): cdef unsigned long long f1, f2, res f1, f2 = 1, 2 res = 2 while f2 <upb: f2, f1 = f1 + f2, f2 if f2% 2 == 0: res + = f2 return res

Let's use our favorite GPP (or what, you haven’t loved it yet?), Recall the knowledge that we acquired during the creation of the first Python / GPP polyglot program, and first write a GPP program:

#ifdef debug def sum_even_fibo_le (upb): #else cpdef unsigned long long sum_even_fibo_le (unsigned long long upb): cdef unsigned long long f1, f2, res #endif f1, f2 = 1, 2 res = 2 while f2 <upb: f2, f1 = f1 + f2, f2 if f2% 2 == 0: res + = f2 return res

Not bad. But if we try to read it as a Python program, counting the lines with sharps for comments, we will see that this is not a polyglot at all. And an invalid program in Python - right after the line with def comes the line with cpdef . Let's think about how to disguise the Python code from Cython interpreter.

Previous experience tells us that the #ifdef debug block should contain only Python code, and the #else block should contain only preprocessor code. Or comments. But how to hide the #else block from Python interpretation? If we comment it out - through GPP we will not be able to uncomment it. If we create any GPP macro that comments on this block in debug-build and decomposes it in release-build, it will be an invalid Python code, without passing GPP.

Let's take a closer look at the junction of languages ​​and try to do something there. A hint can be given to us by the way the unwanted Cython lines look and where they interpret the resulting code as Python functions:

def sum_even_fibo_le (upb): cpdef unsigned long long sum_even_fibo_le (unsigned long long upb): cdef unsigned long long f1, f2, res f1, f2 = 1, 2 res = 2 while f2 <upb: f2, f1 = f1 + f2, f2 if f2% 2 == 0: res + = f2 return res

... They are right under the function signature, in the place where they should be ... docstring-i!

Yeah. The idea is clear. Let's try to fix this code so that the Cython lines look like docstring (or just free strings) when interpreted by Python. And with the interpretation of GPP - no.

#ifdef debug def sum_even_fibo_le (upb): "" " #else cpdef unsigned long long sum_even_fibo_le (unsigned long long upb): cdef unsigned long long f1, f2, res #endif "" " f1, f2 = 1, 2 res = 2 while f2 <upb: f2, f1 = f1 + f2, f2 if f2% 2 == 0: res + = f2 return res

This is still far from working code, this is just the beginning. But the idea is clear: from the point of view of Python (and, which is necessary - simultaneously after preprocessing GPP without -Ddebug ), the code in Cython will become a docstring.

But now we have a problem: the pass by the preprocessor in the -Ddebug mode generates an empty docstring, and the pass by the preprocessor in the release-mode generates only the final triple of double quotes from the docstring. Dead end.

Something needs to be done to get rid of the triplet quotes in preprocessing, but save it without preprocessing?
Or in general, so that the triplets of quotes disappear in the preprocessing mode, but remain without preprocessing? No, of course, you can frame each triplet with your own #if debug , or something like that, but this is cumbersome. Now, if it were possible to hide them in the preprocessing mode ... comment!

And for sure. GPP . , , C: /* ... */ . Python- «» (, - ) Cython- , C-style Cython . C-style , Python-, Python- .

#!/usr/bin/python #mode comment "/*" "*/" #ifdef debug def sum_even_fibo_le(upb): #else /* """ */ cpdef unsigned long long sum_even_fibo_le(unsigned long long upb): cdef unsigned long long f1, f2, res #endif /* """ # */ f1, f2 = 1, 2 res = 2 while f2 < upb: f2, f1 = f1 + f2, f2 if f2 % 2 == 0: res += f2 return res

Everything is fine. , , . Python-, Cython-. .

Down to Earth

... Well, I agree, he is not great. Hand on heart, I have to admit - he sucks. It is absolutely impossible to support him. It can only copy-paste. This confusion of punctuation marks is completely unintuitive. If we need to specify this construction for each function, the code will look like Dresden after the bombing. In a real product this should not be.

Good. In the best traditions of Dr. House, the first diagnosis and the first treatment turned out to be incorrect and dangerous for the patient. We make a differential diagnosis again, based on what we already know.

We need to make two preprocessing branches. A branch #ifdef debugshould generate, without preprocessing, non-executing code when interpreting Python. Branch#elseshould contain just working code. It should all look quite understandable and readable, and it should be easy to use.

Therefore, the variant with commenting on everything and everything and overlapping of the commenting area in different languages ​​is dismissed. It was unfit in any case - what if there was a sequence somewhere in Python /*?

Good. What are our options?

Preprocessor macros. Yes, they are. It makes it all easy. Just as we did in the very first experiment with GPP and logging.

Let's get a macrocython() , , Python-. Cython-. GPP ; Python- . Something like this:

#!/usr/bin/python def cython(text): pass #ifndef debug #define cython(x) x #endif cython("cpdef unsigned long long sum_even_fibo_le(unsigned long long upb):") def sum_even_fibo_le(upb): cython("cdef unsigned long long f1, f2, res") f1, f2 = 1, 2 res = 2 while f2 < upb: f2, f1 = f1 + f2, f2 if f2 % 2 == 0: res += f2 return res

, . . , , , :

#!/usr/bin/python def cython(text): pass "cpdef unsigned long long sum_even_fibo_le(unsigned long long upb):" def sum_even_fibo_le(upb): "cdef unsigned long long f1 f1, f2 = 1, 2 res = 2 while f2 < upb: f2, f1 = f1 + f2, f2 if f2 % 2 == 0: res += f2 return res

-, Cython- , Python- . — , Cython- , #ifdef debug .

-, GPP «». cython , , GPP. …

#!/usr/bin/python def cython(text): pass #ifndef debug #mode string QQQ "\"" "\"" #define cython(x) x #endif cython("cpdef unsigned long long sum_even_fibo_le(unsigned long long upb):") #ifdef debug def sum_even_fibo_le(upb): #endif cython("cdef unsigned long long f1, f2, res") f1, f2 = 1, 2 res = 2 while f2 < upb: f2, f1 = f1 + f2, f2 if f2 % 2 == 0: res += f2 return res

. . — cython() .

Total

, . Python Cython . debug- -. Python-. , all included, :

#!/usr/bin/python #include debug.py from debug import cython, log cython("cpdef unsigned long long sum_even_fibo_le(unsigned long long upb):") #ifdef debug def sum_even_fibo_le(upb): #endif cython("cdef unsigned long long f1, f2, res") log ("Calculating sum_even_fibo_le") assert upb> 2 f1, f2 = 1, 2 res = 2 while f2 <upb: f2, f1 = f1 + f2, f2 if f2% 2 == 0: res + = f2 log (res) return res

Here you can live with it.