Python-> Cython-> C ++, and COM to boot: writing a framework for autotests
I think everyone is aware of the benefits of autotests. They help to keep the code in working condition, even with significant changes. It can also save testers from tedious manual work and allows you to focus on more interesting types of testing.
Despite the fact that some parts of our project are over 25 years old, we are only at the very beginning of the introduction of automatic testing. Nevertheless, we already have some successes, which I want to tell about in this article.
How to write good autotests is a topic for a separate article. And probably not one. I will tell you how we have implemented testing of individual components. Components are written in C ++ and have interfaces very similar to COM. We chose python as the test language and use the very powerful PyTest test framework. In the article I will tell you about the complexity of the C ++ / COM and python bundles, the pitfalls that we stumbled upon and how to solve these problems.
')
In the project I'm working on, we are developing a large and complex module. Several million lines of C ++ code, a dozen large components and a hundred dll under the hood. This module is used in several huge applications.
Historically, this is all tested only through the UI, and mostly manually. It often happens that a change in one area gets out of a bug somewhere in a completely different place. And they find this bug only in a few days, or even weeks. And sometimes something pops up in a few months, when another product decides to integrate a new version of our module.
We have unit tests that chase CI by commits. But the code was over 15 years old when we started talking about TDD. The code is monolithic and it will not work out just to launch it separately. We need a great refactoring for which no resources are given. Therefore, we only have unit tests for simple functions or individual classes.
But since the module has some API, then this module can be tested through this API. You can collect the use cases of all applications that use us and write autotests for this. Then it would be possible to drive these tests directly to CI by commits. So the idea of ​​component testing was born.
But who will write the tests? Testers could come up with good test scenarios and prepare test data, but testers do not know C ++ (and those who know quickly dumped into developers). Programmers could code such tests, but usually fantasy is only enough for a couple of positive scenarios. To cover all negative cases, patience is usually not enough.
We decided to adopt the experience of colleagues from the neighboring team. They made vrappers for their component using cython and put into the python a simplified interface that is used for tests. The threshold for entering the python is much lower than in C ++. Testers can easily master the python in a couple of days and begin to write good autotests.
Before you begin to describe our torment you need to say a few words about our interfaces. We use the technology slicked with COM and ported to Linux, poppy and fryu. There are some infrastructural differences associated with the lack of a registry, but for the article it doesn’t matter.
COM-like technology gives us a bunch of buns, like a ready-made plug-in component infrastructure. We can easily dock modules written by different teams in different countries (including third party plugins). At the same time, we are not concerned with the compatibility issues of different compilers, runtimes, and standard libraries. The same style of interfaces of all modules, agreements on the transfer of parameters and return values, the lifetime of objects - all this is governed by agreements as with COM.
There is a downside. Inside the modules we can use any buns of modern C ++ standards. But in public interfaces, we must adhere to the rules of SOM, only simple types or interfaces inherited from IUnknown. No stl. No exceptions, only HRESULT. Because of this, the code on the borders of the modules is quite cumbersome and not very readable.
To begin with, we identified a dozen interfaces to our module, with which we could implement a small but complete workflow.
But even though these interfaces are part of a public API, in reality they are quite low-level. In order to make a certain operation it is impossible to simply take and call a single function or method. You need to create five objects, connect them with each other, run for execution and wait for the result through the future. All this complexity is needed to organize transactional, asynchronous, Undo / Redo, to organize access to thread-safe insides and even heaps of other things. In short, 2 screens of C ++ code in COM style.
We decided to follow the recommendations of our colleagues and write a small layer that would hide low-level interfaces. In the python, it was proposed to expose several high-level functions.
A cython vrapper just was forwarding calls to c ++:
The C ++ implementation of the CMyModuleObject class was already involved in useful actions: it created objects of our module and called some useful methods (the same 2 screens of code).
Cython is essentially a translator. Based on the source code above, cython generates a ton of generic code. If we compile it as dll / so (and rename it to pyd), we get a Python module. With ++ implementation CMyModuleObject also needs to be lodged in this dll. Now our Python module can be imported from Python (continuing from the import paths first). You can run using the usual Python interpreter, the main thing is that the architecture coincides. When executing the import string, the python will raise our dll itself, initialize and import everything it needs.
The python script looked something like this:
Cool! Much easier than C ++!
At the first stage, we decided not to bother with the test frameworks, but first to work out the approach on ordinary scripts. In this form, you can already write something, But this approach did not differ in flexibility. If you had to change something in the interlayer, then you had to change the code in C ++.
I insisted on trying to wrap up low-level interfaces directly, and write a layer, if necessary, on python. The idea was sold to the authorities that we can wring 1 to 1 interfaces and test at the level of our public API.
No sooner said than done. We quickly came to this scheme. The constructor of the Python object creates the SOM object and owns the link to it. Links, of course, are considered a smart copy of CComPtr.
Usually HRESULT functions are not interesting to us. If the function is successful - well, nice. If you zafeilas, then most likely you do not need to go further. Therefore, we simply check the error code and throw a Python exception. Processing return codes is not made to the level of the client Python code, which makes the client code much more compact and readable.
Note that the hresultcheck function is declared as cpdef. This means that it can be caused by both Python (sometimes hresult is checked in python) and native sish. The second property significantly reduces the error-handling code generated by the siton and speeds up the execution. We have not mastered the call to the SUCCEEDED macro, so we are comparing it with S_OK - for now.
Sometimes we still departed from 1 to 1 vrapping, when it was clear that certain interfaces and their methods should be used only in one particular way and in no other way. For example, if it is assumed that the COM object will be created empty, and then parameters will be crammed into it via the Set * () methods or a call to some Initialize (), in this case, at the python level, we simply made a convenient constructor with parameters.
Or another example. It happens that a request to one object conceptually returns a link to another object (or just a new object). In COM, you have to use output parameters, but in a python you can return an object humanly.
From the point of view of encapsulation, the code is not very good - one object climbs into the guts of another. But in python with encapsulation (or rather, with privacy) and so not very good. But we have not yet come up with a more beautiful way. There is a risk that someone will try to create Class2 with his hands in the client code; nothing good will probably come out. I would be glad if someone tells you the option of a private constructor in python.
The code examples above are located in files with the pyx extension (by the way, you can do a lot of them, rather than shoving everything in one). It's like cpp in the pros - a file with the implementation. But in a siton you still need a file with ads - pxd - a place where all the names that are considered sishnye will be described.
Pay attention to CComPtr :: operator = (). If in the Siton code you try to assign CComPtr directly, nothing happens. He simply can not really parse this syntax. I had to resort to the trick of renaming characters. So assign is how a character will look in a siton, and in quotes it is set what exactly needs to be called in a sish code.
The trick is useful if you need to call the Python class or function in the same way as a sishin.
pxd:
pyx:
Returning to our project. Python code is simpler and more compact, but still too low-level for most users. Therefore, we still decided to leave the interlayer, rewriting it in python. As a result, those 2 pages of cumbersome COM code have turned into this.
So our code has become much more compact and clearer, it was possible to use both high-level interfaces like do_operation (), and, if necessary, go down to "sishnyh" interfaces.
There was a feeling of flexibility, it was not necessary to recompile the C ++ part each time. Moreover, to start, we needed to build up only 10 interfaces, and for each subsequent feature, it was necessary to do some 1-2, which really added strength and faith to the chosen approach.
In this form, the technology may already be suitable for most projects, but we have come up against several fundamental limitations.
So, our COM host (the object that provides the COM infrastructure, everyones CoCreateInstance there, etc.) is a normal plus object. So, someone must create it (analogous to CoInitialize) and then delete it (CoFinalize). But here's the problem, the Python module has no main (). In any case, in the form as we needed.
Therefore, we created a plus Application object and put the initialization / finalization of the host into this object. We wrote a wrapper who allowed us to create this object from a python (at the beginning of each script).
But very quickly we began to catch kreshi at the exit. It turned out that, unlike C ++ (the first created object will be removed last), the order of destruction of objects in python is not defined. Well, in any case, there is no way to influence it. Depending on the phase of the moon, the python nailed the Application object first, it put out the COM infrastructure and forcibly unloaded all the components. Then the python deleted some other object that had a reference to some COM object. An attempt to call Release () from a dll that has already been unloaded resulted in a crash.
The second problem is the location of the executable file. It turned out that in our large component there are a lot of places that are trying to open files with data along a certain path relative to the executable file. This is normal if the final application is installed in the system in some known way. This is also normal if we work with the application compiled in the working directory. But this method stops working if the executable file suddenly becomes a python in the system directory.
There was even a function that allows you to override the application directory. It worked in most cases. But, unfortunately, there were cyclists who ignored this redefinition and continued to calculate the path independently relative to the executable. It would be correct to take it and fix it, but it would require considerable labor costs. We decided that this is still lying in backlog.
Finally, the third problem is the event loop. The fact is that our module is very complex and interactive. This is not just a “call function - get result” library. This is a huge combine. Inside, there are a couple of hundreds of threads that exchange messages. Some parts of the code were written in the time of the Mesozoic and are intended for execution only in the main thread (otherwise it will not work). In other places, a hardcode is sent to the main thread, expecting there to know how to process this message. And we also have our own subsystem of threads and messages, which also implies that the message processing cycle will necessarily spin in the main thread and conduct all of this. Without it in any way.
The easiest solution at the start was to insert the run_event_loop () method into our Application class, which twisted the message loop. The process stopped when our useful work was completed (as I understand it now, it was by coincidence :))
In general, scripts of the following type worked normally for us: we start some work using a non-blocking function (which does not wait for the end), after which we head off to the event loop
But with the scenarios that required some interactivity, there was a problem. We could not, for example, start the work, and then after a couple of seconds, try to stop it. In fact, the work did not start until the message loop in the main thread started. And if the cycle has started, then we will not return to the python.
Of course, it would be possible to fence something asynchronous at the python level, but this is clearly not what we would like. After all, the approach was supposed to push people who are not tempted by asynchronous systems. They would like to write just like that and not to bathe in some event loops
Without thinking, we tried to start processing in another thread, and in the main thing, to twist the message loop. But we immediately came up against the following problem - GIL (Global Interpreter Lock) . It turned out that Python threads are not actually running in parallel. Only one stream is running at a time, and the streams are switched every 100 commands. All this regulates this very GIL, stopping all threads except one.
It turned out that if the main thread went to the app.run_event_loop () function and did not return (it should hang there by design), then other Python commands in other threads are not executed. It’s just that there were not 100 more teams in the main thread, and the interpreter decided that it was too early to switch.
The solution was found in the nogil seedon keyword. A piece of code tagged nogil first releases the GIL, then performs a sish call. At the end of the GIL is captured again. So the main thread released GIL and went into the message loop. The second thread got control and did everything it needed there.
Cython, by the way, is a very capricious thing. It does not always allow to interfere with the Python and C code in one line. It also does not allow you to call some Python constructions and create new variables in the nogil sections (logically, this requires access to the python intestines, which are protected by GIL). It is necessary to be perverted like this, in order to correctly declare variables and make the necessary calls.
Everything seems to start working, but very unstable. We constantly caught some kind of crashes and hangs, and also constantly ran across a non-working functional (the file did not open in a relative path, but no one threw the error).
For several weeks we tried to defeat these 3 problems, tried different approaches. But every time there was another insoluble problem. Most of all delivered GIL, but we didn’t even imagine how to defeat the unloading of a COM host.
We even thought about jumping onto lua, but some of the restrictions would still remain. Only in this case would still have to go all the way from scratch.
But here came a bold idea. And what if we run not a code from a python, but rather a python from a code? Let's write your application that will do the following:
This approach solves all 3 problems in one fell swoop:
And you know? This approach worked! There were, however, a few minor problems that were eventually solved.
Another problem that had to be tricky was the sys.exit () function. We needed it to catch the return code from unittest and pass it to the output, and then process it with CI.
It works like this. If someone in the script calls sys.exit (), a SystemExit exception is actually generated. This exception is caught by the python itself and, like any other exception globally, must be printed to the console along with the stack trace. But the Py_PrintEx function knows that there is such a special case, and if we are offered to print a SystemExit exception, then we need to call the exit exit ()
Yes, yes, like this! A function called Print makes an exit () call. And this exit honestly works out - just takes and cuts down the entire application. And he didn’t want to spit that the application has unallocated handles, unfinished threads, unclosed files, unfinalized modules, a million active threads and all that jazz.
But the python (in any case, 2.7.6. Old time, I know) does not allow it to turn at the API level. I had to just copy several functions from the python sources (starting with PyRun_SimpleFileExFlags () and a few private ones that it calls) to my project and finish them by myself. So, our version in the case of SystemExit correctly exits and returns a return code. So after the completion of the Python part, the test application can properly clean and extinguish itself.
At first we had 2 projectors - one build test application with built-in python, and the second, as before, loadable module for python. But later we combined it all into one projector. The test application initialized the python, after which it called the initialization function of our python module (generated by a siton). So Python already at the start already knew about our module (although I still had to make imports).
In this form, the test application showed itself very well. We screwed the test framework (standard unittest) and testers began to write tests little by little. We ourselves, meanwhile, continued to drag the interfaces.
Screwing another piece of functionality, we stumbled upon the fact that in some cases we need to be able to accept callbacks. Those. Python synchronously calls a function, and in its interior it calls a callback to a python.
Positive interface looks like this:
The Python class cannot be inherited from the plus interface under any sauce. Therefore, in the project part of the project we had to do our own implementation, which threw calls to the python.
.h
.cpp
A special moment in this code is the capture of GIL. Otherwise, at best, the python will break on checking that the GIL is captured, but most likely it will either hang or cross.
We have a console application, so the error output to the console is the same. Even if the Python code throws an exception, our handler will catch it and print out the traceback.
From the side of the siton it looks like this:
When calling the MethodWithCallback () method, you must release the GIL, otherwise the callback will not be able to capture it.
With the client Python code, everything should be simple and clear.
, . , API, cython.
, . , , . , GIL ( , ), . PyGILState_Ensure() , , , .
, , , . , , , . .
. , , . PyRun_FileExFlags(), , PyArena_Free()
The python-> cython-> C ++ sandwich has been very successful as a framework for the AutoTest API. The threshold for entering the python is very small compared to other programming languages. Any competent tester in a couple of days will master the python at a level sufficient for writing autotests. The main thing in this matter is to figure out how to test this or that functional, and it is a matter of technology to express it in code.
We managed to build a user-friendly layer that exposes a simple and intuitive interface to the python. Calls of the low-level C ++ / COM code are hidden behind the wrappers and understanding of this code is not necessary for most tasks. But if necessary, it is easy to go down there.
Recently we screwed PyTest as a framework for writing tests. Very taxis! Tests have become even easier, clearer and faster.
Now there are no serious architectural flaws in sight, but there are still some bugs. For example, now there are a couple of cyclic dependencies, which is why several key objects do not want to collapse. But where to break the cyclical dependence correctly, we have not yet invented.
As for the siton himself. The developers of cython go to contact and have already fixed a couple of bugs for us. They release releases regularly.
Cython is quite capricious. It does not always intelligibly explain what exactly is not true in the source file, and sometimes also indicates the wrong place. Some things we have not mastered.
We also screwed the interactive python mode. Sometimes it is more convenient to debug Python code directly in the python console than to edit the script and run it again and again. It turned out all simple - just call it:
Developers of the C ++ module actively launch tests directly from the IDE. You can simply take a test that fell on the CI and otdebezhit. All breakpoints in C ++ code work as needed. But we have not yet invented how to debug the Python part. However, there was no special need - with PyTest, the tests are very simple and there is practically nothing to debug there.
Now it is the realization that the technology has taken place. Extend functionality easily and conveniently. People liked to write autotests, although for now they are treated with suspicion.
I hope the article will be useful, and you too can organize something like this.
Despite the fact that some parts of our project are over 25 years old, we are only at the very beginning of the introduction of automatic testing. Nevertheless, we already have some successes, which I want to tell about in this article.
How to write good autotests is a topic for a separate article. And probably not one. I will tell you how we have implemented testing of individual components. Components are written in C ++ and have interfaces very similar to COM. We chose python as the test language and use the very powerful PyTest test framework. In the article I will tell you about the complexity of the C ++ / COM and python bundles, the pitfalls that we stumbled upon and how to solve these problems.
')
Disclaimer
- I can not just copy the code of our project because of the NDA. Therefore, all the examples in the article are written from scratch and never compiled. Therefore, there may be minor inaccuracies, syntax errors or non-compliance with the code design rules. But I tried to convey the main meaning.
- I am not an expert in python. To tell you the truth, I started learning python around the middle of working on a project. Therefore, some statements regarding python may be either not entirely correct, or not fully worked out.
- The Python code in the examples may not correspond to pep8, since often reflects the SOMI prototype and borrows its style.
- This article is not a manual on cython, many things were left overs
Prehistory
In the project I'm working on, we are developing a large and complex module. Several million lines of C ++ code, a dozen large components and a hundred dll under the hood. This module is used in several huge applications.
Historically, this is all tested only through the UI, and mostly manually. It often happens that a change in one area gets out of a bug somewhere in a completely different place. And they find this bug only in a few days, or even weeks. And sometimes something pops up in a few months, when another product decides to integrate a new version of our module.
We have unit tests that chase CI by commits. But the code was over 15 years old when we started talking about TDD. The code is monolithic and it will not work out just to launch it separately. We need a great refactoring for which no resources are given. Therefore, we only have unit tests for simple functions or individual classes.
But since the module has some API, then this module can be tested through this API. You can collect the use cases of all applications that use us and write autotests for this. Then it would be possible to drive these tests directly to CI by commits. So the idea of ​​component testing was born.
But who will write the tests? Testers could come up with good test scenarios and prepare test data, but testers do not know C ++ (and those who know quickly dumped into developers). Programmers could code such tests, but usually fantasy is only enough for a couple of positive scenarios. To cover all negative cases, patience is usually not enough.
We decided to adopt the experience of colleagues from the neighboring team. They made vrappers for their component using cython and put into the python a simplified interface that is used for tests. The threshold for entering the python is much lower than in C ++. Testers can easily master the python in a couple of days and begin to write good autotests.
What does the COM?
Before you begin to describe our torment you need to say a few words about our interfaces. We use the technology slicked with COM and ported to Linux, poppy and fryu. There are some infrastructural differences associated with the lack of a registry, but for the article it doesn’t matter.
COM-like technology gives us a bunch of buns, like a ready-made plug-in component infrastructure. We can easily dock modules written by different teams in different countries (including third party plugins). At the same time, we are not concerned with the compatibility issues of different compilers, runtimes, and standard libraries. The same style of interfaces of all modules, agreements on the transfer of parameters and return values, the lifetime of objects - all this is governed by agreements as with COM.
There is a downside. Inside the modules we can use any buns of modern C ++ standards. But in public interfaces, we must adhere to the rules of SOM, only simple types or interfaces inherited from IUnknown. No stl. No exceptions, only HRESULT. Because of this, the code on the borders of the modules is quite cumbersome and not very readable.
The first experience with cython
To begin with, we identified a dozen interfaces to our module, with which we could implement a small but complete workflow.
But even though these interfaces are part of a public API, in reality they are quite low-level. In order to make a certain operation it is impossible to simply take and call a single function or method. You need to create five objects, connect them with each other, run for execution and wait for the result through the future. All this complexity is needed to organize transactional, asynchronous, Undo / Redo, to organize access to thread-safe insides and even heaps of other things. In short, 2 screens of C ++ code in COM style.
We decided to follow the recommendations of our colleagues and write a small layer that would hide low-level interfaces. In the python, it was proposed to expose several high-level functions.
A cython vrapper just was forwarding calls to c ++:
cdef class MyModuleObject(): cdef CMyModuleObject * thisptr # wrapped C++ object def __init__(self): self.thisptr = new CMyModuleObject() def __dealloc__(self): del self.thisptr def DoSomething1(self): self.thisptr.DoSomething1() def DoSomething2(self): self.thisptr.DoSomething2() def GetResult(self): return self.thisptr.GetResult() The C ++ implementation of the CMyModuleObject class was already involved in useful actions: it created objects of our module and called some useful methods (the same 2 screens of code).
Cython is essentially a translator. Based on the source code above, cython generates a ton of generic code. If we compile it as dll / so (and rename it to pyd), we get a Python module. With ++ implementation CMyModuleObject also needs to be lodged in this dll. Now our Python module can be imported from Python (continuing from the import paths first). You can run using the usual Python interpreter, the main thing is that the architecture coincides. When executing the import string, the python will raise our dll itself, initialize and import everything it needs.
The python script looked something like this:
from my_module import * obj1 = MyModuleObject() obj1.DoSomething1() obj1.DoSomething2() print obj1.GetResult() Cool! Much easier than C ++!
At the first stage, we decided not to bother with the test frameworks, but first to work out the approach on ordinary scripts. In this form, you can already write something, But this approach did not differ in flexibility. If you had to change something in the interlayer, then you had to change the code in C ++.
We wound COM interfaces
I insisted on trying to wrap up low-level interfaces directly, and write a layer, if necessary, on python. The idea was sold to the authorities that we can wring 1 to 1 interfaces and test at the level of our public API.
No sooner said than done. We quickly came to this scheme. The constructor of the Python object creates the SOM object and owns the link to it. Links, of course, are considered a smart copy of CComPtr.
cdef class PyComponent: cdef CComPtr[IComponent] thisptr def __cinit__(self): # Get the COM host cdef CComPtr[IComHost] com_host result = GetCOMHost(IID_IComHost, <IUnknown**>&(com_host)) hresultcheck (result) # Create an instance of the component result = com_host.inArg().CoCreateInstance( CLSID_Component, NULL, IID_IComponent, <void**>self.thisptr.outArg() ) hresultcheck( result ) def SomeMethodWithParam(self, param): result = self.thisptr.inArg().SomeMethodWithParam(param) hresultcheck (result) def GetStringFromComponent(self): cdef char [1024] buf result = self.thisptr.inArg().GetStringFromComponent(buf, sizeof(buf)-1) hresultcheck(result) return string (buf) Usually HRESULT functions are not interesting to us. If the function is successful - well, nice. If you zafeilas, then most likely you do not need to go further. Therefore, we simply check the error code and throw a Python exception. Processing return codes is not made to the level of the client Python code, which makes the client code much more compact and readable.
class HRESULT_EXCEPTION(Exception): def __init__(self, result): super(HRESULT_EXCEPTION, self).__init__("Exception code: " + str(hex(result & 0xffffffff))) cpdef hresultcheck(HRESULT result): if result != S_OK: raise HRESULT_EXCEPTION(result) Note that the hresultcheck function is declared as cpdef. This means that it can be caused by both Python (sometimes hresult is checked in python) and native sish. The second property significantly reduces the error-handling code generated by the siton and speeds up the execution. We have not mastered the call to the SUCCEEDED macro, so we are comparing it with S_OK - for now.
Sometimes we still departed from 1 to 1 vrapping, when it was clear that certain interfaces and their methods should be used only in one particular way and in no other way. For example, if it is assumed that the COM object will be created empty, and then parameters will be crammed into it via the Set * () methods or a call to some Initialize (), in this case, at the python level, we simply made a convenient constructor with parameters.
Or another example. It happens that a request to one object conceptually returns a link to another object (or just a new object). In COM, you have to use output parameters, but in a python you can return an object humanly.
cdef class Class2: cdef CComPtr[IClass2] thisptr cdef class Class1: cdef CComPtr[IClass1] thisptr def GetClass2 (self): class2 = Class2() result = self.thisptr.inArg().GetClass2( class2.thisptr.outArg() ) hresultcheck ( result ) return class2 From the point of view of encapsulation, the code is not very good - one object climbs into the guts of another. But in python with encapsulation (or rather, with privacy) and so not very good. But we have not yet come up with a more beautiful way. There is a risk that someone will try to create Class2 with his hands in the client code; nothing good will probably come out. I would be glad if someone tells you the option of a private constructor in python.
The code examples above are located in files with the pyx extension (by the way, you can do a lot of them, rather than shoving everything in one). It's like cpp in the pros - a file with the implementation. But in a siton you still need a file with ads - pxd - a place where all the names that are considered sishnye will be described.
from libcpp cimport bool from libcpp.vector cimport vector from libcpp.string cimport string from libc.stdlib cimport malloc, free cdef extern from "mytypes.h": ctypedef unsigned short int myUInt16 ctypedef unsigned long int myUInt32 ctypedef myUInt32 HRESULT ctypedef struct GUID: pass ctypedef enum myBool: kMyFalse kMyTrue kMyBool_Max cdef extern from "hresult.h": cdef HRESULT S_OK cdef HRESULT S_FALSE cdef extern from "Iunknown.h": cdef cppclass IUnknown: HRESULT QueryInterface (const IID & iid, void ** ppOut) HRESULT AddRef () HRESULT Release () cdef extern from "CComPtr.h": cdef cppclass CComPtr [T]: # this is trick, to assign pointer into wrapper T& assign "operator="(T*) T* inArg() T** outArg() cdef extern from "comhost.h": cdef extern IID IID_IComHost cdef cppclass IComHost(IUnknown): HRESULT CoCreateInstance ( const GUID& classid, IUnknown* pUnkOuter, const IID& iid, void** x ) Pay attention to CComPtr :: operator = (). If in the Siton code you try to assign CComPtr directly, nothing happens. He simply can not really parse this syntax. I had to resort to the trick of renaming characters. So assign is how a character will look in a siton, and in quotes it is set what exactly needs to be called in a sish code.
The trick is useful if you need to call the Python class or function in the same way as a sishin.
pxd:
cdef extern from "MyFunc.h": int CMyFunc "MyFunc" () pyx:
def MyFunc(): CMyFunc() Returning to our project. Python code is simpler and more compact, but still too low-level for most users. Therefore, we still decided to leave the interlayer, rewriting it in python. As a result, those 2 pages of cumbersome COM code have turned into this.
def do_operation(param1, param2): operation = DoSomethingOperation(param1, param2) engine = TransactionEngine() future = engine.Submit(operation) future.Wait() return future.GetResult() So our code has become much more compact and clearer, it was possible to use both high-level interfaces like do_operation (), and, if necessary, go down to "sishnyh" interfaces.
There was a feeling of flexibility, it was not necessary to recompile the C ++ part each time. Moreover, to start, we needed to build up only 10 interfaces, and for each subsequent feature, it was necessary to do some 1-2, which really added strength and faith to the chosen approach.
Problems begin
In this form, the technology may already be suitable for most projects, but we have come up against several fundamental limitations.
So, our COM host (the object that provides the COM infrastructure, everyones CoCreateInstance there, etc.) is a normal plus object. So, someone must create it (analogous to CoInitialize) and then delete it (CoFinalize). But here's the problem, the Python module has no main (). In any case, in the form as we needed.
Therefore, we created a plus Application object and put the initialization / finalization of the host into this object. We wrote a wrapper who allowed us to create this object from a python (at the beginning of each script).
But very quickly we began to catch kreshi at the exit. It turned out that, unlike C ++ (the first created object will be removed last), the order of destruction of objects in python is not defined. Well, in any case, there is no way to influence it. Depending on the phase of the moon, the python nailed the Application object first, it put out the COM infrastructure and forcibly unloaded all the components. Then the python deleted some other object that had a reference to some COM object. An attempt to call Release () from a dll that has already been unloaded resulted in a crash.
The second problem is the location of the executable file. It turned out that in our large component there are a lot of places that are trying to open files with data along a certain path relative to the executable file. This is normal if the final application is installed in the system in some known way. This is also normal if we work with the application compiled in the working directory. But this method stops working if the executable file suddenly becomes a python in the system directory.
There was even a function that allows you to override the application directory. It worked in most cases. But, unfortunately, there were cyclists who ignored this redefinition and continued to calculate the path independently relative to the executable. It would be correct to take it and fix it, but it would require considerable labor costs. We decided that this is still lying in backlog.
Finally, the third problem is the event loop. The fact is that our module is very complex and interactive. This is not just a “call function - get result” library. This is a huge combine. Inside, there are a couple of hundreds of threads that exchange messages. Some parts of the code were written in the time of the Mesozoic and are intended for execution only in the main thread (otherwise it will not work). In other places, a hardcode is sent to the main thread, expecting there to know how to process this message. And we also have our own subsystem of threads and messages, which also implies that the message processing cycle will necessarily spin in the main thread and conduct all of this. Without it in any way.
The easiest solution at the start was to insert the run_event_loop () method into our Application class, which twisted the message loop. The process stopped when our useful work was completed (as I understand it now, it was by coincidence :))
In general, scripts of the following type worked normally for us: we start some work using a non-blocking function (which does not wait for the end), after which we head off to the event loop
app = Application() start_some_processing_async() app.run_event_loop() But with the scenarios that required some interactivity, there was a problem. We could not, for example, start the work, and then after a couple of seconds, try to stop it. In fact, the work did not start until the message loop in the main thread started. And if the cycle has started, then we will not return to the python.
Of course, it would be possible to fence something asynchronous at the python level, but this is clearly not what we would like. After all, the approach was supposed to push people who are not tempted by asynchronous systems. They would like to write just like that and not to bathe in some event loops
start_some_processing_async() time.sleep(3) cancel_processing() Without thinking, we tried to start processing in another thread, and in the main thing, to twist the message loop. But we immediately came up against the following problem - GIL (Global Interpreter Lock) . It turned out that Python threads are not actually running in parallel. Only one stream is running at a time, and the streams are switched every 100 commands. All this regulates this very GIL, stopping all threads except one.
It turned out that if the main thread went to the app.run_event_loop () function and did not return (it should hang there by design), then other Python commands in other threads are not executed. It’s just that there were not 100 more teams in the main thread, and the interpreter decided that it was too early to switch.
The solution was found in the nogil seedon keyword. A piece of code tagged nogil first releases the GIL, then performs a sish call. At the end of the GIL is captured again. So the main thread released GIL and went into the message loop. The second thread got control and did everything it needed there.
def Func(self): result = 0 cdef IComponent * component = self.thisptr.inArg() with nogil: result = component.Func() hresultcheck(result) Cython, by the way, is a very capricious thing. It does not always allow to interfere with the Python and C code in one line. It also does not allow you to call some Python constructions and create new variables in the nogil sections (logically, this requires access to the python intestines, which are protected by GIL). It is necessary to be perverted like this, in order to correctly declare variables and make the necessary calls.
Everything seems to start working, but very unstable. We constantly caught some kind of crashes and hangs, and also constantly ran across a non-working functional (the file did not open in a relative path, but no one threw the error).
Design opposite
For several weeks we tried to defeat these 3 problems, tried different approaches. But every time there was another insoluble problem. Most of all delivered GIL, but we didn’t even imagine how to defeat the unloading of a COM host.
We even thought about jumping onto lua, but some of the restrictions would still remain. Only in this case would still have to go all the way from scratch.
But here came a bold idea. And what if we run not a code from a python, but rather a python from a code? Let's write your application that will do the following:
- Initialize COM host
- Start the second thread
- In the main thread, run the event loop
- In the second thread, run the python interpreter (it has an appropriate API for embedding in other applications)
This approach solves all 3 problems in one fell swoop:
- We control the life of the COM host and can guarantee its destruction after the Python thread has finished its work.
- Our test application will live next to the main product, which means that all relative paths will work.
- Finally, no problems with GIL. Python executes a single-threaded script, which means you don’t need to share resources with anyone.
And you know? This approach worked! There were, however, a few minor problems that were eventually solved.
- some calls must still be made in the main thread. Well, nothing, we started the message in the main thread, with a request to perform what is needed.
- I had to tinker with the installation of PYTHON_HOME and PYTHON_PATH. The nontrivial moment was that neither the Python function Py_SetPythonHome (), nor the standard setenv () copy the passed string to itself, but simply remember the pointer. In our case, it was a pointer to a temporary variable.
- in order not to depend on the version of the python, they decided to carry all the entrails with them. Including the standard library (which, as it turned out, is perfectly readable directly from the zip) and several additional libraries
Another problem that had to be tricky was the sys.exit () function. We needed it to catch the return code from unittest and pass it to the output, and then process it with CI.
It works like this. If someone in the script calls sys.exit (), a SystemExit exception is actually generated. This exception is caught by the python itself and, like any other exception globally, must be printed to the console along with the stack trace. But the Py_PrintEx function knows that there is such a special case, and if we are offered to print a SystemExit exception, then we need to call the exit exit ()
Yes, yes, like this! A function called Print makes an exit () call. And this exit honestly works out - just takes and cuts down the entire application. And he didn’t want to spit that the application has unallocated handles, unfinished threads, unclosed files, unfinalized modules, a million active threads and all that jazz.
But the python (in any case, 2.7.6. Old time, I know) does not allow it to turn at the API level. I had to just copy several functions from the python sources (starting with PyRun_SimpleFileExFlags () and a few private ones that it calls) to my project and finish them by myself. So, our version in the case of SystemExit correctly exits and returns a return code. So after the completion of the Python part, the test application can properly clean and extinguish itself.
At first we had 2 projectors - one build test application with built-in python, and the second, as before, loadable module for python. But later we combined it all into one projector. The test application initialized the python, after which it called the initialization function of our python module (generated by a siton). So Python already at the start already knew about our module (although I still had to make imports).
Callbacks
In this form, the test application showed itself very well. We screwed the test framework (standard unittest) and testers began to write tests little by little. We ourselves, meanwhile, continued to drag the interfaces.
Screwing another piece of functionality, we stumbled upon the fact that in some cases we need to be able to accept callbacks. Those. Python synchronously calls a function, and in its interior it calls a callback to a python.
Positive interface looks like this:
class ICallback : public IUnknown { virtual HRESULT CallbackFunc() = 0; }; Class IComponent : public IUnknown { virtual HRESULT MethodWithCallback(ICallback * cb) = 0; }; The Python class cannot be inherited from the plus interface under any sauce. Therefore, in the project part of the project we had to do our own implementation, which threw calls to the python.
.h
//Forward declaration struct _object; typedef struct _object PyObject; class CCallback : public ICallback { //COM stuff ... CCallback * Create(); // ICallback virtual HRESULT CallbackFunc(); public: void SetPythonCallbackObject(PyObject * callback_handler); private: PyObject * m_pPythonCallbackObject; }; .cpp
const char PythonMethodName[] = "PythonCallbackMethod"; void CCallback::SetPythonCallbackObject(PyObject * callback_handler) { // Do not addref to avoid cyclic dependency m_pPythonCallbackObject = callback_handler; } HRESULT CCallback::CallbackFunc() { if(!m_pPythonCallbackObject) return S_OK; // Acquire GIL PyGILState_STATE gstate = PyGILState_Ensure(); if ( gstate == PyGILState_UNLOCKED ) { // Call the python method char * methodName = const_cast<char *>(PythonMethodName); //Py_Api doesn't work with constant char * PyObject * ret = PyObject_CallMethod(m_pPythonCallbackObject, methodName, NULL); if (!ret) { if (PyErr_Occurred()) { PyErr_Print(); } std::cout<<"cannot call"<<PythonMethodName<<std::endl; } else Py_DecRef(ret); } // Release the GIL PyGILState_Release(gstate); return S_OK; } A special moment in this code is the capture of GIL. Otherwise, at best, the python will break on checking that the GIL is captured, but most likely it will either hang or cross.
We have a console application, so the error output to the console is the same. Even if the Python code throws an exception, our handler will catch it and print out the traceback.
From the side of the siton it looks like this:
cdef class PyCallback (object): cdef CComPtr[ICallback] callback def __cinit__(self): self.callback.assign( CCallback.Create() ) self.callback.inArg().SetPythonCallbackObject(<PyObject *> self) def PythonCallbackMethod(self): print "PythonCallbackMethod called" cdef class Component: cdef CComPtr[IComponent] thisptr def __cinit__(self): // Create IComponent instance ... def CallMethodWithCallback(self, PyCallback callback): cdef IComponent * component = self.thisptr.inArg() cdef ICallback * cb = callback.callback.inArg() hresult = 0 with nogil: hresult = component.MethodWithCallback(cb) hresultcheck(hresult) When calling the MethodWithCallback () method, you must release the GIL, otherwise the callback will not be able to capture it.
With the client Python code, everything should be simple and clear.
component = Component() callback = PyCallback() component.CallMethodWithCallback(callback) , . , API, cython.
, . , , . , GIL ( , ), . PyGILState_Ensure() , , , .
, , , . , , , . .
. , , . PyRun_FileExFlags(), , PyArena_Free()
PyThreadState *_save = PyEval_SaveThread(); while (GetCurrentlyActiveCallbacks() > 0) ; // semicolon here is correct PyEval_RestoreThread(_save); Conclusion
The python-> cython-> C ++ sandwich has been very successful as a framework for the AutoTest API. The threshold for entering the python is very small compared to other programming languages. Any competent tester in a couple of days will master the python at a level sufficient for writing autotests. The main thing in this matter is to figure out how to test this or that functional, and it is a matter of technology to express it in code.
We managed to build a user-friendly layer that exposes a simple and intuitive interface to the python. Calls of the low-level C ++ / COM code are hidden behind the wrappers and understanding of this code is not necessary for most tasks. But if necessary, it is easy to go down there.
Recently we screwed PyTest as a framework for writing tests. Very taxis! Tests have become even easier, clearer and faster.
Now there are no serious architectural flaws in sight, but there are still some bugs. For example, now there are a couple of cyclic dependencies, which is why several key objects do not want to collapse. But where to break the cyclical dependence correctly, we have not yet invented.
As for the siton himself. The developers of cython go to contact and have already fixed a couple of bugs for us. They release releases regularly.
Cython is quite capricious. It does not always intelligibly explain what exactly is not true in the source file, and sometimes also indicates the wrong place. Some things we have not mastered.
We also screwed the interactive python mode. Sometimes it is more convenient to debug Python code directly in the python console than to edit the script and run it again and again. It turned out all simple - just call it:
PyRun_InteractiveLoop(stdin, "<stdin>"); Developers of the C ++ module actively launch tests directly from the IDE. You can simply take a test that fell on the CI and otdebezhit. All breakpoints in C ++ code work as needed. But we have not yet invented how to debug the Python part. However, there was no special need - with PyTest, the tests are very simple and there is practically nothing to debug there.
Now it is the realization that the technology has taken place. Extend functionality easily and conveniently. People liked to write autotests, although for now they are treated with suspicion.
I hope the article will be useful, and you too can organize something like this.
Source: https://habr.com/ru/post/273401/
All Articles