Bot-teller at pywinauto, or GUI automation for payment gateway
I work in a small Israeli startup, our product is a platform for ordering food from restaurants, cafes and shops. Unlike dozens of similar services, we are monopolists in the US student market. We process about hundreds of thousands of orders per day at peak and one of the production gateways is built on GUI automation for a Win32 application using the pywinauto library.
First, why do we work only in universities? Many ideas often look good only in perspective. For example, the same idea of ​​food delivery. It seems to be great to write an application so that you can order from any restaurant, not to bathe with payment, etc. Everyone who tries to implement this idea stumbles upon the so-called Chicken or the egg problem. While there are few restaurants in the annex, it is meaningless; in the meantime, there are few users; restaurants do not rush to integrate the new application into their workflow. Usually at this moment everything dies. This situation is not unique to the food ordering application.
The solution to this problem is usually a kind of hybrid model, when, in addition to the future benefits of socialization, there is a benefit for each individual user. For example, Waze. Now this is an application that knows about all traffic jams, accidents and police on the road, but they started as a free GPS application, which was useful for everyone. There are other solutions, about one of them I will tell on our example.
Resting on this problem, the same fate awaited us. We rather unsuccessfully tried to break through to the San Francisco market, which is full of such decisions, and before we die quietly, we decided to try another idea. It consisted in not trying to negotiate with all the restaurants, but only with a certain platform that has leverage on local businesses.
')
The choice fell on universities. Offering a percentage of income and service for students, we were an excellent offer for the university, problems of local restaurants related to the integration of the application into their business process, they were not interested, for them we were a direct and indirect benefit. They required a business on campus to work with our application. We integrated into all the restaurants on the campus - for students, our application became very convenient: you can order lunch during lectures and not wait for your order at recess. We instantly capture most of the campus, and the viral effect is very large. When a person lonely waits in a queue, and people come alongside, take their ready order and immediately leave, most often our application is downloaded right on the spot.
At some point we had a problem. The fact is that in America it is very difficult to get a credit card, and most students do not have it. On the other hand, student IDs are issued in the form of plastic cards with a magnetic stripe and are a payment tool inside the campus. The university charges them with some sort of scholarship. These funds are more structured: there are separate parts that can be spent freely, but there are funds only for food. And these cards are only accepted on campus.
We, as a company that works with students, of course wanted to serve these cards. But not everything is so simple. The company that produces these cards is not a bank. The student card service market is almost monopolized and divided between three companies into hundreds of universities across America.
One of these companies tried to launch a service similar to ours, and, losing the competition, “offended” in the best traditions of sandy grounds and refused to conduct any cooperation with us. The fact is that under an agreement with a university, a business that has the right to accept payment via student cards, the issuing company must provide a terminal with a desktop application where you can enter the student card number and withdraw money. It is very convenient for all kiosks and sausages. But it is not necessary to provide the API for mass service under the contract, and they did not want to open it. I had to work with what is. As a result, we found a rather unusual solution for a payment gateway, which in the first few months of its existence is quite successfully operating in 6 universities: tens of thousands of transactions were carried out for hundreds of thousands of dollars, and in the next academic year this solution will be scaled to several dozen American universities with a turnover funds in tens of millions of dollars.
About our decision. I must say that it was very important for us to find a solution that is absolutely legal. For example, Reverse Enginering is illegal; it can also be problematic to sniff traffic and simulate the frontend of this POS terminal, but it is absolutely legal to automate clicks on the desktop application. On campus, we are allocated a Windows virtual machine with a desktop cash register installed. The Flask + Tornado bundle is also installed there. When a user makes an order from a phone, a request comes to the virtual machine with all the necessary parameters: amount, student card number, etc. Next, using pywinauto, we enter the amount, card number, all the necessary parameters (there is quite a complicated logic in terms of discounts, free lunches in a certain part of the day, etc.). We carry out a transaction, check the result and return the response to the server. It took about 20 seconds to process one transaction, but in the end it was reduced to 3.
When profiling, some features of the library were found.
Calls to Application (). Connect () for connecting to an application have many different parameters and, for example, identifying an application or a window by class_name works 20 times faster than it does by title_re (a regular window title expression).
The second unexpected point was that if the window had not yet opened, the call to Connect () takes a lot of time (up to a few seconds) before throwing an exception. It should not be called without being sure that the window is already open, it is better to find a heuristic model that will allow you to understand that the call will be successful. In my case, when opening a new window, the form of the main application stopped being 'active', this could be caught by calling form.WaitNot ('active'), after returning which you can safely call Connect ().
Another of the problems we encountered: automation via the Win32 API does not work without an open user session on the machine. This is a common problem for all such tools. For example, the ClickInput and TypeKeys methods do not work in the pywinauto library if the session is locked or the RDP connection is minimized. This was my first acquaintance with automating the GUI and Win32 API, so the solution to the problem could be very long. Fortunately, the library has a wonderful maintainer vasily-v-ryabov . I express my deep gratitude to him for the detailed consultations.
We found a solution to all problems, mainly many issues were solved with alternative challenges. For example, when a non-standard tab switching widget was encountered in an application that did not respond to any standard call, a solution was found by analyzing Windows Messages in Spy ++ and sending the same messages using the PostMessage method in pywinauto .
The library is actively developing and, as practice has shown, is stable enough even to use it in production for the implementation of a payment gateway. Almost all the problems associated with reduced functionality in the absence of the session will be solved in the next release (0.6.0), which is preparing to be released this year (the UIA branch on the github). For example, the send_chars method will allow you to successfully transfer almost any combination of characters to an inactive or even a minimized window. And most importantly, in the UIA branch there is support for Microsoft UI Automation technology (for WinForms, WPF, StoreApps, Qt and browsers), but this is a topic for a separate article.
If it seemed to you that the implementation of a payment gateway by automating the GUI is not the top of surrealism, then I can add that one of the universities in Texas has very strict restrictions on access to the internal network, so the deployment, debug and upgrade module that processes payments of tens of thousands People go on Skype - "Now write git fetch" - and stuff like that. By the way uptime we have> 99.6%. Thank you pywinauto ).
First, why do we work only in universities? Many ideas often look good only in perspective. For example, the same idea of ​​food delivery. It seems to be great to write an application so that you can order from any restaurant, not to bathe with payment, etc. Everyone who tries to implement this idea stumbles upon the so-called Chicken or the egg problem. While there are few restaurants in the annex, it is meaningless; in the meantime, there are few users; restaurants do not rush to integrate the new application into their workflow. Usually at this moment everything dies. This situation is not unique to the food ordering application.
The solution to this problem is usually a kind of hybrid model, when, in addition to the future benefits of socialization, there is a benefit for each individual user. For example, Waze. Now this is an application that knows about all traffic jams, accidents and police on the road, but they started as a free GPS application, which was useful for everyone. There are other solutions, about one of them I will tell on our example.
Resting on this problem, the same fate awaited us. We rather unsuccessfully tried to break through to the San Francisco market, which is full of such decisions, and before we die quietly, we decided to try another idea. It consisted in not trying to negotiate with all the restaurants, but only with a certain platform that has leverage on local businesses.
')
The choice fell on universities. Offering a percentage of income and service for students, we were an excellent offer for the university, problems of local restaurants related to the integration of the application into their business process, they were not interested, for them we were a direct and indirect benefit. They required a business on campus to work with our application. We integrated into all the restaurants on the campus - for students, our application became very convenient: you can order lunch during lectures and not wait for your order at recess. We instantly capture most of the campus, and the viral effect is very large. When a person lonely waits in a queue, and people come alongside, take their ready order and immediately leave, most often our application is downloaded right on the spot.
At some point we had a problem. The fact is that in America it is very difficult to get a credit card, and most students do not have it. On the other hand, student IDs are issued in the form of plastic cards with a magnetic stripe and are a payment tool inside the campus. The university charges them with some sort of scholarship. These funds are more structured: there are separate parts that can be spent freely, but there are funds only for food. And these cards are only accepted on campus.
We, as a company that works with students, of course wanted to serve these cards. But not everything is so simple. The company that produces these cards is not a bank. The student card service market is almost monopolized and divided between three companies into hundreds of universities across America.
One of these companies tried to launch a service similar to ours, and, losing the competition, “offended” in the best traditions of sandy grounds and refused to conduct any cooperation with us. The fact is that under an agreement with a university, a business that has the right to accept payment via student cards, the issuing company must provide a terminal with a desktop application where you can enter the student card number and withdraw money. It is very convenient for all kiosks and sausages. But it is not necessary to provide the API for mass service under the contract, and they did not want to open it. I had to work with what is. As a result, we found a rather unusual solution for a payment gateway, which in the first few months of its existence is quite successfully operating in 6 universities: tens of thousands of transactions were carried out for hundreds of thousands of dollars, and in the next academic year this solution will be scaled to several dozen American universities with a turnover funds in tens of millions of dollars.
How it works?
About our decision. I must say that it was very important for us to find a solution that is absolutely legal. For example, Reverse Enginering is illegal; it can also be problematic to sniff traffic and simulate the frontend of this POS terminal, but it is absolutely legal to automate clicks on the desktop application. On campus, we are allocated a Windows virtual machine with a desktop cash register installed. The Flask + Tornado bundle is also installed there. When a user makes an order from a phone, a request comes to the virtual machine with all the necessary parameters: amount, student card number, etc. Next, using pywinauto, we enter the amount, card number, all the necessary parameters (there is quite a complicated logic in terms of discounts, free lunches in a certain part of the day, etc.). We carry out a transaction, check the result and return the response to the server. It took about 20 seconds to process one transaction, but in the end it was reduced to 3.
When profiling, some features of the library were found.
Calls to Application (). Connect () for connecting to an application have many different parameters and, for example, identifying an application or a window by class_name works 20 times faster than it does by title_re (a regular window title expression).
The second unexpected point was that if the window had not yet opened, the call to Connect () takes a lot of time (up to a few seconds) before throwing an exception. It should not be called without being sure that the window is already open, it is better to find a heuristic model that will allow you to understand that the call will be successful. In my case, when opening a new window, the form of the main application stopped being 'active', this could be caught by calling form.WaitNot ('active'), after returning which you can safely call Connect ().
Another of the problems we encountered: automation via the Win32 API does not work without an open user session on the machine. This is a common problem for all such tools. For example, the ClickInput and TypeKeys methods do not work in the pywinauto library if the session is locked or the RDP connection is minimized. This was my first acquaintance with automating the GUI and Win32 API, so the solution to the problem could be very long. Fortunately, the library has a wonderful maintainer vasily-v-ryabov . I express my deep gratitude to him for the detailed consultations.
We found a solution to all problems, mainly many issues were solved with alternative challenges. For example, when a non-standard tab switching widget was encountered in an application that did not respond to any standard call, a solution was found by analyzing Windows Messages in Spy ++ and sending the same messages using the PostMessage method in pywinauto .
The library is actively developing and, as practice has shown, is stable enough even to use it in production for the implementation of a payment gateway. Almost all the problems associated with reduced functionality in the absence of the session will be solved in the next release (0.6.0), which is preparing to be released this year (the UIA branch on the github). For example, the send_chars method will allow you to successfully transfer almost any combination of characters to an inactive or even a minimized window. And most importantly, in the UIA branch there is support for Microsoft UI Automation technology (for WinForms, WPF, StoreApps, Qt and browsers), but this is a topic for a separate article.
If it seemed to you that the implementation of a payment gateway by automating the GUI is not the top of surrealism, then I can add that one of the universities in Texas has very strict restrictions on access to the internal network, so the deployment, debug and upgrade module that processes payments of tens of thousands People go on Skype - "Now write git fetch" - and stuff like that. By the way uptime we have> 99.6%. Thank you pywinauto ).
Source: https://habr.com/ru/post/307420/
All Articles