Using SikuliX in the background on Cloud9
SikuliX (old name Sikuli) is a tool that allows you to automate what you see on the screen using visual recognition (pictures). On Habré already have enough information on it , but all the examples involve the use of a real screen.
Here I will describe my experience of using SikuliX in the background on Cloud IDE Cloud9 . In the process of working with this tool I got some bumps, so maybe my experience will be useful to someone.
There are quite a few options for using SikuliX, here are some of the actions you can automate:
In my case, it was necessary to automate the routine work with the flash site, log in, select a certain item in the drop-down list and press a few buttons. It took about 1-3 minutes several times a day. After activating SikuliX, this can be done through a simple launch of the script, and spending free time on more useful things (for example, learning English).
')
SikuliX depends on tools such as OpenCV (computer vision) and Tesseract (text recognition). Therefore, we will install them. Let's start with OpenCV.
Installing OpenCV was not very simple, so I used this script, which installs and knocks down everything you need. Clone it, give the right to perform and run.
I must say that the installation of OpenCV “eats away” almost all the resources that cloud9 provides in free mode. With a paid plan it would be easier, but what was there was enough for me. The build process was 2.15 hours, at this time you can go about your business, problems should not arise. After installation, the folder 3d1fce8dae65e765f7c4 can be deleted (it will free up almost all the memory).
Installing Tesseract is easy:
To install SikuliX itself, we still need a real screen (Java restriction). The solution was the launch of X11. To put the necessary components, we can use this script. The structure includes supervisor, xvfb, fluxbox, x11vnc. To start, we can transfer the run.sh and supervisord.conf files to the root, or correct the path to supervisord.conf in the run.sh file
If everything went well, after the launch of run.sh, the terminal will display a link. Go over it in the browser. Something like your-workspace / vnc.html . There, click “Connect” and get into the fluxbox window manager.
To install, we will need SikuliX 1.1.1 (did not try the version above), since in version 1.1.0 there was a bug that did not allow running sikuli in the background. We need to download sikulixsetup, take it here .
In fluxbox, open the terminal and launch the downloaded file for installation. I had this:
A window should open with options for installing SikuliX.
I chose the first option that installs sikulix.jar, which includes the SikuliX IDE. Scripts written in it can then be run in the terminal without having to open the IDE. As a sub, I chose Python. You can choose another installation option based on your needs.
At the Jython installation stage, there may be problems with installing Jython 2.7.0 (I had them, the script just didn’t run). Therefore it is better to click the “NO” button and install Jython 2.5.4rc1.
If everything went well, then you should see this pop-up window:
Now we can run IDE. In it, we will write and debug our scripts to run them further in the background.
For example, let's write a simple script that checks whether the habr is in the top of Google issue on the request “how to feed the cats”. We include user logs and write our actions to a file.
The script was written, debugged.
In the process of debugging, some errors occurred in the terminal and the log, such as:
Errors are not critical and we can simply ignore them.
After visual debugging, we can close the fluxbox and go to the IDE terminal Cloud9.
Xvfb (X virtual framebuffer) will help us run the script in the background. This is a virtual X server that performs all graphical operations in memory. You can read here . Let's write the parameters of the virtual screen, the path to the saved project, the name of the log file and try to start it.
If everything went smoothly, then in our log file you can read “Success. Habrahabr is in Google's first page. ” Profit
Then you can write a small API and run our scripts / autotests, for example, from the console of the working computer.
The official site of the project.
Here I will describe my experience of using SikuliX in the background on Cloud IDE Cloud9 . In the process of working with this tool I got some bumps, so maybe my experience will be useful to someone.
There are quite a few options for using SikuliX, here are some of the actions you can automate:
- Work with flash sites (and any other)
- Automate browser games (harvest at the farm)
- Work with desktop applications
- Computer settings
- Various bots
- Any kind of routine work with sites / documents
In my case, it was necessary to automate the routine work with the flash site, log in, select a certain item in the drop-down list and press a few buttons. It took about 1-3 minutes several times a day. After activating SikuliX, this can be done through a simple launch of the script, and spending free time on more useful things (for example, learning English).
')
Installing dependencies
SikuliX depends on tools such as OpenCV (computer vision) and Tesseract (text recognition). Therefore, we will install them. Let's start with OpenCV.
Installing OpenCV was not very simple, so I used this script, which installs and knocks down everything you need. Clone it, give the right to perform and run.
git clone git@gist.github.com:3d1fce8dae65e765f7c4.git cd 3d1fce8dae65e765f7c4 chmod +x install-opencv-2.4.11-in-ubuntu.sh ./install-opencv-2.4.11-in-ubuntu.sh I must say that the installation of OpenCV “eats away” almost all the resources that cloud9 provides in free mode. With a paid plan it would be easier, but what was there was enough for me. The build process was 2.15 hours, at this time you can go about your business, problems should not arise. After installation, the folder 3d1fce8dae65e765f7c4 can be deleted (it will free up almost all the memory).
Installing Tesseract is easy:
sudo apt-get install tesseract-ocr To install SikuliX itself, we still need a real screen (Java restriction). The solution was the launch of X11. To put the necessary components, we can use this script. The structure includes supervisor, xvfb, fluxbox, x11vnc. To start, we can transfer the run.sh and supervisord.conf files to the root, or correct the path to supervisord.conf in the run.sh file
git clone git@github.com:fjakobs/cloud9-vnc.git cd cloud9-vnc/ ./install.sh ./run.sh If everything went well, after the launch of run.sh, the terminal will display a link. Go over it in the browser. Something like your-workspace / vnc.html . There, click “Connect” and get into the fluxbox window manager.
SikuliX installation
To install, we will need SikuliX 1.1.1 (did not try the version above), since in version 1.1.0 there was a bug that did not allow running sikuli in the background. We need to download sikulixsetup, take it here .
In fluxbox, open the terminal and launch the downloaded file for installation. I had this:
java -jar sikulixsetup-1.1.1-20151126.001204-18-forsetup.jar A window should open with options for installing SikuliX.
I chose the first option that installs sikulix.jar, which includes the SikuliX IDE. Scripts written in it can then be run in the terminal without having to open the IDE. As a sub, I chose Python. You can choose another installation option based on your needs.
At the Jython installation stage, there may be problems with installing Jython 2.7.0 (I had them, the script just didn’t run). Therefore it is better to click the “NO” button and install Jython 2.5.4rc1.
If everything went well, then you should see this pop-up window:
Sikuli IDE launch
Now we can run IDE. In it, we will write and debug our scripts to run them further in the background.
java -jar sikulix.jar Example
For example, let's write a simple script that checks whether the habr is in the top of Google issue on the request “how to feed the cats”. We include user logs and write our actions to a file.
The script was written, debugged.
In the process of debugging, some errors occurred in the terminal and the log, such as:
[error] command 'wmctrl' is not executable
command 'xdotool' is not executable
Failed to initialize libdc1394
Errors are not critical and we can simply ignore them.
After visual debugging, we can close the fluxbox and go to the IDE terminal Cloud9.
Xvfb (X virtual framebuffer) will help us run the script in the background. This is a virtual X server that performs all graphical operations in memory. You can read here . Let's write the parameters of the virtual screen, the path to the saved project, the name of the log file and try to start it.
xvfb-run --server-num=10 --server-args="-screen 0 1280x864x16" ./runsikulix -r project.sikuli/ -f SikuliLog.txt If everything went smoothly, then in our log file you can read “Success. Habrahabr is in Google's first page. ” Profit
What's next:
Then you can write a small API and run our scripts / autotests, for example, from the console of the working computer.
Sources:
The official site of the project.
Source: https://habr.com/ru/post/275717/
All Articles