Nirvana for Testers - Nerrvana
More than four years ago, we attended to the creation of a convenient system for the automatic execution of Selenium tests of our products on different browsers. No one had heard of Sauce Labs , and everyone was inventing something of their own. We also invented - we made our own system, which is able to independently run Selenium tests on selected browsers and OS on a schedule. Well, and stores the results of implementation, of course.
At some point we decided that the system could be useful not only for us, and decided to make the service accessible to all.
Not so long ago, this service - Nerrvana - was opened to all comers.
')
For each test run, the right amount of Selenium RC / node is created in the cloud, which are combined into Selenium Grids. In some ways, the service is similar to Sauce Labs, but it has a key difference: it not only provides Selenium nodes, but also performs custom tests on them. At the same time, in order to run tests on different browsers, you don’t need to change anything in the test code - all you have to do is tick the boxes in the web interface.
Working with the service looks like this: you have the Selenium tests that you upload via FTP to Nerrvana. Through the web-interface (or through the API) you choose (tick) which browsers the tests should run on, and choose whether this is a one-time run or whether the tests need to be run on a schedule.
If your tests can work in several streams - you can specify how many Selenium nodes you want to use for each of the browsers.
This is where your work ends, then Nerrvana does everything.
It will run your tests on selected browsers on a predetermined schedule, saving the results of each execution so that you can view them later. Tests will work simultaneously for selected browsers (but, of course, independently of each other).
If you wish, you can easily let your tests report errors to Nerrvana — then Nerrvana can notify you, for example, by mail.
You can also use the Jenkins plugin , which can update the tests and run them in Nerrvana for each new build. You can also add to the Selenium IDE Nerrvana formatter , which will create tests that are optimized for working with Nerrvana ( more ).
How it all happens, you can see for yourself if you have five minutes of free time.
(on a click - pictures in full resolution)
You can read more about the reports themselves here ( example , example ), and if you prefer, you can use it for your tests.
In the end, you can look at the code of the test itself.
Here are some screenshots made tests for different browsers: FF , Chrome , Safari , IE10 .
If you want to play again, you can change the test code via FTP or put another test. In the tests you can use both Selenium1 and WD - in the folder simple_test_java you will find examples.
In fact, we do not quite know where to go next, and how to use our differences from Sauce Labs. For our purposes, Nerrvana is enough for the eyes, but perhaps you will be able to say how the service could be useful to you, should it be developed and how? Some of our thoughts on this:
In general, you can do a lot, but for now it is not entirely clear what will really be interesting and in demand.
I apologize in advance if habraeffect will happen and it will cause some problems with the service. Besides us , there are almost no real users now, and therefore we use only one physical server. If necessary, we will add servers in the future.
We will be happy if Nerrvana is helpful to one of you.
And I will be glad to hear comments and answer questions. Perhaps some answers are already here .
______
* Selenium1 in the test used to at the same time show our reporting framework, we have not yet had time to rewrite it under WD. Found that my test for Selenium1 somehow does not work well in Opera. On WD in Opera, the same works ok - if you try, keep in mind. And in general, according to my observations, WD is already running more stable than the first Selenium. An example of a WD test is in simple_test_java.
At some point we decided that the system could be useful not only for us, and decided to make the service accessible to all.
Not so long ago, this service - Nerrvana - was opened to all comers.
')
What can do the service
For each test run, the right amount of Selenium RC / node is created in the cloud, which are combined into Selenium Grids. In some ways, the service is similar to Sauce Labs, but it has a key difference: it not only provides Selenium nodes, but also performs custom tests on them. At the same time, in order to run tests on different browsers, you don’t need to change anything in the test code - all you have to do is tick the boxes in the web interface.
Working with the service looks like this: you have the Selenium tests that you upload via FTP to Nerrvana. Through the web-interface (or through the API) you choose (tick) which browsers the tests should run on, and choose whether this is a one-time run or whether the tests need to be run on a schedule.
If your tests can work in several streams - you can specify how many Selenium nodes you want to use for each of the browsers.
This is where your work ends, then Nerrvana does everything.
It will run your tests on selected browsers on a predetermined schedule, saving the results of each execution so that you can view them later. Tests will work simultaneously for selected browsers (but, of course, independently of each other).
If you wish, you can easily let your tests report errors to Nerrvana — then Nerrvana can notify you, for example, by mail.
You can also use the Jenkins plugin , which can update the tests and run them in Nerrvana for each new build. You can also add to the Selenium IDE Nerrvana formatter , which will create tests that are optimized for working with Nerrvana ( more ).
How it works in practice
How it all happens, you can see for yourself if you have five minutes of free time.
(on a click - pictures in full resolution)
- You get to Nerrvana (email is needed, account details will come to it).
- Click Add new test runScreenshot
- Fill in the “Test run name” field, tick the required platform (temporarily you can select only two platforms at a time, or two Node on a platform - a measure against habraeffekt)
- Choose habratest / rebuildAndRun.sh as the executable file. I wrote an elementary test for Selenium1 *, which opens Habr and does something on it, and then shows the results in Nerrana.
- Choose "Option 1", "Run" (in option 2, i.e., scheduled launch, the ability to set the periodicity is disabled due to the habraeffect too).Screenshot
- Click “Go ahead” and on the Dashboard watch how the test is performed.Screenshot
- You can go to the page with the results of execution and see what the test wished to inform Nerrvana. You can click on the eye (you will need a login and password that came to the post office) and see what report he created. The report is in _files_out / habratest / logs / html / habraReport.html, our small framework is used to create it.Performance report
FTP results
Test generated report
You can read more about the reports themselves here ( example , example ), and if you prefer, you can use it for your tests.
In the end, you can look at the code of the test itself.
Here are some screenshots made tests for different browsers: FF , Chrome , Safari , IE10 .
If you want to play again, you can change the test code via FTP or put another test. In the tests you can use both Selenium1 and WD - in the folder simple_test_java you will find examples.
What will happen next
In fact, we do not quite know where to go next, and how to use our differences from Sauce Labs. For our purposes, Nerrvana is enough for the eyes, but perhaps you will be able to say how the service could be useful to you, should it be developed and how? Some of our thoughts on this:
- Nerrvana supports running tests in Java, PHP, Ruby, Python, Perl, including working with tests that use frameworks like ucumber , Robot Framework , Thucydides and others (we plan to write about how to use these frameworks in Nerrvana).
- Tests are on Nerrvana, i.e. you can run them from anywhere, or your client can.
- Since the tests run on Nerrvana on a schedule, with its help, you can monitor some of the key functions of your sites.
- Nerrvana servers can be spread around the world and given the opportunity to choose where to run the tests.
- Tests are faster than tests through Sauce Labs, since there are no delays in communicating the test with the Selenium node.
- Nerrvana instance can be deployed inside the corporate network if security rules or other reasons do not allow running tests outside
- The disadvantage that the tests are performed by us is that the software suite is limited to what is preinstalled by us (if this is not a jar library, of course). We are considering the option of using Chef so that the configuration needed by the tests themselves can be run - OS version, Java / PHP version, Selenium itself, etc. Such a configurable machine will run longer, but will be more flexible.
- If you implement the previous item, Nerrvana can become a CI server in the cloud (such as TravisCI ), capable of performing not only Unit, but also Selenium tests.
In general, you can do a lot, but for now it is not entirely clear what will really be interesting and in demand.
I apologize in advance if habraeffect will happen and it will cause some problems with the service. Besides us , there are almost no real users now, and therefore we use only one physical server. If necessary, we will add servers in the future.
We will be happy if Nerrvana is helpful to one of you.
And I will be glad to hear comments and answer questions. Perhaps some answers are already here .
______
* Selenium1 in the test used to at the same time show our reporting framework, we have not yet had time to rewrite it under WD. Found that my test for Selenium1 somehow does not work well in Opera. On WD in Opera, the same works ok - if you try, keep in mind. And in general, according to my observations, WD is already running more stable than the first Selenium. An example of a WD test is in simple_test_java.
Source: https://habr.com/ru/post/171733/
All Articles