Automated functional testing of Windows applications using Ranorex

Last year, not only the Web, but also the Windows client appeared in our company. Since we didn’t use the test automation tools we used for the Web (for example, Selenium ) in this situation, we had to search for automated functional testing technology for Windows applications.

The criteria for our search were as follows:

• The functional test development environment should support both automatic recording of test cases (for use by testers with minimal programming experience) and the ability to write individual parts of tests manually.
• Since the client is being developed in C #, I would also like to write functional tests either in C # or in one of the .NET languages.
• Must have Data-Driven Testing support
• XPath should preferably be used to identify user interface elements.
• The code and all testing artifacts should be stored in the same version control system in which our main code is stored (in our case, Git)
• This automated functional testing technology should be easily integrated into our continuous integration system ( Jenkins ).

After preliminary market research, we stopped at 2 products :

1. HP QuickTest Professional , which is one of the most used products on the market, but has an extremely high price.
2. Ranorex is a newer product (although I first heard about it in 2007, but then it was still quite raw), but often already mentioned as an alternative in professional forums

During the month I tested both products (HP QTP 11 and Ranorex 4) and now I would like to share my conclusions about the capabilities of Ranorex, comparing it with HP QuickTest Professional, and also consider integrating Ranorex with Jenkins CI server (but integration with other servers, for example TeamCity , will look similar).

Ranorex features

• What types of applications can be tested : here the set of Ranorex is comparable to HP QTP: Windows native (WinForms, WPF, Win32), Java, Qt, Delphi, Flex + of course HTML (browsers - IE, Firefox, Chrome, Safari). The latest version also includes support for testing mobile clients on Android and iOS, but I have not yet taken advantage of this opportunity.
• Recording actions and item repositories : Ranorex supports both recording actions using the built-in recorder (similar to HP QTP) and identifying user interface elements using the Ranorex Spy component.
  
  All identified elements are stored in an XML repository, where each element is written using XPath notation. For example, a button on a toolbar can be described as:
  

  /form[@ title~'^MyApp']//toolbar[@ controlname='mainToolbar']/button[@ controlname='addUser'] 

  Based on the recording, a code is automatically generated (in C # or VB.NET). Any step can also be written manually (for example, repo.MyApp.MainPain.BtnOk.Click(); )
• Identification of user interface elements : Using XPath (or, as it is called here, RanoreXPath) makes searching for the necessary elements extremely easy in most cases. For example, regular expressions ( form[@ title~'^MyApp.*'] ), Search for all descendants ( .//button ), logical operators ( @ text='OK' OR @ controlname='buttonOK' ), built-in functions are @ text='OK' OR @ controlname='buttonOK' ( table/row/cell[first()='True'] ) and more. Which is very convenient, each item in the repository has a search-timeout property. For example, if we work with the button from the example above and at the window ( form[@ title~'^MyApp'] ) search-timeout is 30 seconds, then Ranorex will wait for 30 seconds to wait for this window to appear (for example, when the application starts) after which it will proceed to performing actions with the button. This allows you to almost completely abandon the use of the wait() function in tests.
• Elements in the repository : in Ranorex we work with different types of elements (for example: form , toolbar , container , tree , list , etc.). Since we work with elements not directly, but with element adapters, the button element is in the case of a Windows native application, in the case of a Java offer and in the case of Flex, etc. will have a similar set of properties and methods (for example text , pressed , click() )
  
• IDE : Ranorex Studio is based on free SharpDevelop . As a result, the organization of the code and test cases most closely resembles MS Visual Studio: All test cases are decomposed into projects. Projects that are used together can be combined into solutions. At the output after compilation, we get one EXE file per solution and 1 DLL per project. It is enough to copy them to the test machine and run as a normal Windows application.
  
• Organization of test cases : Each solution (solution - see above) has one test suite. In this test suite we add the test cases we need and drag the recordings from our projects. Recordings can either be recorded automatically or written in one of the supported languages ​​manually. Here in the manual modules we can use any code we need in C # / VB.NET, for example, the code for accessing the Windows registry, uploading files via FTP, etc.
• Data-Driven Testing : any test case can be associated with data from a CSV / Excel file or from a database. Each module (record) can have variables associated with the data or populated in the module itself.
  
• Programming language : for writing tests in Ranorex, either C # or VB.NET is used. In the case of HP QTP, VB Script is used. Probably for people with no programming experience VB.NET or especially C # are more difficult to learn than VB Script. Script languages, as a rule, are more suitable for test automation, but personally, choosing between C # - a full OO language with full support for .NET and a script and rather primitive VB Script, of course I prefer C #. For some, the VB.NET language support in Ranorex will be the best choice.
  
• Code storage and versioning : HP QTP stores code and repositories partly in binary format, which makes its versioning by third-party tools like SVN or Git rather problematic (and my own versioning system based on HP QC seemed to me extremely primitive, for example, I did not find an opportunity commits multiple resources at once, tagging, etc.). In contrast, Ranorex stores all code (including that generated by auto-generation) and all repositories in text form (for example, XML format is used for repositories) - this gave us the opportunity to use Git with all its features for versioning and collaborative development. like a local repository, tagging, etc.

Using Ranorex with Continuous Integration System

Unlike HP QTP, Ranorex does not have a system that allows running tests on a schedule or on an external trigger on remote computers. On the one hand, this is a minus, on the other hand, we were able to integrate Ranorex tests with our Jenkins server without any problems (but any server, for example, the same TeamCity, will do ):

1. Compiling Ranorex tests: On Jenkins, we have a build Build_Ranorex_Tests. As soon as one of the testers makes a push code to our Git repository, one of the Git hooks launches this jab. It consists of two phases: compilation of all solutions (test suites) using MS Build ( /t:Clean,Build /p:Configuration=Release ). At the end of the MS Build phase, all the files needed for testing (namely, EXE (one for each solution), DLL (one for the project), CSV / Excel and others) are archived (step - Archive the artifacts)
2. Test execution: Next, the following “matrix” job is launched on all test machines (we are testing under Windows XP, Vista and Windows 7) on which Jenkins Slaves are installed. This job first installs the latest version of our application (using MSI), then sequentially launches EXE files of test suites compiled by the previous job (see above). After their execution, we receive one ZIP file per test suite, each of which contains 1 XML file with results and a directory with screenshots
3. Presentation of the results: After this, the script (it can be made as a separate small suite in Ranorex, which will be launched last) translates the Ranorex XML format to xUnit format. Thanks to this, for each client we have both a report in Ranorex format and a format that Jenkins understands, so Jenkins can draw a graph of test results. Jenkins also sends by mail to all interested persons test results in Ranorex format.
   

')

Price

Unfortunately, unlike Selenium, Ranorex is not a free open source product. But comparing the automatic testing system under Windows is necessary not with Selenium, but with the same HP QTP or IBM Rational Functional Tester. In this case, Ranorex ne seems so expensive. 1 license linked to a workplace costs 1 480 euros for a purchase + 290 euros per year (starting from the second) for further updates and support. It is several times cheaper than HP QTP and IBM Rational Functional Tester.

findings

Based on our comparative testing of Ranorex and HP QTP, we chose to use Ranorex in the future. The main advantages for us are: a convenient and understandable development environment, full support for .NET development and C # and VB.NET languages, using XPath to identify elements in the repository, the ability to store tests in our standard versioning system (Git), ease of integration with the CI system (Jenkins).