CodedUI or Ranorex? Automation of functional testing. NET applications
The author of the article is Tatiana Kurnosova, but she went to conquer the mountains of Kyrgyzstan. Therefore, the honor to publish this post fell to me.
In addition to the well-known Desktop , Mobile and Online applications, 2GIS has developed many internal Enterprise products. These products are hidden from the eyes of users, however, it is with their help that tremendous work is done to provide the entire infrastructure of 2GIS with cartographic and reference data. Processing this data is labor intensive and requires error-free calculations, so all products are thoroughly tested before “hiring”.
In this article I will tell you how we chose the tools, how we analyzed their technical capabilities, so that they fit into our infrastructure, and where we left off.
')
At one time, we were faced with the task of choosing a package suitable for testing the user interface of the products used. Everything that amenable to automation, we cover with automated tests, and the GUI is no exception. For this, as you know, there are tools that allow you to simulate user actions.
Taking into account the fact that the tested products are implemented on .NET, and their testers are familiar with the C # programming language, it was decided to develop tests in C # as well. From a large set of tools on the market, we stopped at two. The first is CodedUI, which comes with Microsoft Visual Studio actively used by us. The second is Ranorex, as it is compatible with Visual Studio, has a user-friendly interface and positive testers reviews.
Coded UI is one of the solutions that come with Microsoft Visual Studio Premium / Ultimate, provides access to libraries for test development.
Ranorex Automation Tools is a complete development environment, as well as a set of tools and libraries for writing tests.
Next, we had to make a choice in favor of one of these products.
We formulated the main criteria by which packages were compared:
For comparison, a test application was developed with a GUI based on the Windows Presentation Foundation (WPF). It contained all the key GUI elements that may be involved in testing real products in the future.
CodedUI
CodedUI provides a UIMap class object that implements access to all UI elements of the application. You can dynamically get any element of the interface, knowing its name, or by passing through a chain of nested elements, for example:
"The first text field on the first tab"
However, such a record can be formed only directly in the test code.
Ranorex
In Ranorex, an analogue of UIMap is the GUI Object Repository. XPath is used to access the elements, so you can select an element by generating a corresponding query using the built-in recorder.
"The first text field on the first tab"
CodedUI
Ranorex
According to this criterion, Ranorex looks preferable, since it has a simpler (easily mastered) system for working with controls.
CodedUI
In CodedUI, by default, it is not possible to work with XPath to search for controls, but you can connect a special library .
Ranorex
In Ranorex XPath, expressions are called RanoreXPath. If an element does not have a unique name, then it can be accessed in one action, knowing the XPath expression for it (see the previous paragraph).
CodedUI
Ranorex
An extra point in favor of Ranorex for XPath support by default.
Data Driven Testing is a test automation approach based on the use of test datasets located in a repository separate from the source code of the tests.
CodedUI
When developing CodedUI tests, the MSTest framework, which contains the functionality to create Data Driven tests, is used to organize the test structure itself and run the test. Any test can be linked to data sets from a CSV, XML file, or a database sample.
Ranorex
Ranorex also has the ability to develop Data Driven tests, but excludes XML for data. Apparently, the developers of Ranorex considered it quite difficult for the tester.
CodedUI
Ranorex
According to this criterion, the advantage on the side of CodedUI, as in our products the XML format is very common, and in many cases it is more convenient for the tester to work with him.
The experience of test automation shows that, in any test project, sooner or later, some basic functionality is provided that allows for reducing the test code. This functionality then develops into a small test framework. At this stage, tests can already be constructed from various methods of this framework.
In the long term, we are going to use the tool chosen by us within the framework of such a framework. The framework encapsulates the work with CodedUI or Ranorex necessary for accessing user interface controls, and provides a set of methods that can be associated with user actions, for example, “Open Card”, “Fill Form”. Any UI test consists of a set of calls to such methods and checks for correct processing of the data of the application under test.
The purpose of this criterion is to understand how convenient it is by choosing a particular tool, you can develop a test framework.
CodedUI
When used, CodedUI tests are developed in Visual Studio in C #. Tests can be extended, refined, refactored to infinity in the traditional way, to attract developers.
Ranorex
Ranorex provides its IDE - Ranorex Studio, which is based on SharpDevelop and therefore has a very similar interface and working principle with Visual Studio. Ranorex has a convenient element inspector and the ability to create tests with just a few mouse clicks. In addition, it is possible to develop additional software modules, which will then be used to create more complex test scenarios.
However, if the module is developed not by a tester, but by a programmer, or an automation specialist (as in our case), it is more convenient for him to use Visual Studio, since it has a better debugging system, ReSharper, etc.
CodedUI
Ranorex
Development of the framework is, in general, not the task of the tester, therefore, the tool needs to be more suitable for development. In general, Ranorex Studio is inferior to CodedUI by this indicator.
Test applications deployed on the server Continuous Integration (Build-server), which is selected as TeamCity. Under team agreements, all tests are committed to the repository of the tested product. This allows us to use joint code, simultaneously update and collect application code and tests on the Build server. For GUI tests, the same workflow should remain.
CodedUI
TeamCity allows you to easily connect tests developed using frameworks MSTest, NUnit.
Ranorex
Ranorex, in turn, is able to compile Test Suits into an executable file, which can then be simply launched as a separate TeamCity build task.
CodedUI
Ranorex
By this criterion, tools are conveniently used equally.
CodedUI
MSTest prints results in TRX format. This is an XML document, but it is rather difficult to read it with the naked eye, and it isn’t necessary, since TeamCity provides the same report in CSV format after running the tests.
Ranorex
The IDE Ranorex provides excellent reports with illustrations and explanations. However, for this, tests must be run from within Ranorex Studio itself. Compiled tests after the run return a Ranorex XML report, which can then be converted to xUnit.
CodedUI
Ranorex
Both cases do not provide the final desired type of test report. However, we have already implemented a utility that can create test runs in our test management system (TMS) and import the results of automated tests passed. Moreover, the original report format can be either xUnit or CSV. In TMS, the report looks vividly, understandable and shows the general picture of testing at the current moment.
In addition to importing test results into a test management system, we also import test cases themselves.
How does this happen:
The tester, developing an automated test, writes a step-by-step description directly in the test class code, using special attributes. Also with the help of attributes it indicates which suite and section belongs to this case. Then, with the help of a special utility, imports all test case cases into the corresponding TMS project. Import can be done manually or as part of the TeamCity build task. At the entrance of this utility is supplied a test project DLL file.
We would like to use this opportunity when developing GUI tests. So, you need the ability to connect their libraries and edit tests.
CodedUI
Visual Studio has the ability to view and edit the CodedUI test code, even if it is recorded using the built-in recorder. This makes it possible to put down the necessary attributes for synchronization with TMS. A default test project is assembled into a DLL, which can then be used to import cases.
Ranorex
In Ranorex Studio, a test project is compiled into an exe file. That is, with the current implementation of our utility, it is not possible to use it as a source of test cases.
CodedUI
Ranorex
Thus, using CodedUI, it will be easier for us to tie our integration system to new tests.
CodedUI
1. After creating several examples on our test application, problems immediately emerged when the built-in recorder recorded actions related to the consciousness of dynamic controls. When the test was restarted, the name of the control changed and the test failed. To avoid this and make tests more versatile, we had to edit the test code manually.
2. Develop tests using CodedUI is possible only in Visual Studio. But VS for a non-programmer is a very complex and cumbersome tool.
Ranorex
Ranorex is specifically created as a tester environment. With it, it is convenient to create, run and view test results. In addition, after we developed several examples for a test application, problems with the search for dynamic elements never occurred. Ranorex also successfully worked with .NET WinForms applications for which we were unable to launch CodedUI.
CodedUI
Ranorex
Ranorex has a more user-friendly interface and system for working with elements.
As a conclusion, I will give a summary table according to the criteria described above:
opportunity implemented
opportunity not implemented
opportunity implemented and easy to use
By the comparison criteria we chose, Ranorex was rated higher, mainly due to convenience.
In the process of analysis, we attracted testers to develop small test scenarios and it seemed much more convenient and clearer to work with Ranorex.
If you are a tester and you need to test a Windows application from scratch and you choose between buying a license to use Visual Studio with CodedUI or Ranorex support, then choosing Ranorex would be more logical.
As for our projects.
Starting to test the project, we conditionally break the product into independent parts that can be tested separately. For example, an application has an interface that processes database data through an API, integration services synchronize data with other products, and stored procedures at the database level work on a schedule and calculate the values ​​of some fields. And for each such part, its own set of tests is developed as a separate application (test project = test framework, adapted to the needs of product testing + many tests combined into test sets). All test projects are part of the solution for the entire product being tested.
In this structure, we needed to add GUI testing.
Since Ranorex and CodedUI did not differ much in terms of critical parameters for us, we thought about the prospect of test development.
Ultimately, the choice was between a tool in which testers would write GUI tests separately from the product and developers, and the time spent on developing everything needed (framework, documentation) for quick and easy development of tests using CodedUI.
Since in the second case such a system can be made more universal, we chose CodedUI. Yes, this is more development hours, but at the same time we had some universal solution for automating the functional GUI testing of all Enterprise products of the company.
In addition to the well-known Desktop , Mobile and Online applications, 2GIS has developed many internal Enterprise products. These products are hidden from the eyes of users, however, it is with their help that tremendous work is done to provide the entire infrastructure of 2GIS with cartographic and reference data. Processing this data is labor intensive and requires error-free calculations, so all products are thoroughly tested before “hiring”.
In this article I will tell you how we chose the tools, how we analyzed their technical capabilities, so that they fit into our infrastructure, and where we left off.
')
At one time, we were faced with the task of choosing a package suitable for testing the user interface of the products used. Everything that amenable to automation, we cover with automated tests, and the GUI is no exception. For this, as you know, there are tools that allow you to simulate user actions.
Taking into account the fact that the tested products are implemented on .NET, and their testers are familiar with the C # programming language, it was decided to develop tests in C # as well. From a large set of tools on the market, we stopped at two. The first is CodedUI, which comes with Microsoft Visual Studio actively used by us. The second is Ranorex, as it is compatible with Visual Studio, has a user-friendly interface and positive testers reviews.
Coded UI is one of the solutions that come with Microsoft Visual Studio Premium / Ultimate, provides access to libraries for test development.
Ranorex Automation Tools is a complete development environment, as well as a set of tools and libraries for writing tests.
Next, we had to make a choice in favor of one of these products.
Comparison criteria
We formulated the main criteria by which packages were compared:
- Support for dynamically generated graphical controls (controls)
- Customizable search system controls
- Simple support for data-based tests (Data Driven Testing)
- The ability to develop your own modules (frameworks) and use them when developing tests in C #
- Support for running tests on the Continuous Integration server (TeamCity)
- Generating informative reports on test run results
- Ability to integrate tests with test cases of the test management system (TMS)
- Ease of learning and use by testers
Test layout
For comparison, a test application was developed with a GUI based on the Windows Presentation Foundation (WPF). It contained all the key GUI elements that may be involved in testing real products in the future.
Compare products by criteria
1. Support for dynamically generated controls
CodedUI
CodedUI provides a UIMap class object that implements access to all UI elements of the application. You can dynamically get any element of the interface, knowing its name, or by passing through a chain of nested elements, for example:
"The first text field on the first tab"
this.UIViewmodeltitleWindow.UIItemTabList.Tabs[0].GetChildren()[1].GetChildren()[1] However, such a record can be formed only directly in the test code.
Ranorex
In Ranorex, an analogue of UIMap is the GUI Object Repository. XPath is used to access the elements, so you can select an element by generating a corresponding query using the built-in recorder.
"The first text field on the first tab"
/form[1]/tabpagelist[1]/tabpage[1]/element[@classname='CardEditView']/text[4] CodedUI
Ranorex
According to this criterion, Ranorex looks preferable, since it has a simpler (easily mastered) system for working with controls.
2. Flexible and customizable search control system
CodedUI
In CodedUI, by default, it is not possible to work with XPath to search for controls, but you can connect a special library .
Ranorex
In Ranorex XPath, expressions are called RanoreXPath. If an element does not have a unique name, then it can be accessed in one action, knowing the XPath expression for it (see the previous paragraph).
CodedUI
Ranorex
An extra point in favor of Ranorex for XPath support by default.
3. Simple Data Driven Testing Support
Data Driven Testing is a test automation approach based on the use of test datasets located in a repository separate from the source code of the tests.
CodedUI
When developing CodedUI tests, the MSTest framework, which contains the functionality to create Data Driven tests, is used to organize the test structure itself and run the test. Any test can be linked to data sets from a CSV, XML file, or a database sample.
Ranorex
Ranorex also has the ability to develop Data Driven tests, but excludes XML for data. Apparently, the developers of Ranorex considered it quite difficult for the tester.
CodedUI
Ranorex
According to this criterion, the advantage on the side of CodedUI, as in our products the XML format is very common, and in many cases it is more convenient for the tester to work with him.
4. Ability to develop frameworks for testing
The experience of test automation shows that, in any test project, sooner or later, some basic functionality is provided that allows for reducing the test code. This functionality then develops into a small test framework. At this stage, tests can already be constructed from various methods of this framework.
In the long term, we are going to use the tool chosen by us within the framework of such a framework. The framework encapsulates the work with CodedUI or Ranorex necessary for accessing user interface controls, and provides a set of methods that can be associated with user actions, for example, “Open Card”, “Fill Form”. Any UI test consists of a set of calls to such methods and checks for correct processing of the data of the application under test.
The purpose of this criterion is to understand how convenient it is by choosing a particular tool, you can develop a test framework.
CodedUI
When used, CodedUI tests are developed in Visual Studio in C #. Tests can be extended, refined, refactored to infinity in the traditional way, to attract developers.
Ranorex
Ranorex provides its IDE - Ranorex Studio, which is based on SharpDevelop and therefore has a very similar interface and working principle with Visual Studio. Ranorex has a convenient element inspector and the ability to create tests with just a few mouse clicks. In addition, it is possible to develop additional software modules, which will then be used to create more complex test scenarios.
However, if the module is developed not by a tester, but by a programmer, or an automation specialist (as in our case), it is more convenient for him to use Visual Studio, since it has a better debugging system, ReSharper, etc.
CodedUI
Ranorex
Development of the framework is, in general, not the task of the tester, therefore, the tool needs to be more suitable for development. In general, Ranorex Studio is inferior to CodedUI by this indicator.
5. Support for running tests on a CI server
Test applications deployed on the server Continuous Integration (Build-server), which is selected as TeamCity. Under team agreements, all tests are committed to the repository of the tested product. This allows us to use joint code, simultaneously update and collect application code and tests on the Build server. For GUI tests, the same workflow should remain.
CodedUI
TeamCity allows you to easily connect tests developed using frameworks MSTest, NUnit.
Ranorex
Ranorex, in turn, is able to compile Test Suits into an executable file, which can then be simply launched as a separate TeamCity build task.
CodedUI
Ranorex
By this criterion, tools are conveniently used equally.
6. Generation of informative reports on the result of the run
CodedUI
MSTest prints results in TRX format. This is an XML document, but it is rather difficult to read it with the naked eye, and it isn’t necessary, since TeamCity provides the same report in CSV format after running the tests.
Ranorex
The IDE Ranorex provides excellent reports with illustrations and explanations. However, for this, tests must be run from within Ranorex Studio itself. Compiled tests after the run return a Ranorex XML report, which can then be converted to xUnit.
CodedUI
Ranorex
Both cases do not provide the final desired type of test report. However, we have already implemented a utility that can create test runs in our test management system (TMS) and import the results of automated tests passed. Moreover, the original report format can be either xUnit or CSV. In TMS, the report looks vividly, understandable and shows the general picture of testing at the current moment.
7. Ability to integrate tests with TMS test cases
In addition to importing test results into a test management system, we also import test cases themselves.
How does this happen:
The tester, developing an automated test, writes a step-by-step description directly in the test class code, using special attributes. Also with the help of attributes it indicates which suite and section belongs to this case. Then, with the help of a special utility, imports all test case cases into the corresponding TMS project. Import can be done manually or as part of the TeamCity build task. At the entrance of this utility is supplied a test project DLL file.
We would like to use this opportunity when developing GUI tests. So, you need the ability to connect their libraries and edit tests.
CodedUI
Visual Studio has the ability to view and edit the CodedUI test code, even if it is recorded using the built-in recorder. This makes it possible to put down the necessary attributes for synchronization with TMS. A default test project is assembled into a DLL, which can then be used to import cases.
Ranorex
In Ranorex Studio, a test project is compiled into an exe file. That is, with the current implementation of our utility, it is not possible to use it as a source of test cases.
CodedUI
Ranorex
Thus, using CodedUI, it will be easier for us to tie our integration system to new tests.
8. Easy to learn and use by testers.
CodedUI
1. After creating several examples on our test application, problems immediately emerged when the built-in recorder recorded actions related to the consciousness of dynamic controls. When the test was restarted, the name of the control changed and the test failed. To avoid this and make tests more versatile, we had to edit the test code manually.
2. Develop tests using CodedUI is possible only in Visual Studio. But VS for a non-programmer is a very complex and cumbersome tool.
Ranorex
Ranorex is specifically created as a tester environment. With it, it is convenient to create, run and view test results. In addition, after we developed several examples for a test application, problems with the search for dynamic elements never occurred. Ranorex also successfully worked with .NET WinForms applications for which we were unable to launch CodedUI.
CodedUI
Ranorex
Ranorex has a more user-friendly interface and system for working with elements.
Conclusion
As a conclusion, I will give a summary table according to the criteria described above:
opportunity implemented opportunity not implemented opportunity implemented and easy to use| Criterion | CodedUI | Ranorex |
|---|---|---|
| Support for dynamically generated controls | | |
| Customizable search system controls | | |
| Simple support for data driven testing | | |
| Ability to develop your own modules | | |
| Support for running tests in TeamCity | | |
| Generating informative reports on test run results | | |
| Ability to integrate tests with TMS test cases | | |
| Ease of learning and use by testers | | |
By the comparison criteria we chose, Ranorex was rated higher, mainly due to convenience.
In the process of analysis, we attracted testers to develop small test scenarios and it seemed much more convenient and clearer to work with Ranorex.
If you are a tester and you need to test a Windows application from scratch and you choose between buying a license to use Visual Studio with CodedUI or Ranorex support, then choosing Ranorex would be more logical.
As for our projects.
Starting to test the project, we conditionally break the product into independent parts that can be tested separately. For example, an application has an interface that processes database data through an API, integration services synchronize data with other products, and stored procedures at the database level work on a schedule and calculate the values ​​of some fields. And for each such part, its own set of tests is developed as a separate application (test project = test framework, adapted to the needs of product testing + many tests combined into test sets). All test projects are part of the solution for the entire product being tested.
In this structure, we needed to add GUI testing.
Since Ranorex and CodedUI did not differ much in terms of critical parameters for us, we thought about the prospect of test development.
Ultimately, the choice was between a tool in which testers would write GUI tests separately from the product and developers, and the time spent on developing everything needed (framework, documentation) for quick and easy development of tests using CodedUI.
Since in the second case such a system can be made more universal, we chose CodedUI. Yes, this is more development hours, but at the same time we had some universal solution for automating the functional GUI testing of all Enterprise products of the company.
Source: https://habr.com/ru/post/188302/
All Articles