Automate Web Application Testing

Test automation is a meeting place for two disciplines: development and testing. Probably therefore, I attribute this practice to difficult, but interesting.

Through trial and error, we arrived at the following technology stack:

1. SpecFlow (optional): DSL
2. NUnit: test framework
3. PageObject + PageElements: UI-abstrakitsya
4. Test context (information about the target environment, system users)
5. Selenium.WebDriver

To run tests on a schedule, we use TFS 2012 and TeamCity.
In the article I will describe how we arrived at this, typical mistakes and ways to solve them.

Why so hard?

Obviously, test automation has many advantages. Automatic solution:

1. Saves time
2. Eliminates the human factor when testing
3. Removes the burden of routine regression testing

Everyone who has ever been involved in automated testing knows about the downside. Automatic tests can be:

1. Fragile and “break” due to UI change
2. Incomprehensible, contain the code "with a nice smell"
3. Unreliable: to test the wrong behavior or depend on the characteristics of the environment

For example, consider the following code. The title shows that we are testing the fact that, upon the request of “gangnam style”, Google will give out the YouTube channel of the Korean popular artist PSY by the first result.
')

[Test] public void Google_SearchGangnamStyle_PsyYouTubeChanelIsOnTop() { var wd = new OpenQA.Selenium.Firefox.FirefoxDriver {Url = "http://google.com"}; try { wd.Navigate(); wd.FindElement(By.Id("gbqfq")).SendKeys("gangnam style"); wd.FindElement(By.Id("gbqfb")).Click(); var firstResult = new WebDriverWait(wd, TimeSpan.FromSeconds(10)).Until( w => w.FindElement(By.CssSelector("h3.r>a"))); Assert.AreEqual("PSY - YouTube", firstResult.Text); Assert.AreEqual("http://www.youtube.com/user/officialpsy", firstResult.GetAttribute("href")); } finally { wd.Quit(); } } 

There are a lot of problems in this test:

1. Mixed application layers (driver, locators, results)
2. Strings sewn in dough
3. In order to change the web driver, for example on IE you will have to change all the tests.
4. Locators are protected in the test and will be duplicated in each test again.
5. Duplicate the code to create a web driver
6. Assert is not accompanied by an error message.
7. If the first Assert "falls", then the second condition will not be checked at all.
8. When you first look at the test, it is not clear what is going on in it, you will have to read carefully and spend time understanding code

If we approach automation “in the forehead,” instead of getting rid of the routine repetition of the same actions, we will get an additional headache with the support of tests, false alarms and a spaghetti code to boot.

Application Layers in Automated Testing

Your tests are code too. Treat them as reverently as the code in the application. The theme of business application layers is already well covered. What layers can be distinguished in tests?

1. Technical driver (WebDriver, Selenium RC, etc)
2. Test context (target environment, users, data)
3. UI abstraction - pages, widgets, page components (PageObject pattern)
4. Tests (test framework: NUnit, xUnit, MSTest)
5. DSL

Let's do an evolutionary refactoring and fix our test.

Technical driver

In our case, this is Selenium.WebDriver. By itself, WebDriver is not a test automation tool, but only a browser control tool. We could automate testing at the level of HTTP requests and save a lot of time. To test web services, we don’t need a web driver at all: a proxy is enough.
Using a web driver is a good idea because:

1. Modern applications are more than just a request-response. Sessions, cookies, java-script, web sockets. All this can be pretty damn hard to repeat programmatically.
2. Such testing is as close as possible to user behavior.
3. The complexity of writing code is much lower.

The technical driver layer includes:

1. All web driver settings
2. Logic create and destroy web driver
3. Error control

To begin with we will take out settings in a config. Here it looks like this:

 <driverConfiguration targetDriver="Firefox" width="1366" height="768" isRemote="false" screenshotDir="C:\Screenshots" takeScreenshots="true" remoteUrl="…"/> 

Let's create a separate class that will take on the logic of reading the config, creating and destroying the web driver.

 [Test] public void WebDriverContextGoogle_SearchGangnamStyle_PsyYouTubeChanelIsOnTop() { var wdc = WebDriverContext.GetInstance(); try { var wd = wdc.WebDriver; wd.Url = "http://google.com"; wd.Navigate(); wd.FindElement(By.Id("gbqfq")).SendKeys("gangnam style"); wd.FindElement(By.Id("gbqfb")).Click(); var firstResult = new WebDriverWait(wd, TimeSpan.FromSeconds(10)).Until( w => w.FindElement(By.CssSelector("h3.r>a"))); var expected = new KeyValuePair<string, string>( "PSY - YouTube", "http://www.youtube.com/user/officialpsy"); var actual = new KeyValuePair<string, string>( firstResult.Text, firstResult.GetAttribute("href")); Assert.AreEqual(expected, actual); } finally { wdc.Dispose(); } } 

It got a little better. Now we are sure that only one web driver will always be used. All settings are in the config, so we can change the driver and other settings without recompiling.

Testing context

For black-box testing of the application, we need some amount of input data:

1. Target environment - url, ports of the tested applications
2. Users with different role set

This information does not relate to the testing logic, so we will move this to the configuration section. We describe all the environments in the config.

 <environmentsConfiguration targetEnvironment="Google"> <environments> <environment name="Google" app="GoogleWebSite"> <apps> <app name="GoogleWebSite" url="http://google.com/" /> </apps> <users> <user name="Default" login="user" password="user" /> </users> </environment> </environmentsConfiguration> 

Instead of wd.Url = " google.com "; wd.Url = EnvironmentsConfiguration.CurrentEnvironmentBaseUrl;

1. We do not have to duplicate the URL in all tests.
2. To test another environment, it is enough to build a project with a different configuration and add a transformation.

 <environmentsConfiguration targetEnvironment="Google-Test" xdt:Transform="SetAttributes"> 

Page Objects

The Page Objects pattern is well established in test automation.
The main idea is to encapsulate the logic of page behavior in a page class. Thus, tests will not work with a low-level code of a technical driver, but with a high-level abstraction.

The main advantages of Page Objects:

1. Separation of powers: all the “business logic” of the page should be placed in Page Objects, test classes just call public methods and check the result
2. DRY - all locators are located in one place. If, when the UI changes, we will only change the locator in one place.
3. Hiding technical driver layer. Your tests will work with high-level abstraction. In the future, you may want to change the driver: for example, use PhantomJS, or even refuse to use WebDriver for some sites to improve performance. In this case, you have to replace only the Page Objects code. Tests will remain unchanged
4. Page Objects allows you to write locators in a declarative style.

What is missing in Page Objects

The canonical pattern involves creating one class per page of your application. This can be inconvenient in some cases:

1. Customizable and / or dynamically changeable layout
2. Widgets or other elements present on many pages.

Partially these problems can be solved with the help of inheritance, but aggregation seems to be preferable, both from a technical point of view and from the point of view of understanding code.
Therefore, it is better to use the enhanced version of the pattern - Page Elements. Page Elements - allows you to split the page into smaller components - blocks, widgets, etc. Then these blocks can be reused in several pages.
Create a page:

 [FindsBy(How = How.Id, Using = "gbqfq")] public IWebElement SearchTextBox { get; set; } [FindsBy(How = How.Id, Using = "gbqfb")] public IWebElement SubmitButton { get; set; } public GoogleSearchResults ResultsBlock { get; set; } public void EnterSearchQuery(string query) { SearchTextBox.SendKeys(query); } public void Search() { SubmitButton.Click(); } 

And a “widget” with results

 public class GoogleSearchResults : PageElement { [FindsBy(How = How.CssSelector, Using = "h3.r>a")] public IWebElement FirstLink { get; set; } public KeyValuePair<string, string> FirstResult { get { var firstLink = PageHelper.WaitFor<GoogleSearchResults>(w => w.FirstLink); return new KeyValuePair<string, string>(firstLink.Text, firstLink.GetAttribute("href")); } } } 

NuGet has a WebDriver.Support package with an excellent PageFactory.InitElements method .
The method is good, but has side effects. PageFactory from the WebDriver.Support package returns a proxy and does not wait for the item to load. In this case, if all synchronization methods work with the class By , which does not know how to work with the FindsBy attribute yet .
This problem is solved by creating a base Page class.

 /// <summary> /// Get Page element instance by type /// </summary> /// <typeparam name="T">Page element type</typeparam> /// <param name="waitUntilLoaded">Wait for element to be loaded or not. Default value is true</param> /// <param name="timeout">Timeout in seconds. Default value=PageHelper.Timeout</param> /// <returns>Page element instance</returns> public T GetElement<T>(bool waitUntilLoaded = true, int timeout = PageHelper.Timeout) where T : PageElement /// <summary> /// Wait for all IWebElement properies of page instance to be loaded. /// </summary> /// <param name="withElements">Wait all page elements to be loaded or just load page IWebElement properties</param> /// <returns>this</returns> public Page WaitUntilLoaded(bool withElements = true) 

In order to implement the WaitUntilLoaded method, it is enough to collect all public properties with FindBy attributes and use the WebDriverWait class. I will omit the technical implementation of these methods. It is important that the output we get a simple and elegant code:

 var positionsWidget = Page.GetElement<GoogleSearchResults>(); 

There was the last inconvenient case. There are some widgets that hide / show some of the elements depending on the state. To break such a widget into several with one property each is impractical.
The solution was also found.

 public static IWebElement WaitFor<TPage>( Expression<Func<TPage, IWebElement>> expression, int timeout = Timeout) var firstLink = PageHelper.WaitFor<GoogleSearchResults>(w => w.FirstLink); 

I will not bore the technical implementations of these methods. Let's see what the code will look like after refactoring.

 [Test] public void Google_SearchGangnamStyle_PsyYouTubeChanelIsOnTop() { try { var page = WebDriverContext.CreatePage<GooglePage>(EnvironmentsConfiguration.CurrentEnvironmentBaseUrl); page.EnterSearchQuery("gangnam style"); page.Search(); var expected = new KeyValuePair<string, string>( "PSY - YouTube", "http://www.youtube.com/user/officialpsy"); var actual = page.GetElement<GoogleSearchResults>().FirstResult; Assert.AreEqual(expected, actual); } finally { WebDriverContext.GetInstance().Dispose(); } } 

At this stage, it became much better:

1. The testing class overtook driver management and delegated these responsibilities to the page class.
2. We got rid of duplicate locators
3. Readability tests improved

Tests

After we brought out the locators and logic in Page Objects, the test code became laconic and cleaner. However, a few things are still not very good:

1. The logic of creating a web driver is duplicated from test to test
2. The logic of creating a page in each method is also redundant.
3. Magical lines, "gangnam style", "PSY - YouTube", ”http://www.youtube.com/user/officialpsy” blister eyes
4. The test script itself is quite fragile: the indexing results may change and we will have to change the code

Create a base test class

 public class WebDriverTestsBase<T> : TestsBase where T:Page, new() { /// <summary> /// Page object instance /// </summary> protected T Page { get; set; } /// <summary> /// Relative Url to target Page Object /// </summary> protected abstract string Url { get; } [SetUp] public virtual void SetUp() { WebDriverContext = WebDriverContext.GetInstance(); Page = Framework.Page.Create<T>( WebDriverContext.WebDriver, EnvironmentsConfiguration.CurrentEnvironmentBaseUrl, Url, PageElements); } [TearDown] public virtual void TearDown() { if (WebDriverContext.HasInstance) { var instance = WebDriverContext.GetInstance(); instance.Dispose(); } } } 

Rewrite the test again

 public class GoogleExampleTest : WebDriverTestsBase<GooglePage> { [Test] public void Google_SearchGangnamStyle_PsyYouTubeChanelIsOnTop() { Page.EnterSearchQuery("gangnam style"); Page.Search(); var expected = new KeyValuePair<string, string>( "PSY - YouTube", "http://www.youtube.com/user/officialpsy"); var actual = Page.GetElement<GoogleSearchResults>().FirstResult; Assert.AreEqual(expected, actual); } } 

Already almost perfect. We'll take out the magic lines in the TestCase attribute and add a comment to the Assert.

 [TestCase("gangnam style", "PSY - YouTube", "http://www.youtube.com/user/officialpsy")] public void Google_SearchGoogle_FirstResult(string query, string firstTitle, string firstLink) { Page.EnterSearchQuery(query); Page.Search(); var expected = new KeyValuePair<string, string>(firstTitle, firstLink); var actual = Page.ResultsBlock.FirstResult; Assert.AreEqual(expected, actual, string.Format( "{1} ({2}) is not top result for query \"{0}\"", firstTitle, firstLink, query)); } 

1. The test code has become clear
2. Duplicate operations moved to base class
3. We have provided enough information, in case of a test crash, everything will be clear from the test runner logs.
4. You can add as many input and output parameters as you like without changing the test code using the TestCase attribute

DSL

This code still has problems:

1. The code has become clear and clean, but in order to maintain it in this state, the qualifications of specialists supporting the tests must be appropriate
2. The QA department most likely has its own test plan, and our auto tests do not correlate with it at all.
3. Often the same steps are repeated in several scenarios at once. Code duplication can be avoided using inheritance and aggregation, but this already seems like a difficult task, especially since the order of the steps may be different.
4. Google_SearchGangnamStyle_PsyYouTubeChanelIsOnTop () : CamelCase is hard to read

Using the SpecFlow plugin, we can solve these problems. SpecFlow allows you to record test scripts in Given When Then style, and then automate them.

 Feature: Google Search As a user I want to search in google So that I can find relevent information Scenario Outline: Search Given I have opened Google main page And I have entered <searchQuery> When I press search button Then the result is <title>, <url> Examples: |searchQuery |title |url |gangnam style |PSY - YouTube |http://www.youtube.com/user/officialpsy [Binding] public class GoogleSearchSteps : WebDriverTestsBase<GooglePage> { [Given("I have opened Google main page")] public void OpenGooglePage() { // Page is already created on SetUp, so that's ok } [Given(@"I have entered (.*)")] public void EnterQuery(string searchQuery) { Page.EnterSearchQuery(searchQuery); } [When("I press search button")] public void PressSearchButton() { Page.Search(); } [Then("the result is (.*), (.*)")] public void CheckResults(string title, string href) { var expected = new KeyValuePair<string, string>(title, href); var actual = Page.GetElement<GoogleSearchResults>().FirstResult; Assert.AreEqual(expected, actual); } } 

In this way:

1. Each step can be implemented only once.
2. Attributes Given When Then support regular expressions - you can create reusable "functional" steps
3. QA department can record scripts in auto-test projects.
4. Testers can write DSL, and automation can be assigned to programmers.
5. At any time, a report on the passed tests, and thus the number of developed functionality, is available on the CI server.

More information about SpecFlow and requirements management with Given When Then can be found in this article .

Automating Guideline

1. Avoid fragile and difficult locators.
   Not properly:
   

    [FindsBy(How = How.XPath, Using = "((//div[@class='dragContainer']/div[@class='dragHeader']" + "/div[@class='dragContainerTitle'])[text()=\"Account Info\"])" + "/../div[@class='dragContainerSettings']")] public IWebElement SettingsButton { get; set; } 

   
   Right:
   

    [FindsBy(How = How.Id, Using = "gbqfb")] public IWebElement SubmitButton { get; set; } 

   
   It is best to use id. The id may depend on the java-script and the front-end players will change it less likely. If you cannot use id (dynamic markup), use data-aid (automation-id), or a similar attribute
   
2. Encapsulate your application logic in page classes (for example, LogonPage, RegistrationPage, HomePage, OrderPage, etc.).
3. Select widgets and repeating blocks in “widgets” (Page Elements), for example: Header, Footer, LogonLogoff
4. Group elements into widgets depending on their display, for example: ConfirmationPopup, EditPopup, AddPopup
5. Avoid magic lines in the test code, move them to the properties of pages and widgets or helpers, for example OrderSuccessMessage, RegistrationSuccessMessage, InvalidPasswordMessage. This will avoid unnecessary code waiting load / appearance of elements
6. Bring repeated operations to the base classes of tests, use SetUp, TearDown: create and destroy the driver, create screenshots with errors, authenticate, improve the readability of the test
7. Take out the Page Objects in a separate assembly, it will help to avoid duplication of widgets / pages. Bundles with tests can be many
8. Use Assert only in test code, Asserts should not be contained in pages and widgets
9. Use the Assertions that best describe what you are testing. This will improve the readability of the test.
   Not properly:
   

    var actual = Page.Text == “Success” Assert.IsTrue(actual); 

   
   Right:
   

    Assert.AreEqual(MessageHelper.Success, Page.Text) 

   
   
10. Use Assert Error Messages
    

     Assert.AreEqual(MessageHelper.Success, Page.Text, “Registration process is not successfull”); 

    
    
11. Avoid using Thread.Sleep as a timeout for loading items. Use high-level abstractions to ensure that the necessary DOM elements are loaded, for example:
    

     Page.GetElement<GoogleSearchResults>(); var firstLink = PageHelper.WaitFor<GoogleSearchResults>(w => w.FirstLink); 

    
    
12. If you do not use DSL, write the name of the tests in the format of [Page_] Scenario_Seed Behavior . This will help people understand exactly what behavior is being tested. This can be especially important when changing requirements.
13. In DSL, group the steps semantically and so as to maximize the reuse of steps.
    Not properly:
    

     I have logged as a user with empty cart 

    
    Right:
    

     I have logged in And my cart is empty 

    
    
14. In DSL, compare tuples in one step, this will allow you to write more concise code and get more information when the test fails.
    Not properly:
    

     When I open Profile page I can see first name is “Patrick” And I can see last name is “Jane” And I can see phone is “+123 45-67-89” 

    
    Right
    

     When I open Profile page I can see profile info: Patrick Jane +123 45-67-89 

    
    
15. Use black-box testing, make initialization of test data from the test code. For this, for example, SSDT project is well suited.
16. Use CI to run tests regularly, make sure that test results are publicly available and understandable.

A very good eBay report on test automation can be found here: www.youtube.com/watch?v=tJ0O8p5PajQ