Visual Studio Online Testing Tools, Experience and Comparison with the Manual Approach

This article continues the cycle begun by our friends from Kaspersky Lab and describes the actual experience of using Microsoft's testing tools with recommendations and conclusions. The author is a testing engineer, Igor Shcheglovitov (@ ins4n3 on Habré). Our testing articles are tagged with #mstesting .

Introduction

In a previous post, we looked at creating an infrastructure for cloud load testing on Microsoft Azure virtual machines. Now we will look at the load testing tool - Visual Studio Online - and compare it with the manual approach.

Load testing using Visual Studio Online allows you to automatically create and configure all the necessary infrastructure in the cloud - deploying the controller and the required number of agents with specific settings. The results of a test run always remain in the VS Online cloud database, and can be accessed at any time.


Previously, I did not use this approach for 2m main reasons:
- the infrastructure was unstable and often fell
- it was not possible to remove performance counters from the system being loaded,
so test results are limited by bandwidth data.
')
Update 3 in Visual Studio 2013 now has the ability to dynamically load the necessary performance counters from Application Insights telemetry right during load testing (more details can be found here , however, for general education only - there have been many changes since writing this article).

We will look at setting Application Insights for a simple WCF service Calculator (from the previous article), and also test it for workload using Visual Studio Online.

[ServiceContract] public interface ICalculator { [OperationContract] int Sum(int a, int b, int timeOutInMiliseconds); } 

We will also compare the new approach with the one described in the previous post.
We will need:

- Visual Studio Ultimate 2013 (Update 4),
- Visual Studio Online account
- One virtual machine in Microsoft Azure.
- Test data

Download the project used in the article .
(now the host for WCF service will be not Windows Service, but IIS).

Configuring the environment

We create through Azure the console the new virtual computer (I experimented on image of Windows Server 2012 R2). When creating a virtual machine, open port 80y (HTTP).

After successful creation of a virtual machine, we connect to it via RDP. In order to run a WCF HTTP service on our WCF virtual machine, you should put additional components there (IIS, etc.). This is done very simply:

• Open Service Manager and in the Manage menu select Add Roles and Feature.
• In the Installation Type step, select the Role-based or feature based installation option .
• In the Features step, reveal the list of features. NET Framework 4.5 Features => WCF Services and tick the checkbox HTTP Activation. The installer itself will connect the remaining dependent components of IIS, ASP.NET, and so on.
• Start the installation process

After successful configuration of the infrastructure, our WCF service should be covered.
For this:

• From the folder with test data, copy the Service folder to an arbitrary directory on the virtual machine.
• Open IIS Management Console ( Server Manager => Tools => Internet Information Service (IIS))
• Create a new Application inside the default website with a calculator service

Further, in order to make sure that everything is working correctly, you can open the service directly in your browser - http: //localhost/applicationName/Calculator.svc

A page should open with a link for generating wsdl

Configuring Application Insights

After successfully setting up a virtual machine through the Azure portal, you need to create a new Visual Studio Online account (or use an existing one).

Next, go to the VS Online management menu in the browser and click on the Application Insights link.

By default, if no applications have been added to Application Insights, the setup page for a simple test will open to check if a certain URL is available. You can skip this and click the link I want to monitor something else .
After that, for further configuration, you will need to answer a few simple questions (see screen)



Then click Click here to show instructions.

On the next page, you will need to specify an arbitrary application name and click Create (to be more precise, the name of the virtual Application Insights container where the diagnostic information will be stored).



After this, the portal will automatically generate ApplicationInsights.config (a config file with personal diagnostic settings) and the Microsoft Monitoring Agent installer.
Next, copy ApplicationInsights.config to the root folder of the Calculator service on the virtual machine and install the agent.

Please note that during installation you need to tick



After installation, restart IIS.

By default, ApplicationInsights.config is configured so that statistics will be collected from only a few basic performance counters. But we can add additional counters to this file - to do this, it’s enough to list them in a special form inside the XML section of the PerformanceCounters, for example:


In order to find out the correct name of the necessary counters, I recommend to unload them through the perfmon utility . To do this, add the necessary counters to Performance Monitor, select them and save with the right mouse button in html format.



If you open the saved file in Notepad, then you can see the correct name of the counters in this form:

Running tests

Open in Visual Studio test solyushin (from test data).
In the Team explorer menu, connect to the created account of Visual Studio Online.



Next, open the Remote.testsetting file in a solyushin and select Run tests using Visual Studio Online on the first tab.


Since we have a very simple test, you can skip the additional steps in the testsettings file. Using these steps, you can configure virtual test agents (for example, install certificates, which is important for HTTPS services, run any batch files, etc.).

Next, in the RemoteTestProject project appconfig, you need to register the correct URI that listens to the WCF Service Calculator.

Now you can proceed to configure the load test.

By default, the load test will run in 250 threads on 2 single-core agents, i.e. each agent will generate a load of 125 threads.
To increase the number of agents, use the Run Settings => Agent Count ( Total Cores ) setting . You can see some additional settings, as well as possible problems, by the link .

To connect performance counters from Application Insights , right-click on Run Settings => Get Performance Data from Application Insights and select the counters of interest.



During startup load tests, Visual Studio Online will begin to allocate resources for agents.



After the resources are allocated, the load test will begin.



Performance counters from Application Insights during the load test will be on the Application tab


After completion of the test report can be downloaded and viewed in the usual way.

Method comparison

For the experiment, I conducted load testing of this WCF service using both approaches.

In the first case, I deployed 2e single-core virtual machines and installed Visual Studio Agents on them. Put the local controller and run the test. As a result, two agents were able to generate a load of approximately 500 calls per second, while the CPU load on both agents reached 100%, while the processor load on the virtual machine with test service was approximately 15-20% (the main contribution was made by the process w3wp).

I repeated this test, but using Visual Studio Online + Application Insights (Agent total count = 2). The result was identical to the first, 2a mononuclear cloud agents generated a continuous load of approximately 500 requests per second. After waiting for the test to finish, I downloaded the report and found that the virtual agents of the CPU were also 100% loaded



The increase in the number of agent cores to 4x showed an almost linear increase in load on average to almost 800 requests per second (server load at the same time averaged 40%).
Thus, I assumed that our simple WCF service Calculator is able to withstand a load of 3000 parallel requests per second (working on a single-core machine), taking into account the fact that we also send a 1 ms timeout in the request.
In fact, everything was wrong. When I launched a load test on 10 agents (250 threads), the maximum performance was about 1000 requests per second, while the agents were underloaded (CPU load was 30%), and the server was overloaded (CPU was 100%).

I began to understand and rewrote my load test, so that I could see the increase in server CPU dependency on the increasing load. Every 10 seconds the load increased by 5 users.

At about 100 users, a maximum performance of ~ 1000 requests per second is observed, a further increase in load leads to a rapid degradation of the CPU Calculator CPU.

Conclusion

VS Online allows you to raise the cloud load testing infrastructure, the required power, almost by switching one button. And it is worth noting that this infrastructure is not inferior to the classical one (with agents and a controller). Now about the price. Application Insights is currently in Preview and is provided free of charge. One minute of a virtual user is worth 1 kopeck, so a load test of 1000 virtual users for 10 minutes will cost you about 100 rubles (regardless of one agent, you generate a load or 10).