Impressions of the best presentations at the SQA Days conference

Along with other areas in IT, testing is developing rapidly. And here it is just as important to keep your finger on the pulse. One of the available ways to obtain new information is to attend conferences / seminars. And we successfully use this method.

On May 20-21, 2016, the 19th international conference “SQA Days” was held in St. Petersburg. The event was held in three streams, and over the two days of the conference, 57 presentations were made on various topics: the testing process itself, the principles of building a team of testers, professional growth and the motivation of a QA-specialist.

Reports on test automation went a separate stream. Many performances were devoted to such trends as DevOps, Continuous Delivery. We talk about the most interesting reports of SQA Days-19.

Jenkins 2.0: We organize testing as part of Continuous Delivery (Oleg Nenashev)

Oleg is a member of the core team in the open-source project Jenkins, works for CloudBees, where he develops the CloudBees Jenkins Platform. Oleg touched upon the topic of Continuous Delivery, as well as tasks arising from the implementation of this methodology.
')


With Continuous Delivery, product testing takes place in short cycles at all stages, tests are automated as much as possible. If necessary, Continuous Delivery involves a quick rollback to a previous version of the product. In this methodology, the infrastructure becomes critical for the project. As soon as a failure occurs, we immediately begin to incur losses, especially if we need to roll back the changes, and not release a new functionality. The infrastructure is changing with the product and should be tested along with it. As part of the Continuous Delivery - this, of course, short cycles, the division of the system into modules. What is needed for the organization of such testing? Parallelization of processes in the project, code versioning and assembly procedures, reproducibility of tests and assemblies, testing on a real release.

To implement Continuous Delivery, an automation system is required. Oleg spoke about the new features of Jenkins 2.0 and focused on the new technology Pipeline as code. The classic Jenkins interaction model is a web user interface. The user is required to create a new task manually, and then add details. In this regard, additional efforts are required to create and manage projects, in addition, settings for Jenkins tasks are stored separately from the project code. This approach makes it impossible to apply the best methods of Continuous Integration / Continuous Delivery. Jenkins Pipeline allows you to describe the build process in the form of code, so that the configuration of Jenkins tasks can be stored in the version control system along with the project code, track changes and test along with the project.



In conclusion, Oleg spoke about the successful use of the Pipeline in the CloudBees Jenkins Platform, and also concluded that the Pipeline as code reduces the cost of supporting automation and can be used for Continuous Delivery.

How to stop being afraid and start automating. Or not to start (Alexey Lyazgunov)

This report will first be interesting:

• testing managers who want to implement automation in their project,
• to the teams whose automation “seems to be working”, but it is not clear whether it brings results,
• as well as engineers who automate some of their actions and want to bring it to the command level.

The report turned out to be detailed, such points were touched upon: what automation is, roles in automation and its customers, what is expected from automation, why automation is expensive and time consuming, goals and expectations from automation, causes of failures. The speaker described in detail the goals of automation and conditionally divided them into good, bad and dubious.

• Reduce testing time . At first glance, the goal is good, but it all depends on what needs to be automated. For example, if this is a new functionality, the test duration will increase significantly. Therefore, you need to understand that if you need to test quickly and right now, then you probably should not be automated.
• Increase the number of checks / coverage .
• Provide more frequent checks . One of the main goals of automation.
• Reduce the influence of the "human factor" . According to Alexei, a dubious goal, but in some cases - a good goal. Indeed, sometimes it is necessary for the test to be performed in predetermined successive steps. But it is necessary to understand that quite often the tests do not catch the defects that people can detect.
• Cheaper testing . Automation is expensive, you have to pay for a new practice, and constantly.
• Introduce new types of testing . Automation allows you to do what we can not do with our hands.
• Reduce the time of release of the new release . A good goal, but at first to reach it is somewhat difficult, since there must be some basic set of tests.
• Replace manual tester auto tests .
• Get current status quickly . Great goal. For example, we want to know whether it is worth giving a build to testing, and for this we run a specific set of tests.
• Automate manual scripts . From the point of view of the speaker, this is a bad goal. Of course, it is good to automate a set of acceptance scenarios, however, if we cover auto-tests in more detail and widely, we will encounter a number of difficulties. First, manual scripts are mostly UI tests, which is the most expensive and unreliable automation. Further, different test specification, different partitioning of checks by scenarios. Sometimes it is better to use different interfaces (for example, creating a user is better through the API than filling out forms at the front end). Also, not all manual tests are automated, as well as not all possible checks have manual scripts.
• Automate regression testing . If we want to automate regression testing, then we need to understand that the pace of writing new autotests should be the same as the pace of product development. If there is a possibility, then you can set yourself such a goal, otherwise it is better to automate the regression testing only at the module or component level.
• Automation will simplify testing .

Summarizing the goals, Alexey suggests first focusing on the goals highlighted in green, and only then set “blue” goals.

Next, Alexey introduces the concept of “Three TO”: Automation is Good, Expensive, Long. This is a list of questions, but which is worth answering when introducing automation.

Automation is welcome:

• What problems do I want to solve by implementing automation?
• My expectations from automation are not delusions?
• Is there support and understanding of management and teams?
• Do I know how to integrate automation into the process?

Automation is Expensive:

• Am I willing to pay for automation?
• Am I ready to hire new people?
• Am I willing to pay for infrastructure, tools, and training?
• Am I willing to pay constantly?

Automation is Long:

1. I understand that the benefits of automation are not instantaneous?
2. I know that all participants in the development will be involved and need to change the approach to development?
3. I understand that, starting, I can not stop?
4. Is the project long enough to enjoy the benefits of automation?
5. Am I willing to spend time researching in automation?

As a conclusion, Alexey proposes to compile a checklist and, if on the basis of the information collected, the question “is it worth introducing automation or not?”, You answer “Yes”, this does not mean that you will be able to implement it. This means that maybe you will succeed. But if your answer is “No,” then, most likely, you will fail. But if you do not try, then definitely will not work.

Automation Methodology Assessment - MBT (Dmitry Himion)

Technically, the report was strong and of high quality. It was devoted to the Model Based Testing (MBT) methodology and its applicability in the framework of Agile.

In short, MBT is one of the ways to organize tests within its automated testing system. Any application is a state machine.



In the classical sense for this methodology there are no test cases, there is only a model of the system under test. There is no such thing as an automated test, test scripts are generated during the autotest run.

Dmitry compares MBT with the traditional approach to automated testing. Any state, like any transition from one state to another, is described once and used once during test generation by the combinatorial mechanism. The scope of change is about the same as that of the developers. In the case of using linear scenarios, even when we have a good modular automated testing system, when the common code is rendered, and it is necessary, for example, to add new steps, then we need to fix all the tests that use the new functionality. With the increase in the number of tests, the duration of their development and updating will increase. MBT is beneficial for flexible software development. Agile-methodology always assumes a working product, short sprints. All teams that work within the framework of Agile ideally seek to complete testing at the time of development. Considering that the generated test suite can be flexibly changed by modifying the model, and the time costs are approximately equal to the development of functionality, then with proper organization, MBT automation either keeps up with the development or, if the development increases the time, does not have time, an increase in the duration of the sprint.

Further, Dmitry described the concept of the MBT framework, the architectural scheme of which is shown below. Source - the external source from which the model is built. Test Model - a model of the system being tested, which is transferred from an external source to a programmatic form. Generator reads the model as a graph and forms paths (test sequences). Restriction tool - a mechanism for limiting the generation of test sequences. Test Manager is responsible for system performance and aggregation of results. Logger - a module for logging.



Summing up, Dmitry highlighted the positive aspects of using MBT. This methodology works to minimize the amount of code. MBT allows to provide high test coverage at the expense of combinatorics. Medotology is applicable in Agile, since the time for updating the model, in most cases, will correspond to the development time of the new functionality. Among the disadvantages of using MBT is the limited choice of tools. In addition, the modules that are grayed out in the previous figure will have to be implemented on their own.