“It is important to prioritize”: about testing at Sberbank Technologies

If you think about which Russian companies testing can be a particularly hardcore task, Sberbank-Technologies immediately comes to mind. Firstly, there is a huge scale, secondly, a big responsibility (financial operations are not photosharing with geolocation), and thirdly, the course is set for the fastest release cycle: that is, you need to test a lot, very high quality and very fast.

How do you live with such almost mutually exclusive installations? Following the conference Heisenbag , where the company was a sponsor, we asked Anton Romanov several questions from her Quality Department - the deputy head of the corporate credit systems testing department.

- Before Sberbank Technologies, you worked at Aplana, where you also tested Sberbank systems, but from an external contractor. And how do Sbertech generally share testing tasks between the company and the contractors?
')
- Historically, contractors were initially attracted mainly for regression testing. And I worked at Aplana just when mainly regression testing was transferred to a contract, and Sbertech controlled. And now, as the amount of work increases, contractors are also involved in testing new projects.

- It is obvious to all that Sberbank-Technologies is a big company, and can you use any numbers to give an idea how large it is with respect to testing?

- The Department of Quality employs about 1,500 people, these are 38 divisions. There are about five departments that are divided according to functional competencies: Load Testing Management, Automated Banking Systems Testing Management, Frontal Systems Testing Management, Administration Administration. And there are about five departments in each department.

- When there are so many departments, it becomes interesting: how much is all this unified? Do you have strict uniform standards in testing that everyone obeys, or do departments have a high degree of autonomy?

- I would say we have a happy medium. For example, the testing organization department sets some guiding practices. Criteria are defined for them that are fulfilled or not fulfilled under certain conditions (testing at early stages, coordination of their tests with analysts, etc.). That is, the teams work on a common set of rules, but depending on the characteristics of the system and the development process, the team independently determines which practices are applicable and how they will be implemented.

Naturally, the processes are transparent and open: with active feedback, coordination and understanding of working processes.

- Can the team use some innovative practices that have not yet become mainstream, or do you always need time-tested in your case?

- Of course, we need reliability, but this does not mean that it is impossible to work with something new. For example, mind maps (mental maps) were until recently considered innovative practice. We use them, but as an additional tool. And using only mind maps instead of any other test documentation is, of course, not allowed. Thus, using something new, we calculate and level risks, we often carry out similar tests in pilot mode. And only if we see efficiency and results, such decisions are practiced and replicated.

- Do we understand correctly that the high responsibility of financial transactions affects the approach to testing?

- There, where money concerns, there is a huge responsibility - and undoubtedly, this affects our approaches to testing. The process is quite strict, with several stages of coordination - transfer from development to testing, from system testing to integration testing, from integration testing to acceptance. That is, everything is rechecked several times, there are strict criteria for the transfer of such a distribution kit, the implementation of regression testing and the quality of testing of new functionality are necessarily controlled.

- And still, probably, in your case load testing is especially important?

- Yes it is. For example: in Sberbank about 150 different systems interacting with each other. Our department has one large system, which receives approximately 500 payment documents per second per operational day. Accordingly, the volume per territorial bank is about four million documents per day. These are payments to customers, and tax and utility payments - the mass of everything.

Thus, load testing carries a very important role, because for Sberbank, the processing time of these documents is critical - they should not accumulate in queues and slow down banking operations. We pay great attention to how the system behaves after the next updates - it is very important that the level of service does not subside. Therefore, in load testing, we try to create an environment that most closely matches the industrial one.

“At the same time, you have huge amounts of already written code, which should complicate regression testing, and at the same time the goal is to speed up releases as much as possible.” What helps to cope with such a challenge?

- Good old practices help - testing based on risks.

The system that we are testing passed to us by inheritance: it was originally developed by a third-party company. The documentation remained somewhere seven years old, and the new changes were scattered in different places. That is, the first task is to understand what, where and how it works.

And the next moment is volumes. In this system, there are about 15,000 transactions; all payments flow to it - accounting reports, transactions, and everything else. Our releases have tight deadlines (about two weeks for everything), so it is important to determine what to check in the first and second turn, and to what extent. Defining priorities, we attract colleagues from the development, maintenance and Sberbank - we conduct a comprehensive dialogue with stakeholders.

It is very important to automate processes and thus save time. But all the same, there remain checks that can only be done manually, and here we use the Lead-Time metric (meaning the time from the beginning to the end of the regression) - the time it takes to perform regression testing. We use data on test run times that are stored in our instrument, and based on this we estimate how long it will take us to perform the regression, we compare the same time with the calculation by date, just the end date - the end date (two calculation methods - through actual dates of completion and start, both in terms of test volume and average execution time; so far, “fact” exceeds the estimated value). As long as we have differences and we are thinking about how to reduce the time taken to complete individual tests.

- It seems to Sbertech about a company of scale that there should be a lot of tools developed inside of such - do you encounter such in your work?

- It depends on what to say: we do not have our own JIRA, but many utilities are being developed for specific testing needs. For example, the creation of special messages in a specific format, test data generators, SQL scripts are very common.

- Earlier, Sbertech reported that DevOps is being implemented in the Quality Department - and what does this mean in practice for your department?

- With the advent of devops, practices such as Continuous Integration and Continuous Deployment appear. In the department where I work, they are not there yet, because our system belongs to the legacy class, which will be replaced by a new platform (it is already being developed as a web application). But colleagues from other departments are already working in DevOps, which really saves time and speeds up the whole process. Because everything gathers automatically, rolls in and starts at night, and you just have to see the results in the morning.

- About Sbertech it is known that he actively hires developers - and in testing, too, constantly attract new people?

- A new IT platform of Sberbank is being actively developed, which will combine a large number of systems that currently exist separately. This will be a single platform consisting of a large number of modules that will perform individual functions. But in parallel, we process applications for existing systems. Therefore, of course, we need people to perform this really large-scale task.

You are a certified tester up to the level of ISTQB Full Advanced - and how much does the gained knowledge and the fact of obtaining a certificate affect Sbertech?

- In my work, this knowledge is very useful. Often there are some tasks from the series “a business process is being designed, it is necessary to estimate in some way the number of tests in order to roughly plan the work, approximately to understand what the volume of regress will be”. Here we need test design methods, which are discussed in ISTQB. They help, having even not very detailed documentation and based on assumptions about existing risks and quality criteria, to determine the possible scope - maybe not in terms of tests, but in terms of checks.

Also a risk based strategy is very applicable to our regression model. How to assess risks, how to prioritize them, who to bring to this is all very useful and helped me in my work more than once. Well, in general, it seems to me, the knowledge obtained in preparation for certification determines the future approach to work. The installation gradually fits in my head that you cannot test everything: you need to proceed from risks, prioritize and ensure maximum quality with the available resources. With this approach to solving problems, a completely different result is obtained.