How is the testing of "heavy" banking software in a German company

Good afternoon, Habr. In this article I want to show the life cycle of testing the client portal developed initially for the largest German bank (Deutsche Bank) and further for the leading banks in German-speaking Europe (UBS - Switzerland, Raifeissen - Austria), as well as for other banks operating under the European EBICS standard.

First, a little background.

My name is Alexander. In 2009 I graduated from the Moscow Engineering and Physics Institute, Department of Theoretical and Experimental Physics. During my studies, like many techies, I worked actively in the field of IT: first, manual testing (Microsoft Office and Windows Vista), and then 1C (Programmer-Consultant). All this quickly bored me and I decided to turn to physics again. Having preliminarily taught English, having passed the TOEFL and GRE exams, I began to enter the USA in graduate school at PhD. I managed to send only documents to 1 (medium-sized) university, from where a positive answer came quickly enough. At the same time, my friend, who already found a PhD position in France, recommended me to write an announcement on the European job site for young scientists - and I unexpectedly received 2 invitations from Germany. As a result, my choice fell in favor of Deutschland. Let me list the main factors that determined my choice: the proximity of the country to Russia (2-hour flight Berlin-Moscow), a long vacation compared to the USA, PhD lasts an average of 3-4 years instead of 5-6, plus I liked German, thanks to Rammstein.

PhD flew fairly quickly (just over 3 years). Despite some achieved results and 4 published articles, I didn’t see myself further in science due to: the unpopularity of my topic (solid state physics), very high competition in the scientific community (30 postdocs on average for 1 professor’s place) also the constant need to move around the world in search of positions. After studying in Germany and learning more or less German (the labor costs for studying according to my estimates are 2 times higher than English), I decided to turn to the local labor market (my doctoral degree was defended as obtaining German education, that is, I could live in Germany and look for a job within 1 year). However, it turned out to be difficult to find a job: in many cases I was “retrained”, I also did not have certificates and local experience. I did not move to purely developer positions, because I could only program a little on Python, and 1C is still a bad start, especially in Germany. As a result, the choice was narrowed enough: Data Analysis or, bearing in mind their experience, testing as a first step to jump into the IT sphere. Despite more than a hundred resumes sent out, the job search lasted more than six months, possibly also because there are few new employees in the summer and autumn.
')
As a result, in the winter of 2015, I got a position in an average-sized IT firm in Hamburg (about 500 employees throughout Germany, 200 in Hamburg). Then I issued the Blue Card (there was recently a post about this so-called Blue / Blue Card: by the way, Germany issued 87% of all such cards from the entire European Union, which indicates the relative availability of the labor market for foreigners and the fact that there is a job) which will allow early next year to get a residence permit. Registration is very simple, it is enough to bring a copy of the contract and the German certificate (level B1).

From the very beginning, I was thrown on a new project: a client portal, then KundenPortal , working according to SEPA standards.

It is important to mention that KundenPortal works in tandem with another product of our company: BankRechner , aka Bank Clearing Center: a program that accumulates all payments, including from client portals of other companies, and carries out subsequent calculations for analysis and tax reporting. The “heaviness” of the software results from the fact that the largest banks work with many thousands of clients, each of which can enter a huge array of documents into the system. Naturally, all this works on very powerful servers (Unix), with terabyte databases (Oracle). As a result, load testing in this case is critical.

The project originally submitted to the client (almost 2 years ago) had only basic functionality. Then began an active interaction between the managers of our company and client banks. Clients make their wishes in the form of so-called CR s (Change Requests), which describes the required functionality. Naturally, this is not just a Wishlist, but everything is coordinated by competent people in the field of Zahlungsverkehr (Payment Transport / Transactions) by people from both sides based on the EBICS standard and with the development of the legal side and security concepts. These aspects are not in my area of ​​competence, so I will leave them outside the brackets.

Further, these CRs (from different clients) are accumulated into a single list. The execution of this list determines the iteration (the same version of the product). In our case, the typical version cycle lies between the 3rd and 6th months. Based on this list, the technical project manager creates a Concept , where he describes how it should be implemented and divides everything into topics that programmers choose in the so-called “Fair of topics” (usually 2 programmers fall into 1 topic for backend and experience sharing, At the completion of the topic, they are again mixed up and reunited in pairs. Programmers based on the Concept and consistent GUI templates ( Mockups ) implement everything in code. When the code is ready for some acceptable percentage, the topics are distributed among the testers. Since there are fewer testers, we get several topics, also by choice. The idea is similar, as in the case of developers: we need to know the maximum functionality and insure each other. On the one hand, such a method is difficult in terms of knowing everything about the program and being able to switch quickly, but on the other, it doesn’t get boring, you constantly learn something new (even about finances and national banking systems), and you can also look at testing problems from different angles, which increases efficiency and even intuition when searching for bugs. In this iteration, I have already about 10 topics.

At the initial stage in the project there were about 10 developers and 3 testers (and 2 of them just came to work, another tester, however, had an IT education). Now there are about 30 developers and 6 testers in the project, plus incoming students who perform manual testing in various browsers. The only tester with experience (about 15 years already) was Frau Frauke (that name), which became our Timlid. She quickly brought me up to speed, despite my initial problems in German, as well as my lack of experience in automated testing, which will be discussed. The main language of the company is German, primarily due to the national composition: I was one of the first foreigners in the office. Therefore, screenshots will be shown in German with explanations if necessary. So let's go.


From the point of view of methodology, it is impossible to say specifically what we have. Such a hodgepodge of the V model, Agile, etc., and the cycles are approximately as follows: preparation, development of test cases, implementation of test cases, execution of tests and detection of bugs, repeated and regression testing.

To give an idea of ​​what a bank client portal is, I will show a screenshot of the old version (sorry, copyright). Now there are already a bunch of menu items, depending on the licensing option. Themes and colors of the interface are usually tailored to the general style of the client bank.



This shows the introduction of a certain type of payment data (DTAZV). Payment is entered in the section having the same name, which in turn is in the Erfassung section (entering / registering data in the system). The subsection above is called Kontoinformationen (Billing Motions). Below are the EBICS features.

All interfaces are designed in the same style and are united by similar logic in terms of controls, the possibility of repeating the same action in different ways, and so on. The program allows you to track the full life cycle of paperwork, including: their very design, sending to the bank, receiving a response and the status of documents, analyzing the state of bank accounts, generating and uploading reports. All this, of course, in compliance with the rules and formats of the Eurozone and the national characteristics of each bank, plus seasoned with cryptography and encryption. This leads to a greater number of settings, licensing options, database options, software options on servers, as well as a brain-feeding system of administrative and other roles (at least 3 levels of administrators with several subspecies at each level and a couple of hundred rights and roles which in turn are divided into levels of applicability).

The selected type of payment document: Auftragsart (AZV) is one of dozens of possible options for this type of payment only (DTAZV), so we have a wide field for combinatorics. Naturally, in such a case, automatic testing becomes paramount, the main phases of which, as well as the preparation and accompanying software, I will describe.

Training

Before programmers start the first modules ready for testing, our department is not idle. We test already released previous versions (which may be different for different clients) primarily through automated Smoke tests (when basic functionality is tested daily).

Next, for the new iteration (the recall cycle lasts an average of 3-6 months), we need to prepare virtual machines for testing, install the necessary software (new versions of browsers, plug-ins, etc.), read and analyze the Concept, estimate what is important for testing, find possible weak points, generate the first test data, or prepare the received data from the client.

Design Test Cases

Here we are fairly conservative: we form all our ideas into test cases, trying to cover the main aspects of testing and briefly enter them into an Excel document with notes and questions. Then either a personal meeting is organized (about 1 hour), or a video conference (we partially have a team located in another city) between the project manager, the testing team leader, programmers and testers who are related to the topic. The task of the tester is to present this document, explain what and how it is going to be tested. If necessary, we accompany the presentation by viewing the GUI mockups (Mockups) or launching the program itself. Answers to questions, explanations from both sides, instructions of programmers to important or vulnerable places, suggestions and suggestions are fixed and it is time to form test cases in reality.

Implementation / Programming Test Cases

She / it consists of 2 main parts:
a) A specific description of test cases in the TestLink program, that is, in fact, documentation.
b) Automation of test cases in the QF Test program .
Description again in German, please do not judge strictly.



As you can see: there is a splitting into versions and themes, at the moment we have almost 700 test cases.

Previously, the well-known HP Quality Center was the main product for testing at the company. Now the standard has become QF Test>, which essentially belongs to the Captcha-Replay family, that is, it is possible to record user actions in the program (click, enter data into the GUI, etc., and reuse it). A certain set of actions is collected in the procedure. Typically, interface elements have their IDs, which we use to refer to an element. IDs can also be dynamically generated depending, for example, on the line number in the list of items. In this case, in order to gain access to the elements, we use regular expressions as well. Test cases themselves are already formed, as a rule, from procedures like a constructor. Here is the interface of this program:



Testfallsatz (group of test cases) is a reflection of what we are testing (from top to bottom): Currency balances, Account movements for Austria, Self-administration, the Category Assignment Codes and CFONB (French Standards). We have several test cases in each topic, which in turn may contain a training unit (Vorbereitung), test steps, sequences, and most importantly, a call to the executable procedures themselves. Below we can track how variables change during execution.

Thus, the main task of the tester at this stage is to design executable procedures, which should work quickly and cover the main paths used by end users in the graphical interface, and form them into test cases on the basis of an Excel document / presentation agreed upon. Sometimes clicking on the interface for too long or it does not make sense to do this in each procedure, then we can use scripts or auxiliary Java classes to, for example, create a quick user or bank account.

The big advantage of QF Test is the ability to write scripts either in Jython (Java + Python) or in Groovy. Personally, I only write in Jython, since I like Python, which is very similar to it. Only now I will mention that KundenPortal consists of 2 parts: a GUI and the so-called Web Services: shells, where many actions can be performed without an interface, which is much faster than clicking in the program itself. That's just a piece of the script, so as not to anger advanced programmers, where a user is created directly in Web Services.

Test case execution, search and error logging

The construction and execution of test cases occurs simultaneously, until we receive their ready, stable version. Stability here means that the test case will be automatically executed to the end and only an error or some technical problem can interrupt its execution. We found the errors found in the error management system, Trac :

Error Correction Testing and Regression Tests

You can see that more than 3000 have already been found, this is despite the fact that programmers are also testing before passing the baton to us. Such a high number is primarily due to the many different formats and sufficient complexity and versatility of the interfaces (despite the fact that they, as I said, are built on a similar principle), and, of course, our attentiveness and corrosiveness =) So far we manage catch almost all the bugs. As far as I know, clients have so far found only once something relatively serious in the testing stage already by them.

After programmers fix bugs, we re-run our automated tests, including leaving work at night. In the morning you can already analyze errors and problems. We also perform regression tests, that is, we retest places where errors might not have been detected, but changes in the code could lead to changes in the operation of the program, including the appearance of bugs.

And finally, I will show the general working environment:



The project itself (KundenPortal) and BankRechner, with which it interacts, is located in the Eclipse Workspace with the version control system (SVN), where we ask:

a) Java Auxiliary Classes. We program them ourselves (maybe not by Feng Shui, but they perform their task), which are called from QF Test scripts.
b) Current parameters of test virtual machines: at a time, only one KundenPortal and one BankRechner are possible (configfig).
c) CSV files (the example on the right is a list of users), which contains information about users, customers, accounts, payment templates, and so on (again for both systems), aka test data (completely generated by us, but having the correct parameters, or intentionally incorrect for negative testing).
d) Test Suites, actually QF Test files (collections of test cases or procedures for execution): common (for a specific version or theme) and individual for each tester, so that you can safely deal with your own topic and not make an extra number of commits to common Test Suites .
e) Some scripts, where, for example, it is determined which Test Suites will be launched, the launch configuration, partitioning the hard disk into subsections, etc.

After all the tests are in the green zone and the iteration is successfully completed, we will organize a joint meeting (all programmers and testers), where there is a general discussion: what was good, what was not, what difficulties and problems were during the project. Every employee should at least talk about 2 positive and 2 negative aspects. Based on this, we adapt the construction of the workflow for the next iteration. Also, based on the experience gained, we fill in our internal Wikipedia, which describes the main steps for setting up systems and Best Practices:



Well, a couple of words about the nature of the work.

Working time, human resources and iterations are so planned and distributed that at the moment we managed to come to the finish line at the same time with the developers (up to a day): that is, from our side all the tests in the green zone, from their side all the functionality ready, all bugs fixed, all ToDos made. The most important thing is that this time always coincided with the promised client. , , -. , , , . , 40 – , , ( , 2 ). 8-9 17-18. 30 , 6 ( ), 1 , 2 ( !), 2 . IT , : , , .

, ( KundenPortal ). Deutsche Bank 15 ( 100 ). ( , ), .