End-to-end testing of microservices with Catcher

Good day! I would like to present a new tool for end-to-end testing of microservices - Catcher

Why test?

Why do you need e2e testing? Martin Fowler recommends avoiding it in favor of simpler tests.

However, the higher the tests are - the less rewritten. Unit tests are almost completely rewritten. Functional tests also have to waste your time in the event of a serious refactoring. End-to-end tests should check business logic, and it changes least of all.

Moreover, even full coverage of all microservices with tests does not guarantee their correct interaction. Developers may incorrectly implement the protocol (errors in the name / data type).

Or implement a new functionality relying on the data schema from the documentation, and on the product environment get a surprise in the form of schema mismatch errors: a mess in the data or someone forgot to update the data schema.

And the tests of each of the involved services will be green.

Why automatic testing?

Really. At my previous place of work, it was decided that it was too long, difficult and expensive to spend time on deploying automatic tests. The system is not large (10-15 microservices with a common hitch). CTO decided that "tests are not important, the main thing is that the system worked." Tested manually on multiple environments.

What it looked like (general process):

1. Agree with other developers (rollout of all microservices participating in the new functionality)
2. Roll out all services
3. Connect to remote kafka (double ssh in dmz)
4. Connect to k8s logs
5. Manually create and send a message (thank God json)
6. Watch the logs, trying to figure out whether it worked or not.

And now a little tar in this barrel with chocolate: most of the tests needed to create users, because it is difficult to reuse existing ones.

Firstly, due to the fact that the system is distributed - several services had their own databases, which contained information about users.

Secondly, Kafka was used for permanent data storage. Those. even if the information is deleted / changed in the database, the service will still read it back on restart.

How did the registration of the new test user look (approximately):

1. Enter any data (name, mail, etc.)
2. Enter personal data (address, phone number, any tax information)
3. Input of bank data (actually, bank data)
4. Answer 20-40 questions (do you already feel pain?)
5. Get IDNow identification (on dev environment, thank God it was turned off, on stage it's around 5 minutes or more, because their sandbox is sometimes overloaded)
6. At this step, opening an account in a third-party system is required and you can’t do anything through the front-end. You need to go through ssh to the kafka and work as a mock server (send a message that the account is open)
7. Next you need to go to another front-end in the personal account of the moderator and confirm the user.

Great, user is registered! Now a little more tar: for some tests you need more than 1 test user. And sometimes from the first time tests do not pass.

And how is the verification of the new functionality and confirmation from the business team?
All the same need to be repeated in the next environment.

Do I have to say that after a while you begin to feel like a monkey, which only does what presses the buttons, registering users.

Some other developers (usually the front-end) had problems connecting to the kafka. And with a bug in the terminal with a line of 80+ characters (not everyone knew about tmux).

Pros :

• No need to configure / write. Test directly on the running environment.
• does not require high qualification (cheaper specialists can do)

Cons :

• takes a lot of time (the farther, the more)
• usually only new functionality is tested (it is not clear if the existing one is broken)
• Often skilled developers are busy in manual testing (expensive specialists do cheap work).

How to automate?

If you read up to here nodding your head and saying: “yes, an excellent process, guys know what they are doing,” then you will not be interested further.

Self-made e2e tests are of two types and depend on which of the programmers was more free:

• the backend that lives in your test environment. It contains the logic of testing, which is twitching through endpoints. It can even be partially automated due to its interaction with CI.
• The script, with the same wired logic. The only difference is that you need to go somewhere and from there start it. If you trust your CI, then you can even run it automatically.

Sounds good. Problems?

Yes, such tests are written on what the person who writes them knows. Usually these are scripting languages ​​like Ruby or Python, which allow you to quickly and simply write this kind of thing. However, sometimes you can stumble upon a bunch of bash scripts, C or something more exotic (I spent a week rewriting the bike on bash scripts to a python, because the scripts were no longer extensible and no one really knew how they work or what they test) .
Sample project here

Pros :

• automation

Cons :

• additional requirements for the qualifications of developers are possible (if they are developed in Java, and the tests were written in Python)
• writing code to test written code (who will test the tests?)

Is there anything ready?

Of course, just look in the direction of BDD . There is Cucumber , there is Gauge .

In short, the developer describes the business scenario in a special language, then it implements the script steps in the code. The language is generally human readable and it is assumed that it will be read / written not only by developers, but also by project managers.

The scenarios along with the implementation of the steps are also in a separate project and run by third-party products (Cucumber / Gauge / ...).

The script looks like this:

Customer sign-up ================ * Go to sign up page Customer sign-up ---------------- tags: sign-up, customer * Sign up a new customer with name "John" email "jdoe@test.de" and "password" * Check if the sign up was successful 

And implementation:

 @Step("Sign up as <customer> with email <test@example.com> and <password>") public void signUp(String customer, String email, String password) { WebDriver webDriver = Driver.webDriver; WebElement form = webDriver.findElement(By.id("new_user")); form.findElement(By.name("user[username]")).sendKeys(customer); form.findElement(By.name("user[email]")).sendKeys(email); form.findElement(By.name("user[password]")).sendKeys(password); form.findElement(By.name("user[password_confirmation]")).sendKeys(password); form.findElement(By.name("commit")).click(); } @Step("Check if the sign up was successful") public void checkSignUpSuccessful() { WebDriver webDriver = Driver.webDriver; WebElement message = webDriver.findElements(By.className("message")); assertThat(message.getText(), is("You have been signed up successfully!")); } 

Complete project here

Pros :

• business logic is described in human readable language and is stored in one place (can be used as documentation)
• ready-made solutions are used, developers only need to know how to use them

Cons :

• managers will not read and write scripts
• you have to keep track of both the specifications and their implementation (and this is writing code and editing specifications)

So why then the catcher?

Of course, to simplify the process.

The developer writes only scripts in json / yaml, and Catcher executes them. The script consists of sequentially executed steps, for example:

 steps: - http: post: url: '127.0.0.1/save_data' body: {key: '1', data: 'foo'} - postgres: request: conf: 'dbname=test user=test host=localhost password=test' query: 'select * from test where id=1' 

Catcher supports jinja2 patterns, so you can use variables instead of wired values ​​in the example above. Global variables can be stored in inventory files (as in an edible), pull up from the environment and register new ones:

 variables: bonus: 5000 initial_value: 1000 steps: - http: post: url: '{{ user_service }}/sign_up' body: {username: 'test_user_{{ RANDOM_INT }}', data: 'stub'} register: {user_id: '{{ OUTPUT.uuid }}' - kafka: consume: server: '{{ kafka }}' topic: '{{ new_users_topic }}' where: equals: {the: '{{ MESSAGE.uuid }}', is: '{{ user_id }}'} register: {balance: '{{ OUTPUT.initial_balance }}'} 

Additionally, you can run verification steps:

 - check: # check user's initial balance equals: {the: '{{ balance }}', is: '{{ initial_value + bonus }}'} 

You can also run some scripts from other scripts, which has a great effect on cleanliness and reuse of the code (including starting only parts of the steps through the tag system, deferred launch, etc. buns).

 include: file: register_user.yaml as: sign_up steps: # .... some steps - run: include: sign_up # .... some steps 

Inserting and using scripts can solve the problem of waiting for a resource (wait for the service while it starts).

In addition to ready-made built-in steps and an additional repository , it is possible to write your own modules on python (simply by inheriting ExternalStep ) or in any other language:

 #!/bin/bash one=$(echo ${1} | jq -r '.add.the') two=$(echo ${1} | jq -r '.add.to') echo $((${one} + ${two})) 

and use:

 --- variables: one: 1 two: 2 steps: - math: add: {the: '{{ one }}', to: '{{ two }}'} register: {sum: '{{ OUTPUT }}'} 

Scripts are placed in the docker file and run through CI.

This image can also be used in Marathon / K8s to test an existing environment. At the moment I am working on a backend (analogue of AnsibleTower) to make the testing process even easier and more convenient.

Pros :

• no need to write code (only in case of custom modules)
• switching environments through inventory files (as in ansible)
• you can use your own modules (in any language, even sh)

Cons :

• not human readable syntax (compared to BDD tools)

Instead of conclusion

When I wrote this tool, I just wanted to reduce the time I usually spend on tests. It so happened that in every new company you have to write (or rewrite) such a system.

However, the tool turned out more flexible than I expected. If someone is interested in an article (or the tool itself), I can tell you how to use Catcher for organizing centralized migrations and updating microservice systems.

Upd

As I pointed out in the comments, the topic is not disclosed.
I will try to indicate here the most controversial theses.

• end-to-end tests are not unit tests. I already referred to M. Fowler in this article. Unit tests are in the test backend project (standard tests directory) and run every time the code changes to CI. And e2e tests are a separate project, they usually run longer, test the interaction of all participating services and do not know anything about the code of your project (black box).
• do not use the Catcher for integration (and below) tests. It is expensive. It is much faster to write a test on your PL for the current backend. You need end-to-end tests only if your business logic is spread over 2 or more services.
• Catcher is also BDD. From my point of view, the main advantage over Gauge / Cucumber is ready-made modules and the ease of adding them. Ideally, only the test is written. In the last company I wrote all 4 tests on standard components, without programming anything. Accordingly, the requirements for qualification (and the price of such a specialist) will be lower. You only need to know json / yaml and the ability to read specifications.
• For writing Catcher tests, you will have to study Catcher-DSL. Alas, it is true. At first I wanted to make the tests themselves written directly from the microphone. But then I thought that they would then dismiss me as unnecessary;) As mentioned above, the Catcher DSL is a standard json / yaml and specification of steps. Nothing fundamentally new.
• You can use standard technology and write something of your own. However, we are talking about microservices. This is a large number of different technologies and PL and a large number of teams. And if for the java-command junit + testcontainers is the obvious choice, the erlang command will choose something else. In a large company with 30+ teams at the top, they will decide that all tests should be given to the new infrastructure / qa team. Present how they will be delighted to this zoo?
• If you have 4-5 e2e tests, then you can write everything in any scripting language and forget about it. However, if over time the logic will change, then in 2-4 years you will have to refactor, passing directly the business logic of the tests and the implementation of access methods to the components under test. So in the end you will write your Catcher, just not so flexible. It took me 4 implementations to understand this;)