Testing in Yandex. Himself a web-service over SSH, or how to make a stub for the whole service
You are a practicing magician manager. Or a combat developer. Or a professional tester. Or maybe just a person who is not indifferent to the development and use of systems that include client-server components. I am sure you even know that the port is not only the place where the ships come, and “ssh” is not only the sound made by the snake. And you know that services located on one or several machines actively communicate with each other. Most often via HTTP. And from version to version the format of this communication needs to be controlled.
I think each of you asked the following release with the next release: “Do we send the correct request exactly?” Or “Did we exactly pass all the necessary parameters to this service?”. Everyone should be aware of the existence of negative scenarios for the development of events along with positive ones. This knowledge should actively generate questions from the “What if ..?” Series. What if the service starts to process connections with a delay of 2 hours? What if the service responds to abracadabra instead of data in json format?
')
Such things are often forgotten during the development process. Due to the complexity of testing problems of this kind, the low probability of such situations, and a thousand other reasons. But a strange error or application crash at a crucial moment may scare the user away forever, and he will no longer return to your product. We at Yandex constantly keep such questions in mind and strive to optimize the testing process as much as possible using useful ideas. How we have made such checks easy, visual, automatic will be discussed in this article.
There are a number of long-known ways to find out how and what is transmitted from service to service - from a mobile application to the server, from one part to another.
The first of the most popular and not requiring serious training - to connect a special program to one of the sources of data transmission or reception. Such programs are called traffic analyzers or, more often, sniffers.
The second is to replace with artificial implementation entirely one of the parties. In this approach, it is possible to define a clear scenario of behavior in certain cases and save all the information that comes to this service. This approach is called using stubs ( mock objects ). We will consider both.
A new release or debugging of changes affecting inter-service communication comes. We are armed with the necessary interceptor programs - WireShark or Tcpdump . We launch traffic interception to the necessary node, imposing filters of a host, port and interface. We do “dark deeds”, initiating the communication we need. We stop interception. We begin to disassemble it. Each service has its own process of parsing, but usually it always resembles convulsive searches in a heap of text of the cherished GET, POST, PUT, etc. Found? Then we repeat it from release to release. This is now a regression test! Not found? Repeat this with different combinations of filters to understand the reasons.
You can do this manually once. Or two. Well, maybe three. And then just get bored. And on the fifth release, this communication will take and break. It is especially difficult to notice this indirectly when communication is a callback call with some kind of notification. Repetitive from release to release, mechanical actions that take a lot of time and effort should be automated. How to do it in JAVA? I am sure that in other languages ​​this can be done in a similar way, but specifically the JUnit4 + Maven bundle works perfectly and works well for testing automation in Yandex.
Suppose that we test the service integrationally, which means that most likely it looks like a battle mode when it is raised on a separate machine, and we are connected to it via SSH. We take the library to work on SSH , connect to the server , run tcpdump and catch everything we can in the file (everything is exactly like hands). After the test, we forcefully end the process and look for what is needed in the file, using grep, awk, sed, etc. Then the resulting process in the test. Did you do it? Not? And do not need!
You can do it. Simply, there are ways easier, because:
We tried this method and we are in Yandex. At first it seemed convenient - all we had to do was start and stop tcpdump via ssh, and then find the necessary substring in its output. The first problems with support began almost immediately: the order of the query-parameters was random in the required queries. I had to break the reference line into several and check the occurrence of each. The error messages were also disappointing in the event that the request was not found - tons of text did not give an adequate way to structure oneself. Headache added asynchronous requests that could appear in a few minutes after vigorous action. It was necessary to build dizzying constructions by waiting for the necessary substring in issuance with a certain delay. The test code sometimes became more complicated than the code we tested. Then we began to look for another way to test this part.
Since we are talking about testing, then, most likely, we have all the opportunities not only to put the mesh in the form of a sniffer, but also to replace one of the services entirely. With this approach, “pure” requests will reach artificial service, and we will be able to control the behavior of the destination for messages. For such purposes, there is a wonderful library WireMock . Its code can be viewed on the project's GitHub page . The bottom line is that a web service comes up with a good REST-api, which can be configured in almost any way. This is a JAVA-library, but it has the ability to run as a standalone application, it would be available jre. And then simple setup with detailed documentation.
Here you can find arbitrary response codes, arbitrary content, and transparent redirection of the request to real services with the ability to save responses and send them then yourself. Of particular note is the ability to recreate negative behavior: timeouts, communication breaks, invalid responses. Beauty! The library can also work as a WAR, which can be loaded into Jetty, Tomcat, etc. And, most importantly, this library can be used directly in tests like JUnit Rule! She will take care of parsing the request, dividing the body, address, parameters and headers. We will only have at the right time to get a list of all the visitors and meeting the criteria.
Before proceeding, you need to decide which steps you need to go through to ensure easy, visual, and automatic testing using the second option considered - the mock-object.
What are we checking?
More precisely, the scheme will look like this:
So we need to do a few things:
How to raise the service manually is described in detail on the wiremock website in the Running standalone section. How to use in JUnit, too, however described. But we will need this in the future, so I will give a little bit of code.
Create a JUnit rule that will raise the service on the correct port at the start of the test and finish after the end:
The beginning of the test will look like this:
Here we set up a raised web service so that it responds to any requested address with code 200 with the “OK” body. After some simple configuration steps, there are several options for the development of events. First - we have no problems with access to any port from the client to the machine on which the test is performed. In this case, we simply perform the necessary actions within the framework of the test case, then we proceed to validation. Second, we only have ssh access. Yet the ports are covered with a firewall. This is where ssh port forwarding (or ssh-tunneling) comes to the rescue. This is discussed below.
We need REMOTE (which is with the -R key) and, accordingly, ssh access to the typewriter. This will allow the test service to access its local port, and for us to listen to ours. And everything will work.
Since we are engaged in the automation of this process, we will consider in detail how to make the use of this mechanism convenient in tests. Let's start from the top level - the junit-rule interface. It will allow to connect the
Remembering the Ganymed SSH2 library. We connect it using maven:
(The release version can always be seen in Maven Central .)
Open an example using this library to raise the tunnel via ssh. We understand that we need four parameters. We assume that the test "talks" through his local port, so the
There are three parameters that need to be specified:
Here
It is convenient to do the junit rule as an inheritor of the
The connection itself should look something like this:
Successfully catching requests with a muffled service, it remains only to check them. The easiest way is to use the WireMock built-in tools:
A much more flexible way is to simply get a list of the necessary requests, and then, after reaching certain parameters, apply checks to them:
All of the above is a serious general approach. It can be used in many places, as a whole, and in parts. Now the use of plugs at the integration level works fine in a number of large projects to replace various functions of services. For example, we in Yandex have a file download service that records information about files not independently, but through another service. Began to download the file - sent the request. Loaded, counted checksums - another request. We checked the file for viruses, are ready to work with the file further - one more. Each next stage continues depending on the response to the previous ones, while the number of connections between the services is limited.
How to check that requests really go away and contain all the information about the file in the right format? How to check what will happen if the request was accepted, but there was no response? First, we check the positive scenario of the development of events - we replace the service writing to the database with an artificial one, accept and analyze the traffic. (The code examples above are a copy of what is happening in the tests.) The tunnel via ssh was required so that autotests, without superuser privileges, could be tied to a specific port on the local machine, the address of which is always arbitrary, and in the download service you could specify your local address and port on a permanent basis at the point of request.
Having successfully tested the positive scenario, it was not difficult for us to add checks for negative ones . Simply increasing the response delay time in WireMock to a value greater than the waiting time in the file download service, it turned out to initiate several attempts to send a request.
Having checked that in 120 seconds while waiting for the answer on the service in 60 seconds two requests came, we made sure that the file download service would not hang at the crucial moment.
This means that the developers have foreseen such a development of events, and in this place, in such a situation, information about the download will not be exactly lost. Similarly, a bug was found on one of the services. He concluded that if the service was not answered immediately, the connection remained open for several hours, until it was forcibly closed by outside monitoring services. This could lead to the fact that if there were failures in the network, the connection limit could be completely exhausted in a short time and the rest of the customers would have to wait in the queue for several hours. Good thing we checked it before!
There are a number of limitations in this approach:
In addition to the remote, there is also LOCAL (local), with the
In tests, in addition to WireMock, analogs can be interesting: github.com/jadler-mocking/jadler or github.com/robfletcher/betamax .
I think each of you asked the following release with the next release: “Do we send the correct request exactly?” Or “Did we exactly pass all the necessary parameters to this service?”. Everyone should be aware of the existence of negative scenarios for the development of events along with positive ones. This knowledge should actively generate questions from the “What if ..?” Series. What if the service starts to process connections with a delay of 2 hours? What if the service responds to abracadabra instead of data in json format?
')
Such things are often forgotten during the development process. Due to the complexity of testing problems of this kind, the low probability of such situations, and a thousand other reasons. But a strange error or application crash at a crucial moment may scare the user away forever, and he will no longer return to your product. We at Yandex constantly keep such questions in mind and strive to optimize the testing process as much as possible using useful ideas. How we have made such checks easy, visual, automatic will be discussed in this article.
Salt
There are a number of long-known ways to find out how and what is transmitted from service to service - from a mobile application to the server, from one part to another.
The first of the most popular and not requiring serious training - to connect a special program to one of the sources of data transmission or reception. Such programs are called traffic analyzers or, more often, sniffers.
The second is to replace with artificial implementation entirely one of the parties. In this approach, it is possible to define a clear scenario of behavior in certain cases and save all the information that comes to this service. This approach is called using stubs ( mock objects ). We will consider both.
Using sniffers
A new release or debugging of changes affecting inter-service communication comes. We are armed with the necessary interceptor programs - WireShark or Tcpdump . We launch traffic interception to the necessary node, imposing filters of a host, port and interface. We do “dark deeds”, initiating the communication we need. We stop interception. We begin to disassemble it. Each service has its own process of parsing, but usually it always resembles convulsive searches in a heap of text of the cherished GET, POST, PUT, etc. Found? Then we repeat it from release to release. This is now a regression test! Not found? Repeat this with different combinations of filters to understand the reasons.
From release to release?
You can do this manually once. Or two. Well, maybe three. And then just get bored. And on the fifth release, this communication will take and break. It is especially difficult to notice this indirectly when communication is a callback call with some kind of notification. Repetitive from release to release, mechanical actions that take a lot of time and effort should be automated. How to do it in JAVA? I am sure that in other languages ​​this can be done in a similar way, but specifically the JUnit4 + Maven bundle works perfectly and works well for testing automation in Yandex.
Suppose that we test the service integrationally, which means that most likely it looks like a battle mode when it is raised on a separate machine, and we are connected to it via SSH. We take the library to work on SSH , connect to the server , run tcpdump and catch everything we can in the file (everything is exactly like hands). After the test, we forcefully end the process and look for what is needed in the file, using grep, awk, sed, etc. Then the resulting process in the test. Did you do it? Not? And do not need!
Why not to do so?
I want to note that "not necessary" does not mean "it is not possible."
You can do it. Simply, there are ways easier, because:
- Parsing large lines is always a lot of specific code that is difficult to maintain.
- Parsing HTTP messages has long been done in hundreds of libraries. Why do one hundred and first?
We tried this method and we are in Yandex. At first it seemed convenient - all we had to do was start and stop tcpdump via ssh, and then find the necessary substring in its output. The first problems with support began almost immediately: the order of the query-parameters was random in the required queries. I had to break the reference line into several and check the occurrence of each. The error messages were also disappointing in the event that the request was not found - tons of text did not give an adequate way to structure oneself. Headache added asynchronous requests that could appear in a few minutes after vigorous action. It was necessary to build dizzying constructions by waiting for the necessary substring in issuance with a certain delay. The test code sometimes became more complicated than the code we tested. Then we began to look for another way to test this part.
Using stubs instead of web services
Since we are talking about testing, then, most likely, we have all the opportunities not only to put the mesh in the form of a sniffer, but also to replace one of the services entirely. With this approach, “pure” requests will reach artificial service, and we will be able to control the behavior of the destination for messages. For such purposes, there is a wonderful library WireMock . Its code can be viewed on the project's GitHub page . The bottom line is that a web service comes up with a good REST-api, which can be configured in almost any way. This is a JAVA-library, but it has the ability to run as a standalone application, it would be available jre. And then simple setup with detailed documentation.
Here you can find arbitrary response codes, arbitrary content, and transparent redirection of the request to real services with the ability to save responses and send them then yourself. Of particular note is the ability to recreate negative behavior: timeouts, communication breaks, invalid responses. Beauty! The library can also work as a WAR, which can be loaded into Jetty, Tomcat, etc. And, most importantly, this library can be used directly in tests like JUnit Rule! She will take care of parsing the request, dividing the body, address, parameters and headers. We will only have at the right time to get a list of all the visitors and meeting the criteria.
Automate checks using the stub
Before proceeding, you need to decide which steps you need to go through to ensure easy, visual, and automatic testing using the second option considered - the mock-object.
It is worth noting that each of the stages is also possible to do by hand without much difficulty.
Scheme
What are we checking?
{} in the schema:_ -> {} -> _.More precisely, the scheme will look like this:
_ -> _ :( ): {}.So we need to do a few things:
- Raise the service stub and make it accept certain messages, responding OK (or neOK - depends on the script). This will do WireMock.
- Ensure the delivery of messages to the stub service (in the diagram this
->). Let's talk about this stage separately. - Validate what has come. There are two options - using WireMock tools for validation, or by receiving a list of requests from it, applying matchers to them.
Raise an artificial web service
How to raise the service manually is described in detail on the wiremock website in the Running standalone section. How to use in JUnit, too, however described. But we will need this in the future, so I will give a little bit of code.
Create a JUnit rule that will raise the service on the correct port at the start of the test and finish after the end:
@Rule public WireMockRule wiremock = new WireMockRule(LOCAL_MOCKED_PORT); The beginning of the test will look like this:
@Test public void shouldSend3Callbacks() throws Exception { // stubFor(any(urlMatching(".*")).willReturn(aResponse() .withStatus(HttpStatus.OK_200).withBody("OK"))); ... Here we set up a raised web service so that it responds to any requested address with code 200 with the “OK” body. After some simple configuration steps, there are several options for the development of events. First - we have no problems with access to any port from the client to the machine on which the test is performed. In this case, we simply perform the necessary actions within the framework of the test case, then we proceed to validation. Second, we only have ssh access. Yet the ports are covered with a firewall. This is where ssh port forwarding (or ssh-tunneling) comes to the rescue. This is discussed below.
Reduce the road packages
We need REMOTE (which is with the -R key) and, accordingly, ssh access to the typewriter. This will allow the test service to access its local port, and for us to listen to ours. And everything will work.
In a nutshell, ssh port forwarding (or ssh-tunneling) is a pipe thrown through an ssh connection from a port on a remote machine to a port on a local one. Good instructions for use can be found at www.debianadmin.com
Since we are engaged in the automation of this process, we will consider in detail how to make the use of this mechanism convenient in tests. Let's start from the top level - the junit-rule interface. It will allow to connect the
__: -> ssh -> ____:_ before the start of the test and close the tunnel after its completion.Making a port forwarding junit rule
Remembering the Ganymed SSH2 library. We connect it using maven:
<!--https://code.google.com/p/ganymed-ssh-2/--> <dependency> <groupId>ch.ethz.ganymed</groupId> <artifactId>ganymed-ssh2</artifactId> <version>${last-ganymed-ssh-ver}</version> </dependency> (The release version can always be seen in Maven Central .)
Open an example using this library to raise the tunnel via ssh. We understand that we need four parameters. We assume that the test "talks" through his local port, so the
__ equal to 127.0.0.1 .There are three parameters that need to be specified:
@Rule public SshRemotePortForwardingRule forward = onRemoteHost(props().serviceURI()) .whenRemoteUsesPort(BIND_PORT_ON_REMOTE) .forwardToLocal().withForwardToPort(LOCAL_MOCKED_PORT); Here
.forwardToLocal() is: public SshRemotePortForwardingRule forwardToLocal() { try { hostToForward = InetAddress.getLocalHost().getHostAddress(); } catch (UnknownHostException e) { throw new RuntimeException("Can't get localhost address", e); } return this; } It is convenient to do the junit rule as an inheritor of the
ExternalResource , overriding before() for authorization and raising the tunnel, and after() for closing the tunnel and connection.The connection itself should look something like this:
logger.info(format("Try to create port forwarding: `ssh %s -l %s -f -N -R %s:%s:%s:%s`", connection.getHostname(), SSH_LOGIN, hostOnRemote, portOnRemote, hostToForward, portToForward )); connection.requestRemotePortForwarding(hostOnRemote, portOnRemote, hostToForward, portToForward); Validating
Successfully catching requests with a muffled service, it remains only to check them. The easiest way is to use the WireMock built-in tools:
// , verify(3, postRequestedFor(urlMatching(".*callback.*")) .withRequestBody(matching("^status=.*"))); A much more flexible way is to simply get a list of the necessary requests, and then, after reaching certain parameters, apply checks to them:
List<LoggedRequest> all = findAll(allRequests()); assertThat(" 1 ", all, hasSize(greaterThan(0))); assertThat(" ", all.get(0).getBodyAsString(), containsString("Wow, it's callback!")); How it worked in Yandex
All of the above is a serious general approach. It can be used in many places, as a whole, and in parts. Now the use of plugs at the integration level works fine in a number of large projects to replace various functions of services. For example, we in Yandex have a file download service that records information about files not independently, but through another service. Began to download the file - sent the request. Loaded, counted checksums - another request. We checked the file for viruses, are ready to work with the file further - one more. Each next stage continues depending on the response to the previous ones, while the number of connections between the services is limited.
How to check that requests really go away and contain all the information about the file in the right format? How to check what will happen if the request was accepted, but there was no response? First, we check the positive scenario of the development of events - we replace the service writing to the database with an artificial one, accept and analyze the traffic. (The code examples above are a copy of what is happening in the tests.) The tunnel via ssh was required so that autotests, without superuser privileges, could be tied to a specific port on the local machine, the address of which is always arbitrary, and in the download service you could specify your local address and port on a permanent basis at the point of request.
Having successfully tested the positive scenario, it was not difficult for us to add checks for negative ones . Simply increasing the response delay time in WireMock to a value greater than the waiting time in the file download service, it turned out to initiate several attempts to send a request.
// , 61 //( 1) stubFor(any(urlMatching(".*")).willReturn(aResponse() .withFixedDelay((int) SECONDS.toMillis(61)) .withStatus(HttpStatus.OK_200).withBody(""))); Having checked that in 120 seconds while waiting for the answer on the service in 60 seconds two requests came, we made sure that the file download service would not hang at the crucial moment.
waitFor(120, SECONDS); verify(2, postRequestedFor(urlMatching(".*service/callback.*")) .withRequestBody(matching("^status_xml.*"))); This means that the developers have foreseen such a development of events, and in this place, in such a situation, information about the download will not be exactly lost. Similarly, a bug was found on one of the services. He concluded that if the service was not answered immediately, the connection remained open for several hours, until it was forcibly closed by outside monitoring services. This could lead to the fact that if there were failures in the network, the connection limit could be completely exhausted in a short time and the rest of the customers would have to wait in the queue for several hours. Good thing we checked it before!
What else to say
There are a number of limitations in this approach:
- Requires ssh access to the machine.
- Port forwarding must be enabled on this machine.
- It will be necessary to stop the services if you need to take their port and replace with a stub. This means that we need the user rights to stop services without a password. This also applies to ports with numbers up to 1024.
- In some organizations, it is not possible to forward a port without administrative sanctions.
Local Port Forwarding
In addition to the remote, there is also LOCAL (local), with the
-L key. It allows the mirror described above, referring to a port on your local machine, to get to the internal port of the remote machine, hidden behind the firewall. Such an approach can be an alternative in tests running on ssh on the server being tested and calling curl, wget.Alternatives
In tests, in addition to WireMock, analogs can be interesting: github.com/jadler-mocking/jadler or github.com/robfletcher/betamax .
Source: https://habr.com/ru/post/228691/
All Articles