Two approaches to performance testing. Choose
This article describes the most common approaches to application performance testing; Using analogies from “life” and examples from the author’s experience, he shows why it is impossible to do this; and, finally, trying to sow a spark of understanding the importance of load testing in the bright minds of developers, managers and other good people.
Let's start with a couple of stories.
History A.
')
The other day I talked with one programmer. He writes in C ++, works for a large company in Chicago; The software, to which creation he is involved, is actively used by financial and trading companies. We have, he says, 600 thousand lines of code in the product. It all started with a small application for analyzing stock statistics, and for 20 years this monster has been hoisted. Nice talking. It inspires respect. And how do you test it, your monster? For this, the programmer answers me, there is a special Hindu. He performs some test cases, writes reports. And before him, one manager was engaged in this, but he did more and more manual testing. New features tested, for example. Now here is a Hindu. Okay, I keep trying, this is functional testing. Do you test your performance? No, he says, if customers start complaining about slow work, then we ourselves look for bottlenecks and fix them ourselves. Who develops a product, he knows it better. Which tester can handle this?
A clever man, I thought, but speaks nonsense.
History B.
The day before yesterday I go to Habrahabr, I come across the article “History of development and optimization of one highly loaded resource” . For those who have not read, the author writes how he started working as an administrator of a certain resource. Soon this resource ceased to cope with the increased number of users and he, the author, improved server performance and reduced the load on the database several times, using some technologies. I read the phrase “The load on MySQL has decreased 4 times,” I ask a question in the comments - how did all your optimizations affect the response time for users? ”From the answer it follows that the author is more concerned not with user experience, but with the number of open connections and the number of SQL queries .
A smart man, I thought again, but I don’t fully understand why he is doing something.
These two cases describe the most typical approaches to load testing. Quite briefly, they can be described as follows:
A: “When there will be problems with productivity, then we will solve it. Themselves. We don't need any special testers. ”
B: "... And then we saw that the CPU on the server could not cope with the database! Googled the" MySQL + CPU problem ", found a wonderful library, installed it, and the load decreased N times. Now everything is fine."
Let's start with a critique of approach B (let's call it “admin's” in order to abuse the alphabet less). So, the key problem of the “admin” approach is the wrong point of view on the performance problem .
As can be seen from the story above, the admin looks through the eyes of the admin. The indicator of the problem for him is the load on the server, indicators by which he judges a change in the situation for the better - the number and speed of requests, connections, etc. The question is - how much more a person in the world sees the same problem on a particular site in the same way? The correct answer is zero. Users visiting the site have no desire to worry about indexes in the database. They are not interested in the graphics of the CPU and disk subsystem. The maximum that they can say: "This site slows down." And also: “I’ll look for something similar, but fast”. The ratio of those who measure the speed, based on the number of SQL queries and those who are only important "so that the page does not slow down" - 1 to N, where N is all site visitors, and 1 is a very smart, but lonely administrator.
... When I say all this to the admins, they usually answer me with something like "but users do not know how it all works. And we know. And then, based on the opinion of users, it is almost impossible to determine the source of the problem. ” To this I have a ready answer that works flawlessly. When you come to the doctor, what is he asking you? About well-being. Pressure and temperature are measured only later; the direction to the analyzes is not immediately given. And they start with questions like - “Where does it hurt?”, Or “When is it worse - in the morning or in the evening?”. The questions are simple, it is not always possible to give a mathematically exact answer to them, but it is for them that the symptoms of the disease are determined. And, ultimately, it is the relief of the symptoms that makes the patient most happy. "Write me something so that it does not hurt." Or, in our case - “make at least 200 requests per page, but so as not to slow down”. Of course, a good doctor will never diagnose only according to the description of the patient's state of health. But in the same way, he will never underestimate him or, which is worse, ignore him.
Total - “admin's approach” is bad, because it does not consider performance problems from the point of view of the actual users of the application or site. Servers may have this option, but clients are not particularly. However, if the number of clients is approximately equal to the number of servers ... I hope this is not your case.
In addition, in my practice, I repeatedly encountered a situation where optimization, using as input data graphs of server load and database statistics, yielded a result close to zero.
Example 1 : Oracle statistics analysis showed that a certain query will be executed twice as fast if an index is built. The index was created, the request was executed 0.5 instead of 1 second, the response time of the operation decreased from 35 to 34.5 seconds.
Example 2 : a certain operation was optimized so that when it was executed, the load on the server's CPU decreased by 30%. Unfortunately, the share of this operation among all the other ones performed by users is 0.1%, so there has been no noticeable change in the picture of the load on the server, and users have not noticed an improvement.
Etc. Returning to the doctor-patient analogy:
-Patient, you now have a normal temperature and blood composition, you are healthy.
“But doctor, I still feel bad! ..”
Now let's talk about the approach A. We will call it, as you can already guess, “developer”: the point of view voiced by my friends is typical enough for developers. “Who, except us, is able to understand our application?”, “Testing is when they check that the buttons on the form are pressed”, etc. Disregard for QA is generally one of the ills of the industry, but this topic is worthy of a separate article, we will not be distracted.
What is the main drawback of the “developer” approach? The problem is realized only after users have encountered it. In other words - until the thunder clap. Alas, it often turns out that it is too late to be baptized by this point. The system went into operation and the client will be unhappy if it turns out that something needs to be patched and docked; Team lead development went to Tibet; it is impossible to determine the cause of poor performance, because literally everything is slowly running; There is no money for a new server - the list of possible problems can be continued indefinitely. Usually, in such a situation, some of the most talented (at best) or least loaded (at worst) developers urgently create a crutch that keeps the lame system on the go until the next update / peak load for the New Year / quarterly report. After that, even more people move to Tibet. Not always on their own, ho-ho.
A small remark. In principle, as a consultant in performance and load testing, the “developer” approach usually hinders as little as the “admin” approach. When the system starts to “burn,” the words of programmers and admins - “we are fine” or “we will fix everything now” - cease to convince project managers. The decision is made to hire a fire-fighting specialist for the load, and - voila – I have a new client. So, dear programmers and administrators - keep up the good work, thank you very much. The end of the commercial break.
For those who do not want to lose customers and money, or face the need to send out your resume, I have a new analogy as a gift. You can go to the gym both to improve well-being and to prevent its deterioration. Not everyone chooses the second option, but, you see, it is preferable. Both for health and for a wallet.
In addition to the fact that it is better to avoid problems than to fight them heroically, there is another argument in favor of testing with forces, in the language of my acquaintance, “special testers”. Anyone subconsciously thinks that everything he does is flawless. Or almost flawless. Developers are no exception. "Why test if I know for sure that everything will be fine, only we will lose time." However, experience shows that a) a person from the outside is more inventive and successful in finding errors, b) a waste of time on QA turns into a big saving of both time and money in the future. In all industries, not excluding software development.
So, we considered two approaches to load testing applications: from the point of view of system administrators and from the side of developers. Both of them are imperfect, both lose sight of the most important fact: applications are written, systems are deployed, sites open primarily for users . And it is better to take care of these users in advance.
About where to start performance testing, what are the types of load testing, what methods are better to use, etc. - enough has been written about all this, and I do not set it as my task to retell everything in my own words. Instead, I plan from time to time to write about one or another aspect of performance testing, do reviews of utilities for testing, and so on. Of course, if this topic will be interesting to the audience. Thanks for attention.
Let's start with a couple of stories.
History A.
')
The other day I talked with one programmer. He writes in C ++, works for a large company in Chicago; The software, to which creation he is involved, is actively used by financial and trading companies. We have, he says, 600 thousand lines of code in the product. It all started with a small application for analyzing stock statistics, and for 20 years this monster has been hoisted. Nice talking. It inspires respect. And how do you test it, your monster? For this, the programmer answers me, there is a special Hindu. He performs some test cases, writes reports. And before him, one manager was engaged in this, but he did more and more manual testing. New features tested, for example. Now here is a Hindu. Okay, I keep trying, this is functional testing. Do you test your performance? No, he says, if customers start complaining about slow work, then we ourselves look for bottlenecks and fix them ourselves. Who develops a product, he knows it better. Which tester can handle this?
A clever man, I thought, but speaks nonsense.
History B.
The day before yesterday I go to Habrahabr, I come across the article “History of development and optimization of one highly loaded resource” . For those who have not read, the author writes how he started working as an administrator of a certain resource. Soon this resource ceased to cope with the increased number of users and he, the author, improved server performance and reduced the load on the database several times, using some technologies. I read the phrase “The load on MySQL has decreased 4 times,” I ask a question in the comments - how did all your optimizations affect the response time for users? ”From the answer it follows that the author is more concerned not with user experience, but with the number of open connections and the number of SQL queries .
A smart man, I thought again, but I don’t fully understand why he is doing something.
* * *
These two cases describe the most typical approaches to load testing. Quite briefly, they can be described as follows:
A: “When there will be problems with productivity, then we will solve it. Themselves. We don't need any special testers. ”
B: "... And then we saw that the CPU on the server could not cope with the database! Googled the" MySQL + CPU problem ", found a wonderful library, installed it, and the load decreased N times. Now everything is fine."
* * *
Let's start with a critique of approach B (let's call it “admin's” in order to abuse the alphabet less). So, the key problem of the “admin” approach is the wrong point of view on the performance problem .
As can be seen from the story above, the admin looks through the eyes of the admin. The indicator of the problem for him is the load on the server, indicators by which he judges a change in the situation for the better - the number and speed of requests, connections, etc. The question is - how much more a person in the world sees the same problem on a particular site in the same way? The correct answer is zero. Users visiting the site have no desire to worry about indexes in the database. They are not interested in the graphics of the CPU and disk subsystem. The maximum that they can say: "This site slows down." And also: “I’ll look for something similar, but fast”. The ratio of those who measure the speed, based on the number of SQL queries and those who are only important "so that the page does not slow down" - 1 to N, where N is all site visitors, and 1 is a very smart, but lonely administrator.
... When I say all this to the admins, they usually answer me with something like "but users do not know how it all works. And we know. And then, based on the opinion of users, it is almost impossible to determine the source of the problem. ” To this I have a ready answer that works flawlessly. When you come to the doctor, what is he asking you? About well-being. Pressure and temperature are measured only later; the direction to the analyzes is not immediately given. And they start with questions like - “Where does it hurt?”, Or “When is it worse - in the morning or in the evening?”. The questions are simple, it is not always possible to give a mathematically exact answer to them, but it is for them that the symptoms of the disease are determined. And, ultimately, it is the relief of the symptoms that makes the patient most happy. "Write me something so that it does not hurt." Or, in our case - “make at least 200 requests per page, but so as not to slow down”. Of course, a good doctor will never diagnose only according to the description of the patient's state of health. But in the same way, he will never underestimate him or, which is worse, ignore him.
Total - “admin's approach” is bad, because it does not consider performance problems from the point of view of the actual users of the application or site. Servers may have this option, but clients are not particularly. However, if the number of clients is approximately equal to the number of servers ... I hope this is not your case.
In addition, in my practice, I repeatedly encountered a situation where optimization, using as input data graphs of server load and database statistics, yielded a result close to zero.
Example 1 : Oracle statistics analysis showed that a certain query will be executed twice as fast if an index is built. The index was created, the request was executed 0.5 instead of 1 second, the response time of the operation decreased from 35 to 34.5 seconds.
Example 2 : a certain operation was optimized so that when it was executed, the load on the server's CPU decreased by 30%. Unfortunately, the share of this operation among all the other ones performed by users is 0.1%, so there has been no noticeable change in the picture of the load on the server, and users have not noticed an improvement.
Etc. Returning to the doctor-patient analogy:
-Patient, you now have a normal temperature and blood composition, you are healthy.
“But doctor, I still feel bad! ..”
* * *
Now let's talk about the approach A. We will call it, as you can already guess, “developer”: the point of view voiced by my friends is typical enough for developers. “Who, except us, is able to understand our application?”, “Testing is when they check that the buttons on the form are pressed”, etc. Disregard for QA is generally one of the ills of the industry, but this topic is worthy of a separate article, we will not be distracted.
What is the main drawback of the “developer” approach? The problem is realized only after users have encountered it. In other words - until the thunder clap. Alas, it often turns out that it is too late to be baptized by this point. The system went into operation and the client will be unhappy if it turns out that something needs to be patched and docked; Team lead development went to Tibet; it is impossible to determine the cause of poor performance, because literally everything is slowly running; There is no money for a new server - the list of possible problems can be continued indefinitely. Usually, in such a situation, some of the most talented (at best) or least loaded (at worst) developers urgently create a crutch that keeps the lame system on the go until the next update / peak load for the New Year / quarterly report. After that, even more people move to Tibet. Not always on their own, ho-ho.
A small remark. In principle, as a consultant in performance and load testing, the “developer” approach usually hinders as little as the “admin” approach. When the system starts to “burn,” the words of programmers and admins - “we are fine” or “we will fix everything now” - cease to convince project managers. The decision is made to hire a fire-fighting specialist for the load, and - voila – I have a new client. So, dear programmers and administrators - keep up the good work, thank you very much. The end of the commercial break.
For those who do not want to lose customers and money, or face the need to send out your resume, I have a new analogy as a gift. You can go to the gym both to improve well-being and to prevent its deterioration. Not everyone chooses the second option, but, you see, it is preferable. Both for health and for a wallet.
In addition to the fact that it is better to avoid problems than to fight them heroically, there is another argument in favor of testing with forces, in the language of my acquaintance, “special testers”. Anyone subconsciously thinks that everything he does is flawless. Or almost flawless. Developers are no exception. "Why test if I know for sure that everything will be fine, only we will lose time." However, experience shows that a) a person from the outside is more inventive and successful in finding errors, b) a waste of time on QA turns into a big saving of both time and money in the future. In all industries, not excluding software development.
* * *
So, we considered two approaches to load testing applications: from the point of view of system administrators and from the side of developers. Both of them are imperfect, both lose sight of the most important fact: applications are written, systems are deployed, sites open primarily for users . And it is better to take care of these users in advance.
* * *
About where to start performance testing, what are the types of load testing, what methods are better to use, etc. - enough has been written about all this, and I do not set it as my task to retell everything in my own words. Instead, I plan from time to time to write about one or another aspect of performance testing, do reviews of utilities for testing, and so on. Of course, if this topic will be interesting to the audience. Thanks for attention.
Source: https://habr.com/ru/post/106493/
All Articles