Measuring the quality of the mobile Internet: where we are sharing a container of pink Samsung Galaxy S II
About six months ago, a container of Samsung Galaxy S II phones came in bright pink , which we immediately sent to our employees. Then a lot of interesting things happened to them: someone carried them in a car, someone wore them in a case, and some of them collected high-tech devices of this type:
Device for "information intelligence"
Why do we need all this? It's pretty simple. The fact is that in 2012, my unit was tasked with setting up a regular collection of statistics on the quality of the mobile Internet throughout the territory of the Beeline network.
')
At the beginning of the project, the introductory was this : quite often the client calls the call-center, says that he has problems with mobile Internet (low speed, no connection, etc.), but from the point of view of network infrastructure everything looks fine. In general, if there are several dozen of such situations, a mobile measuring laboratory can be sent to the site, but each of her visits is, in essence, nailing the microscope with nails.
We were assigned the task of constantly making measurements of the quality of the mobile Internet in those places where there are clusters of subscribers in order to have a representative picture of statistics.
Why do we need such measurements if there is internal monitoring of the network infrastructure? It's simple: you should always look at the network not only from the point of view of infrastructure, but also from the point of view of subscribers. That is, to simulate their usual use of the mobile Internet.
The task of my team was to organize the collection of data and provide a variety of tools for their subsequent analysis and use. We are like an encapsulated object in a huge company structure - we have been given the task, we need to give a stream of reliable data at the exit. What will happen next - a question for other units.
World experience has suggested that the most effective option in such cases is to enable customers themselves to collect information and notify about problems. Some foreign operators offer mobile applications that essentially log all technical parameters and actions on the subscriber’s device and share statistics with the operator. This is a good decision that I immediately had to give up for three simple reasons:
Our solution was extremely simple and effective: to distribute devices to company employees. Here on the scene and a container of pink phones appeared.
We have several hundred people in our company in every major city, we are constantly moving to the places where subscribers are crowded (because they themselves are), and if necessary, we can go with such a test phone to any place where there is a suspicion of problems.
It turns out sufficient representativeness (this is the most important) and good regularity. Looking ahead, I will say that in six months we have made about 4 million measurements, which would be simply impossible by other means.
So, to begin with, we determined the quality indicators to be measured. It should be not only speed. As a result, we are looking at Accessibility, Retainability, Access Time, Speed, MeanDataRate and Latency. That is, what together determines the comfort of the subscriber and, for example, the ability to check mail in seconds. They can be obtained by means of a mobile OS using specialized software for a smartphone.
Then we found, customized and implemented a mobile Android application that runs the test every 5 minutes. If at the moment of the measurement start the smartphone is registered in the 3G network - a 1.5-megabyte picture is uploaded. If, at the time of the measurement, the smartphone is registered in the 2G network, the picture is only 200 kilobytes. The picture itself is on a server in the Moscow data center, and is not cached on our media servers or other types of proxies. Regardless of the region of Russia, downloading takes place from a centralized resource using the http protocol, which ensures the same conditions and allows you to evaluate the quality of the mobile Internet service in real subscriber conditions.
The measurement results are regularly sent (https) to the technological web server (also located in the Moscow data center) and stored in a database for further processing, creating reports, graphs and other analytical information. If the smartphone went out of the access zone or lost the Internet connection, the data is written to the memory card and wait for the next communication session to be sent.
Of course, the project which was named Mobile Quality Analyzer (MQA) was not without surprises. An important point was the choice of a smartphone. As you know, the calculation of the selected quality indicators requires a sufficiently large computing power. It turned out that many modems (radio modules) of budget smartphones simply cut the speed when they rest on their ceiling. Tests have shown that the cheapest model available at the time on the market that does not distort measurements is Samsung Galaxy S II (it supports HSDPA, HSPA +, HSUPA technologies). Here we began to look for them in sufficient quantity - and found it at one retail chain, which with great pleasure made us discount on the unsold pink model. Fortunately, we didn’t care what color the future device would be.
Distributed phones by region in accordance with the number of broadcasting sectors of our network in order to have the desired representativeness.
To conduct beta testing, the phones were distributed to those who travel to the mobile measurement lab.
When conducting combat measurements, the usual picture was the presence of three phones in the car, serving base stations with connected charges, or a special bag, or a miracle device, as in the picture at the top of the topic. It was very important that all possible measurement errors (of which there are quite a lot) were smoothed out due to a multitude of tests. Cities with a large number of subscribers per day accounted for several hundred measurements - this excluded all “moon phases”.
The first measurement results were disassembled by hands, they calculated abnormal situations and looked at why they were occurring. It was necessary to introduce special data processing, which allows to find and discard the distorted results, for example, not to take into account the data with a low (less than 3%) battery charge.
All phones were preconfigured like this: the 2G / 3G network auto selection was turned on, the GPS was turned off (they were originally planning to work with it, but it’s very quickly putting the battery on), Wi-Fi was turned off. The mobile application on the smartphone also ensures that in no case is any data channel other than a mobile one used - otherwise the results would have come with inaccuracies. Also, the mobile application is able to follow the updates itself and receive remote configuration settings.
Here is a table of measurements from January 1, 2013 to mid-summer:
The servers are running RedHat OS. Apache + TomCat was chosen as Web server.
On the MQA Gate server, besides https, there is protection against DDoS attacks, plus all phones are authorized by IMEI.
Oracle 11g was chosen as the database.
What is nice, the cost - at least.
We deployed together with a contractor in the cloud, first on a test server in Germany, then in a Moscow data center by our standards, while all parts of the processing system were virtualized.
Why is that? Because we did not know whether the project would be successful or not, and we could not immediately predict the necessary computing power. It has now become clear that the project is more than successful, and will be expanded. Horizontal scaling is done very simply with the capabilities of virtual environments.
What was the result? Our team has implemented a project that, according to our data, has no analogues from other Russian operators. Now the technical divisions of the company have accurate data on the quality of the provision of mobile Internet from the point of view of subscribers.
From my own experience, it is very interesting to do such large-scale projects, roughly speaking, sitting in my room in the office. We did not even go to the fields - all the time we worked either with the application, or with the organization of the whole process, or with the server part. From the point of view of an IT project, there is a lot of useful experience, especially in terms of organizing a large number of people and working with statistics. From a merely human point of view, it’s nice that we have developed a tool that will allow us to actually improve the network over the next couple of years (even though we are not planning and optimizing the network).
Device for "information intelligence"
Why do we need all this? It's pretty simple. The fact is that in 2012, my unit was tasked with setting up a regular collection of statistics on the quality of the mobile Internet throughout the territory of the Beeline network.
')
At the beginning of the project, the introductory was this : quite often the client calls the call-center, says that he has problems with mobile Internet (low speed, no connection, etc.), but from the point of view of network infrastructure everything looks fine. In general, if there are several dozen of such situations, a mobile measuring laboratory can be sent to the site, but each of her visits is, in essence, nailing the microscope with nails.
We were assigned the task of constantly making measurements of the quality of the mobile Internet in those places where there are clusters of subscribers in order to have a representative picture of statistics.
Why do we need such measurements if there is internal monitoring of the network infrastructure? It's simple: you should always look at the network not only from the point of view of infrastructure, but also from the point of view of subscribers. That is, to simulate their usual use of the mobile Internet.
The task of my team was to organize the collection of data and provide a variety of tools for their subsequent analysis and use. We are like an encapsulated object in a huge company structure - we have been given the task, we need to give a stream of reliable data at the exit. What will happen next - a question for other units.
Search for a solution
World experience has suggested that the most effective option in such cases is to enable customers themselves to collect information and notify about problems. Some foreign operators offer mobile applications that essentially log all technical parameters and actions on the subscriber’s device and share statistics with the operator. This is a good decision that I immediately had to give up for three simple reasons:
- The code of business conduct of our company does not include the collection of such data, it is simply prohibited by ethical standards.
- The implementation for those who give voluntary consent (for example, paid or “ideological” beta testers from among subscribers) unfortunately lies in the area of ​​not only technical, but requires the connection of a large number of participants, including lawyers, as it affects the contract and personal data.
- One of the operators has already launched a similar system, but, according to our data, it was not representative, and the interest of subscribers to it, let's say, has died away.
Our solution was extremely simple and effective: to distribute devices to company employees. Here on the scene and a container of pink phones appeared.
We have several hundred people in our company in every major city, we are constantly moving to the places where subscribers are crowded (because they themselves are), and if necessary, we can go with such a test phone to any place where there is a suspicion of problems.
It turns out sufficient representativeness (this is the most important) and good regularity. Looking ahead, I will say that in six months we have made about 4 million measurements, which would be simply impossible by other means.
So, to begin with, we determined the quality indicators to be measured. It should be not only speed. As a result, we are looking at Accessibility, Retainability, Access Time, Speed, MeanDataRate and Latency. That is, what together determines the comfort of the subscriber and, for example, the ability to check mail in seconds. They can be obtained by means of a mobile OS using specialized software for a smartphone.
Then we found, customized and implemented a mobile Android application that runs the test every 5 minutes. If at the moment of the measurement start the smartphone is registered in the 3G network - a 1.5-megabyte picture is uploaded. If, at the time of the measurement, the smartphone is registered in the 2G network, the picture is only 200 kilobytes. The picture itself is on a server in the Moscow data center, and is not cached on our media servers or other types of proxies. Regardless of the region of Russia, downloading takes place from a centralized resource using the http protocol, which ensures the same conditions and allows you to evaluate the quality of the mobile Internet service in real subscriber conditions.
The measurement results are regularly sent (https) to the technological web server (also located in the Moscow data center) and stored in a database for further processing, creating reports, graphs and other analytical information. If the smartphone went out of the access zone or lost the Internet connection, the data is written to the memory card and wait for the next communication session to be sent.
Project progress
Of course, the project which was named Mobile Quality Analyzer (MQA) was not without surprises. An important point was the choice of a smartphone. As you know, the calculation of the selected quality indicators requires a sufficiently large computing power. It turned out that many modems (radio modules) of budget smartphones simply cut the speed when they rest on their ceiling. Tests have shown that the cheapest model available at the time on the market that does not distort measurements is Samsung Galaxy S II (it supports HSDPA, HSPA +, HSUPA technologies). Here we began to look for them in sufficient quantity - and found it at one retail chain, which with great pleasure made us discount on the unsold pink model. Fortunately, we didn’t care what color the future device would be.
Distributed phones by region in accordance with the number of broadcasting sectors of our network in order to have the desired representativeness.
To conduct beta testing, the phones were distributed to those who travel to the mobile measurement lab.
When conducting combat measurements, the usual picture was the presence of three phones in the car, serving base stations with connected charges, or a special bag, or a miracle device, as in the picture at the top of the topic. It was very important that all possible measurement errors (of which there are quite a lot) were smoothed out due to a multitude of tests. Cities with a large number of subscribers per day accounted for several hundred measurements - this excluded all “moon phases”.
The first measurement results were disassembled by hands, they calculated abnormal situations and looked at why they were occurring. It was necessary to introduce special data processing, which allows to find and discard the distorted results, for example, not to take into account the data with a low (less than 3%) battery charge.
All phones were preconfigured like this: the 2G / 3G network auto selection was turned on, the GPS was turned off (they were originally planning to work with it, but it’s very quickly putting the battery on), Wi-Fi was turned off. The mobile application on the smartphone also ensures that in no case is any data channel other than a mobile one used - otherwise the results would have come with inaccuracies. Also, the mobile application is able to follow the updates itself and receive remote configuration settings.
Measurements
Here is a table of measurements from January 1, 2013 to mid-summer:
Architecture
The servers are running RedHat OS. Apache + TomCat was chosen as Web server.
On the MQA Gate server, besides https, there is protection against DDoS attacks, plus all phones are authorized by IMEI.
Oracle 11g was chosen as the database.
What is nice, the cost - at least.
We deployed together with a contractor in the cloud, first on a test server in Germany, then in a Moscow data center by our standards, while all parts of the processing system were virtualized.
Why is that? Because we did not know whether the project would be successful or not, and we could not immediately predict the necessary computing power. It has now become clear that the project is more than successful, and will be expanded. Horizontal scaling is done very simply with the capabilities of virtual environments.
What was the result? Our team has implemented a project that, according to our data, has no analogues from other Russian operators. Now the technical divisions of the company have accurate data on the quality of the provision of mobile Internet from the point of view of subscribers.
From my own experience, it is very interesting to do such large-scale projects, roughly speaking, sitting in my room in the office. We did not even go to the fields - all the time we worked either with the application, or with the organization of the whole process, or with the server part. From the point of view of an IT project, there is a lot of useful experience, especially in terms of organizing a large number of people and working with statistics. From a merely human point of view, it’s nice that we have developed a tool that will allow us to actually improve the network over the next couple of years (even though we are not planning and optimizing the network).
Source: https://habr.com/ru/post/191968/
All Articles