We make our firmware for an IP camera on Rust and fight fraud

This is a continuation of the story about the best ( in the opinion of the participants ) HighLoad ++ 2017 reports. In a previous article I wrote about the two leaders in the rating. Go ahead. In this material - the report of Vadim Antonyuk about fraud and Max Lapshin - about the firmware for the IP camera on Rust.

Fraud in mobile applications: Vadim Antonyuk (4.73 points)

Honorable third place. Vadim is not the first time talking about the anti-fraud on Highload ++. A year earlier, he talked about the anti-counterfeit in RTB-systems in general, this time "got" mobile applications. Looking ahead, I will say that there are still disputes about what to consider fraud in mobile applications, unlike the web, where everything is more or less clear. Nevertheless, the segment itself is growing, and with it the problem. So, it's time to start solving it - “not yet started.”
')

In the meantime, if you do not know or still doubt that RTB is Haylod, how do you figure 400 billion requests per day? At the same time, according to statistics, the share of mobile applications accounts for about 40% - more than on the desktop web. IPONWEB (one of the major players in the market of RTB advertising) for a long time and successfully detected fraud in the web segment, but they only got to applications.

So. Classical methods (bots, AdStackng, Domains Spoofing, Ghost sites), which have long been used in the classic web, do not work in mobile applications explicitly. As a result, methods of dealing with them are not suitable, and something new needs to be invented.
By the way, the industry has not yet agreed on what to consider fraud in mobile applications. It would seem that stories with ads in mobile phones are already decent, but the fact remains a fact, and Vadim speaks about it.

The report discusses two (and in fact almost three) approaches to detect fraud:

• Backgroundness - display ads in the background. The essence of this type of fraud is in its name. That is, there is advertising, but the user does not see it. How to find her? Everything is not very difficult, but there are nuances. The essence of the original method is to search in a stream of incoming requests for all requests from the “application + user” pair, which send requests continuously. Such intervals are calculated and the analysis for realism is done.
  
  For example, on the slide below, the T1 signal flow with the interval between the signals t1 can be quite long, 10 hours, which almost certainly indicates that it is fraud. After all, the user can not look at the screen for 10 hours without a break. If you collect statistics on this application, it is possible to calculate the villains with a high degree of probability. But, as usual, there is a nuance - the method depends on at least two parameters, and both of them are calculated empirically.
  
  
  
  Can it be made easier? It turns out that you can only rely on the value of T (the period of continuous signal emission) and analyze the intervals between adjacent time intervals (in our example, the desired interval between T1 and T2. If there is no realistic break during the day (let's say 6 hours), then Most likely, something went wrong. As a result, with a large number of measurements from a single application, the accuracy of the method increases.
• The second method is Entropy Score. Search for scoundrels through abnormal entropy. To make it clearer, I recommend pre- reading the wiki . The method is based on a sequential numerical analysis of user and application behavior. The Entropy Score is calculated, due to the large number of measurements, the allowable interval for the value of this parameter is determined — when, with a high degree of probability, the behavior of the application and user are real — and all other applications are analyzed.
  
  A slightly more complicated method is used in IPONWEB as a supplement to the first one. It allows you to catch applications that have a small number of users, even though there is no continuous stream of requests. As an example, are applications for artificial viewing of advertising.
• The third method is not explicitly highlighted. It analyzes a large number of requests from a single device. In fact, this is an analogue of a bot on the web. So if you go back to the beginning of the conversation, Vadim was a little cunning when he said that fraud analysis methods on the web are not suitable.
  
  
  
  As a result, due to the use of a combination of fraud identification methods, the guys from IPONWEB filter out more than 15% of the traffic. Very solid indicator, taking into account the total number of requests. All methods are probabilistic, and there is always the likelihood of error.
  
  Dialogue on this topic is constantly being conducted when, as a result of the analysis, good applications are cut, and therefore the system is constantly being adjusted and adjusted. However, it must be understood that bad traffic still remains. Returning to the question of the use of certain methods in the web and mobile applications, Vadim noted that in a number of cases nothing prevents them from combining them and getting good results.

If you are interested in this topic, you can look through the links under the spoiler or simply google the terms that Vadim mentions. The only thing that is a bit annoying is the lion's share of information on this topic is semi-advertising articles in exchange for mail from teams that deal with both advertising and anti-fraud.

Making our firmware for an IP camera on Rust, Max Lapshin (4.70 points)

Max is a regular participant of IT events, a member of the Highload ++ program committee, the author of very fast and productive video streaming solutions that are used all over the planet, an erlang expert and just a great speaker. His story was not included in the top three, but it is necessary to listen to him - if only because his company is one of the few in Russia that is successfully promoting its software product on the world market.


Directly about the streaming and its nuances, Max told at past conferences. This time it was about how to change software on IP cameras, how Rust appeared in this story, and what came of it. The report has two parts: the first is directly about the specifics of working with IP cameras, the second is devoted to Rust and his features.

Despite the fact that IP cameras are widespread, cheap, and the hardware platform every year becomes more reliable, the software for IP cameras leaves much to be desired. That is, together with the camera, you get 10-12 year old software with all defects, lack of features, and most importantly - the inability to make changes. As a result, it is most reasonable to write your software in order to get a set of necessary features and reduce the complexity of working with the device. By the way, there was also an option to make my own camera - but it was abandoned due to its high cost and duration, as well as the need to work with existing cameras, of which there are millions.



In fact, an IP camera is a computer with a processor, memory and other nodes, and therefore it is possible to work with it, for example, by analogy with mobile phones. Just like in mobile phones, there are a lot of options for running internal software: somewhere you can write something, only read somewhere, there is a different U-boot (a powerful bootloader with a bunch of functions needed for uploading) with different software for rewriting, there is a large set of different plumes, tongs and soldering irons. Well, as a result, all the delights of flashing your favorite android get out: the power jumped - the camera turned into a brick, made a mistake in the firmware - again a brick. Fortunately, the cost of such a "brick" is currently in the range of ten dollars apiece.

To make your life easier, you can, as soon as you manage to get documentation on the camera and hack it, put software on it, allowing you to make further updates in a more familiar way.

After assembly, you can safely proceed to the firmware. It turns out that what you used to think of as a “zoo” of technologies is babbling compared to when it comes to cameras. Different versions of the kernel, libraries, SDK - the absolute norm for them. The norm should be considered the complete lack of the ability to drive tests without pouring onto the camera, a severe lack of space (8 megabytes, yeah), as well as constant work with hardware and accurate memory management. To summarize - the camera is very interesting and at the same time a cheap device (you can safely experiment and turn cameras into bricks), with which you can interact very well, achieving your goals.

The second part of the story is entirely devoted to Rust and its features. An attentive listener will definitely note for himself the reasons why Rust was chosen, and not the good old C.

The following two slides show the results that were achieved when using this not very common language for working with iron.





I will not dwell in detail, I will only note that Max explains to the HYIP lovers very intelligibly that you have to pay for everything, and Rust is no exception. In conclusion, it is noted that, despite the external complexity, it is possible to use Rust systems for semi-embedded systems, which is fresh and cool, although you will have to reformat your brain a little.

From myself, I note that many of the things that Max talks about are characteristic of developing most embedded systems. If you are aiming at this direction - be sure to listen.

---

Soon you will see four more reports. If you yourself are ready to share experience or interesting cases in the field of Hyloca, management and teambuilding, programming or mobile development, we invite you to become the speaker of the corresponding section of RIT ++ 2018. Applications that can be left here are already open in the program committee. Perhaps your report will be the best this year.