Magic formula or how to see the threat
Every system works according to a unique algorithm, without an algorithm it is not a system. Flexible, rigid, linear, branching, deterministic, stochastic - it does not matter. It is important that in order to achieve the best result, the system obeys certain rules. We are often asked a question about the algorithms of our product, in particular: how do we manage to better competitors to calculate future threats? For obvious reasons, all the details of this magic formula cannot be revealed, but you can slightly open the door of our technological kitchen and learn something.
In general, we use the deductive method, that is, we go from the general to the particular. In our case, it could look something like this: all malware does this -> this file does this too -> therefore, the file is malicious. But in practice, not everything is so simple and smooth.
First of all, it is impossible to identify specific actions that clearly indicate the harmfulness of the object. The classic example with access to the MBR is that everything that pulls this command malicious cannot be considered, there are many other applications for peaceful purposes. The same applies to all other actions. After all, in the original, all the teams were created to perform some useful things.
In other words, the task “to separate the wheat from the chaff” is irrelevant here. Another thing is to estimate the proportions and composition of grains and chaff, estimate the same parameters in the adjacent barn, analyze the results and then make an informed decision about setting a comma in the phrase “treat not to be missed”. To do this, we use technology with a straightforward name SR (Security Rating) - a sort of spreading self-learning system of weights, which helps to better understand the true nature of an object in the process of its formal evaluation and emulation .
')
The analysis takes into account the composition and density of the events generated by the object and its external attributes (name, size, location, compression, etc.). Based on a set of rules, each of the parameters receives an individual hazard rating (0-100%). The first package of rules (and there are more than 500 of them now) was the result of a “manual” study of more than 80 thousand unique malware of various families. Now the rules are developed mainly automatically, and expert analysts only “twist” self-learning algorithms.
For ease of testing and maintenance, the rules are divided into groups (for example, the Internet, Passwords, Registry, etc.) and if the object is “inherited” by appropriate behavior in one or more of them, appropriate sanctions are taken to it.
Examples of the simplest rules:
The rule “Driver loading via the low-level ntdll.dll API”
API function: Driver loading (NtLoadDriver)
Argument 1: *
Argument 2: *
Argument 3..N: *
Evaluation: single operation - 40% , 2-3 operations - 50% ,> 3 operations - 60%
Harmfulness: No
Rule "Analysis of machine kernel code (removal of hooks)"
API function: Create / open file (CreateFile)
Argument 1: Contains the entry "ntoskrnl.exe"
Argument 2: *
Argument 3..N: *
Evaluation: single operation - 100% , 2-3 operations - 100% ,> 3 operations - 100%
Harmfulness: Yes
The final object rating is the sum of all private ratings after checking the entire database of rules. In other words, this is a typical neural network that collects signals from a variety of sensors, analyzes their qualitative and quantitative characteristics, examines connections and issues a verdict. Approximately in this form the SR took over the watch in 2007. (patent US7530106 ). As you guessed, we immediately counted the technology to feed, educate and pump.
Problem Number One was that the analyzed file can generate a huge number of minor events that can not affect the final verdict of its harmfulness. For example, a Delphi application generates up to 500 such events at startup. They will be identical in any application written in this language and at the same time do not carry any useful information about the real intentions of the file. This "noise" not only consumed computer resources, but also made analysis difficult.
To eliminate such noise, we made a filter. Moreover, unlike the usual rules, there is a sufficiently Boolean feature. Thus, the rule is greatly simplified and, accordingly, its work is accelerated. As a result, the rules contain only the name of the API function and masks for its arguments.
For example:
SysAllocString (*, - NULL -, - NULL-)
Here, if the first argument of the function has any value, and the rest are missing, the event will be considered insignificant.
For automatic generation of rules for filtering minor events, we used three methods.
The first is the "fruit fly method". We are preparing an application of the “Hello World” type using the X development environment, where possible we use its most popular libraries. We feed the compiled application to the emulator, and write all the events generated by the “Drosophila” into the “insignificant” column.
The second is the "packed fruit fly method". Similar to the first, except that we are interested in the behavior of the packer / protector. To do this, we process the soother assembly in turn with all sorts of packers and protectors, feed the emulator and ... well, then you guessed it.
The third method is statistical. We analyze the behavior of a large number of legitimate and malicious files, and select the API calls that are often found in the behavior of those and others. This method complements the first two and is effective if it is not possible to create a “fruit fly”. A typical example of functions identified in this way is functions for working with GUI and memory allocation.
But it was one of the smallest Challenges. Further - more interesting.
The first version of SR worked on a specific protected computer practically in isolation. We did not have a global picture, we did not understand what the rules are, how often and how exactly they work and we could not quickly change their ratings. The result is a large unused potential for improving efficiency :) At the same time, the KSN cloud was developing at full speed and we had already hooked the Astraea expert system to it to analyze the huge volume of signals from protected computers and to give reasonable conclusions about the epidemiological situation in the world.
In 2009, to everyone's joy, the next version of SR (SR2, US8640245 ) merged with KSN and Astraea.
And a bigdat with a good expert superstructure is in our field a "materiel" of the formula for success!
In essence, we obtained a lever for (i) “shooting off” ineffective and “dead” rules, (ii) temporarily turning off or testing rules, (iii) practically real-time correction of rule ratings using coefficients. At the same time, the size of the coefficient base amounted to funny kilobytes and its updating, even in those years, practically did not burden the Internet channel.
Astraea also expanded the statistical base for calculating ratings: this process involved not only signals from various emulators, but also many other sensors, which also reported to KSN. In addition, the product could receive from the cloud an already well-known verdict and make a decision, without bringing the emulation process to the end (and in some cases not launching it at all). And another nice plus is that we can, with a high degree of certainty, pull out of the stream for manual research of unknown “animals” who did not gain the threshold value of the rating, but were very suspicious.
It is important that Astraea makes the correction of the rules automatically - it is up to the person to regularly evaluate the effectiveness of the mathematical model used and optimize it (patent application US20140096184 ).
The presence of global bigdats immediately rolled our lip into new ideas for solving old problems. First of all - falssy . In principle, we twisted this topic from the very first day of the implementation of SR in products. But in 2011. rolled out several new features at once to minimize false positives.
There are many operations that are performed by legitimate software with completely peaceful purposes. For example, installers delete files in the System32 folder. Auto-adjusting the rating of this operation will lead to its unreasonable degradation and we will start to skip real malware. Some compromise is required here so that the wolves are full and the sheep are safe. And then we decided to divide the rating calculation mechanism into 3 parts.
First, the calculation described above: the more dangerous the behavior and the more common, the higher the rating. Secondly, a kind of whitelist rules that cancel or correct the actions of ordinary rules apply to specific situations or files. Thirdly, the rules of detecting legitimate applications that reduce the hazard rating when detecting characteristic behavior and can even form a safety rating.
Example:
Rule "Create startup key"
API function: Registry: setting a parameter value (RegSetValueEx)
Argument 1: Contains the entry "\ Registry \ Machine \ Software \ Classes \ * \ shellex \ ContextMenuHandlers \ Notepad ++"
Argument 2: *
Argument 3..N: *
Evaluation: single operation - 1% , 2-3 operations - 1% ,> 3 operations - 1%
Harmfulness: No
Here you can clearly see that the registry key is twitching, but this is just Notepad ++ throwing its library. The rule argument removes the Fols, while the main rule remains unchanged and for other attempts to change the key will work as expected.
And in 2011, we introduced another useful feature. As mentioned above, in the SR, the rules worked autonomously from each other, and thus we could not study complex dependencies such as “file upload” - “save file to disk” - “prescription to autorun”. After all, if you trace such a dependence, you can give a rating much more than the sum of the ratings of individual events. Or less :) And then we decided to implement event correlation in SR2 for more accurate detection of unknown malware.
Made it in two ways. First, we created bit masks that distinguish groups of rules or individual rules on OR and AND. The basic description is a bit classification index of behavior. Initially, this was invented for clustering malware according to the characteristics of their behavior, but a similar approach can be used to refine the rating score - using masks to implement a function like (RULE76 or RULE151) and (RULE270 or RULE540). The advantage of such masks is compactness and speed of work, the disadvantage is low flexibility.
Secondly, they made special scripts that conduct a global analysis after calculating the SR (patent US8607349 ). Scripts can be launched one by one, both independently and when a rule is triggered. Each of them has access to a database of accumulated statistics of previously triggered rules and groups of rules. As a result, we have the opportunity to (i) use complex logic - conditions, calculations, cycles, subroutine calls; (ii) maximize the use of neural networks; (iii) to form a script not only a clarification to the SR rating, but also new knowledge that will be applied by subsequent scripts.
For example, the first script, based on an analysis of a dozen rules, concludes that “the application is trying to obtain passwords for other programs”. The second script likewise concludes that “the application is transmitting something to the Internet.” The third script makes a conclusion like “if the application is interested in passwords AND transmits something to the Internet, THEN + 100% rating”.
In addition, the scripts can be “hooked” to any rule, in total, the rule becomes a kind of “trigger” for some algorithm.
Sample script:
In this example, the script evaluates the file creation operation. We check the fact of creating the file in the root of the disk, and for this we immediately charge 20% SR. Further, depending on the file extension, we make an additional rating charge with a “+” or “-” sign.
The given example demonstrates the main plus of scripts - the possibility of complex differentiated evaluation of function arguments with setting an individual SR rating based on the results of various checks. Moreover, some checks can increase the rating, some - lower, which allows you to do complex checks and complex analysis, aimed at additional suppression of falss.
As a bonus - the hand of the author of the technology described above. As it is, almost on a napkin:
And now a little about the future.
The personal product line 2015 is coming out very soon. In general, we thought-weighed and decided to completely abandon the local SR and completely transfer the calculation of the rating to the cloud. This approach has many advantages at once. The quality of analysis will not suffer, but the resource consumption of the protected computer will decrease by an order of magnitude - all calculations will occur in our infrastructure. At the same time, the delay in delivery of the verdict will be ... mneee ... yes, practically nothing will be: only framed software will be able to notice these fractions of milliseconds.
Well, like this briefly about our magic formula, only 10 thousand characters. And this is really only a “slightly opened door”: in fact, acquaintance with the detailed description of the technology will take several days and a lot of clarifications.
By the way, you can clarify right now, below, in the comments.
In general, we use the deductive method, that is, we go from the general to the particular. In our case, it could look something like this: all malware does this -> this file does this too -> therefore, the file is malicious. But in practice, not everything is so simple and smooth.
First of all, it is impossible to identify specific actions that clearly indicate the harmfulness of the object. The classic example with access to the MBR is that everything that pulls this command malicious cannot be considered, there are many other applications for peaceful purposes. The same applies to all other actions. After all, in the original, all the teams were created to perform some useful things.
In other words, the task “to separate the wheat from the chaff” is irrelevant here. Another thing is to estimate the proportions and composition of grains and chaff, estimate the same parameters in the adjacent barn, analyze the results and then make an informed decision about setting a comma in the phrase “treat not to be missed”. To do this, we use technology with a straightforward name SR (Security Rating) - a sort of spreading self-learning system of weights, which helps to better understand the true nature of an object in the process of its formal evaluation and emulation .
')
The analysis takes into account the composition and density of the events generated by the object and its external attributes (name, size, location, compression, etc.). Based on a set of rules, each of the parameters receives an individual hazard rating (0-100%). The first package of rules (and there are more than 500 of them now) was the result of a “manual” study of more than 80 thousand unique malware of various families. Now the rules are developed mainly automatically, and expert analysts only “twist” self-learning algorithms.
For ease of testing and maintenance, the rules are divided into groups (for example, the Internet, Passwords, Registry, etc.) and if the object is “inherited” by appropriate behavior in one or more of them, appropriate sanctions are taken to it.
Examples of the simplest rules:
The rule “Driver loading via the low-level ntdll.dll API”
API function: Driver loading (NtLoadDriver)
Argument 1: *
Argument 2: *
Argument 3..N: *
Evaluation: single operation - 40% , 2-3 operations - 50% ,> 3 operations - 60%
Harmfulness: No
Rule "Analysis of machine kernel code (removal of hooks)"
API function: Create / open file (CreateFile)
Argument 1: Contains the entry "ntoskrnl.exe"
Argument 2: *
Argument 3..N: *
Evaluation: single operation - 100% , 2-3 operations - 100% ,> 3 operations - 100%
Harmfulness: Yes
The final object rating is the sum of all private ratings after checking the entire database of rules. In other words, this is a typical neural network that collects signals from a variety of sensors, analyzes their qualitative and quantitative characteristics, examines connections and issues a verdict. Approximately in this form the SR took over the watch in 2007. (patent US7530106 ). As you guessed, we immediately counted the technology to feed, educate and pump.
Problem Number One was that the analyzed file can generate a huge number of minor events that can not affect the final verdict of its harmfulness. For example, a Delphi application generates up to 500 such events at startup. They will be identical in any application written in this language and at the same time do not carry any useful information about the real intentions of the file. This "noise" not only consumed computer resources, but also made analysis difficult.
To eliminate such noise, we made a filter. Moreover, unlike the usual rules, there is a sufficiently Boolean feature. Thus, the rule is greatly simplified and, accordingly, its work is accelerated. As a result, the rules contain only the name of the API function and masks for its arguments.
For example:
SysAllocString (*, - NULL -, - NULL-)
Here, if the first argument of the function has any value, and the rest are missing, the event will be considered insignificant.
For automatic generation of rules for filtering minor events, we used three methods.
The first is the "fruit fly method". We are preparing an application of the “Hello World” type using the X development environment, where possible we use its most popular libraries. We feed the compiled application to the emulator, and write all the events generated by the “Drosophila” into the “insignificant” column.
The second is the "packed fruit fly method". Similar to the first, except that we are interested in the behavior of the packer / protector. To do this, we process the soother assembly in turn with all sorts of packers and protectors, feed the emulator and ... well, then you guessed it.
The third method is statistical. We analyze the behavior of a large number of legitimate and malicious files, and select the API calls that are often found in the behavior of those and others. This method complements the first two and is effective if it is not possible to create a “fruit fly”. A typical example of functions identified in this way is functions for working with GUI and memory allocation.
But it was one of the smallest Challenges. Further - more interesting.
The first version of SR worked on a specific protected computer practically in isolation. We did not have a global picture, we did not understand what the rules are, how often and how exactly they work and we could not quickly change their ratings. The result is a large unused potential for improving efficiency :) At the same time, the KSN cloud was developing at full speed and we had already hooked the Astraea expert system to it to analyze the huge volume of signals from protected computers and to give reasonable conclusions about the epidemiological situation in the world.
In 2009, to everyone's joy, the next version of SR (SR2, US8640245 ) merged with KSN and Astraea.
And a bigdat with a good expert superstructure is in our field a "materiel" of the formula for success!
In essence, we obtained a lever for (i) “shooting off” ineffective and “dead” rules, (ii) temporarily turning off or testing rules, (iii) practically real-time correction of rule ratings using coefficients. At the same time, the size of the coefficient base amounted to funny kilobytes and its updating, even in those years, practically did not burden the Internet channel.
Astraea also expanded the statistical base for calculating ratings: this process involved not only signals from various emulators, but also many other sensors, which also reported to KSN. In addition, the product could receive from the cloud an already well-known verdict and make a decision, without bringing the emulation process to the end (and in some cases not launching it at all). And another nice plus is that we can, with a high degree of certainty, pull out of the stream for manual research of unknown “animals” who did not gain the threshold value of the rating, but were very suspicious.
It is important that Astraea makes the correction of the rules automatically - it is up to the person to regularly evaluate the effectiveness of the mathematical model used and optimize it (patent application US20140096184 ).
The presence of global bigdats immediately rolled our lip into new ideas for solving old problems. First of all - falssy . In principle, we twisted this topic from the very first day of the implementation of SR in products. But in 2011. rolled out several new features at once to minimize false positives.
There are many operations that are performed by legitimate software with completely peaceful purposes. For example, installers delete files in the System32 folder. Auto-adjusting the rating of this operation will lead to its unreasonable degradation and we will start to skip real malware. Some compromise is required here so that the wolves are full and the sheep are safe. And then we decided to divide the rating calculation mechanism into 3 parts.
First, the calculation described above: the more dangerous the behavior and the more common, the higher the rating. Secondly, a kind of whitelist rules that cancel or correct the actions of ordinary rules apply to specific situations or files. Thirdly, the rules of detecting legitimate applications that reduce the hazard rating when detecting characteristic behavior and can even form a safety rating.
Example:
Rule "Create startup key"
API function: Registry: setting a parameter value (RegSetValueEx)
Argument 1: Contains the entry "\ Registry \ Machine \ Software \ Classes \ * \ shellex \ ContextMenuHandlers \ Notepad ++"
Argument 2: *
Argument 3..N: *
Evaluation: single operation - 1% , 2-3 operations - 1% ,> 3 operations - 1%
Harmfulness: No
Here you can clearly see that the registry key is twitching, but this is just Notepad ++ throwing its library. The rule argument removes the Fols, while the main rule remains unchanged and for other attempts to change the key will work as expected.
And in 2011, we introduced another useful feature. As mentioned above, in the SR, the rules worked autonomously from each other, and thus we could not study complex dependencies such as “file upload” - “save file to disk” - “prescription to autorun”. After all, if you trace such a dependence, you can give a rating much more than the sum of the ratings of individual events. Or less :) And then we decided to implement event correlation in SR2 for more accurate detection of unknown malware.
Made it in two ways. First, we created bit masks that distinguish groups of rules or individual rules on OR and AND. The basic description is a bit classification index of behavior. Initially, this was invented for clustering malware according to the characteristics of their behavior, but a similar approach can be used to refine the rating score - using masks to implement a function like (RULE76 or RULE151) and (RULE270 or RULE540). The advantage of such masks is compactness and speed of work, the disadvantage is low flexibility.
Secondly, they made special scripts that conduct a global analysis after calculating the SR (patent US8607349 ). Scripts can be launched one by one, both independently and when a rule is triggered. Each of them has access to a database of accumulated statistics of previously triggered rules and groups of rules. As a result, we have the opportunity to (i) use complex logic - conditions, calculations, cycles, subroutine calls; (ii) maximize the use of neural networks; (iii) to form a script not only a clarification to the SR rating, but also new knowledge that will be applied by subsequent scripts.
For example, the first script, based on an analysis of a dozen rules, concludes that “the application is trying to obtain passwords for other programs”. The second script likewise concludes that “the application is transmitting something to the Internet.” The third script makes a conclusion like “if the application is interested in passwords AND transmits something to the Internet, THEN + 100% rating”.
In addition, the scripts can be “hooked” to any rule, in total, the rule becomes a kind of “trigger” for some algorithm.
Sample script:
Var S : string; begin if copy(ExtractFilePath(APIArg[1]), 2, 100)=':\' then begin AddSR(20); s := LowerCase(ExtractFileExt(APIArg[1])); if S = '.exe' then AddSR(40); if S = '.com' then AddSR(50); if S = '.pif' then AddSR(60); if S = '.zip' then AddSR(-20); end; end. In this example, the script evaluates the file creation operation. We check the fact of creating the file in the root of the disk, and for this we immediately charge 20% SR. Further, depending on the file extension, we make an additional rating charge with a “+” or “-” sign.
The given example demonstrates the main plus of scripts - the possibility of complex differentiated evaluation of function arguments with setting an individual SR rating based on the results of various checks. Moreover, some checks can increase the rating, some - lower, which allows you to do complex checks and complex analysis, aimed at additional suppression of falss.
As a bonus - the hand of the author of the technology described above. As it is, almost on a napkin:
And now a little about the future.
The personal product line 2015 is coming out very soon. In general, we thought-weighed and decided to completely abandon the local SR and completely transfer the calculation of the rating to the cloud. This approach has many advantages at once. The quality of analysis will not suffer, but the resource consumption of the protected computer will decrease by an order of magnitude - all calculations will occur in our infrastructure. At the same time, the delay in delivery of the verdict will be ... mneee ... yes, practically nothing will be: only framed software will be able to notice these fractions of milliseconds.
Well, like this briefly about our magic formula, only 10 thousand characters. And this is really only a “slightly opened door”: in fact, acquaintance with the detailed description of the technology will take several days and a lot of clarifications.
By the way, you can clarify right now, below, in the comments.
Source: https://habr.com/ru/post/228427/
All Articles