The problem of continuous protection of web applications. View from the side of researchers and operators. Part 2
So, we continue the topic started a week ago. Last time, the developer of WAF shared his views on continuous protection of web applications. Today I would like to talk about how this problem is solved in the Solar JSOC cyber attack monitoring and response center. Under the cut - everything about the combat operation of WAF: what methods of work we consider the most effective, how our clients' web services attack and how WAF works (and sometimes does not work).
One of the most important aspects related to WAF, in addition to the choice of vendor and product implementation, is the further operation. At the same time, the classic approach to the operation of the Web Application Firewall is very conditional and has a number of important features.
Firstly, due to the location of WAF and the type of resources it protects, promptly updating signatures and installing patches on it is a much higher priority than in the case of other equipment. The website always sticks out, which makes it particularly vulnerable to various newly published vulnerabilities. A delay of several days can threaten serious losses and hacking of the protected resource. Vendors release signatures quickly, but the process of launching them into lockout mode usually takes place in several stages. First, the learning process takes place when we observe the number of draws and analyze the packets for which the signature worked. Further, it is transferred to another mode. But in the case of critical vulnerabilities, you must either immediately transfer the signature into the block, or, without waiting for the vendor to update, write the signature yourself.
')
Secondly, proactive exploitation comes to the fore with WAF - response to incidents in a mode close to online, analysis and operational tuning of policies and profiles. This is especially true in the case of a hybrid mode of operation, when part of the workouts are not blocked, but only fixed by WAF. This is done to minimize the risk of blocking legitimate requests.
Thirdly, due to the reduction of the release cycle, the problem of dynamic content and profiling requires the presence of specialists working in full-time mode.
WAF Operation
What is the task of operation? I identified four main areas:
However, only the first of the above points can be performed independently from the development department. All other operating functions are inextricably linked with the developers of the protected website or application. Another important role is played by testing collected profiles and created policies, including in query blocking mode.
I think that there is no doubt that the “come-configured-enabled-works” option is not viable in 99% of cases and does not work on dynamically changing content. Exceptions are those customers who block requests for static signatures, exclude testing of updates arriving on WAF and immediately roll them into production. That is, if we are not talking about the learning functionality of WAF, the cycle of updates, installing patches over RFC, creating policies for a specific protected resource - you may not need a full time specialist or service manual.
WAF mode
The first option: only blocking - by signatures and profile. Two points must be taken into account here: 1) there is a fairly high risk of blocking legitimate requests; 2) and, conversely, WAF can skip requests that exploit vulnerabilities and holes. It may happen that with a large-scale scan, 99.9% of requests will be blocked, and 0.1% will pass, but this will be enough for a successful attack. Upon analysis, 0.1% new custom signatures are written and added to the block.
Second option: notification only. In this case, upon the triggering of the signatures, it is necessary to conduct a comprehensive investigation and correlate data with other sources, for example, with databases in the case of using SQL injections. Given the huge amount of WAF responses, it’s almost impossible to manually analyze them all.
That is why the third variant is most often used - hybrid.
In this case, the analysis of static signatures is carried out, and in the absence or minimal number of false positives, they are transferred to the blocking mode.
In parallel, analysts profile the work of the site and user activities in order to further identify anomalies. It should be noted here that the translation of a policy created on the basis of a site profile into a blocking mode is impossible without serious analytics and interaction with the development. Otherwise it threatens to block legitimate requests, which will result in reputational loss for the customer.
At the same time, due to both dynamic content and newly emerging situations related to customer activities, the refinement of policies, custom signatures, the list of exceptions never ends — this is a continuous process.
Below is an example of setting up groups of signatures of one of our clients. As part of the service, we connected the WAF and web server logs (IIS) to the Solar JSOC SIEM system. Critical signatures are placed in blocking mode, logged and added to the training list, in case you need to make exceptions:
Content settings:
Invalid request headers are not blocked yet, but there is training and logging of such requests:
Whichever option you choose, it is important to keep in mind the need to analyze both the requests that caused the triggers and the missing requests to the application. And forcing WAF to log the missed requests is quite laborious from the point of view of load, so it is more expedient to look at the logs of the application itself or the web server.
Custom signatures
One of the most important moments in the operation of Web Application Firewall is to write custom signatures. This is required in several cases: the emergence of a new vulnerability, the registration of anomalies that require additional analysis, and the collection of long-term statistics of authentication in the application.
In the case of a serious vulnerability, such as shellshock, for example, operational policy refinement is required. Usually, the WAF operation service can implement it within 1-2 hours, while the vendor can take up to several days. Delay in such situations threatens the successful exploitation of the vulnerability, since the attackers work almost instantly, launching a scan of all the white addresses on the Internet for vulnerability.
An example of the detection of anomalies using manual analysis is the collection of statistics on the large volume of traffic sent and received. In general, a large session size does not indicate an incident, but when viewing the overall statistics, you can find really important triggers. Similarly, with monitoring file downloads with non-standard extensions - in general, blocking such attachments is impractical, but analyzing them, including using a sandbox, is recommended.
Work with dynamic content
The correct cycle of working with constantly changing content, it seems to me, should look like this:
In an ideal world there are pre-productive environments, functional and stress tests lasting about 2 weeks. But life sometimes puts things differently. Approximately in 30% of cases we have to work with a reduced development mode of applications and web sites at customers, when the update immediately “rolls” onto productive servers. In this case, it is necessary to quickly adjust the WAF policy to the new working conditions. For this, profiling is started for a short period of time - half a day-day. Due to the large amount of traffic, this is often sufficient for training. After this time, the “pre-profiled” updated policy is transferred to the blocking mode.
Both methods require additional attention from the operational service during the transition to the blocking mode. This is done for manual analysis of blocked requests, since no intelligent system of training and profiling can work perfectly without false positives.
Here are two vivid examples of the development cycle of our customers:
The first case - the customer releases one release per week. At the same time, the interaction between the customer and the contractor does not provide for a detailed release notes, according to which it can be understood whether a policy adjustment is necessary. The only option is to work “hot” with a short and operational cycle of tuning WAF policies:
In our other client, part of the site changes several times a day as part of the placement of new products, promotions, etc. It is impossible to include any profiling and blocking on the collected profile for this part of the site content. Therefore, we are working on it in the mode of monitoring anomalies and creating custom signatures.
Integration of WAF with SIEM: how and what is needed?
If a company uses its own or outsourced Security Operations Center, connecting WAF to a SIEM system will allow for prompt notification and analysis of all types of incidents, including web attacks, from a single point - the SIEM system.
To fix and further investigate incidents in the SIEM system, you need to connect the following sources:
The important point in collecting logs of a web server or application is not only the completeness of the collection of information, but also the analysis of the retrospective. SIEM systems are sharpened for storing large amounts of data with the ability to quickly search in retrospect with a depth of several months, including using regular expressions for queries. This is another powerful argument for integrating WAF with SIEM.
When connecting WAF, you need to understand which fields, headers and metadata will be useful in the investigation of incidents. As an example, consider connecting the F5 ASM to ArcSight.
Of the required fields is to highlight:
Date and time of the event
Signature name — for example, “Illegal request length”
Attack type - for example, “Session Hijacking”
Requested URL:
Request method - GET, POST, etc.
Address from which the request came
Address to which the request came
Connection protocol
Site response code
The name of the policy on which the signature worked
Date and time of this policy
Despite the limitation of 4096 characters in the customstring-fields of ArcSight, we "zamapi" there the entire request, the analysis of which very often comes in handy in the investigation:
First line monitoring specialists handle potential incidents that are configured for different scenarios:
After the first line specialists notify the WAF operation service, the latter have the opportunity to connect to the GIS interface and “tune” policies in real time to counter the attack. Connecting to a SIEM system allows you not only to set up prompt notification of potential incidents, but also to enrich information about attacks with information from other systems, as well as to produce complex correlation along chains of events. For example, to understand the success of an attack, information from a Web Application Firewall can be correlated with queries in SQL databases or local logs of a web server.
Our approach to handling incidents:
For example, in one of our customers, the key business component is an online store. All existing business processes are tied to the logic of working with clients and processing orders. As a loyalty program, the company uses virtual points, which can be used to calculate the store, including partial. When making a purchase at the stage of choosing a payment method, the customer has the opportunity to enter the number of points in the field that will be used instead of real money.
The customer needs to control suspicious transactions - too many bonuses or transactions with bonuses for a certain time. A separate table (Active List in ArcSight) is used as a tool for collecting summary statistics. Daily unloading from this sheet is sent to the analyst to study the anomalies that do not fall into the current correlation response.
An entry was found in one of these unloadings:
When analyzing this anomaly, it turned out that one of the clients tried using POST requests, containing various variations of bonuses required for writing off, to deceive the logic of the website. And got his way! When sending a negative value, the system compared the number of available bonuses with the number of bonuses to the write-off and, when performing mathematical inequality, approved the process.
But this is not all: with the further processing of the order in specialized systems, to which information was transferred from the website, the value of the bonuses required for write-off was taken modulo! So the client has successfully received a discount of 20 thousand rubles, with 18 points on the account.
The Web Application Firewall did not see anything abnormal because the request structure was correct and the number of requests was not suspicious. Only the subsequent collection of statistics and viewing of events through the eyes of a live analyst revealed a successful fraud attempt in the client’s key business application.
From here we can draw the following conclusion: any important requests to a key application, such as redemption of points, cash, etc., should be logged using the custom policy settings on WAF or by connecting the application logs and further analysis in the SIEM system.
Application Environment Protection
Not all hackers are attacking the application "in the forehead." In some cases, it is much easier to penetrate the infrastructure of the company and attack the application or its database from the inside in order to make changes directly. In addition, there are internal intruders, including those with access to the application, and they need no less control.
To solve this problem by WAF is impossible, this is work for the SIEM system. The case for the protection of a key application is solved in a complex and usually the sources connected to the SIEM include:
Next, an analysis of current risks and vectors of potential attacks, fraudulent operations and violations in the application:
On the basis of the vectors defined above, the sources and points of control necessary for the connection are determined in the form of correlation rules that are configured in the SIEM system.
The composition of the rules for identifying incidents usually includes three blocks:
The overwhelming part of the rules for detecting incidents in the environment of the application does not change much in the basic logic, and customization consists of changing configurations in the Active List. The same applies to the profiling rules.
The content associated with the work of the application itself is created in most cases individually and includes a survey of the system, conducting interviews with owners and developers, and further analytics to find “bottlenecks”.
As a summary of the article, I would like to emphasize that the security of key applications and sites is only partly related to external direct threats, and one should always take into account a wider vector of potential attacks and fraudulent operations.
One of the most important aspects related to WAF, in addition to the choice of vendor and product implementation, is the further operation. At the same time, the classic approach to the operation of the Web Application Firewall is very conditional and has a number of important features.
Firstly, due to the location of WAF and the type of resources it protects, promptly updating signatures and installing patches on it is a much higher priority than in the case of other equipment. The website always sticks out, which makes it particularly vulnerable to various newly published vulnerabilities. A delay of several days can threaten serious losses and hacking of the protected resource. Vendors release signatures quickly, but the process of launching them into lockout mode usually takes place in several stages. First, the learning process takes place when we observe the number of draws and analyze the packets for which the signature worked. Further, it is transferred to another mode. But in the case of critical vulnerabilities, you must either immediately transfer the signature into the block, or, without waiting for the vendor to update, write the signature yourself.
')
Secondly, proactive exploitation comes to the fore with WAF - response to incidents in a mode close to online, analysis and operational tuning of policies and profiles. This is especially true in the case of a hybrid mode of operation, when part of the workouts are not blocked, but only fixed by WAF. This is done to minimize the risk of blocking legitimate requests.
Thirdly, due to the reduction of the release cycle, the problem of dynamic content and profiling requires the presence of specialists working in full-time mode.
WAF Operation
What is the task of operation? I identified four main areas:
- Update - tracking patches, including closing errors. This was mentioned above, so we move on.
- Writing private signatures:
- Tracking new critical signatures, joint testing with the customer and preparation of countermeasures for the period until the official updates are released (if it is not possible to write custom ones).
- writing signatures as compensation measures to close application source code vulnerabilities:
- vulnerabilities known to developers, the closure of which requires large-scale modifications of the source code;
- vulnerabilities identified by analyzing the source code of applications.
- Profiling of requests, references to pages of the site, application.
- Validation - checking and analyzing the parameters of requests / responses to the protected resource / application.
However, only the first of the above points can be performed independently from the development department. All other operating functions are inextricably linked with the developers of the protected website or application. Another important role is played by testing collected profiles and created policies, including in query blocking mode.
I think that there is no doubt that the “come-configured-enabled-works” option is not viable in 99% of cases and does not work on dynamically changing content. Exceptions are those customers who block requests for static signatures, exclude testing of updates arriving on WAF and immediately roll them into production. That is, if we are not talking about the learning functionality of WAF, the cycle of updates, installing patches over RFC, creating policies for a specific protected resource - you may not need a full time specialist or service manual.
WAF mode
The first option: only blocking - by signatures and profile. Two points must be taken into account here: 1) there is a fairly high risk of blocking legitimate requests; 2) and, conversely, WAF can skip requests that exploit vulnerabilities and holes. It may happen that with a large-scale scan, 99.9% of requests will be blocked, and 0.1% will pass, but this will be enough for a successful attack. Upon analysis, 0.1% new custom signatures are written and added to the block.
Second option: notification only. In this case, upon the triggering of the signatures, it is necessary to conduct a comprehensive investigation and correlate data with other sources, for example, with databases in the case of using SQL injections. Given the huge amount of WAF responses, it’s almost impossible to manually analyze them all.
That is why the third variant is most often used - hybrid.
In this case, the analysis of static signatures is carried out, and in the absence or minimal number of false positives, they are transferred to the blocking mode.
In parallel, analysts profile the work of the site and user activities in order to further identify anomalies. It should be noted here that the translation of a policy created on the basis of a site profile into a blocking mode is impossible without serious analytics and interaction with the development. Otherwise it threatens to block legitimate requests, which will result in reputational loss for the customer.
At the same time, due to both dynamic content and newly emerging situations related to customer activities, the refinement of policies, custom signatures, the list of exceptions never ends — this is a continuous process.
Below is an example of setting up groups of signatures of one of our clients. As part of the service, we connected the WAF and web server logs (IIS) to the Solar JSOC SIEM system. Critical signatures are placed in blocking mode, logged and added to the training list, in case you need to make exceptions:
Content settings:
Invalid request headers are not blocked yet, but there is training and logging of such requests:
Whichever option you choose, it is important to keep in mind the need to analyze both the requests that caused the triggers and the missing requests to the application. And forcing WAF to log the missed requests is quite laborious from the point of view of load, so it is more expedient to look at the logs of the application itself or the web server.
Custom signatures
One of the most important moments in the operation of Web Application Firewall is to write custom signatures. This is required in several cases: the emergence of a new vulnerability, the registration of anomalies that require additional analysis, and the collection of long-term statistics of authentication in the application.
In the case of a serious vulnerability, such as shellshock, for example, operational policy refinement is required. Usually, the WAF operation service can implement it within 1-2 hours, while the vendor can take up to several days. Delay in such situations threatens the successful exploitation of the vulnerability, since the attackers work almost instantly, launching a scan of all the white addresses on the Internet for vulnerability.
An example of the detection of anomalies using manual analysis is the collection of statistics on the large volume of traffic sent and received. In general, a large session size does not indicate an incident, but when viewing the overall statistics, you can find really important triggers. Similarly, with monitoring file downloads with non-standard extensions - in general, blocking such attachments is impractical, but analyzing them, including using a sandbox, is recommended.
Work with dynamic content
The correct cycle of working with constantly changing content, it seems to me, should look like this:
- Formed policy is unloaded from the combat server to the test environment.
- In the process of testing the updated content, compatibility with the current WAF policy is simultaneously tested, if necessary, it is adjusted.
- In an ideal scheme, an application or service update is tested on a pilot group (region). At this stage, the WAF policy is transferred to the blocking mode and tested in combat conditions. Tuning and tweaking policy configuration is also performed.
- Further update and policy is applied globally to all customers of the customer and becomes a reference until the next update.
In an ideal world there are pre-productive environments, functional and stress tests lasting about 2 weeks. But life sometimes puts things differently. Approximately in 30% of cases we have to work with a reduced development mode of applications and web sites at customers, when the update immediately “rolls” onto productive servers. In this case, it is necessary to quickly adjust the WAF policy to the new working conditions. For this, profiling is started for a short period of time - half a day-day. Due to the large amount of traffic, this is often sufficient for training. After this time, the “pre-profiled” updated policy is transferred to the blocking mode.
Both methods require additional attention from the operational service during the transition to the blocking mode. This is done for manual analysis of blocked requests, since no intelligent system of training and profiling can work perfectly without false positives.
Here are two vivid examples of the development cycle of our customers:
The first case - the customer releases one release per week. At the same time, the interaction between the customer and the contractor does not provide for a detailed release notes, according to which it can be understood whether a policy adjustment is necessary. The only option is to work “hot” with a short and operational cycle of tuning WAF policies:
In our other client, part of the site changes several times a day as part of the placement of new products, promotions, etc. It is impossible to include any profiling and blocking on the collected profile for this part of the site content. Therefore, we are working on it in the mode of monitoring anomalies and creating custom signatures.
Integration of WAF with SIEM: how and what is needed?
If a company uses its own or outsourced Security Operations Center, connecting WAF to a SIEM system will allow for prompt notification and analysis of all types of incidents, including web attacks, from a single point - the SIEM system.
To fix and further investigate incidents in the SIEM system, you need to connect the following sources:
- Directly WAF itself.
- Web server logs. As mentioned above, enabling logging of all requests for WAF can cause excessive load, so it is easier to connect logging to IIS or Apache.
- Logs of the WAF protected application for tracking requests.
- Database logs for tracking requests.
The important point in collecting logs of a web server or application is not only the completeness of the collection of information, but also the analysis of the retrospective. SIEM systems are sharpened for storing large amounts of data with the ability to quickly search in retrospect with a depth of several months, including using regular expressions for queries. This is another powerful argument for integrating WAF with SIEM.
When connecting WAF, you need to understand which fields, headers and metadata will be useful in the investigation of incidents. As an example, consider connecting the F5 ASM to ArcSight.
Of the required fields is to highlight:
Date and time of the event
Signature name — for example, “Illegal request length”
Attack type - for example, “Session Hijacking”
Requested URL:
/login?_*********=/sitebuilder/****/myaccount/login/*******form Request method - GET, POST, etc.
Address from which the request came
Address to which the request came
Connection protocol
Site response code
The name of the policy on which the signature worked
Date and time of this policy
Despite the limitation of 4096 characters in the customstring-fields of ArcSight, we "zamapi" there the entire request, the analysis of which very often comes in handy in the investigation:
First line monitoring specialists handle potential incidents that are configured for different scenarios:
- The abnormal number of different signatures that worked on a WAF from an external host is a potential scan for vulnerabilities.
- Triggering signatures for several days in a row from a single external host is a slow scan.
- Triggering the threshold value of signatures in a short period of time from different hosts - distributed scanning / application-level DDoS.
- Attempt to exploit critical vulnerabilities. In the case of the drawdown of this scenario, our specialists, together with customers, analyze not only the WAF logs, but also successful requests from the same address to the website. Further, according to the results of the analysis and when the need arises, the WAF policy is finalized.
After the first line specialists notify the WAF operation service, the latter have the opportunity to connect to the GIS interface and “tune” policies in real time to counter the attack. Connecting to a SIEM system allows you not only to set up prompt notification of potential incidents, but also to enrich information about attacks with information from other systems, as well as to produce complex correlation along chains of events. For example, to understand the success of an attack, information from a Web Application Firewall can be correlated with queries in SQL databases or local logs of a web server.
Our approach to handling incidents:
For example, in one of our customers, the key business component is an online store. All existing business processes are tied to the logic of working with clients and processing orders. As a loyalty program, the company uses virtual points, which can be used to calculate the store, including partial. When making a purchase at the stage of choosing a payment method, the customer has the opportunity to enter the number of points in the field that will be used instead of real money.
The customer needs to control suspicious transactions - too many bonuses or transactions with bonuses for a certain time. A separate table (Active List in ArcSight) is used as a tool for collecting summary statistics. Daily unloading from this sheet is sent to the analyst to study the anomalies that do not fall into the current correlation response.
An entry was found in one of these unloadings:
modify_order_bonus ":"{\"bonusAvailable\":18.0,\"bonus\":-20000}" reason: success When analyzing this anomaly, it turned out that one of the clients tried using POST requests, containing various variations of bonuses required for writing off, to deceive the logic of the website. And got his way! When sending a negative value, the system compared the number of available bonuses with the number of bonuses to the write-off and, when performing mathematical inequality, approved the process.
But this is not all: with the further processing of the order in specialized systems, to which information was transferred from the website, the value of the bonuses required for write-off was taken modulo! So the client has successfully received a discount of 20 thousand rubles, with 18 points on the account.
The Web Application Firewall did not see anything abnormal because the request structure was correct and the number of requests was not suspicious. Only the subsequent collection of statistics and viewing of events through the eyes of a live analyst revealed a successful fraud attempt in the client’s key business application.
From here we can draw the following conclusion: any important requests to a key application, such as redemption of points, cash, etc., should be logged using the custom policy settings on WAF or by connecting the application logs and further analysis in the SIEM system.
Application Environment Protection
Not all hackers are attacking the application "in the forehead." In some cases, it is much easier to penetrate the infrastructure of the company and attack the application or its database from the inside in order to make changes directly. In addition, there are internal intruders, including those with access to the application, and they need no less control.
To solve this problem by WAF is impossible, this is work for the SIEM system. The case for the protection of a key application is solved in a complex and usually the sources connected to the SIEM include:
- Logs of operating systems for application servers, databases, auxiliary systems, terminal servers.
- Logs of firewalls separating the application segment from the rest.
- WAF logs.
- Application logs (front-end, back-end).
- Web server logs.
- Application database logs, at least at the level of authentication, and better at the level of requests to the database itself.
- VPN logs in case of contractors working with the application via remote connections.
Next, an analysis of current risks and vectors of potential attacks, fraudulent operations and violations in the application:
- Monitoring the work of contractors with the application.
- Supervision of the work of privileged employees.
- Identify external attacks on the application.
On the basis of the vectors defined above, the sources and points of control necessary for the connection are determined in the form of correlation rules that are configured in the SIEM system.
The composition of the rules for identifying incidents usually includes three blocks:
- Control of user actions - calls to the application / database, work in it for all types of users, including privileged ones.
- Control the integrity of key tables, files, registry branches, environment settings, running services and processes.
- Anomaly detection - collecting profiles based on statistics and further monitoring deviations from it.
The overwhelming part of the rules for detecting incidents in the environment of the application does not change much in the basic logic, and customization consists of changing configurations in the Active List. The same applies to the profiling rules.
The content associated with the work of the application itself is created in most cases individually and includes a survey of the system, conducting interviews with owners and developers, and further analytics to find “bottlenecks”.
As a summary of the article, I would like to emphasize that the security of key applications and sites is only partly related to external direct threats, and one should always take into account a wider vector of potential attacks and fraudulent operations.
Source: https://habr.com/ru/post/332846/
All Articles