Optimization and automation of web application testing

In this article I will discuss how to optimize and automate the processes of penetration testing using specialized utilities and their extensions.

Penetration testing can be divided into two stages:

• automated testing;
• manual testing.

Pros and cons of automated testing

Pros:

With automated testing, as a rule, testing time is significantly saved, it is possible to cover a large area of ​​a web application in less time.

A large number of checks. Automated systems contain a huge number of attack patterns, signs of vulnerabilities, and are usually expandable.

Browsing files and folders, selecting passwords - here, I think everything is clear and true.

Routine scanning and inventory procedures - for these purposes, automated systems are best suited.

Minuses:

False positive trigger. Very often, scanners, guided by formal signs, reveal vulnerabilities that are not there. Classics of the genre - when scanning Single Page Application, the scanner receives the response code 200 to all its requests and displays a long list of vulnerabilities that are not actually present.

They are "very noisy." When a site is scanned, a lot of events are created in the web server logs for which it is easy to identify the attack.

Load on web server. Sometimes an automated scan can put a significant load on a web server, which can lead to an unstable web application (although this minus is related to configuring the web server).

Blocking means of protection. As a rule, signs of automated systems are well known to developers and they take them into account when designing. As a result, blocking occurs (by User Agent, scanner markers, or request frequency).

Do not take into account errors of logic.

Requires manual validation of vulnerabilities.

Building effective automated systems optimized for web application

In order for the automated system to be as efficient as possible, it must have the following capabilities:

• expansion of functionality, including with the help of third-party modules;
• multi-format test results;
• import / export test results;
• standardized test results;
• the ability to compare results;
• ability to integrate the system into a more complex one.

These factors will allow you to build a system that meets your requirements and goals.

As an example of a "finished system" I can cite the Sparta that I described earlier. In order for testing to be as efficient, controlled as possible, as well as for comfortable validation of vulnerabilities, it is necessary to take into account all system components, the architecture of the application under test, and the connectivity of solutions.

We pump utilities

As an optimal basis, I propose to stop at two cross-platform systems for testing web applications (both in manual and automated mode): OWASP ZAP (free version) and BurpSuite (free + paid version).

The most important difference between these systems and classic scanners is the principle of operation: the “site-wide” scanner directly, identifying certain signs of vulnerabilities, often skipping huge sections of a web application. And Zap and Burp work as a proxying mechanism, allowing you to add all areas of the site (both with built-in spiders and when you manually surf the app). Also, an important feature is the ability to parse each request "on the fly".

A huge advantage of these applications is the ability to expand using plug-ins / components:

• application store (ready-made solutions);
• integration of third-party modules (outside the app store);
• writing your own (customized for a specific project).

As an example of an application, I propose to consider a vulnerable web application with the following characteristics:

• known CMS;
• plugin / component support;
• contains vulnerabilities (including OWASP A1 - sql injection);
• contains remedies.

Based on this data, we need to select and use the following components (minimum set).

Owasp ZAP:

Installed:

You need to install:

Here it is necessary to choose those tools that will help identify and exploit the vulnerabilities identified in the list above.

Burp Suite:
Frankly, I like Burp more than Owasp Zap, so I’ll stop on it in more detail.

We need to perform the task of identifying the CMS and installed components, identify the outdated versions, try to bypass the WAF, and exploit the SQL injection.

First of all, you need to adhere to the methodology of testing a web application. This will help us presented at the latest Def Con HUNT Burp Suite Extension :

• We define common parameters for certain classes of vulnerabilities.
• We organize the testing methodology inside the Burp Suite.

Next, we need a plugin (already added to the BApp store) from Vulners.com (@isox, avleonov - thanks for the great tool).

Also, an extended set of fuzzing for sql injections (which is not in the free version - sql and not only) can be useful.

Next, sql map will help us speed up the "unwinding" of the injection - to integrate it, you need to use the SQLiPy plugin:

We know that the web application is protected by WAF - what the WAF and Waf bypass extensions can be useful.

I also recommend the plugin from Vladimir Ivanov: burp-xss-sql-plugin .

As an automation tool, you can use the built-in scanner, or use the Burp Automator tool:

Requirements:

• burp-rest-api
• Burp Suite Professional
• slackclient

This tool will automate checks using the Burp Suite as the basis:

$ python burpa.py -h ################################################### __ / /_ __ ___________ ____ _ / __ \/ / / / ___/ __ \/ __ `/ / /_/ / /_/ / / / /_/ / /_/ / /_.___/\__,_/_/ / .___/\__,_/ /_/ burpa version 0.1 / by 0x4D31 ################################################### usage: burpa.py [-h] [-a {scan,proxy-config}] [-pP PROXY_PORT] [-aP API_PORT] [-rT {HTML,XML}] [-r {in-scope,all}] [--include-scope [INCLUDE_SCOPE [INCLUDE_SCOPE ...]]] [--exclude-scope [EXCLUDE_SCOPE [EXCLUDE_SCOPE ...]]] proxy_url positional arguments: proxy_url Burp Proxy URL optional arguments: -h, --help show this help message and exit -a {scan,proxy-config}, --action {scan,proxy-config} -pP PROXY_PORT, --proxy-port PROXY_PORT -aP API_PORT, --api-port API_PORT -rT {HTML,XML}, --report-type {HTML,XML} -r {in-scope,all}, --report {in-scope,all} --include-scope [INCLUDE_SCOPE [INCLUDE_SCOPE ...]] --exclude-scope [EXCLUDE_SCOPE [EXCLUDE_SCOPE ...]] 

As a result, you will receive a scan report:

Conclusion

I deliberately did not consider ready-made scanners like w3af (or paid counterparts), allowing to carry out such work, where use boils down to existing functionality (and setting checkboxes for options), but I tried to uncover interesting plugins and improvements for the proxy utility Burp Suite - as the most popular effective tool.

In order to use automated testing tools as efficiently as possible, it is necessary to have a manual verification basis for more precise system setup.