How to write a vulnerability scanner and earn exploits

TLDR : About how I quit my job as a security man and wrote my vulnerability scanner.
And how to make money on it. At the end of the scanner code pieces and specification.

Who and what hacks

In the 1970s, a long time ago, the Internet looked like this.

The main threat to him was presented by researchers and enthusiasts who were the first to reach the documentation and networks of large corporations.

Since then, the Internet has changed a lot. Attacks on networks, services and web applications have become commonplace. Every day, developers and researchers find dozens of vulnerabilities. In parallel, tens of thousands of sites are hacked.

This is clearly seen on the attack map of Kaspersky Lab. It displays IDS, vulnerability scans and botnet attacks. The map looks like such a beautiful picture and there are millions of attacks a day only in Russia.

Cyber-cross involved not by amateurs, but by organized groups of dozens of people. They do it out of interest. This is a business. They are not guys in black hoods writing "Black C0d" at night in front of the monitor. This is a socially active people working in offices around the world.

My career was such that for several years I was responsible for the security of the network perimeter and the protection of web applications. I will discuss in what article my experiences from that period.

If you work \ worked in a telecom, bank, provider or you are a developer in a large company, and especially if you are a security person, then you know that security in a company with more than 10-15 sites is trash, hell, death.

Let's try to improve the situation. They say that security is a process.
And a lot of processes.

Briefly list them:

• Threat management
• Vulnerability Management
• Risk management
• Patch management
• Identity & Acess Management
• Security awarness
• Security Provider Management
  Other.

The listed species in the article will not be considered, hurray.
There are many of them, and the benefits vary according to the size of the organization, the types of assets and their criticality.

We will talk about a process that will benefit even a small business.
From an online store with hundreds of thousands of rubles in revenue to a huge corporation with dozens of data centers.

Vulnerability Detection

In order to understand what are the ways to hack us, let's look at ourselves through the eyes of an attacker.

Security analysis consists of several stages. For each stage I will give a basic set of utilities that you can use.

• Gathering information about which IP, domains, subnets our purpose has. Instead of knocking on the front door, it’s better to find less protected entry points.
  

  dnsrecon, Fierce, Censys, reverse-PTR-lookup. 

• Service Analysis for known vulnerabilities (CVE). For them, public exploits are common.
  

   Vulners, exploit-DB, NIST, CVEdetails 

• Analysis of services for weak passwords.
  

   potator, hydra 

• Web application analysis for vulnerabilities
  

   Zap, Burp, w3af, Arachni 

cool

Okay, cool. We have a way and tools to check out 1 website.
A check + analysis of one service will take several days to understand thoroughly - a week or more.

Not cool

The problem is that we may have more than one service. And for example / 20 subnet.
4096 addresses. Each may have 65535 services. Not cool.

Yes, there are automated solutions. Qualys, Acunetix, Nessus, there are also domestic vendors. Analysis of what they are good and bad, I suggest to leave for another article.

They did not solve my problems. Decided to fix it. He quit his job and about a year he and his friends went to work out.

How to write a vulnerability scanner

Let's start with the requirements for the system we wanted to receive:

• Explores and locates target domains and ip
• Monitors the network perimeter. Sees when a new address appears in the range or a subdomain appears on the protected site.
• Constantly checks addresses, sees when network ports open and close.
• Finds vulnerabilities and known banner / CPE exploits
• Picks up weak passwords
• Finds application and OS configuration errors
• Finds vulnerabilities in CMS and their plugins.
• Interactively finds XSS, SQLi, XXE, RCE, LFI, RFI and those from OWASP to 10 that are amenable to automated search
• It does this more than once, but constantly re-checks my resources and reacts if new vulnerabilities arise.

Users are owners of large network ranges, that is, those who have 10 or more web applications.
It is necessary to provide daily parallel scanning for web vulnerabilities, weak passwords, configuration errors, and to show outgoing exploits for hundreds of ip and websites.

To do this, we use a horizontally scalable system. You can add new nodes and new types of scanners on the fly. Now the scanner uses 7 nodes and is divided into 2 interacting locations in Germany and the USA. If interested, we will write about it in another article.

Thought about how to write such a scanner. They understood that to write from scratch such a system makes little sense. It required tremendous experience and understanding of different platforms, its own database of network fingerprints, its own CVE database and exploits for them, and a whole huge system dedicated to analyzing the security of web applications.

Where the product license allows it, we decided to use open-source developments. There are components, closed source, but free for commercial use. There are a lot of samopisny and forks open-source projects.

So, we have about 20 different applications, which are the components necessary to cover the L3-L7 levels for automated pentest.

Scanner Interface

Each script, PoC program or binary accepts different parameters to start scanning. Not very comfortable. I wanted to get a unified format for running any possible scanner.

Thought that you need to know any possible scanner to identify the target. Made a table.

After the initial verification of nmap and a plugin that accesses CVEdetails for each host, we get a set of such objects. We call them Metadata.

JSON representation:

            {             "cves": [],             "service": "mysql",             "protocol": "tcp",             "target": "example.com",             "time": "1513941789",             "product": "mysql",             "vendor": "mysql",             "version": "5.1.63-community",             "port": 3306,             "uri": ""           } 

For each port on Target we get a set of such metadata. Now we need to understand which of our scanners want to work with such a goal.

All scanners have a parent - ScanManager, which makes it elegant:

 product(scanners, metadata_as_list) 

Takes all the services and for each of them checks all possible scanners.
Each scanner provides a function that determines whether it wants to work with such a Target.

Let's talk about what is a scanner.
Considered three options:

• Implement each scanner as a microservice that supports our RPC
• Implement each scanner as a service and use a common message bus
• Third option

But we are very lazy and realized that in the case of the first two options, in addition to the scanners, we also need some kind of infrastructure for deployment, writing RPC, catching the bugs of this RPC. Plus we wanted to write everything in Python.

We tried several solutions. We looked at similar projects on the management of scanners, such as Yandex Molly and Minion from Mozilla. We looked at how the work of w3af, Zap. Burp has a very interesting plugin architecture.

We decided that each scanner should be implemented as a serializable function on python. This will allow them to be stored directly in rabbit-mq, ​​quickly delivered to different nodes, atomically executed. We can add new scanners without the need to build and distribute applications. In fact, we received a service for distributed and asynchronous execution of functions on python.

Scanner Format

The scanner launched on our platform must be an object inherited from the base class Scanner.

Any scanner must accept a Metadata object as input and return Vulnerability objects.

 class Scanner(object):   name = "scanner_base"   vuln_type = "default_vuln_type"   user_options = {}   Vulnerability_body_fields_to_web_interface = []   def __init__(self, opts, target, metadata):       self.metadata = metadata       self.opts = opts       self.target = target   @staticmetod   def circuit(Metadata):       '''         .            Metadata.             CVE.       '''       return [Vulnerability(), Vulnerability()]   def check_start_condition(self):       '''        ,    Target          True,    .    False.       '''       return True   class ScannerError(Exception):       def __init__(self, value):           self.value = value       def __str__(self):           return repr(self.value) 

Vulnerability Format

The next task we faced was to create a unified format in which we can store the output of any scanner. Whether it is a web combine, a directory brutilka or a PoC exploit for memcached.

We decided not to fence our formats, but to use the industry standard - the CVE format. We will enrich CVE with our vulnerability metadata for easy storage and retrieval. To store scanner-specific data, add a body field. Which keys from the body should be displayed in the web interface is determined by the Scanner.

 class Vulnerability(object):   '''           body     .   '''   def __init__(self, target, port, scanner, text, VulnerabilityTypes, Score, protocol,                UpdateDate=None, scan_date=None, Complexity=None, Access=None, CWEID=None, Authentication=None,                Integ=None, PublishDate=None, Conf=None, ofExploits=0, Avail=None,                CVEID=None, references=None, GainedAccessLevel=None, false_positive=False,                fixed=None, body=None):       scan_date = scan_date if scan_date is not None else calendar.timegm(gmtime())       references = references or []       body = body or {}       self.name = self.construct_cve_name(VulnerabilityTypes, protocol, target, port, credentials, uri, params)       self.data = {           "target" : target,           "port" : int(port),           "Scanner": scanner,           "Scan_date": scan_date,           "Name": name,           "UpdateDate": UpdateDate,           "VulnerabilityTypes": VulnerabilityTypes,           "Complexity": Complexity,           "text": text,           "Access": Access,           "CWEID": CWEID,           "Hash": sha1(self.name.encode('utf-8')).hexdigest(),           "Authentication": Authentication,           "Integ": Integ,           "PublishDate": PublishDate,           "Conf": Conf,           "ofExploits": ofExploits,           "Score": Score,           "Avail": Avail,           "CVEID": CVEID,           "References": references,           "GainedAccessLevel": GainedAccessLevel,           "FalsePositive": false_positive,           "Fixed": fixed,           "Body": body       }   @staticmethod   def construct_cve_name(VulnerabilityTypes, protocol, target, port, credentials, uri, params):       # vuln_type:host:port:protocol:credentials:uri:params       return "{}:{}:{}:{}:{}:{}:{}".format(           VulnerabilityTypes,           target,           port,           protocol,           credentials,           uri,           params)   def __str__(self):       return self.data   def __repr__(self):       return self.name 

We tried to give the authors of scanners a maximum of freedom in implementation.
And we invite you to participate in the development.

How to make money on exploits.

We want people who are interested in security, authors of utilities and scripts, researchers, to be able to absolutely legally monetize the results of their work.

The scanner is called METASCAN. If you have your own scripts or 1-day PoC-exploits, or maybe you are the author of the scanner. Send us modules in Scanner format to david.ordyan@metascan.ru!

We will pay remuneration to the authors of the modules on a monthly basis and announce the acceptance of the modules by November 30th.
We will test them by running our client database of more than 2000 sites and counting the number of detections.

The authors of the first three scanners by the number of found vulnerabilities will receive:

• 31 337 rubles for the 1st place.
• 13,370 rubles for 2nd and 3rd place.

We will also offer them to enter into contracts and receive a reward for using their scanners on a monthly basis, as long as they find vulnerabilities.

After November we will expand the program for receiving scanners.

Write to david.ordyan@metascan.ru or TG https://t.me/METASCAN
BB8E 3D9B 04FF 70C9 A260 CD45 E0BF 4EB4 9838 A7EC

Examples and code will appear here.