Conference DEFCON 22. "Mass scanning of the Internet through open ports." Robert Graham, Paul McMillan, Dan Tantler
My name is Rob Graham, I am the head of Errata Security, a company that deals with Internet consulting. Today we will talk about how to scan the entire Internet and why it is needed. Until today, there were few tools to solve this problem, so we created our own tools. The Internet is small enough - there are only about 4 billion addresses.
Scanning the Internet is quite simple - you sit down in front of your computer, launch the console with the command line and enter the subnet address. And you watch how your screen is filled with data, and the lines all run and run on. As a result, you get a list of open ports of devices with different IP addresses.
')
Why do I need to scan the Internet in the context of protection? If you are concerned about security issues, this should be done to get an answer to the following questions:
How many computer systems are vulnerable to Heartbleed (an error in cryptographic software that allows an attacker to read the memory of a client or server and get the server's encryption key)?
How many computer systems can I use to increase attacks on NTP servers?
How many systems are at risk due to the vulnerability of D-Link routers?
An overview of all SSL certificates used.
The existing tools for finding the vulnerability of specific networks and equipment are rather slow, but mass scanning allows you to get the characteristics of the vulnerability of over 100,000 devices fairly quickly. An important problem that needs to be solved is the identification of equipment that is used to communicate with NTP servers during DDOS attacks. A lot of home equipment is vulnerable due to the fact that D-link routers are not strong enough protection. Just look at the D-link network to see how many botnet systems exploit their vulnerability. Scanning SSL certificates is also useful because it reveals outdated certificates that are prone to errors and vulnerabilities. So scanning everything you can “reach out” is an important task.
Internet scanning is also needed in the context of prevention. It helps to reveal Deepnet - many Internet pages that are not visible to search engines. These pages are generated by user requests and may carry malicious information.
Try scanning random ports by running the "- banners" bulk scan command, and within a few minutes you will find something that can be hacked easily.
In fact, scanning the Internet is useful because:
it's fun;
this is informative (you can see how small the Internet is by running the 0.0.0.0/0 scanning command, there are only 65 thousand ports on the Internet);
It will make you famous: - select a target, for example, a Siemens control system; - scan for it the Internet; - create for it a BlackHat Talk computer security conference; - use the privileges of an expert.
What you need to know in order to scan the Internet? First, you need to know the theoretical part of the physical infrastructure:
data packets have a fixed size: - Ethernet packets contain 44 bytes; - TCP SYN packets contain 40 bytes.
Maximum speeds of 1 Gbit / s Ethernet: - 476 Mbit / s for real traffic; - 524 Mbps for Ethernet connection; - 1,488,000 packets per second.
This means that we are overpaying the provider because it charges us for guaranteed bandwidth, that is, for guaranteed, not actual data. This is due to excessive packet size. If we transmit 22 or 33 bytes, they are still packaged in a 40 or 44 byte packet. The user almost never can reach the full transfer capacity given in 1 gigabit, because in reality it transfers no more than 524 megabits per second. But due to fixed packet overflow, the data is oversized and this stock is not used at all. But we pay for it. Even if we have a perfectly tuned switch, we still will not be able to use the full bandwidth of the network, and I do not know why this is happening. Confusion exists in the system of payment of bills for Internet services.
How are traffic bills generated by Internet providers?
some provide us with a maximum speed of 1 Gbps Internet connection;
some measure the actual bandwidth of the working network, providing us with a speed of about 600 Mbit / s;
Some ISPs do not see small packets, so they only capture inbound traffic, not outbound traffic. For example, we transferred a ton of information, and paid for several megabytes that we downloaded from the network;
Some providers do not measure traffic at all, and this is of particular interest to us!
For example, in Germany there is a CCC club, which provides users with a speed of 100 Gbit / s. I could not test this network, but maybe this year I will take my 10 Gigabit Ethernet card with me and check if this is true. But the problem is that when we send too small packets, we thereby violate the existing agreements between the peers of the same network.
Consider the physical network infrastructure further.
Private VPNs are able to adapt to the load of small packets. Ethernet struggles with small packets, and speeds above 500 Kbps are often difficult. If your switch is able to work at such a speed, this does not mean that the rest of the infrastructure can support it. In this case, it can be helpful to disable the flow control by the Flow Control, in which the transmitter brakes the data transfer if the receiver is not ready to accept it.
In some cases, packets can be lost - transmission at a speed of 500 Kbps does not guarantee that all packets will reach the Internet. Scanning allows you to identify ports that use packet loss. You can use only those ports that provide the same reception and transmission: if you send 10 thousand packets, you will also receive 10 thousand packets. Therefore, I mainly use speeds of up to 150 Kbps, and sometimes even 15 Kbps, this allows not to think about the integrity of the packets.
A big problem is complaints of abuse, Abuse Complaints. This term means that someone has marked you as a source of spam or other malicious activity. Often this happens with companies when the addressee does not want to receive more letters from you, but cannot unsubscribe from the mailing list, because your company did not give him a link for this. It marks your mail as spam, and it harms the overall reputation. This can happen when scanning a network. You can get Abuse Complaints, and your ISP will get upset. Or you break the agreement between the feasts, you will not be allowed to play the role of a feast. However, there are much worse things:
Scanning the Heartbleed threat will generate Abuse Complaints a few weeks later and you will still get a blow to your reputation;
HTTP scanning can send you to the fail2ban ban list, that is, your IP will be blocked;
Snort Threat Rules can violate Abuse Complaints as well.
The existing network monitoring methodology tracks incoming traffic. If you use a scan, your incoming traffic will be large and you will be suspicious. It is believed that hackers can be traced this way, although it’s like looking for keys lost in the bushes under a street lamp just because it's lighter there.
What should Internet service providers take seriously? To the fact that some networks use blackholing (“routing to nowhere” when such routing packets are deleted due to “No route to host”) for the whole autonomous AS network.
The exclusion list is required when scanning, since we do not want to scan other people's mailboxes and private network segments. Creating an exclusion list is done using the scan parameters in the command line:
An important thing is to create a public list of exceptions. We would very much like to create a public list of security experts, but most of those who send us a request to participate in the program are usually asked to remove them from this list. They are afraid that someone will know their IP or corporate network addresses and try to hack them. Fortunately, BGP networks have all this information in public access, which is laid out in a fairly elegant format and is accessible to everyone. People need to understand that scanning the Internet will only benefit them and in no way affect personal information that they do not want to show anyone. Unfortunately, most have to prove this, because they confuse scanning and hacking. And in general, people find it difficult to believe that you can scan the entire Internet.
For example, a company has a certain network, which we scan at their request, but they also have a subnet in which important information is stored. So, after seeing how the scan goes, they get scared and say: “You even scan hidden networks, you see a range of ports and addresses, so you can hack us”!
An interesting story happened six months ago. I scanned the network for one customer, and he was woken at night by a call about an emergency conference called due to the hacking of the network. He called me and I had to calm him down and explain that the scan had nothing to do with the hacker attack, they just found their vulnerabilities before we. Often, clients believe that as soon as they give us permission to scan, some security gaps immediately open up, and hackers immediately climb into it.
Another case was with a guy from Australia. He noticed that when we scanned the network, we sent him a request for a connection in the form of a single SYN packet, and called me, they say, who we are and on what basis we are doing this. I explained everything, told him the address of our site, where there are all the rules and regulations, said that we do it absolutely legally, according to customer orders. He did not want to listen to anything and began to threaten us with the Internet police, that he would immediately call to where he should be and we would all be arrested. It’s just some crazy person who didn’t understand that if we were engaged in scanning illegally, in an hour we would have been caught by everyone, because we are acting completely openly.
Such complainants are often just stupid. They do not understand that the vast majority of processes taking place on the network, all ports, routers, switches, sessions are constantly open and are not protected by any encryption. Otherwise, the Internet could not work at all if permission was required for every action. If a person is afraid that his bank card data may be stolen, it is better not to use the Internet at all. And this is happening against the background of the fact that people are not able to simply configure their devices so as to close the existing gaps. They leave them open to everyone, and then they are surprised that they have become the prey of hackers. I want to show you the letter we received with such content: “The infrastructure of the Woori financial group is classified as“ class A national security equipment ”and unauthorized access to this equipment is prohibited by relevant laws and regulations”. This company is located in Korea, and we first learned about it from this letter, because they not only sent a complaint to us, but also explained their actions by letter. I first met an entire organization that wants to have access to the Internet while securing all its equipment. Why then go on the Internet at all if it cannot be done with closed ports?
An important aspect for our work is close cooperation with Internet providers. We have to be friends with them, otherwise we will not get an effective scan. We offer them free advice on Internet security, they help us correct the list of complaints received. That is, the provider understands who and why has complained about us, and rejects unfounded accusations against us. Together with them, we are creating a SWIP project - “Who is who on the Internet” with a list of verified IP addresses, and we put in our “black list” those who insist that we be banned for scanning.
As an alternative, helping to avoid some misunderstanding, you can create an anonymous virtual dedicated VPS server. It has the following advantages:
VPS provider can pay a small bitcoin amount;
you can scan without any complaints, since you simply disconnect your account from the network after conducting a study, for example, VPS on Linode hosting allows you to delete an account immediately after paying $ 50;
A sufficient number of such providers favorably refer to spammers and scammers operating under the guise of virtual servers.
What does masscan technology look like?
It is similar to the nmap utility, which is designed to scan IP networks with any number of objects and determine the status of ports and their corresponding services:
All nmap options can be parsed in parts, except those that are said: “This nmap option is not supported”;
when using some tools, it is useful that the output data formats are close to nmap;
Many features are supported, such as transmission protocol scanning with SCTP flow control or the use of UDP user data protocol as the nmap payload.
But the masscan is not like nmap:
Port-at-a-time mode instead of Host-at-a-time mode. This means that the results for each port are reported immediately, as it will be detected, and these results are not combined with each other using a host. That is, our program does not need to send a request, receive an answer and spend time on it. There is no need to store in memory a billion requests and a billion answers, so it works faster.
It works asynchronously: the transmitted array is created from queries, the resulting array is created from responses;
she scans 1000 times faster.
Nmap is the best scanner - its NSE scripting engine is very flexible, and scanning multiple hosts without problems. Masscan is designed for large networks, as this program is much faster and better scaled.
Masscan has its own TCP / IP stack:
it works in parallel with the existing stack;
by default has the same address;
serves to duplicate the ARP network layer protocol and TCP RST packets.
This is how hacker attacks are performed with address spoofing, the so-called spoofing attacks. Suppose we have a host A - an attacker, a host V that is being attacked, and a host O, whose IP address the hackers want to use for the attack.
Host A sends a SYN packet to host V, but the return address does not indicate its IP address, but the host address O. The attacked host V responds to host O with a SYN / ACK packet. But Host O did not send anything to Host A and therefore must break the connection with the RST packet. Suppose that host O did not send such a packet because it was overloaded, or turned off, or is protected by a firewall that blocks SYN / ACK packets.
If Host O has not sent the RST packet and has not interrupted the attack that has begun, Hacker Host A can interact with Host V, impersonating Host O. Therefore, any authorizer, captcha checking and so on becomes useless if the user's firewall is not configured correctly.
Thus, RST packets protect IP connections from establishing communications, that is, they respond to packets with a SYN-failure. With their help, we can protect various IP addresses from spoofing, or install a protective filter for a specific range of ports.
Now let's talk about the teams that run masscan.
Scanning multiple devices is done in such a way that security protocols are not violated:
- - shard 1/50 is used when you need to scan multiple computers;
- - source-ip 10.0.0.32 - 10.0.0.63 extends the scanning range to several IP addresses on the same computer;
- - source- ip 0.0.0.0 - 255.255.255.255 should not be used at all! You just do not see any results, and your computer freezes.
Sometimes, to avoid problems, manual TCP / IP connection configuration is performed:
- - source-ip 192.168.10.15;
- - source-port 4444;
- - router mac 00-11-22-33-44-55 with - - router-ip 192.168.10.1.
This is what the banner verification team does:
establishes a TCP connection;
performs heuristic protocol analysis, that is, it scans port 443 for SSH and HTTP, which the Internet addresses to this port.
At the moment I am using something similar to NSE scripting, but I will soon switch to C-based programming.
You can also use load testing. This can “penetrate” the protection of firewalls and is therefore important for testing their ability to provide security. In this case, the commands - - infinite, - - banners, - - sourse-ip <range> are useful for quickly scanning a large number of devices.
Usually nobody uses it, but in our program there is the possibility of using outgoing binary files, for this the command is used:
– oBfoo.scan –oXfoo.xml
Then the conversion is performed:
masscan–readscanfoo.scan –oXfoo.xml
This method provides a more compact scan. In addition, if there are errors in the outgoing data, it is easier to correct them in binary format.
Another useful feature is spoofing scanning. IP spoofing consists in spoofing an IP address in the packet body so that the response packet is intercepted by the hacker address. This technology is used by hackers to intercept traffic between hosts on Ethernet networks.
Spoofing scan consists of the following:
receiving a packet with one IP address, for example, a smartphone running Android;
received packets have low bandwidth;
sending packets from a data center without outgoing filters, the - - source-ip command allows you to scan the spoofing of another IP address.
Here's what the scan results look like. On the first picture you see the window of our program, on the second - the result of its work.
The result of testing for the threat of Heartbleed shows that as of April 10, the vulnerability was found in 600,000 systems, and in July 300,000 systems were still vulnerable, most of which were hardware devices, that is, the computers themselves, routers, webcams and servers. That is, you will not see their vulnerability, if you check them with DNS names, only scanning by IP addresses helps. We also scanned mainframes — large fault-tolerant servers, such as TN3270 Telnet –over-SSL over port 992. You can take a look at @ mainframed767 and see interesting things like the authorization window of the IBM main server user.
The third picture shows the results of scanning banners. Now I will try to show you our program in action. To do this, I open the main window and use the command line to set the address of the server I want to scan. In some cases, the server does not respond.
Now Paul will try to log in and demonstrate the scanning capabilities.
Paul says that if you have any questions about using the program, you can contact him directly and get the necessary explanations. As an example, Paul crawls the Internet through a VNS 5900 server, which takes 15 to 20 minutes. The advantage of our program is the ability to obtain a list of vulnerabilities without the need for authorization on the network or on each network device. We test the system from the outside, not from the inside. Using scaling allows you to check huge amounts of Internet networks, including clouds, and it costs less than 16 cents per hour.
Right now I am asking for a slow defcon network scan on port 80 at 10 packets per second, and the result immediately appears on the screen.
At the moment, we see how many open unprotected ports in the network are devices with corresponding IP addresses. And a hacker can use these IP addresses for his spoofing attack.
This procedure does not interfere with the network, which is subjected to scanning, the user can run any application. So, defcon network scanning took a little more than a minute, and we identified all existing vulnerabilities by simply scanning port 80. By setting the packet size, you can speed up or slow down the scanning. We told you everything you need to know about the masscan program, and if you have any questions, email us or twitter @erratarob and paulm .
Thank you for staying with us. Do you like our articles? Want to see more interesting materials? Support us by placing an order or recommending to friends, 30% discount for Habr users on a unique analogue of the entry-level servers that we invented for you:The Truth About VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps $ 20 or how to share the server? (Options are available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).