ARP: Cisco equipment features and interesting cases. Part 2

Hi, Habr! In the previous part of this article, we looked at the features of ARP operation on Cisco routers related to NAT rules and the Proxy ARP feature. In this post I will try to make out the differences in ARP work between Cisco routers and Cisco ASA firewalls. In conclusion, I will share a few interesting cases from practice related to the work of ARP.

NAT and Proxy ARP on Cisco ASA Firewalls
')
The behavior of the Cisco ASA firewall in terms of ARP for NAT rules and proxy ARP settings is different from Cisco routers. By default, when you configure NAT rules on a Cisco ASA, the device will respond to ARP requests corresponding to the internal global addresses of the NAT rules. However, this behavior does not depend on the belonging of the ASA's internal global IP address to the IP subnet. This behavior is provided by setting the noproxyarp <interface name> sysopt option.

Consider examples based on the following simplest topology:



Cisco ASA Interface Settings:

interface Vlan1 nameif inside security-level 100 ip address 192.168.20.1 255.255.255.0 ! interface Vlan2 nameif outside security-level 0 ip address 198.18.0.1 255.255.255.0 

Settings for the NAT rule and access list on the outside interface:

 object network TEST-PC-tcp3389 host 192.168.20.5 nat (inside,outside) static 198.18.99.2 service tcp 3389 3389 access-list acl-outside-in extended permit tcp any object TEST-PC-tcp3389 eq 3389 access-group acl-outside-in in interface outside 

As you can see from the presented settings, we publish the tcp 3389 (RDP) port under the address 198.18.99.2, which does not belong to the IP subnet of the Cisco ASA interface.

Check the noproxyarp sysopt option:

 asa# sh run all sysopt . . . no sysopt noproxyarp inside no sysopt noproxyarp outside 

It can be seen that the option is not set (noproxyarp is not enabled). Let's try to open tcp-port 3389 of the address 198.18.99.2 and at the same time look at the sniffer:



Success: Cisco ASA responds to the ARP request and the port opens.

Let's try to set the sysopt noproxyarp outside option. Clear the arc-cache on the computer connected to the outside-interface, and try to open the port again:





The ASA no longer responds to ARP requests for the target IP address. However, it does not matter if the target IP address is on the IP subnet of the ASA interface. When the sysopt option is set, the noproxyarp ASA will respond to ARP requests addressed solely to the IP address of the interface.

You should not compare the configuration of the ip proxy-arp interface of the router with the setting of the Cisco ASA noproxyarp sysopt . The Cisco ASA option does not enable proxying of any ARP requests through the firewall. This option is solely responsible for maintaining the internal global IP addresses of NAT rules and static entries in the ASA ARP table configured with the alias keyword.


In some cases, the functionality of ARP responses must be disabled for specific NAT rules. To do this, use the no-proxy-arp keyword in the NAT settings. The most common example is setting up NAT exceptions when using VPN on a Cisco ASA. Suppose the local network for the Cisco ASA is 192.168.20.0/24, the local network on the remote side of Site-to-Site VPN is 192.168.30.0/24. The NAT exception for this VPN can be configured as follows:

 object network local-LAN subnet 192.168.20.0 255.255.255.0 object network remote-LAN subnet 192.168.30.0 255.255.255.0 nat (any,any) source static local-LAN local-LAN destination static remote-LAN remote-LAN 

The presented configuration indicates for the Cisco ASA that the local-LAN IP subnet 192.168.20.0/24 is fully published on each Cisco ASA interface (any, any in the NAT rule) under the internal global addresses of the same local-LAN IP subnet. Therefore, with this configuration, the Cisco ASA will respond to any ARP request to the target IP address from the IP subnet 192.168.20.0/24 received on any interface, including the inside interface.

Imagine a situation, a host with the address 192.168.20.5 wants to contact a host from the same IP subnet with the address 192.168.20.6. Formed ARP request to the target address 192.168.20.6. The ARP request is broadcast and reaches both the target host and inside the Cisco ASA interface. The configured NAT rule causes the Cisco ASA to respond with its MAC address to an ARP request. If the ARP response from the Cisco ASA arrives before the response from the target host, all traffic sent to the target host will go to the Cisco ASA, where it will be safely dropped.

In the example shown, Cisco ASA will work as a “black hole” for the local IP subnet, trying to “suck” all traffic onto itself. At the same time, the NAT policy element (the NAT setting after the destinations static keywords) does not save the situation. When compared with the router, this setting on the Cisco ASA is similar to the joint configuration of ip proxy-arp and ip local-proxy-arp on the interface of the router. To avoid this effect, the no-proxy-arp keyword must be added to the NAT rule on the Cisco ASA:

 nat (any,any) source static local-LAN local-LAN destination static remote-LAN remote-LAN no-proxy-arp 

Note : The described effect can be avoided by specifying exact interfaces in the NAT rule settings, instead of any keywords. For example, nat (inside, outside) ...

Results

Before proceeding to the description of cases from practice, I will highlight the main points of the second part of the article:

1. The Cisco ASA will by default respond to ARP requests to the inside global IP address of the NAT rule, regardless of the IP subnet of the interface. This behavior is controlled by setting the sysopt noproxyarp <interface name> option.
2. Cisco ASA allows you to disable ARP responses for specific NAT rules using the no-proxy-arp keyword. In particular, it is necessary to disable ARP responses for NAT exclusion rules in order to avoid problems with communication in the local network.
3. Proxy ARP functionality is not explicitly configured on the Cisco ASA, but the desired effect can be achieved using NAT rules.

So, cases from practice. Immediately, I note that all the problems described occurred when I or my colleagues connected Cisco network equipment to Internet service providers. In our experience, this particular scenario is most often accompanied by problems with ARP.

Case number 1. Secondary IP address of the provider

Connected to the provider of the Cisco ASA firewall. The connection was successful and all services worked correctly. However, after a while the connection disappeared. Detailed analysis showed that if you initiate an outgoing connection, it works stably (traffic goes both ways). The problem appears only with incoming connections from the Internet (for example, a remote connection to the router). At the same time, the direct connection of incoming connections to outgoing ones is traced: if there are outgoing connections, then the incoming connections begin to work correctly. However, after some time, connecting remotely or “pinging” the device from the Internet fails again.

Since presumably everything is in order with the physical and channel levels, we began to check the operation of ARP. We started debug arp on ASA and tried to clear arp-cache. On debug messages, we saw that the ASA correctly sends an ARP request and receives an ARP response from the provider without any problems. In this example, the IP of our ASA is 80.XX4, its MAC address is a0ec. ****. ****, the IP gateway of the provider 80.XX1, the MAC gateway of the provider aa43. ****. ****:

 arp-send: arp request built from 80.XX4 a0ec.****.**** for 80.XX1 at 978772020 arp-refresh: Trying to refresh ARP for outside 80.XX1 arp-in: response at outside from 80.XX1 aa43.****.**** for 80.XX4 a0ec.****.**** having smac aa43.****.**** dmac a0ec.****.****\narp-set: added arp outside 80.XX1 aa43.****.**** and updating NPs at 978772020 arp-in: resp from 80.XX1 for 80.XX4 on outside at 978772020 

However, after some time, we noticed a message in debug arp on the ASA:

 arp-in: request at outside from 195.YY1 aa43.****.**** for 80.XX4 0000.0000.0000 having smac aa43.****.**** dmac ffff.ffff.ffff\narp-in: Arp packet received from 195.YY1 which is in different subnet than the connected interface 80.XX4/255.255.255.0 

Judging by this message, the ASA receives an ARP request with the correct Sender MAC Address aa43. ****. **** but the wrong Sender IP Address is 195.YY1. This invalid ASA ARP request discards and does not send an ARP response.

Thus, when outbound traffic exists, the ASA sends an ARP request (if necessary, when the corresponding entry in the ASA ARP table requires updates) to the provider side and receives an ARP response. Thanks to the ARP request from the ASA, the provider equipment also receives an ASA entry in its ARP table. However, when outbound traffic from the ASA is absent for a long time and the ASA does not send an ARP request, the ASA entry expires on the provider’s equipment in the ARP table. Equipment provider tries to update the record by sending an ARP request, but does not receive a response. From here and there are "floating" problems with communication.

It remains to understand why the provider’s equipment sends an ARP request with the wrong Sender IP Address. Later, the provider checked this situation on its part and even showed a dump of ARP traffic, which confirmed the ASA debug messages:

 324: 22:03:41.056546 802.1Q vlan#2 P0 arp who-has 80.XX4 tell 195.YY1 325: 22:03:41.937329 802.1Q vlan#2 P0 arp who-has 80.XX4 tell 195.YY1 326: 22:03:42.822909 802.1Q vlan#2 P0 arp who-has 80.XX4 tell 195.YY1 

It turned out that on the provider interface, the address 195.YY1 was configured as primary, and the IP address 80.XX4, which was the default gateway for us, was configured as secondary. This setting fully explained the situation. In this case, the problem was solved by adding a static ARP entry on the equipment of the provider.

We had a completely similar situation on another site when we used a Cisco router to connect to the provider. On the equipment of our provider, our gateway was also configured as a seconday IP address. In this case, to speed up the problem solving process, we added a secondary IP address on the router from the same subnet as the primary IP address on the provider interface. After this, our router began to respond successfully to ARP requests from the provider, since an IP address from the same subnet as the one in the ARP request appeared on our interface.

Case number 2. ARP incompatibility

We were assigned a task in one of the offices to replace the Cisco ISR router with a Cisco ASA firewall. The ASA firewall was previously configured and sent to the installation point. After connecting to the Cisco ASA provider, it was unavailable for remote connection. At first glance, everything worked correctly on the device. The ASA firewall correctly determined the MAC address of the ISP’s router using the standard ARP request / ARP response procedure. The packages went from the external interface of the device to the Internet. At the same time, nothing came to the ASA. This fact was recorded by the built-in packet capture.

After contacting the provider, it was determined that the packages from the ASA were successfully delivered to the equipment of the provider, the provider also saw response packets coming from the superior equipment. After the router was reconnected, the traffic again began to go correctly in both directions. This meant that the problem was somewhere at the junction between the provider and the ASA firewall. After re-contacting the provider with a detailed description of the problem, it was determined that the provider equipment does not see the MAC address of the ASA firewall. The assembled demo stand proved that the ASA was working correctly (the Cisco router played the role of the provider). For some reason, the provider device did not add an ASA entry to the ARP table after receiving an ARP reply from the ASA. Most interesting is that the ARP request coming from the Cisco ASA was not dropped. The provider equipment correctly responded to the ARP request from the ASA, but also did not add the ASA entry to the ARP table.

As a result, the provider was asked to make a static binding in the ARP table. This case showed ARP equipment incompatibility with the Cisco ASA firewall. Unfortunately, the provider has not announced the manufacturer of its equipment.

Case number 3. Gratuitous ARP

And again we connect to the Cisco ASA provider. This time instead of the MS TMG server. The peculiarity of this case is that MS TMG was connected to the provider's device via an L2 switch. The ASA was also supposed to connect via an L2 switch:



So, disable MS TMG and instead connect the Cisco ASA to the same port of the L2 switch. We see the standard situation: traffic leaves the external Cisco ASA port, but there are no response packets. After contacting the provider, it turns out that the provider’s hardware still sees the MAC address of the MS TMG server behind the IP address that was passed to the Cisco ASA.

Further investigation showed that the Cisco ASA firewall does not send a Gratuitous ARP message when the interface transitions from the DOWN state to the UP state. And since the equipment of the provider is connected via an L2 switch, when changing the device on our side, the channel does not drop to the provider, and the interface of the router of the provider always remains on. The only way to promptly inform the provider that the device and the MAC address have changed on our side is Gratuitous ARP. Otherwise, you will have to wait for the provider to have an ARP entry timeout, which is usually 4 hours.
In this case, we made no ip address commands on the Cisco ASA interface, ip address xxxx yyyy After that, the ASA still sent Gratuitous ARP, and everything took off.

Conclusion

In this article, which consists of two parts, I tried to consider the subtleties of ARP operation on Cisco equipment in cases of using Network Address Translation (NAT), as well as the functionality of Proxy ARP. Disassembled the differences in the operation of ARP between Cisco routers and Cisco ASA firewalls. In conclusion, I looked at the problems that arise when connecting to an Internet provider, due to the work of ARP.

The cases described show how important it is to remember to check ARP in the process of solving and eliminating network problems. I hope the material of this article will be useful for a deeper understanding of the work of ARP.

Waiting for comments. Maybe someone can tell their own interesting cases related to the work of ARP.