Ubuntu, KVM and proxy_arp - how to fool an evil provider

One company located a server for internal use at a kolokeyshnik and immediately bought / 30 addresses for their needs. Configured as aliases (eth0: 0, eth0: 1, etc.). Everything worked great, until after some time there was a sensible idea of ​​spreading different services to different virtual machines. Since Ubuntu Server was used as a host, the choice of KVM as a virtualizer happened by itself. Both here and the rest of the net there are already a lot of clever words written about installing and configuring KVM and the network environment, I will not stop here, I’ll tell you only about the little children’s rakes conveniently planted by the provider.
First of all, it should be noted that everything happened on a lively actively used technological machine, where the breaks in the provision of the service were undesirable, because all reconfigurations were done at night with the corresponding brain state;)
So, leaving eth0 intact (so that important production services are not interrupted), extinguish all aliases, create a bridge br0, stick eth0 into it and start virtual machines, just like in all KVM letters written in, tapX into the same bridge.
Voila - IPs are visible, each other's cars are pinging, it's time to open champagne when it turns out that only the host is available from the Internet. Lowering the subtleties of searching for a problem, I immediately turn to the point - at the provider, where the server was located at the collocation, the switch is configured with port security, i.e. released out only nailed when installing the MAC. The provider refused to let in poppies of virtual machines, for which I do not blame him, and this is his right to establish an internal technical policy. In response to the confused question: “What about us?” Was answered again from the KVM primer: leave aliases on the interface, specify the tenth grid on the virtual machines and then “iptables -j DNAT bla-bla-bla”
Having grieved a little about some clumsiness of such a solution and thoughtfully smoked Google, an alternative variant was found with the keyword proxy_arp.
First of all we do apt-get install uml-utilities
In /etc/network/interfaces register:
auto eth0
iface eth0 inet static
address 1.2.3.1
netmask 255.255.255.0
network 1.2.3.0
broadcast 1.2.3.255
gateway 1.2.3.254
post-up sysctl -w net.ipv4.ip_forward=1
post-up sysctl -w net.ipv4.conf.eth0.proxy_arp=1
pre-down sysctl -w net.ipv4.ip_forward=0

auto qtap1
iface qtap1 inet manual
tunctl_user root
uml_proxy_arp 1.2.3.2
uml_proxy_ether eth0
up ip link set qtap1 up
down ip link set qtap1 down

auto qtap2
iface qtap2 inet manual
tunctl_user root
uml_proxy_arp 1.2.3.3
uml_proxy_ether eth0
up ip link set qtap2 up
down ip link set qtap2 down

We start virtual computers:

kvm -m 512 -hda image.img -net nic,macaddr=00:01:02:03:04:05 -net tap,ifname=qtap1,macaddr=00:01:02:03:04:05,script=no -daemonize -vnc :1 , etc.

Further, in virtual machines as gw, we write the address of the host 1.2.3.1 and we get a working bunch of virtual machines, which were hidden from the provider behind the MAC host.
')
For those who do not yet consider themselves to be a network guru, a few words about proxy_arp and how it differs from the bridge.

By default, a regular bridge simply transmits packets from one interface to another unchanged. It only considers the hardware address of the packet to determine in which direction the packet should be sent. This means that Linux can forward any kind of traffic, even one that it does not know, if the packets have a hardware address.

A pseudo-bridge works somewhat differently and is more like a hidden router than a bridge, but like a bridge it has some influence on the network architecture. True, this is not exactly a bridge, since the packets actually pass through the core and can be filtered, modified, redirected, or sent along a different route.

Another advantage of the pseudo-bridge is that it cannot transmit packets of protocols that “do not understand” - which prevents the network from being filled with any kind of “garbage”.

In this scheme, when a provider switch wants to establish a connection with a virtual machine located behind the host, it sends the ARP request packet to the host, which, translated into human language, may sound like this: “Who has the address 1.2.3.2 installed? Report at 1.2.3.254. " In response to the request, 1.2.3.1 will respond with a short package “I am here! My hardware address is xx: xx: xx: xx: xx: xx. "

When 1.2.3.254 receives a response, it will remember the hardware address 1.2.3.2 and will keep it in the cache for a while.

When configuring proxy_arp, we told the eth0 interface to respond to ARP requests. This will force the router to send packets to the bridge, which will then process them and transfer them to the appropriate virtual machine.

Thus, whenever a provider router on one side of the bridge requests the hardware address of a computer on the other side, the bridge responds to the request, as if to say “pass everything through me”, as a result of which all traffic will fall on eth0 and will be transferred to necessary virtualku.

In preparing the article, materials from the Linux Advanced Routing & Traffic Control HOWTO were used.