Floating IP Addresses for Networking in OpenStack Public and Private Clouds
Posted by: Piotr Siwczak
I recently described how VlanManager works and how it provides network scalability and user isolation. But until now, I’ve only talked about networks with a fixed IP address that belong to different users. And although by default, instances are assigned fixed IP addresses, they do not guarantee the immediate availability of instances from outside the network (or from other data centers). Imagine the following scenario:
You run a small website with one www-server, a database server, and a firewall that performs network address translation (NAT) and traffic filtering. Usually you apply the following settings:
-All servers communicate within the network within the private (non-routable) network range (for example, 192.168.0.0/24).
-There is one public routable IP range, which is visible www-server.
')
You do the following:
- Assign a public IP address to the firewall.
-Create a NAT rule on the firewall to route traffic from the public IP address to the private IP address of the www-server.
The fixed IP addresses in OpenStack work the same way as the network range 192.168.0.0/24 in the example above. They guarantee interaction between instances within the same OpenStack cluster. But OpenStack also introduces another pool of IP addresses called “floating IP addresses”. Floating IP addresses are publicly routable IP addresses that you purchase from an Internet service provider (the one that fits into the above firewall). Users can assign IP addresses to their instances, making them accessible from the external network.
Floating IP addresses are not assigned to virtual machines by default. Cloud users must explicitly “take” them from the pool configured by the OpenStack administrator, and then assign them to their virtual machines. As soon as the user takes a floating IP address from the pool, he becomes his “owner” (that is, at any time he can detach the IP address from the virtual machine and attach it to another). If the virtual machine for some reason ceases to exist, the user does not lose the floating IP address - he can then assign it to another instance. Unfortunately, it’s not possible to share a single floating IP address with multiple virtual machines for load balancing, such as elastic load balancing on Amazon EC2.
On the other hand, fixed IP addresses are dynamically assigned by the nova-network component when the VM is booted. There is no way to order OpenStack to assign a specific fixed IP address to a virtual machine. Thus, you may find yourself in a situation where, if the virtual machine is accidentally shut down after it recovers from the snapshot, a new instance is loaded with a new fixed IP address.
System administrators can configure multiple pools of floating IP addresses. However, unlike fixed IP address pools, floating IP address pools cannot be associated with specific users. Each user can “take” a floating IP address from any pool of floating IP addresses. But the main motivation for creating multiple pools of floating IP addresses is for each pool to serve its own Internet access provider. Thus, we can guarantee connectivity, even if one of the suppliers fails.
As a result, the basic functions of floating IP addresses are as follows:
- Floating IP addresses are not automatically assigned to virtual machines by default (they must be assigned to instances manually).
-If the virtual machine ceases to exist, the user can reuse the floating IP address by assigning it to another instance.
- Users can take floating IP addresses from various pools defined by the cloud administrator to ensure that they can connect to virtual machines by different Internet service providers or from external networks.
“Public availability” of a floating IP address is a relative concept. For public clouds, you might want to define a floating IP address pool as a pool of IP addresses publicly accessible from the Internet. Then your clients will be able to assign them to virtual machines to log in through SSH from their home or work computers:
If a corporate cloud is running in your data center, then the floating IP pool can be any IP address range that provides access to OpenStack instances from the rest of the data center.
For the traffic of your data center you can define the following range: 10.0.0.0/16.
Inside OpenStack, you can create the following range of fixed IP addresses: 192.168.0.0/16, divided into user subnets.
To make OpenStack instances available from the entire data center, you can define a floating IP address pool as a subnet 10.0.0.0/8 (for example, 10.0.0.0/16) and register it with OpenStack so that users can take IP addresses from there.
As I mentioned earlier, the system administrator first registers a pool of floating IP addresses with OpenStack:
Thus, the public pool becomes available to users.
Now users follow this procedure:
-Download a copy:
+ -------------------------------------- + --------- + -------- + -------------------------------- +
| ID | Name | Status | Networks |
+ -------------------------------------- + --------- + -------- + -------------------------------- +
| 79935433-241a-4268-8aea-5570d74fcf42 | inst1 | ACTIVE | private = 10.0.0.4 |
+ -------------------------------------- + --------- + -------- + -------------------------------- +
-List the available floating IP address pools:
nova floating-ip-pool-list
+ ------ +
| name |
+ ------ +
| pub |
| test |
+ ------ +
- Take a floating IP address from the “pub” pool (or, if desired, from the “test” pool):
nova floating-ip-create pub
+ --------------- + ------------- + ---------- + ------ +
| Ip | Instance Id | Fixed Ip | Pool |
+ --------------- + ------------- + ---------- + ------ +
| 172.24.4.225 | None | None | pub |
+ --------------- + ------------- + ---------- + ------ +
- Assign a floating IP address to the instance:
(where the first argument is the uuid of the instance, and the second is the floating IP itself)
-Check the correctness of all settings:
nova floating-ip-list
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
| Ip | Instance Id | Fixed Ip | Pool |
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
| 172.24.4.225 | 79935433-241a-4268-8aea-5570d74fcf42 | 10.0.0.4 | pub |
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
The instance should now be visible from outside the OpenStack cluster to a floating IP address.
What happens inside an instance after adding a floating IP address? The correct answer is nothing. If you connect to it over SSH and look at the network configuration, you will see that there is one network interface with a fixed IP address configured.
All configuration is performed on the compute node. All work related to the floating IP address is performed by the nova-network service: a network address translator (NAT) is organized between the fixed and floating IP addresses of the instance. An explanation of how NAT works can be found here .
Take a look at the following chart:
The diagram displays a single compute node configured in network mode with node distribution and VlanManager used to configure fixed IP networks. The computing node is equipped with two network interfaces: the eth0 interface is dedicated to traffic with a fixed IP / VLAN, and eth1 is the interface through which the computing node is connected to an external network; it contains floating IP addresses. (For information on how VlanManager configures fixed IP networks, see the previous article. )
Note that while the eth0 interface (fixed / private) does not have an address configured, the eth1 interface is assigned an IP address, which is the default gateway for the compute node (91.207.15.105).
When a user assigns a floating IP address (91.207.16.144) to VM_1, two things happen:
- The floating IP address is configured as a secondary address on the eth1 interface: this is the output of the “ip addr show eth1" command, containing the following actions:
inet 91.207.15.105/24 scope global eth1 # primary eth1 ip
inet 91.207.16.144/32 scope global eth1 # floating ip of VM_1
- A set of NAT rules for a floating IP address is configured in the iptables tables. Below are all the corresponding entries from the “nat” table of the compute node (with the exception of the “iptables –S -t nat" command. Details on how to configure NAT using iptables tables in Linux can be found here ):
# this rule ensures that packets originating from compute node
# where the instance resides by its floating IP:
-A nova-network-OUTPUT -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
# secures that all external traffic to the floating IP
# is directed to the IP of the instance
-A nova-network-PREROUTING -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
# all the traffic originating from the instance will be SNAT-ted to its floating IP
-A nova-network-float-snat -s 10.0.0.3/32 -j SNAT --to-source 91.207.16.144
In general, nova-network adds additional chains to those that are predefined in the NAT table. The order of the chains with respect to the floating IP traffic is shown below (using the rules shown above):
Chain OUTPUT - Chain nova-network-OUTPUT - Rule: -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
Chain PREROUTING - Chain nova-network-PREROUTING - Rule: -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
Chain POSTROUTING - Chain nova-postrouting-bottom - Chain nova-network-snat - Chain nova-network-float-snat - Rule: -s 10.0.0.3/32 -j SNAT --to-source 91.207.16.144
The code responsible for setting the rules is located in nova / network / linux_net.py in the function:
def floating_forward_rules (floating_ip, fixed_ip):
return [('PREROUTING', '-d% s -j DNAT --to% s'% (floating_ip, fixed_ip)),
('OUTPUT', '-d% s -j DNAT --to% s'% (floating_ip, fixed_ip)),
('float-snat',
'-s% s -j SNAT --to% s'% (fixed_ip, floating_ip))]
We return to the chart. If the user wants to access the virtual machine by its IP address from the external network (for example, “ping 91.20.16.144 ″):
-The traffic reaches the public interface (eth1) of the compute node. In nova-network-PREROUTING, DNAT is executed, changing the destination IP address of the packets from 91.207.16.144 to 10.0.0.3.
- The computing node accesses its routing table and sees that the 10.0.0.0 network is available on the br100 interface (with the exception of the “ip route show” compute node):
10.0.0.0/24 dev br100
So he sends the packet to the br100 interface, then the packet reaches the virtual machine.
If the virtual machine sends the packet to the outside world (for example, “ping 8.8.8.8):
- Since the destination address is not located in the local network of the virtual machine, the packets are sent to the default virtual machine gateway with the IP address 10.0.0.1 (the device address is “br100 ″ on the compute node).
The compute node checks its routing tables and finds that there is no 8.8.8.8 address in directly connected networks, so it forwards the packet to the default gateway (which is the primary address of eth1 - 91.207.15.105 in this case).
-The packet enters the POSTROUTING chain and is transferred to the “nova-network-float-snat” chain, where its source IP address is rewritten to the floating IP address (91.207.16.144).
With OpenStack, system administrators give complete control over iptables tables to nova services. The set of custom rules is very complicated and is easily broken by any outside intervention. Moreover, each time the nova-network daemon is restarted, it applies all the rules in the iptables table chains associated with OpenStack. If there is a need to change the behavior of iptables tables in any way, this can be done by changing the code in the appropriate places of linux_net.py (for NAT rules, these will be floating_forward_rules).
It is also worth saying that nova-network does not track its tables in any way. Thus, if we manually remove any rules from chains associated with OpenStack, they will not be restored the next time the nova-network is started.
Thus, a system administrator can easily accidentally open unwanted access to a compute node. Remember that nova-network placed the floating IP address as secondary on the eth1 interface and configured DNAT rules that direct traffic to the fixed IP address of the virtual machine:
-A nova-network-PREROUTING -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
Thus, all traffic directed to 91.207.16.144 gets to the address 10.0.0.3.
Now let's imagine that the system administrator solved network connection problems at night and accidentally deleted all the NAT rules by typing:
iptables –F –t nat
The above NAT rule has been removed, but the eth1 interface is still assigned the secondary IP address of 91.207.16.144. Thus, we can still access address 91.207.16.144 from the outside, but instead of getting to the virtual machine, we now have access to the compute node itself (the destination IP address is no longer broadcast according to the DNAT rule, since we deleted all NAT rules). This security hole will not be closed until the next restart of the nova-network process, which re-creates the rules.
In the nova.conf service, there are flags that affect the behavior of floating IP addresses:
# the interface to which ips are attached
# as secondary addresses
public_interface = "eth1"
# the pool from which floating IPs are taken by default
default_floating_pool = "pub"
# I can add a message to ip automatically
auto_assign_floating_ip = false
In addition to providing access to virtual machines directly from the Internet, the floating IP address mechanism gives cloud users some flexibility. After they “pick up” a floating IP address, they can change their ownership, that is, on the fly, assign them to various virtual machines and reassign them, which greatly facilitates the release of new code and system updates. For system administrators, this poses a potential security risk, since the underlying mechanism (iptables) works very hard and is not monitored by OpenStack. It is therefore very important that only OpenStack software can change firewall policies and that they do not change manually.
Original article in English
I recently described how VlanManager works and how it provides network scalability and user isolation. But until now, I’ve only talked about networks with a fixed IP address that belong to different users. And although by default, instances are assigned fixed IP addresses, they do not guarantee the immediate availability of instances from outside the network (or from other data centers). Imagine the following scenario:
You run a small website with one www-server, a database server, and a firewall that performs network address translation (NAT) and traffic filtering. Usually you apply the following settings:
-All servers communicate within the network within the private (non-routable) network range (for example, 192.168.0.0/24).
-There is one public routable IP range, which is visible www-server.
')
You do the following:
- Assign a public IP address to the firewall.
-Create a NAT rule on the firewall to route traffic from the public IP address to the private IP address of the www-server.
The fixed IP addresses in OpenStack work the same way as the network range 192.168.0.0/24 in the example above. They guarantee interaction between instances within the same OpenStack cluster. But OpenStack also introduces another pool of IP addresses called “floating IP addresses”. Floating IP addresses are publicly routable IP addresses that you purchase from an Internet service provider (the one that fits into the above firewall). Users can assign IP addresses to their instances, making them accessible from the external network.
Differences between floating and fixed IP addresses
Floating IP addresses are not assigned to virtual machines by default. Cloud users must explicitly “take” them from the pool configured by the OpenStack administrator, and then assign them to their virtual machines. As soon as the user takes a floating IP address from the pool, he becomes his “owner” (that is, at any time he can detach the IP address from the virtual machine and attach it to another). If the virtual machine for some reason ceases to exist, the user does not lose the floating IP address - he can then assign it to another instance. Unfortunately, it’s not possible to share a single floating IP address with multiple virtual machines for load balancing, such as elastic load balancing on Amazon EC2.
On the other hand, fixed IP addresses are dynamically assigned by the nova-network component when the VM is booted. There is no way to order OpenStack to assign a specific fixed IP address to a virtual machine. Thus, you may find yourself in a situation where, if the virtual machine is accidentally shut down after it recovers from the snapshot, a new instance is loaded with a new fixed IP address.
System administrators can configure multiple pools of floating IP addresses. However, unlike fixed IP address pools, floating IP address pools cannot be associated with specific users. Each user can “take” a floating IP address from any pool of floating IP addresses. But the main motivation for creating multiple pools of floating IP addresses is for each pool to serve its own Internet access provider. Thus, we can guarantee connectivity, even if one of the suppliers fails.
As a result, the basic functions of floating IP addresses are as follows:
- Floating IP addresses are not automatically assigned to virtual machines by default (they must be assigned to instances manually).
-If the virtual machine ceases to exist, the user can reuse the floating IP address by assigning it to another instance.
- Users can take floating IP addresses from various pools defined by the cloud administrator to ensure that they can connect to virtual machines by different Internet service providers or from external networks.
Floating IP Addresses — Internal or External Clouds
“Public availability” of a floating IP address is a relative concept. For public clouds, you might want to define a floating IP address pool as a pool of IP addresses publicly accessible from the Internet. Then your clients will be able to assign them to virtual machines to log in through SSH from their home or work computers:
If a corporate cloud is running in your data center, then the floating IP pool can be any IP address range that provides access to OpenStack instances from the rest of the data center.
For the traffic of your data center you can define the following range: 10.0.0.0/16.
Inside OpenStack, you can create the following range of fixed IP addresses: 192.168.0.0/16, divided into user subnets.
To make OpenStack instances available from the entire data center, you can define a floating IP address pool as a subnet 10.0.0.0/8 (for example, 10.0.0.0/16) and register it with OpenStack so that users can take IP addresses from there.
Work with floating IP addresses
As I mentioned earlier, the system administrator first registers a pool of floating IP addresses with OpenStack:
nova-manage floating create --ip_range=PUBLICLY_ROUTABLE_IP_RANGE --pool POOL_NAMEThus, the public pool becomes available to users.
Now users follow this procedure:
-Download a copy:
+ -------------------------------------- + --------- + -------- + -------------------------------- +
| ID | Name | Status | Networks |
+ -------------------------------------- + --------- + -------- + -------------------------------- +
| 79935433-241a-4268-8aea-5570d74fcf42 | inst1 | ACTIVE | private = 10.0.0.4 |
+ -------------------------------------- + --------- + -------- + -------------------------------- +
-List the available floating IP address pools:
nova floating-ip-pool-list
+ ------ +
| name |
+ ------ +
| pub |
| test |
+ ------ +
- Take a floating IP address from the “pub” pool (or, if desired, from the “test” pool):
nova floating-ip-create pub
+ --------------- + ------------- + ---------- + ------ +
| Ip | Instance Id | Fixed Ip | Pool |
+ --------------- + ------------- + ---------- + ------ +
| 172.24.4.225 | None | None | pub |
+ --------------- + ------------- + ---------- + ------ +
- Assign a floating IP address to the instance:
nova add-floating-ip 79935433-241a-4268-8aea-5570d74fcf42 172.24.4.225(where the first argument is the uuid of the instance, and the second is the floating IP itself)
-Check the correctness of all settings:
nova floating-ip-list
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
| Ip | Instance Id | Fixed Ip | Pool |
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
| 172.24.4.225 | 79935433-241a-4268-8aea-5570d74fcf42 | 10.0.0.4 | pub |
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
The instance should now be visible from outside the OpenStack cluster to a floating IP address.
How do floating IPs work
What happens inside an instance after adding a floating IP address? The correct answer is nothing. If you connect to it over SSH and look at the network configuration, you will see that there is one network interface with a fixed IP address configured.
All configuration is performed on the compute node. All work related to the floating IP address is performed by the nova-network service: a network address translator (NAT) is organized between the fixed and floating IP addresses of the instance. An explanation of how NAT works can be found here .
Take a look at the following chart:
The diagram displays a single compute node configured in network mode with node distribution and VlanManager used to configure fixed IP networks. The computing node is equipped with two network interfaces: the eth0 interface is dedicated to traffic with a fixed IP / VLAN, and eth1 is the interface through which the computing node is connected to an external network; it contains floating IP addresses. (For information on how VlanManager configures fixed IP networks, see the previous article. )
Note that while the eth0 interface (fixed / private) does not have an address configured, the eth1 interface is assigned an IP address, which is the default gateway for the compute node (91.207.15.105).
When a user assigns a floating IP address (91.207.16.144) to VM_1, two things happen:
- The floating IP address is configured as a secondary address on the eth1 interface: this is the output of the “ip addr show eth1" command, containing the following actions:
inet 91.207.15.105/24 scope global eth1 # primary eth1 ip
inet 91.207.16.144/32 scope global eth1 # floating ip of VM_1
- A set of NAT rules for a floating IP address is configured in the iptables tables. Below are all the corresponding entries from the “nat” table of the compute node (with the exception of the “iptables –S -t nat" command. Details on how to configure NAT using iptables tables in Linux can be found here ):
# this rule ensures that packets originating from compute node
# where the instance resides by its floating IP:
-A nova-network-OUTPUT -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
# secures that all external traffic to the floating IP
# is directed to the IP of the instance
-A nova-network-PREROUTING -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
# all the traffic originating from the instance will be SNAT-ted to its floating IP
-A nova-network-float-snat -s 10.0.0.3/32 -j SNAT --to-source 91.207.16.144
In general, nova-network adds additional chains to those that are predefined in the NAT table. The order of the chains with respect to the floating IP traffic is shown below (using the rules shown above):
Chain OUTPUT - Chain nova-network-OUTPUT - Rule: -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
Chain PREROUTING - Chain nova-network-PREROUTING - Rule: -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
Chain POSTROUTING - Chain nova-postrouting-bottom - Chain nova-network-snat - Chain nova-network-float-snat - Rule: -s 10.0.0.3/32 -j SNAT --to-source 91.207.16.144
The code responsible for setting the rules is located in nova / network / linux_net.py in the function:
def floating_forward_rules (floating_ip, fixed_ip):
return [('PREROUTING', '-d% s -j DNAT --to% s'% (floating_ip, fixed_ip)),
('OUTPUT', '-d% s -j DNAT --to% s'% (floating_ip, fixed_ip)),
('float-snat',
'-s% s -j SNAT --to% s'% (fixed_ip, floating_ip))]
We return to the chart. If the user wants to access the virtual machine by its IP address from the external network (for example, “ping 91.20.16.144 ″):
-The traffic reaches the public interface (eth1) of the compute node. In nova-network-PREROUTING, DNAT is executed, changing the destination IP address of the packets from 91.207.16.144 to 10.0.0.3.
- The computing node accesses its routing table and sees that the 10.0.0.0 network is available on the br100 interface (with the exception of the “ip route show” compute node):
10.0.0.0/24 dev br100
So he sends the packet to the br100 interface, then the packet reaches the virtual machine.
If the virtual machine sends the packet to the outside world (for example, “ping 8.8.8.8):
- Since the destination address is not located in the local network of the virtual machine, the packets are sent to the default virtual machine gateway with the IP address 10.0.0.1 (the device address is “br100 ″ on the compute node).
The compute node checks its routing tables and finds that there is no 8.8.8.8 address in directly connected networks, so it forwards the packet to the default gateway (which is the primary address of eth1 - 91.207.15.105 in this case).
-The packet enters the POSTROUTING chain and is transferred to the “nova-network-float-snat” chain, where its source IP address is rewritten to the floating IP address (91.207.16.144).
Safety notes
With OpenStack, system administrators give complete control over iptables tables to nova services. The set of custom rules is very complicated and is easily broken by any outside intervention. Moreover, each time the nova-network daemon is restarted, it applies all the rules in the iptables table chains associated with OpenStack. If there is a need to change the behavior of iptables tables in any way, this can be done by changing the code in the appropriate places of linux_net.py (for NAT rules, these will be floating_forward_rules).
It is also worth saying that nova-network does not track its tables in any way. Thus, if we manually remove any rules from chains associated with OpenStack, they will not be restored the next time the nova-network is started.
Thus, a system administrator can easily accidentally open unwanted access to a compute node. Remember that nova-network placed the floating IP address as secondary on the eth1 interface and configured DNAT rules that direct traffic to the fixed IP address of the virtual machine:
-A nova-network-PREROUTING -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
Thus, all traffic directed to 91.207.16.144 gets to the address 10.0.0.3.
Now let's imagine that the system administrator solved network connection problems at night and accidentally deleted all the NAT rules by typing:
iptables –F –t nat
The above NAT rule has been removed, but the eth1 interface is still assigned the secondary IP address of 91.207.16.144. Thus, we can still access address 91.207.16.144 from the outside, but instead of getting to the virtual machine, we now have access to the compute node itself (the destination IP address is no longer broadcast according to the DNAT rule, since we deleted all NAT rules). This security hole will not be closed until the next restart of the nova-network process, which re-creates the rules.
Setting up floating IPs
In the nova.conf service, there are flags that affect the behavior of floating IP addresses:
# the interface to which ips are attached
# as secondary addresses
public_interface = "eth1"
# the pool from which floating IPs are taken by default
default_floating_pool = "pub"
# I can add a message to ip automatically
auto_assign_floating_ip = false
Summary Comments
In addition to providing access to virtual machines directly from the Internet, the floating IP address mechanism gives cloud users some flexibility. After they “pick up” a floating IP address, they can change their ownership, that is, on the fly, assign them to various virtual machines and reassign them, which greatly facilitates the release of new code and system updates. For system administrators, this poses a potential security risk, since the underlying mechanism (iptables) works very hard and is not monitored by OpenStack. It is therefore very important that only OpenStack software can change firewall policies and that they do not change manually.
Original article in English
Source: https://habr.com/ru/post/183264/
All Articles