Diagnosing Namespaces in OpenStack Neutron
Posted by: Damien Igby
Recently, I tentatively introduced you to my Neutron namespace report, prepared for the OpenStack Summit in Hong Kong. One of the comment authors saw a video of my presentation and asked us to place a presentation here. In this post I will show you how:
1. Properly define the namespace.
2. Perform general diagnostics in the specified namespace.
')
In addition, at the end of the post is a presentation with my presentation.
So, let's get straight to the point. If you run two L3 + DHCP services on one node at once, you need to activate namespaces to eliminate route conflicts:
Use_namespaces = True
To disable namespaces, make sure that the neutron server used by the neutron server has the following setting:
allow_overlapping_ips = False
Also, the dhcp_agent.ini and l3_agent.ini files contain:
use_namespaces = False
How are the namespaces implemented in Neutron and how to recognize them? First you need to state two very important facts:
1. Each l2-agent / private network has an associated DHCP namespace.
2. Each l3-agent / router has an associated router namespace.
OK, but what about namespaces? At the beginning of the name of the network namespace goes dhcp- followed by the network ID. By analogy, at the beginning of the namespace of the router's name goes qrouter- followed by the router ID. For example:
stack @ vmnet-mn: ~ $ sudo ip netns list
qdhcp-eb2367bd-6e43-4de7-a0ab-d58ebf6e7dc0
qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd
The above list of namespaces is typical for a simple network consisting of one private network and one router. For a better understanding of Fig. 1 is an illustration of such a simple network.
Figure 1. Illustration of a simple network with a router and DHCP namespaces
Fig. Figure 1 shows how a leased network is built, consisting of a router and DHCP namespaces. For purposes of illustration, one virtual machine (VM1) is included in the network. As networks are built, similar network topologies will be created for each tenant. From fig.1 we can make the following observations:
• The data center network is an external network through which packets arrive on the Internet.
• A router with an IP address of 10.0.0.1 connects the leased network to the data center network. It has an associated router namespace that allows you to isolate the routing tables of one tenant from any other tenant when you deploy OpenStack.
• Bridge denotes a leased network with an IP address of 10.0.0.0/24. A private network is connected to the DHCP server (in this case, the dnsmasq process), and it has an associated IP address (10.0.0.2) just like a physical server connects to the network. Please note that the router is usually assigned the first IP address, the DHCP server the next IP address, and the virtual machines the IP addresses remaining in the pool. Similarly, the DHCP server has a DHCP namespace associated with it, which allows you to distribute IP addresses and transmit all traffic within a given leased network.
Without a proper understanding of network namespaces as a fundamental concept in building an OpenStack network, finding and correcting network problems is akin to finding a needle in a haystack. Imagine a small OpenStack implementation with approximately 1,000 tenants, with each Neutron tenant including at least one router and one private network. The total number of namespaces to be created will be at least 2000, since each rented router is a namespace, and each network will have an associated DHCP namespace (that is, two namespaces per tenant, as shown in Figure 1). In this scenario, for example, some tenant had problems with DHCP distribution, allowing the instance to show the assigned IP addresses that are viewed through the OpenStack Dashboard control panel, but you cannot connect to the instance using the SSH protocol and when viewing through the remote VNC access see that the network interface does not have an IP address.
To solve this problem, you need to check the various components of the topology of the leased network presented in Fig. 1, mainly the DHCP agent. The first logical step is to define the tenant namespace. In this case, we will see how this process differs from troubleshooting when deploying a traditional physical network. Here, without understanding how the namespace works in Neutron, the administrator is completely confused and will not be able to continue troubleshooting with logical procedures.
So, the steps for finding and fixing errors are as follows:
1. Define a namespace correctly.
2. Implement general diagnostics in the specified namespace.
Create two leased networks to illustrate identifying and fixing errors in namespaces.
Figure 2. Network for tenant A
Figure 3. Network for tenant B
In the above schemes, the network for tenant A has two private networks and one router; The network for tenant B is also two networks and one router. Consider these networks. Note that in a multimode configuration that includes a controller node, a network node and compute nodes, the namespace commands will be executed from the network node.
stack @ vmnet-mn: ~ $ sudo ip netns list
[sudo] password for stack:
qdhcp-eb2367bd-6e43-4de7-a0ab-d58ebf6e7dc0
qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922
qrouter-4587f8d0-08a7-434a-aaeb-74e68a50c48d
qdhcp-f5a69443-bf3f-4f9a-b3e2-5625e70514b4
qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd
qdhcp-e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c
We see a list of six namespaces of two leased networks (three namespaces per tenant).
OK, but how do you know which namespace belongs to which tenant, since tenant names are not indicated in namespaces? This can be done in two ways.
First, it is worth noting that, as a rule, only the cloud administrator has access to the network node, so only the administrator can run namespace commands on the network node. One way a cloud administrator can define a namespace is to establish the source of the user credential file associated with a given tenant. After that, all commands with neutron will be executed under this user and only the components of the leased network to which this user belongs will be shown. We illustrate this as follows.
1. Identify the source of the file with the credentials owned by user tenant A.
2. Execute the following commands:
stack @ vmnet-mn: ~ $ neutron router-list
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
| id | name | external_gateway_info |
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
| 4adef4a5-1122-49b8-9da2-3cbf803c44dd | router_proj_one | {"Network_id": "ea00b8c3-7063-4780-ad73-ef0b47518f10"} |
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
The list shows that tenant A has one router with the identifier 4adef4a5-1122-49b8-9da2-3cbf803c44dd
Having found this ID in the ip netns command above, we can easily determine that the qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd router namespace belongs to tenant A.
For DHCP namespace:
stack @ vmnet-mn: ~ $ neutron net-list
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
| id | name | subnets |
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
| e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c | net_proj_one | f02b6df4-11af-411d-b41e-e4ce66d5510d 10.10.1.0/24 |
| ea00b8c3-7063-4780-ad73-ef0b47518f10 | ext_net | 4f7c6c9e-28d8-43b3-b7d4-276920216b31 |
| f5a69443-bf3f-4f9a-b3e2-5625e70514b4 | net2 | f6cd4df6-b37f-4615-94f6-8abcab38ef99 192.168.0.0/24 |
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
In addition, according to this list, you can define two private networks with identifiers 0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c and f5a69443-bf3f-4f9a-b3e2-5625e70514b4.
By checking the command from the ip netns list above, we can also set the following DHCP namespaces: qdhcp-e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c and qdhcp-f5a69443-bf3f-4f9a-b3e2-5625e70514b4
The procedures described above are effective and simplest if the cloud administrator knows the tenant's username / password, but this is not always the case. Therefore, another way that a cloud administrator can define a tenant namespace is to ask the user for the network ID or network name. The user of the leased network can easily find out this information in the Horizon OpenStack Dashboard control panel. Knowing either the name or ID, the cloud administrator can troubleshoot as follows:
1. Trace administrator credentials. Knowing them, the cloud administrator will have access to all namespaces of the leased network.
2.List the namespaces and select the tenant namespace.
stack @ vmnet-mn: ~ $ ip netns list | grep 9f7ec6fd-ec9a-4b1b-b854-a9832f190922
qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922
Now that we’ve figured out the namespace of the tenant, we can begin to fix errors in it.
First, check the IP address.
stack @ vmnet-mn: ~ $ sudo ip netns exec qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922 ifconfig
[sudo] password for stack:
lo Link encap: Local Loopback
inet addr: 127.0.0.1 Mask: 255.0.0.0
inet6 addr: :: 1/128 Scope: Host
UP LOOPBACK RUNNING MTU: 16436 Metric: 1
RX packets: 16 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 16 errors: 0 dropped: 0 overruns: 0 carrier: 0
collisions: 0 txqueuelen: 0
RX bytes: 1344 (1.3 KB) TX bytes: 1344 (1.3 KB)
tap43cb2c73-40 Link encap: Ethernet HWaddr fa: 16: 3e: e6: 1b: 24
inet addr: 10.1.0.3 Bcast: 10.1.0.255 Mask: 255.255.255.0
inet6 addr: fe80 :: f816: 3eff: fee6: 1b24 / 64 Scope: Link
UP BROADCAST RUNNING MULTICAST MTU: 1500 Metric: 1
RX packets: 965 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 561 errors: 0 dropped: 0 overruns: 0 carrier: 0
collisions: 0 txqueuelen: 0
RX bytes: 178396 (178.3 KB) TX bytes: 93900 (93.9 KB)
Then ping the interface:
stack @ vmnet-mn: ~ $ sudo ip netns exec qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922 ping –c 3 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56 (84) bytes of data.
64 bytes from 10.1.0.3: icmp_req = 1 ttl = 64 time = 0.081 ms
64 bytes from 10.1.0.3: icmp_req = 2 ttl = 64 time = 0.035 ms
64 bytes from 10.1.0.3: icmp_req = 3 ttl = 64 time = 0.055 ms
The steps above have helped us establish an appropriate namespace and verify the correctness of an available (i.e., running and running) IP address. Further troubleshooting steps may include the following:
• Check if the dnsmasq process is connected to the subnet for DHCP:
stack @ vmnet-mn: ~ $ ps –aux | grep dhcp
According to the results of the above command, determine that the process in the selected network (for example, 10.0.0.0) is running, otherwise the dnsmasq process connected to this network may not be running.
• Ensure that the firewall does not block the transfer of data to / from the subnet.
Before summarizing, pay attention to the following:
• Creating a router or network does not immediately create namespaces. For the namespace network, DHCP is created only when the VM is connected, and for the router, the namespace is created when the gateway is installed. This means that there must be some activity before creating namespaces.
• When you delete a router or network, the associated namespaces are not deleted. They need to be removed manually.
Just as network namespaces are extremely important for building an OpenStack network, their understanding by the OpenStack cloud administrator is also important. We looked at the implementation of network namespaces in OpenStack, taking into account what data an OpenStack cloud administrator needs to have when solving problems with an OpenStack leased network.
Now, if you wish, you can familiarize yourself with the presentation on this topic.
Neutron Network Namespaces and IPtables - A Technical Deep Dive from Mirantis
Original article in English
Recently, I tentatively introduced you to my Neutron namespace report, prepared for the OpenStack Summit in Hong Kong. One of the comment authors saw a video of my presentation and asked us to place a presentation here. In this post I will show you how:
1. Properly define the namespace.
2. Perform general diagnostics in the specified namespace.
')
In addition, at the end of the post is a presentation with my presentation.
So, let's get straight to the point. If you run two L3 + DHCP services on one node at once, you need to activate namespaces to eliminate route conflicts:
Use_namespaces = True
To disable namespaces, make sure that the neutron server used by the neutron server has the following setting:
allow_overlapping_ips = False
Also, the dhcp_agent.ini and l3_agent.ini files contain:
use_namespaces = False
How are the namespaces implemented in Neutron and how to recognize them? First you need to state two very important facts:
1. Each l2-agent / private network has an associated DHCP namespace.
2. Each l3-agent / router has an associated router namespace.
OK, but what about namespaces? At the beginning of the name of the network namespace goes dhcp- followed by the network ID. By analogy, at the beginning of the namespace of the router's name goes qrouter- followed by the router ID. For example:
stack @ vmnet-mn: ~ $ sudo ip netns list
qdhcp-eb2367bd-6e43-4de7-a0ab-d58ebf6e7dc0
qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd
The above list of namespaces is typical for a simple network consisting of one private network and one router. For a better understanding of Fig. 1 is an illustration of such a simple network.
Figure 1. Illustration of a simple network with a router and DHCP namespaces
Fig. Figure 1 shows how a leased network is built, consisting of a router and DHCP namespaces. For purposes of illustration, one virtual machine (VM1) is included in the network. As networks are built, similar network topologies will be created for each tenant. From fig.1 we can make the following observations:
• The data center network is an external network through which packets arrive on the Internet.
• A router with an IP address of 10.0.0.1 connects the leased network to the data center network. It has an associated router namespace that allows you to isolate the routing tables of one tenant from any other tenant when you deploy OpenStack.
• Bridge denotes a leased network with an IP address of 10.0.0.0/24. A private network is connected to the DHCP server (in this case, the dnsmasq process), and it has an associated IP address (10.0.0.2) just like a physical server connects to the network. Please note that the router is usually assigned the first IP address, the DHCP server the next IP address, and the virtual machines the IP addresses remaining in the pool. Similarly, the DHCP server has a DHCP namespace associated with it, which allows you to distribute IP addresses and transmit all traffic within a given leased network.
Finding and fixing bugs in the Neutron namespace
Without a proper understanding of network namespaces as a fundamental concept in building an OpenStack network, finding and correcting network problems is akin to finding a needle in a haystack. Imagine a small OpenStack implementation with approximately 1,000 tenants, with each Neutron tenant including at least one router and one private network. The total number of namespaces to be created will be at least 2000, since each rented router is a namespace, and each network will have an associated DHCP namespace (that is, two namespaces per tenant, as shown in Figure 1). In this scenario, for example, some tenant had problems with DHCP distribution, allowing the instance to show the assigned IP addresses that are viewed through the OpenStack Dashboard control panel, but you cannot connect to the instance using the SSH protocol and when viewing through the remote VNC access see that the network interface does not have an IP address.
To solve this problem, you need to check the various components of the topology of the leased network presented in Fig. 1, mainly the DHCP agent. The first logical step is to define the tenant namespace. In this case, we will see how this process differs from troubleshooting when deploying a traditional physical network. Here, without understanding how the namespace works in Neutron, the administrator is completely confused and will not be able to continue troubleshooting with logical procedures.
So, the steps for finding and fixing errors are as follows:
1. Define a namespace correctly.
2. Implement general diagnostics in the specified namespace.
Create two leased networks to illustrate identifying and fixing errors in namespaces.
Figure 2. Network for tenant A
Figure 3. Network for tenant B
Step 1: Define a Namespace
In the above schemes, the network for tenant A has two private networks and one router; The network for tenant B is also two networks and one router. Consider these networks. Note that in a multimode configuration that includes a controller node, a network node and compute nodes, the namespace commands will be executed from the network node.
stack @ vmnet-mn: ~ $ sudo ip netns list
[sudo] password for stack:
qdhcp-eb2367bd-6e43-4de7-a0ab-d58ebf6e7dc0
qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922
qrouter-4587f8d0-08a7-434a-aaeb-74e68a50c48d
qdhcp-f5a69443-bf3f-4f9a-b3e2-5625e70514b4
qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd
qdhcp-e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c
We see a list of six namespaces of two leased networks (three namespaces per tenant).
OK, but how do you know which namespace belongs to which tenant, since tenant names are not indicated in namespaces? This can be done in two ways.
First, it is worth noting that, as a rule, only the cloud administrator has access to the network node, so only the administrator can run namespace commands on the network node. One way a cloud administrator can define a namespace is to establish the source of the user credential file associated with a given tenant. After that, all commands with neutron will be executed under this user and only the components of the leased network to which this user belongs will be shown. We illustrate this as follows.
1. Identify the source of the file with the credentials owned by user tenant A.
2. Execute the following commands:
stack @ vmnet-mn: ~ $ neutron router-list
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
| id | name | external_gateway_info |
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
| 4adef4a5-1122-49b8-9da2-3cbf803c44dd | router_proj_one | {"Network_id": "ea00b8c3-7063-4780-ad73-ef0b47518f10"} |
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
The list shows that tenant A has one router with the identifier 4adef4a5-1122-49b8-9da2-3cbf803c44dd
Having found this ID in the ip netns command above, we can easily determine that the qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd router namespace belongs to tenant A.
For DHCP namespace:
stack @ vmnet-mn: ~ $ neutron net-list
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
| id | name | subnets |
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
| e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c | net_proj_one | f02b6df4-11af-411d-b41e-e4ce66d5510d 10.10.1.0/24 |
| ea00b8c3-7063-4780-ad73-ef0b47518f10 | ext_net | 4f7c6c9e-28d8-43b3-b7d4-276920216b31 |
| f5a69443-bf3f-4f9a-b3e2-5625e70514b4 | net2 | f6cd4df6-b37f-4615-94f6-8abcab38ef99 192.168.0.0/24 |
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
In addition, according to this list, you can define two private networks with identifiers 0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c and f5a69443-bf3f-4f9a-b3e2-5625e70514b4.
By checking the command from the ip netns list above, we can also set the following DHCP namespaces: qdhcp-e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c and qdhcp-f5a69443-bf3f-4f9a-b3e2-5625e70514b4
The procedures described above are effective and simplest if the cloud administrator knows the tenant's username / password, but this is not always the case. Therefore, another way that a cloud administrator can define a tenant namespace is to ask the user for the network ID or network name. The user of the leased network can easily find out this information in the Horizon OpenStack Dashboard control panel. Knowing either the name or ID, the cloud administrator can troubleshoot as follows:
1. Trace administrator credentials. Knowing them, the cloud administrator will have access to all namespaces of the leased network.
2.List the namespaces and select the tenant namespace.
stack @ vmnet-mn: ~ $ ip netns list | grep 9f7ec6fd-ec9a-4b1b-b854-a9832f190922
qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922
Step 2: Find and fix errors in the namespace
Now that we’ve figured out the namespace of the tenant, we can begin to fix errors in it.
First, check the IP address.
stack @ vmnet-mn: ~ $ sudo ip netns exec qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922 ifconfig
[sudo] password for stack:
lo Link encap: Local Loopback
inet addr: 127.0.0.1 Mask: 255.0.0.0
inet6 addr: :: 1/128 Scope: Host
UP LOOPBACK RUNNING MTU: 16436 Metric: 1
RX packets: 16 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 16 errors: 0 dropped: 0 overruns: 0 carrier: 0
collisions: 0 txqueuelen: 0
RX bytes: 1344 (1.3 KB) TX bytes: 1344 (1.3 KB)
tap43cb2c73-40 Link encap: Ethernet HWaddr fa: 16: 3e: e6: 1b: 24
inet addr: 10.1.0.3 Bcast: 10.1.0.255 Mask: 255.255.255.0
inet6 addr: fe80 :: f816: 3eff: fee6: 1b24 / 64 Scope: Link
UP BROADCAST RUNNING MULTICAST MTU: 1500 Metric: 1
RX packets: 965 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 561 errors: 0 dropped: 0 overruns: 0 carrier: 0
collisions: 0 txqueuelen: 0
RX bytes: 178396 (178.3 KB) TX bytes: 93900 (93.9 KB)
Then ping the interface:
stack @ vmnet-mn: ~ $ sudo ip netns exec qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922 ping –c 3 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56 (84) bytes of data.
64 bytes from 10.1.0.3: icmp_req = 1 ttl = 64 time = 0.081 ms
64 bytes from 10.1.0.3: icmp_req = 2 ttl = 64 time = 0.035 ms
64 bytes from 10.1.0.3: icmp_req = 3 ttl = 64 time = 0.055 ms
The steps above have helped us establish an appropriate namespace and verify the correctness of an available (i.e., running and running) IP address. Further troubleshooting steps may include the following:
• Check if the dnsmasq process is connected to the subnet for DHCP:
stack @ vmnet-mn: ~ $ ps –aux | grep dhcp
According to the results of the above command, determine that the process in the selected network (for example, 10.0.0.0) is running, otherwise the dnsmasq process connected to this network may not be running.
• Ensure that the firewall does not block the transfer of data to / from the subnet.
findings
Before summarizing, pay attention to the following:
• Creating a router or network does not immediately create namespaces. For the namespace network, DHCP is created only when the VM is connected, and for the router, the namespace is created when the gateway is installed. This means that there must be some activity before creating namespaces.
• When you delete a router or network, the associated namespaces are not deleted. They need to be removed manually.
Just as network namespaces are extremely important for building an OpenStack network, their understanding by the OpenStack cloud administrator is also important. We looked at the implementation of network namespaces in OpenStack, taking into account what data an OpenStack cloud administrator needs to have when solving problems with an OpenStack leased network.
Now, if you wish, you can familiarize yourself with the presentation on this topic.
Neutron Network Namespaces and IPtables - A Technical Deep Dive from Mirantis
Original article in English
Source: https://habr.com/ru/post/215983/
All Articles