Planning the hardware for your OpenStack cluster: answers to your questions
By: Greg Elkinbard
My colleague Anna Friend (Anne Friend) and I recently presented a webinar on “How to handle hardware planning for your OpenStack cloud . ” During the webinar, we promised to give you answers to questions that we did not have time to voice on the air. This article will be devoted to the answers to these questions.
You mentioned adding storage to a rack with a super-loaded switch. Can you talk about how to set it up?
')
A typical central switch does not have the same uplink bandwidth compared to the downlink bandwidth. For example, typically a trident + switch will have 48 10-gigabyte downlink ports with a total bandwidth of 960 Gb / s, but only 4 × 40 gigabyte ports or 320 gigs as upstream bandwidth, so the limit is exceeded by about ratio 3/1.
This means that you should limit the traffic going up through the communication channels. This can be done in two ways. One is to run user CMKs in the domain (L2 segment) of the edge switch in order to reduce traffic between outgoing links.
The second main source of traffic is Cinder traffic between the Cinder node and the Compute node. Concentration of this traffic in one switch will also relieve the uplink. For example, if you use Cinder iSCSI storage, then you can provide one or two switches per rack and make sure that the Cinder scheduler creates volumes from storage located in the same rack as the Compute resources. Both of these filters are custom, you need to create them for the Nova and Cinder planners. This is not really a “turnkey solution”, but it’s a simple change.
I'm trying to understand how we can apply some compromise options that you describe in more private terms. Can you give a numerical example of a compromise when allocating vCPU / VRAM for two different cases?
There are too many examples of use to delve into them, but let's look at real calculations.
CPU requirements
-100 virtual machines
- On average 2 EC2 compute nodes
-Maximum 16 EC2 compute nodes
- Lack of excess of a limit
It corresponds:
-200 GHz CPU Capacity (100 users x 2 GHz / user)
-Maximum number of cores - 5 (16 GHz / 2.4 GHz per core)
Based on calculations:
- The factor for Hyperthreading 1.3
-10-11 E5 2640 cores (200 GHz / 2.4 GHz on CPU / 6 cores)
-5-6 dual core servers (11 cores / 2 cores per server)
-17 VM per server (100 VM / 6 servers)
Memory requirements
-100 virtual machines
-4 GB per virtual machine
-Minimum 512 MB, maximum 32 GB
It corresponds:
-400 GB total (100 VM * 4 GB per VM)
Based on the following calculations:
-The need for four machines of 128GB each (400 GB / 128 GB)
- Balanced with CPU, you need 6 machines for total CPU capacity
- Reduce server memory and work with 6 machines of 64 or 96 GB each (6x64 GB is 384 GB, 6 × 96 is 596 GB)
If you need to have a little more memory, you need machines with a capacity of 96 GB.
When you say that a VLAN is suitable for a small network, how much do you mean by a small network?
A small network has less than 4 thousand virtual networks. However, since Neutron allows each user to have multiple networks, you cannot assume that you can host 4,000 users. Also, do not forget that you have some static infrastructure needs; Remember to save tags for these networks.
How does Fuel help automate network configuration?
Fuel can check the network configuration to ensure that your nodes are properly connected and that all relevant VLAN tags are unlocked on the switch.
Do you think it is better to use hardware from well-known manufacturers such as Dell, HP, etc., or can we achieve the same performance with the help of the software we created? Is it recommended to use the Open Compute Platform?
The short answer is that if you have a large enough company to support your own hardware or a small enough company not to worry about downtime during a hardware failure, then you can use virtual data stores or your own build computer. If you are a medium-sized company, we recommend that you use equipment from well-known manufacturers, since you get the best service level agreements.
Open Compute is a promising platform, but it depends on wider hardware support, which will be soon.
Do you recommend separate software for nodes running separate nova services? For example, should a node that runs nova-api have more memory than a node that runs glance-api?
At Mirantis, we recommend consolidating all the OpenStack infrastructure services into dedicated nodes called controllers. This kind of architecture facilitates high availability.
What about ARM (or Atom) based microservers?
If you have a general-purpose cloud, it will be difficult for you to create a significant load on the CPU on ARM or Atom-based microservers. Try running MsSQL or Oracle server on ARM; you will not achieve much. If you have a special purpose cloud that fits within the constraints of these CPUs, then use them anyway. The cloud does not fully rely on the CPU, and the architecture of many ARM / Atom based processors does not require enough bandwidth or disk space to become a good platform.
What about "blade servers"?
Leave the razor blade. Use normal servers for the cloud. If higher density is needed, use a sleds form factor server (Dell C-class, HP SL-class) instead of blade servers. The central blade server module usually does not have enough bandwidth to work well with the cloud, and not enough local storage space, which puts a double load on your chassis bandwidth requirements. In addition, you pay a premium for such servers. One or two of the blade device schemes began to eliminate at least a bottleneck in the network, but other doubts remain.
Can we provide real-time migration without shared storage?
You can perform real-time migration without shared storage. It just takes more time.
For a small private cloud, do you recommend a fiber-optic channel for shared storage on compute nodes or a shared file system for 1 Gigabit?
Neither one nor the other. Use 10 gigs and Ceph or other block storage. You do not need shared FS or fiber cost.
Can you tell a little more about the swift 6.5x requirement?
This is a separate question with a more detailed answer in the recorded webinar, but here is a simple calculation:
Accept the replication factor of 3.
Add two hand-held devices (needed for extra space for crashes)
In addition, if you exceed 75% of the XFS disk capacity, you will have problems, and you will get this calculation:
(3 + 2) /. 75 = 6.7
After deployment, what tools do you use or use to verify CPU and hardware utilization?
In Mirantis, we used (and successfully) Nagios and Zabbix.
Can I deploy OpenStack on the Open Compute Platform OCP?
Yes. Mirantis Fuel is generally independent of the hardware architecture.
How are diskless hypervisors embedded in the storage equation “local vs sharing vs object”? Is it possible to manage compute nodes as iSCSI diskless clients without disrupting Cinder’s ability to connect to iSCSI targets, or does hardware need another SAN solution?
Let's slightly change the question and ask why you need such difficulties. With Mirantis Fuel, you already have an operating system. Having a few small disks for the OS will make setting up easier. We have tried this before, but there are problems in arrays when several initiators for the OS and Cinder from one node want to turn to the same goal. It's not worth it.
Does Fuel interface support?
Yes, but you need to use the command line interface, not the web interface.
Did you work with Illumos-based hypervisors or with something using Illumos, or was it only Linux work?
ZFS is not so comprehensive that it is worth paying attention to side operating systems like Solaris. Yes, you can run XEN and KVM with warnings and restrictions. If you are rich enough to support your own operating system development team, you can do it, but you will always fall behind in functionality. I have created several OS development teams for various companies from scratch, and I can tell you that if this is your professional field, go ahead. Otherwise it is better to go along the beaten path: it will be more convenient for you than to go through the jungle.
Original article in English
My colleague Anna Friend (Anne Friend) and I recently presented a webinar on “How to handle hardware planning for your OpenStack cloud . ” During the webinar, we promised to give you answers to questions that we did not have time to voice on the air. This article will be devoted to the answers to these questions.
You mentioned adding storage to a rack with a super-loaded switch. Can you talk about how to set it up?
')
A typical central switch does not have the same uplink bandwidth compared to the downlink bandwidth. For example, typically a trident + switch will have 48 10-gigabyte downlink ports with a total bandwidth of 960 Gb / s, but only 4 × 40 gigabyte ports or 320 gigs as upstream bandwidth, so the limit is exceeded by about ratio 3/1.
This means that you should limit the traffic going up through the communication channels. This can be done in two ways. One is to run user CMKs in the domain (L2 segment) of the edge switch in order to reduce traffic between outgoing links.
The second main source of traffic is Cinder traffic between the Cinder node and the Compute node. Concentration of this traffic in one switch will also relieve the uplink. For example, if you use Cinder iSCSI storage, then you can provide one or two switches per rack and make sure that the Cinder scheduler creates volumes from storage located in the same rack as the Compute resources. Both of these filters are custom, you need to create them for the Nova and Cinder planners. This is not really a “turnkey solution”, but it’s a simple change.
I'm trying to understand how we can apply some compromise options that you describe in more private terms. Can you give a numerical example of a compromise when allocating vCPU / VRAM for two different cases?
There are too many examples of use to delve into them, but let's look at real calculations.
CPU requirements
-100 virtual machines
- On average 2 EC2 compute nodes
-Maximum 16 EC2 compute nodes
- Lack of excess of a limit
It corresponds:
-200 GHz CPU Capacity (100 users x 2 GHz / user)
-Maximum number of cores - 5 (16 GHz / 2.4 GHz per core)
Based on calculations:
- The factor for Hyperthreading 1.3
-10-11 E5 2640 cores (200 GHz / 2.4 GHz on CPU / 6 cores)
-5-6 dual core servers (11 cores / 2 cores per server)
-17 VM per server (100 VM / 6 servers)
Memory requirements
-100 virtual machines
-4 GB per virtual machine
-Minimum 512 MB, maximum 32 GB
It corresponds:
-400 GB total (100 VM * 4 GB per VM)
Based on the following calculations:
-The need for four machines of 128GB each (400 GB / 128 GB)
- Balanced with CPU, you need 6 machines for total CPU capacity
- Reduce server memory and work with 6 machines of 64 or 96 GB each (6x64 GB is 384 GB, 6 × 96 is 596 GB)
If you need to have a little more memory, you need machines with a capacity of 96 GB.
When you say that a VLAN is suitable for a small network, how much do you mean by a small network?
A small network has less than 4 thousand virtual networks. However, since Neutron allows each user to have multiple networks, you cannot assume that you can host 4,000 users. Also, do not forget that you have some static infrastructure needs; Remember to save tags for these networks.
How does Fuel help automate network configuration?
Fuel can check the network configuration to ensure that your nodes are properly connected and that all relevant VLAN tags are unlocked on the switch.
Do you think it is better to use hardware from well-known manufacturers such as Dell, HP, etc., or can we achieve the same performance with the help of the software we created? Is it recommended to use the Open Compute Platform?
The short answer is that if you have a large enough company to support your own hardware or a small enough company not to worry about downtime during a hardware failure, then you can use virtual data stores or your own build computer. If you are a medium-sized company, we recommend that you use equipment from well-known manufacturers, since you get the best service level agreements.
Open Compute is a promising platform, but it depends on wider hardware support, which will be soon.
Do you recommend separate software for nodes running separate nova services? For example, should a node that runs nova-api have more memory than a node that runs glance-api?
At Mirantis, we recommend consolidating all the OpenStack infrastructure services into dedicated nodes called controllers. This kind of architecture facilitates high availability.
What about ARM (or Atom) based microservers?
If you have a general-purpose cloud, it will be difficult for you to create a significant load on the CPU on ARM or Atom-based microservers. Try running MsSQL or Oracle server on ARM; you will not achieve much. If you have a special purpose cloud that fits within the constraints of these CPUs, then use them anyway. The cloud does not fully rely on the CPU, and the architecture of many ARM / Atom based processors does not require enough bandwidth or disk space to become a good platform.
What about "blade servers"?
Leave the razor blade. Use normal servers for the cloud. If higher density is needed, use a sleds form factor server (Dell C-class, HP SL-class) instead of blade servers. The central blade server module usually does not have enough bandwidth to work well with the cloud, and not enough local storage space, which puts a double load on your chassis bandwidth requirements. In addition, you pay a premium for such servers. One or two of the blade device schemes began to eliminate at least a bottleneck in the network, but other doubts remain.
Can we provide real-time migration without shared storage?
You can perform real-time migration without shared storage. It just takes more time.
For a small private cloud, do you recommend a fiber-optic channel for shared storage on compute nodes or a shared file system for 1 Gigabit?
Neither one nor the other. Use 10 gigs and Ceph or other block storage. You do not need shared FS or fiber cost.
Can you tell a little more about the swift 6.5x requirement?
This is a separate question with a more detailed answer in the recorded webinar, but here is a simple calculation:
Accept the replication factor of 3.
Add two hand-held devices (needed for extra space for crashes)
In addition, if you exceed 75% of the XFS disk capacity, you will have problems, and you will get this calculation:
(3 + 2) /. 75 = 6.7
After deployment, what tools do you use or use to verify CPU and hardware utilization?
In Mirantis, we used (and successfully) Nagios and Zabbix.
Can I deploy OpenStack on the Open Compute Platform OCP?
Yes. Mirantis Fuel is generally independent of the hardware architecture.
How are diskless hypervisors embedded in the storage equation “local vs sharing vs object”? Is it possible to manage compute nodes as iSCSI diskless clients without disrupting Cinder’s ability to connect to iSCSI targets, or does hardware need another SAN solution?
Let's slightly change the question and ask why you need such difficulties. With Mirantis Fuel, you already have an operating system. Having a few small disks for the OS will make setting up easier. We have tried this before, but there are problems in arrays when several initiators for the OS and Cinder from one node want to turn to the same goal. It's not worth it.
Does Fuel interface support?
Yes, but you need to use the command line interface, not the web interface.
Did you work with Illumos-based hypervisors or with something using Illumos, or was it only Linux work?
ZFS is not so comprehensive that it is worth paying attention to side operating systems like Solaris. Yes, you can run XEN and KVM with warnings and restrictions. If you are rich enough to support your own operating system development team, you can do it, but you will always fall behind in functionality. I have created several OS development teams for various companies from scratch, and I can tell you that if this is your professional field, go ahead. Otherwise it is better to go along the beaten path: it will be more convenient for you than to go through the jungle.
Original article in English
Source: https://habr.com/ru/post/198992/
All Articles