VMware vSphere 5.5 and 6.0 performance - settings, considerations. Performance best practices

After reviewing the Perfomance Best Practices for vSphere 5.5 and Perfomance Best Practices for vSphere 6.0 documents, he didn’t reveal any special differences in configuration, as well as something additionally specific to vSphere 6.0.

Most of the writing fits into the standard recommendations of the format “use compatible and certified equipment” and “when seizing a VM, allocate resources (vCPU, vRAM) in the amount not more than necessary”.

Nevertheless, the basic things decided to issue a separate post, slightly restructured, eliminating the "water" and some references and comments that are too specific and for most implementations are more harmful than useful. On the bottom line are the recommendations, tips and considerations, tested and tested in practice and applicable to 90% of VMware vSphere and standalone ESXi infrastructures. Diluted by general considerations and additions.
')

ESXi Host

General recommendations

• Well, of course, you should use compatible hardware. Processors with hardware virtualization support. Processors in different cluster hosts should be at least one manufacturer (Intel / AMD) and preferably one level of technology and generation. Otherwise, problems with vMotion and performance. And it is desirable to ensure uniformity in the cluster hardware configuration - it is easier to manage (Host Profiles) and diagnose.
• Install the latest BIOS and firmware on all "glands". This will not necessarily affect the productivity growth, but in case of problems at the junction of hardware-iron, the vendor’s technical support will still require firmware updates. IBM and I have been repairing the Blade-basket for almost a year - as long as we updated the firmware on all the components involved and not so much, they released their new versions and had to go through the second round.

Bios

• Ensure that all necessary and useful components and technologies are included - processor cores, Turbo Boost, if any, virtualization technologies (VT-x, EPT, AMD-V, etc.).
• The NUMA Node Interleaving option is disabled or the Enable NUMA option is enabled. This item is often misleading. ESXi is a NUMA-aware OS, moreover, it can translate NUMA architecture into virtual machines, so the inclusion of the ability to recognize NUMA nodes has a positive effect on overall performance in most cases. However, the “NUMA Node Interleaving” option, being in the “Enable” state, actually unites the nodes into a single space, that is, disables the recognition of the NUMA node.
  
  NUMA

  / NUMA - Non-Uniform Memory Access - architecture of server systems, in which the time of access to memory is determined by its location relative to the processor. Each processor has its own set of memory modules, where access is faster, this set forms a NUMA node. /
  
• Disable unused devices. This will free up interrupts and relieve the processor a little.
• Enable Hyper-Threading (if available). In this case, vCPUs are assigned by logical cores and in most cases this leads to an optimized use of processor resources.
• CPU Power Saving - OS Controlled mode. Or something similar. If supported. ESXi - has a full set of tools for server power management. And it is better to give this question to the merchant hypervisor, if you want to be sure that it will not affect the performance negatively.

Hypervisor

Here it should be borne in mind that both the processor and the memory for each virtual machine have a certain overhead - an additional amount of both, necessary for the work of the VM itself:

- for the vmx process (VM eXecutable);
- for the process of vmm (VM Monitoring) - monitoring the state of the virtual processor, mapping - virtual memory, etc .;
- for operation of virtual VM devices;
- for the operation of other subsystems - kernel, management agents.

The overhead of each machine most of all depends on the amount of its vCPU and the amount of memory. By itself, it is not big, but it is worth bearing in mind. For example, if the entire amount of host memory is occupied or reserved by virtual machines, the response time at the hypervisor level may increase, and there will be problems with the operation of, for example, technologies like DRS.

Virtual machines

The main recommendation is minimum sizing. In the sense of allocating to the virtual machine no more memory and processors than it really needs to work. For in a virtual environment, more resources often lead to worse performance than less. It is difficult to understand and accept immediately, but it is. Main reasons:

- the overhead projector described in the previous section;
- NUMA. If the number of vCPUs corresponds to the number of cores in the NUMA socket and the memory size does not exceed the limits of the NUMA node, then the hypervisor tries to localize the VM inside one NUMA node. This means that memory access will be faster;
- CPU Scheduler. If a host has a lot of VMs with a large number of vCPUs (more in total than the number of physical cores), then the likelihood of such a phenomenon as Co-Stop - slowing down some vCPUs due to the inability to ensure their synchronous operation within a single VM, because there are not enough physical cores for a simultaneous cycle;
- DRS. Machines with a small number of processors and memory transfer from host to host easier and faster. In case of a sudden load jump, it will be easier to rebalance the cluster if it consists of small VMs, and not multi-gigabyte monsters;
- Localization cache. Inside the VM, the guest OS can transfer single-threaded processes between different processors and lose the processor cache.

Conclusions and recommendations:

• Better one processor, loaded at 80%, than 4 to 20%.
• If the server has a peak load once a quarter, and the rest of the time it works on 10% of its resources, it is better to cut them (resources) 8 times at once, and add the necessary amount once a quarter.
• Try to fit the VM by the number of vCPU and memory in the NUMA-node borders.
• If the VM goes beyond the NUMA node (wide VM), configure the number of processors multiple to the number of cores in the NUMA node. If we have 4 cores in one socket, then the multiples of processors recommended for VM will be 4, 8, 12 ...
• When using multiple vCPUs, it is better to configure them as separate virtual sockets with one virtual core in each. Well, or with the number of cores, which is a whole divisor of the number of cores in NUMA. If there are 4 cores in a physical socket, then in the virtual the correct value will be 1, 2, 4. But not 3 or 6.
• Disable unused virtual hardware of a virtual machine (COM, LPT, USB ports, Floppy Disks, CD / DVD, network interfaces, etc.)
• Use paravirtual hardware (VMware Paravirtual for SCSI controller and VMXNET for network adapter). This reduces processor load and response time, but may require a driver to install the OS.

Guest OS

• Use the latest versions of VMware Tools. After each update, update ESXi.
• And generally have VMware Tools installed.
• Disable screen savers and generally any animation and prettiness. If possible, disable graphics. This greatly reduces the load on the processor.
• Avoid simultaneous launch of intensive tasks (such as anti-virus scanning, backups and especially defragmentation). Defragmentation is better to turn off. The rest, if not avoided, is smeared for different machines at different times.
• Synchronize guest OS time using ntp services or VMware Tools, but not both tools at once. But at least one. Since it is worth bearing in mind that the time in the guest OS is not the exact value, because it depends on the processor, and the processor resource of the VM can receive not evenly and not always in the right amount.
• vNUMA. It is necessary to take into account that for VM with the number of vCPUs greater than eight, forwarding of the NUMA architecture is activated inside the VM. For some NUMA-aware applications, such as Exchange or MS SQL, this is useful. However, vNUMA is determined when the OS is first booted and does not change until the number of processors changes. Therefore, if there are hosts in a cluster with different number of cores in sockets, which means with different NUMA-architecture, then when a VM moves from a host to a host, performance may drop due to the fact that vNUMA does not coincide with NUMA on a new host.

Storage and Storage

The main thing to consider is that your storage must support the vStorage API for Array Integration (VAAI). In this case, the following will be supported:

- Offload processes of copying, cloning and transferring VMs between LUNs of a single storage or between storages supporting the technology. That is, the process will be executed mostly by the storage itself, not by the host processor and the network.
- acceleration of the zeroing of the blocks when creating Thick Eager Zeroed disks and during the initial filling of Thick Lazy Zeroed and Thin disks.
- Atomic Test and Set (ATS) - blocking not the entire LUN, when changing metadata, but only one sector on the disk. Considering that metadata changes during such processes as switching on / off VM, migration, cloning and expansion of a thin disk, LUN with a large number of VMs on it may not get out of SCSI Lock.
- Unmap - “release” of thin LUN blocks when deleting / transferring data (applies only to LUNs, not vmdk).

Considerations and recommendations:

• Independent Persistent Mode vmdk-disk - the most productive, because the changes are made immediately to disk without logging. But such a disk is not susceptible to snapshots, it can not be rolled back.
• When using iSCSI, it is recommended to configure jumbo frames (MTA = 9000) on all interfaces and network equipment.
• MultiPathing - for most cases RoundRobin - OK. Fixed can provide great performance, but this is after thoughtful planning and manual configuration of each host before each LUN. MRU can be delivered with active-passive configuration, if some paths disappear from time to time - so that it does not jump back and forth.

Virtualization infrastructure

DRS and clusters

• For resource management and control of their use, it is better to use Shares to a greater extent, and Limits and Reservation - to a lesser extent. Limits tightly limit VMs, even if the cluster has free resources. Reservations, on the contrary, eats away many resources, even if they are not used. In addition, when upgrading the physical equipment, the Shares settings automatically distribute the new capacities proportionally. And forgotten limits and reservations can lead to the fact that a part of a machine does not receive enough resources, although there are more than enough of them in a cluster.
• It is not necessary to construct complex multistage hierarchical structures from Resource Pools. For hierarchies there are folders. And also to keep at the same level (for example, at the root) and Resource Pools and virtual machines. Because the calculation of Shares for these types of objects is performed in different ways and unexpected performance differences may appear.
• Once again - the closer the hosts are to each other in configuration, the better. Ideally, in clusters all the hosts are of the same type. Without EVC, even to hosts with processors of the same vendor, but with a different set of VM technologies, they will be able to move only in the off state.

vMotion and Storage vMotion

By default, for each active vMotion process, the hypervisor eats off 10% of one processor core. Both on the receiver and on the source. That is, if all the processor resources on the host are in a reservation, there may be problems with vMotion. (With DRS exactly will).

When Storage vMotion, the source datastore is actively reading, and the target is recording. In addition, both datastores are synchronized to record changes inside the VM. Hence the conclusion - if we move the VM from a slow datastor to a fast one, the effect will be noticeable only at the end of the migration. And if from fast to slow, then degradation of performance will come immediately.

vCenter Server

• Minimum hop between vCenter server and its database. It is desirable that both servers are on the same network.
• And vCenter and its database should not be out of resources.
• In the database, the database itself generates mostly random reading, and the logs write sequentially. From here it is better to put both on different LUNs. It is even better if there are different paths to them (different controllers or arrays). Or different disks, at least.
• Also separately worth putting tempDB.
• It should regularly update the statistics of tables and indexes in the database and get rid of fragmentation.
• To close client sessions (both thick and web), if they are not used, they consume a lot of resources for inventory update.
• If the infrastructure has more than 30 hosts and / or 300 VMs, the Update Manager component database is better to be rendered separately from the vCenter database.
• If there are more than 100 hosts in the infrastructure and / or 1000 VMs, it is worthwhile to render the Update Manager itself separately
• If the infrastructure has more than 32 hosts and / or 4000 VMs, it is worth separating the Web-client Server component.