VM performance analysis in VMware vSphere. Part 3: Storage

Part 1. About CPU
Part 2. About Memory

Today we will analyze the disk subsystem metrics in vSphere. The problem with storage is the most common reason for a slow virtual machine. If in cases with CPU and RAM, troubleshooting ends at the hypervisor level, then in case of problems with the disk, you may have to deal with the data network and storage systems.
')
I will analyze the topic using the example of block access to storage, although with file access the counters are about the same.

Bit of theory

When talking about the performance of the disk subsystem of virtual machines, usually pay attention to three related parameters:

• number of input / output operations (Input / Output Operations Per Second, IOPS);
• bandwidth (Throughput);
• latency of input / output operations (Latency).

The number of IOPS is usually important for random workloads: access to blocks on disk located in different places. Databases, business applications (ERP, CRM), etc. can serve as an example of such a load.

Throughput is important for sequential loads: access to blocks located one after the other. For example, file servers (but not always) and video surveillance systems can generate such a load.

Throughput is related to the number of I / O operations as follows:

Throughput = IOPS * Block size , where Block size is the size of the block.

Block size is a pretty important feature. Modern ESXi versions allow blocks up to 32,767 KB in size. If the block is even larger, it is divided into several. Not all storage systems can work effectively with such large blocks, so Advanced Settings ESXi has the DiskMaxIOSize parameter. Using it, you can reduce the maximum size of a block skipped by the hypervisor (more details here ). I recommend that you consult the manufacturer of storage systems before changing this parameter, or at least test the changes at the laboratory bench.

Large block sizes can adversely affect storage performance. Even if the number of IOPS and throughput are relatively small, high latencies can occur with a large block size. Therefore, pay attention to this parameter.

Latency is the most interesting performance parameter. The I / O latency for a virtual machine is the sum of:

• delays within the hypervisor (KAVG, Average Kernel MilliSec / Read);
• latency given by data network and storage (DAVG, Average Driver MilliSec / Command).

The total delay that is visible in the guest OS (GAVG, Average Guest MilliSec / Command) is the sum of KAVG and DAVG.

GAVG and DAVG are measured, and KAVG is calculated: GAVG – DAVG.


A source

Let us dwell on KAVG . In normal operation, the KAVG should tend to zero, or at least be much smaller than the DAVG. The only case I know of when KAVG is expected to be high is the IOPS limit on the VM disk. In this case, when trying to exceed the limit, KAVG will increase.

The most significant component of KAVG is QAVG - the time in the queue for processing inside the hypervisor. The remaining components of KAVG are negligible.

The queue in the disk adapter driver and the moon queue has a fixed size. For heavily loaded environments this size can be useful to increase. It describes how to increase the queue in the adapter driver (at the same time the queue to the moons will increase). This setting works when only one VM works with the moon, which is rare. If there are several VMs on the moon, you must also increase the Disk.SchedNumReqOutstanding parameter (instructions here ). By increasing the queue, you decrease QAVG and KAVG, respectively.

But, again, first read the documentation from the HBA vendor and test the changes at the laboratory bench.

The size of the queue to the moon can be affected by the inclusion of the SIOC (Storage I / O Control) mechanism. It provides uniform access to the moon from all the cluster servers by dynamically changing the queue to the moon on the servers. That is, if a VM is running on a host that requires a disproportionate amount of performance (noisy neighbor VM), SIOC reduces the length of the queue to the moon on this host (DQLEN). More details here .

KAVG sorted out, now a little about DAVG . Everything is simple here: DAVG is the delay introduced by the external environment (data network and storage). Any modern and not so storage system has its own performance counters. To analyze problems with DAVG, it makes sense to look at them. If everything is fine on the ESXi and storage side, check the data network.

To avoid performance issues, choose the right Path Selection Policy (PSP) for your storage. Almost all modern storage systems support Round-Robin PSP (with or without ALUA, Asymmetric Logical Unit Access). This policy allows you to use all available paths to storage. In the case of ALUA, only the paths to the controller that owns the moon are used. Not all storage systems on ESXi have default rules that establish the Round-Robin policy. If there is no rule for your storage system, use the plug-in from the storage manufacturer that will create the corresponding rule on all the hosts in the cluster, or create the rule yourself. Details here .

Also, some storage vendors recommend changing the number of IOPS per path from the standard value of 1000 to 1. In our practice, this allowed us to “squeeze” more performance out of the storage and significantly reduce the time it takes to failover in the event of a controller failure or update. Check the vendor's recommendations, and if there are no contraindications, then try changing this parameter. Details here .

Basic performance counters of the disk subsystem of a virtual machine

The disk subsystem performance counters in vCenter are collected in the sections Datastore, Disk, Virtual Disk:



The Datastore section contains metrics for vSphere disk storages (datastores) on which VM disks are located. Here you will find standard counters for:

• IOPS (Average read / write requests per second),
• bandwidth (Read / Write rate),
• delays (Read / Write / Highest latency).

From the names of the counters, in principle, everything is clear. Once again, I’ll draw attention to the fact that here the statistics are not for a specific VM (or VM disk), but for the entire datastore. In my opinion, it is more convenient to look at these statistics in ESXTOP, at least on the basis that the minimum measurement period is 2 seconds.

The Disk section contains metrics for block devices used by VMs. There are IOPS counters of the summation type (the number of input / output operations per measurement period) and several counters related to block access (Commands aborted, Bus resets). In my opinion, this information is also more convenient to look at in ESXTOP.

The Virtual Disk partition is the most useful in terms of troubleshooting VM disk subsystem performance problems. Here you can see the performance of each virtual disk. This information is needed to understand if a particular virtual machine has a problem. In addition to standard counters for the number of input / output operations, read / write volume and delays, this section contains useful counters that show the block size: Read / Write request size.

In the picture below, a graph of the performance of the VM disk, where you can see the number of IOPS, latency and block size.



Performance metrics can also be viewed across the entire datastore if SIOC is enabled. Here is some basic information on average Latency and IOPS. By default, this information can only be viewed in real time.

ESXTOP

ESXTOP has several screens that display information about the host disk subsystem as a whole, individual virtual machines and their disks.

Let's start with information on virtual machines. The “Disk VM” screen is called up by the “v” key:



NVDISK is the number of VM disks. To view information on each disk, press “e” and enter the GID of the VM you are interested in.

The meaning of the remaining parameters on this screen is clear from their names.

Another useful screen for troubleshooting is Disk adapter. It is called by the “d” key (in the picture below, the fields A, B, C, D, E, G are selected):



NPTH is the number of moon paths that are visible from this adapter. To get information on each path on the adapter, press “e” and enter the name of the adapter:



AQLEN - maximum queue size on the adapter.

Also on this screen are the delay counters, which I talked about above: KAVG / cmd, GAVG / cmd, DAVG / cmd, QAVG / cmd .

The Disk device screen, which is called by the “u” key, provides information on individual block devices - moons (in the picture below, the fields A, B, F, G, I are selected). Here you can see the status of the queue to the moons.



DQLEN - queue size for a block device.
ACTV is the number of I / O commands in the ESXi core.
QUED - the number of I / O commands in the queue.
% USD - ACTV / DQLEN × 100%.
LOAD - (ACTV + QUED) / DQLEN.

If% USD is high, you should consider expanding the lineup. The more teams in the queue, the higher the QAVG and, accordingly, KAVG.

You can also see on the Disk device screen whether the VAAI (vStorage API for Array Integration) is running on storage. To do this, select the fields A and O.

The VAAI mechanism allows you to transfer part of the work from the hypervisor directly to the storage system, for example, zeroing, copying blocks or blocking.



As you can see in the picture above, VAAI works on this storage system: Zero and ATS primitives are actively used.

ESXi Disk Optimization Tips

• Pay attention to the size of the block.
• Set the optimal queue size on the HBA.
• Remember to enable SIOC on the datastores.
• Choose your PSP according to the manufacturer’s recommendations.
• Make sure VAAI is working.