NetApp ONTAP: UNMAP in the SAN environment
The UNMAP team is standardized as part of the T10 SCSI command set and is used to free space from thin moons back to the data store in the SAN environment. As I wrote earlier , the SAN and NAS protocols gradually borrow from each other all the best. One of the useful things that appeared quite a long time ago is the possibility of feedback between the storage system and the host in order to “return” remote blocks to a thin moon, which was not enough in the SAN before. UNMAP is still not widely used in SAN environments, although it is very useful in combination with both virtualized and non-virtualized environments.
Without the support of the UNMAP team, any thin moon created on the storage side could always only increase in size . His growth was a matter of time, which unconditionally always ended with the fact that such a thin moon would eventually occupy its full volume, which it was supposed to, i.e. in the end he will become fat.
Imagine you have 3 virtual machines live on your datastore, each one takes up 100GB. Your datastore is on a thin moon that occupies 300GB. The storage space on the storage and ESXi side is the same: 300GB. After deleting one VM, the size of your moon on the storage system side is still 300GB, and on the hypervisor side, the space occupied on the datastor on this moon is 200GB.
')
This is due to the fact that the wound did not have a host feedback mechanism with the storage system. Namely, when the host recorded a block of information in the moon, the storage system in turn marked this block as used. Then the host could delete this block, but the storage system did not already know about it. The UNMAP team is this feedback that patches the unwanted block with the moon. Finally, our thin moon has learned not only to gain, but also to reduce its weight since the firmware version (Clustered Data) ONTAP 8.2 .
In version 5.0, the UNMAP function was first introduced, turned on by default and automatically started when the set value of remote blocks was reached, in subsequent versions the mechanism is disabled by default and started manually. Starting with VMFS-6 (vSphere 6.5), space is automatically released within 12 hours, if necessary, a manual mechanism for starting the release of space is also available. It is important to note that the release of space, which will be discussed now, takes place at the level of the hypervisor, i.e. the release of remote blocks occurs only after removing the virtual machine or virtual disks as a whole , and not inside the guest OS.
If you already have ESXi and ONTAP storage with UNMAP support, but the feature is not enabled
It is necessary to include it from the storage system and from the hypervisor. Let's start with the fact that we will transfer the moon to offline and enable the space-allocation function on the storage system (if the VM remained there in a neglected state, they will shut down incorrectly and may be damaged, so they should be either temporarily turned off or temporarily migrated):
After that, unmap must be enabled by ESXi, for this, a datastor needs to be mapped and primed, so that ESXi detects UNMAP support (if the VMs are left in a running state, they will shut down incorrectly and may be damaged, so they should be either temporarily disabled or migrated temporarily ):
After that, to free up space, you will need to periodically run the command:
It is important to note that UNMAP works only for the moon formatted with offset partition multiple of 1 MB . What does it mean? This means that if you once converted VMFS3 to VMFS5, UNMAP will not work. This is easy to check, the converted VMFS3 have an MBR breakdown table, and the VMFS5 that were created anew have a GPT breakdown.
Pay attention to the Layout column.
Check the offset is also not difficult, we look at the length of the sector. Let me remind the sector is 512 bytes.
Pay attention to the "Start Sector" and "End Sector" columns. So, the last device naa.60a98000486e574f6c4a5052537a7375 has an offset of 1MB (2048 sectors * 512 = 1048576 byte = 1024KB). But the second device naa.60a98000486e574f6c4a5778594e7456 has an offset that is not a multiple of 1MB, it is clearly smaller, UNMAP will not work on it.
Check if UNMAP ( Delete Status ) is supported:
Automatic release of space back to thin moon on storage is supported starting with vSphere 6.5. For each VMFS-6 datastor, you can assign the High / Mid / Slow space release priority, which will be returned to the repository within 12 hours. You can also start the release of space and configure prioritization for a VMFS-6 datastor manually from the GUI or from the command line.
So, earlier we considered removal of virtual computers from datastor. It would be logical to do the same when deleting data blocks from inside the guest OS, and not the entire virtual machine or its disks. UNMAP is supported on the storage side with ONTAP and in order for the UNMAP mechanism to work when deleting data from the VMDK, i.e. from the inside of the guest OS, from the storage side, nothing is needed to be implemented additionally, it is enough that UNMAP is turned on. It is necessary for the hypervisor to be able to transmit this information from the virtual machine to the storage that is executed completely at the SCSI level. So, starting with ESXi 6.0, it is now possible to transfer information about remote blocks within the guest OS.
For UNMAP to work from inside a virtual machine, you must meet the following conditions and have:
For many years, all storage vendors have said not to create thin virtual disks on thin moons. But now it has changed and in order to release the blocks from within the virtual machine, it is necessary to have thin virtual disks and thin moons on the storage system. In vSphere version 6.0, the functionality of returning remote blocks from within the guest OS was implemented, but it had a number of limitations when using UNMAP , for example, were not supported by Linux machines. In vSphere 6.0 and earlier versions with VMFS, the UNMAP function does not automatically start , you need to run the command manually.
In order for the release of space to work from inside the Windows guest OS, the NTFS file system must be formatted with an allocation unit size of 64KB.
Linux virtual machines supporting SPC -4 and running on vSphere 6.5 will now also be able to return the freed space from inside the virtual machine back to storage.
How to check whether your Linux machine supports the release of space?
It is important to remember that the VMware hypervisor starts UNMAP asynchronously , that is, with a delay. This means that in practice you will most likely never have 1: 1 occupied space, inside the guest OS / on the hypervisor datastor and on the storage system moon.
VVOL technology since version vSphere 6.0, VM Hardware version 11 and Windows 2012 GOS with the NTFS file system supports automatic release of space inside the thin moon on the external storage where this virtual machine is located.
Linux GOS- enabled SPC- 4s installed on VM Hardware version 11 on vSphere 6.5 will also be able to return space from inside the virtual machine to a thin storage system.
This significantly improves utilization of the usable space in the SAN infrastructure using Thing Provisioning and Hardware Assistant snapshots.
Read more about tuning ESXi 6.x for ONTAP .
UNMAP command support for this OS family began with Windows Server 2012 with the NTFS file system. To enable automatic UNMAP, use Windows Host Utility 6.0.2 and higher or ONTAP DSM 4.0 and higher. To check if UNMAP is enabled, run
Read more about tuning Windows Server for ONTAP .
Linux distros support UNMAP. NetApp officially supports RHEL version 6.2 and higher with the –o discard key of the mount command and the fstrim utility. Read more in the RHEL6 Storage Admin Guide .
Read more about tuning Linux Server for ONTAP .
Before ONTAP 9, file deletion operations were generated and so-called Zombie messages.
Let me remind you that ONTAP 9 was released in May 2016. In this version of Zombie software, messages are no longer generated to free space, instead the file is simply marked for deletion. In this connection, the logic of the work of freeing space has changed significantly:
Implemented UNMAP support, both on the guest OS side, the host, and the NetApp storage side, allows you to more efficiently utilize the space in the SAN environment using Thin Provisioning and as a result, will allow you to more efficiently use the storage space with hardware snapshots. UNMAP support at the hypervisor level, and even more so inside the guest OS, will greatly facilitate the use of these two popular technologies.
I ask to send messages on errors in the text to the LAN . Comments, additions and questions on the article on the contrary, please in the comments.
Without the support of the UNMAP team, any thin moon created on the storage side could always only increase in size . His growth was a matter of time, which unconditionally always ended with the fact that such a thin moon would eventually occupy its full volume, which it was supposed to, i.e. in the end he will become fat.
Imagine you have 3 virtual machines live on your datastore, each one takes up 100GB. Your datastore is on a thin moon that occupies 300GB. The storage space on the storage and ESXi side is the same: 300GB. After deleting one VM, the size of your moon on the storage system side is still 300GB, and on the hypervisor side, the space occupied on the datastor on this moon is 200GB.
')
This is due to the fact that the wound did not have a host feedback mechanism with the storage system. Namely, when the host recorded a block of information in the moon, the storage system in turn marked this block as used. Then the host could delete this block, but the storage system did not already know about it. The UNMAP team is this feedback that patches the unwanted block with the moon. Finally, our thin moon has learned not only to gain, but also to reduce its weight since the firmware version (Clustered Data) ONTAP 8.2 .
VMware ESXi & UNMAP
In version 5.0, the UNMAP function was first introduced, turned on by default and automatically started when the set value of remote blocks was reached, in subsequent versions the mechanism is disabled by default and started manually. Starting with VMFS-6 (vSphere 6.5), space is automatically released within 12 hours, if necessary, a manual mechanism for starting the release of space is also available. It is important to note that the release of space, which will be discussed now, takes place at the level of the hypervisor, i.e. the release of remote blocks occurs only after removing the virtual machine or virtual disks as a whole , and not inside the guest OS.
If you already have ESXi and ONTAP storage with UNMAP support, but the feature is not enabled
It is necessary to include it from the storage system and from the hypervisor. Let's start with the fact that we will transfer the moon to offline and enable the space-allocation function on the storage system (if the VM remained there in a neglected state, they will shut down incorrectly and may be damaged, so they should be either temporarily turned off or temporarily migrated):
lun modify -vserver vsm01 -volume vol2 -lun lun1_myDatastore -state offline lun modify -vserver vsm01 -volume vol2 -lun lun1_myDatastore -space-allocation enabled lun modify -vserver vsm01 -volume vol2 -lun lun1_myDatastore -state online After that, unmap must be enabled by ESXi, for this, a datastor needs to be mapped and primed, so that ESXi detects UNMAP support (if the VMs are left in a running state, they will shut down incorrectly and may be damaged, so they should be either temporarily disabled or migrated temporarily ):
esxcli storage filesystem unmount -l myDatastore esxcli storage filesystem mount -l myDatastore esxcli storage vmfs unmap -l myDatastore After that, to free up space, you will need to periodically run the command:
esxcli storage vmfs unmap -l myDatastore It is important to note that UNMAP works only for the moon formatted with offset partition multiple of 1 MB . What does it mean? This means that if you once converted VMFS3 to VMFS5, UNMAP will not work. This is easy to check, the converted VMFS3 have an MBR breakdown table, and the VMFS5 that were created anew have a GPT breakdown.
# esxcli storage core device partition showguid Example output: Device Partition Layout GUID ------------------------------------------------------------- naa.60a98000486e574f6c4a5778594e7456 0 MBR N/A naa.60a98000486e574f6c4a5778594e7456 1 MBR N/A naa.60a98000486e574f6c4a5052537a7375 0 GPT 00000000000000000000000000000000 naa.60a98000486e574f6c4a5052537a7375 1 GPT aa31e02a400f11db9590000c2911d1b8 Pay attention to the Layout column.
Check the offset is also not difficult, we look at the length of the sector. Let me remind the sector is 512 bytes.
# esxcli storage core device partition list Example output: Device Partition Start Sector End Sector Type Size ------------------------------------------------------------------------------------- naa.60a98000486e574f6c4a5778594e7456 0 0 3221237760 0 1649273733120 naa.60a98000486e574f6c4a5778594e7456 1 128 3221225280 fb 1649267277824 naa.60a98000486e574f6c4a5052537a7375 0 0 3221237760 0 1649273733120 naa.60a98000486e574f6c4a5052537a7375 1 2048 3221237727 fb 1649272667648 Pay attention to the "Start Sector" and "End Sector" columns. So, the last device naa.60a98000486e574f6c4a5052537a7375 has an offset of 1MB (2048 sectors * 512 = 1048576 byte = 1024KB). But the second device naa.60a98000486e574f6c4a5778594e7456 has an offset that is not a multiple of 1MB, it is clearly smaller, UNMAP will not work on it.
Check if UNMAP ( Delete Status ) is supported:
# esxcli storage core device vaai status get -d naa Example output: naa.60a98000486e574f6c4a5052537a7375 VAAI Plugin Name: VMW_VAAIP_NETAPP ATS Status: supported Clone Status: supported Zero Status: supported Delete Status: supported Automatic space release in vSphere 6.5
Automatic release of space back to thin moon on storage is supported starting with vSphere 6.5. For each VMFS-6 datastor, you can assign the High / Mid / Slow space release priority, which will be returned to the repository within 12 hours. You can also start the release of space and configure prioritization for a VMFS-6 datastor manually from the GUI or from the command line.
esxcli storage vmfs reclaim config get -l DataStoreOnNetAppLUN Reclaim Granularity: 248670 Bytes Reclaim Priority: low esxcli storage vmfs reclaim config set -l DataStoreOnNetAppLUN -p high UNMAP from guest OS.
So, earlier we considered removal of virtual computers from datastor. It would be logical to do the same when deleting data blocks from inside the guest OS, and not the entire virtual machine or its disks. UNMAP is supported on the storage side with ONTAP and in order for the UNMAP mechanism to work when deleting data from the VMDK, i.e. from the inside of the guest OS, from the storage side, nothing is needed to be implemented additionally, it is enough that UNMAP is turned on. It is necessary for the hypervisor to be able to transmit this information from the virtual machine to the storage that is executed completely at the SCSI level. So, starting with ESXi 6.0, it is now possible to transfer information about remote blocks within the guest OS.
For UNMAP to work from inside a virtual machine, you must meet the following conditions and have:
- Virtual Hardware Version 11
- VMFS6
- vSphere 6.0 * / 6.5
- moon storage should be thin
- virtual virtual disks should be thin
- guest OS file system must support UNMAP
- for vSphere 6.0 CBT must be turned off
- enable UNMAP on hypervisor, if necessary: esxcli storage vmfs unmap -l myDatastore
- enable UNMAP on storage
Never Thin on Thin
For many years, all storage vendors have said not to create thin virtual disks on thin moons. But now it has changed and in order to release the blocks from within the virtual machine, it is necessary to have thin virtual disks and thin moons on the storage system. In vSphere version 6.0, the functionality of returning remote blocks from within the guest OS was implemented, but it had a number of limitations when using UNMAP , for example, were not supported by Linux machines. In vSphere 6.0 and earlier versions with VMFS, the UNMAP function does not automatically start , you need to run the command manually.
Windows Guest OS support
In order for the release of space to work from inside the Windows guest OS, the NTFS file system must be formatted with an allocation unit size of 64KB.
Linux Guest OS SPC-4 support
Linux virtual machines supporting SPC -4 and running on vSphere 6.5 will now also be able to return the freed space from inside the virtual machine back to storage.
How to check whether your Linux machine supports the release of space?
Verifying SPC-4 Support in a Linux Virtual Machine
To do this, run the command
The Unmap command supported (LBPU) parameter set to 1 means that a thin disk is used, which is what we need. A value of 0 means that the type of virtual disk is thick (eager or sparse), with UNMAP function will not work with thick disks.
Version version = 0x02 [SCSI-2] says that UNMAP will not work, we need version of SPC-4:
sg_vpd -p lbpv Logical block provisioning VPD page (SBC): Unmap command supported (LBPU): 1 Write same (16) with unmap bit supported (LBWS): 0 Write same (10) with unmap bit supported (LBWS10): 0 The Unmap command supported (LBPU) parameter set to 1 means that a thin disk is used, which is what we need. A value of 0 means that the type of virtual disk is thick (eager or sparse), with UNMAP function will not work with thick disks.
sg_inq /dev/sdc -d standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x02 [SCSI-2] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: VMware Product identification: Virtual disk Product revision level: 1.0 Version version = 0x02 [SCSI-2] says that UNMAP will not work, we need version of SPC-4:
sg_inq -d /dev/sdb standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x06 [SPC-4] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: VMware Product identification: Virtual disk Product revision level: 2.0 Let's check that UNMAP is up and running:
To do this, run the command
a value of 1 indicates that the guest OS notifies the SCSI level of remote blocks from the file system.
Check if space is freed up using UNMAP:
If you get the error "blkdiscard: / dev / sdb: BLKDISCARD ioctl failed: Operation not supported" then UNMAP does not work. If there is no error, then it remains to mount the file system with the discard key :
grep . /sys/block/sdb/queue/discard_max_bytes 1 a value of 1 indicates that the guest OS notifies the SCSI level of remote blocks from the file system.
Check if space is freed up using UNMAP:
sg_unmap --lba=0 --num=2048 /dev/sdb # blkdiscard --offset 0 --length=2048 /dev/sdb If you get the error "blkdiscard: / dev / sdb: BLKDISCARD ioctl failed: Operation not supported" then UNMAP does not work. If there is no error, then it remains to mount the file system with the discard key :
mount /dev/sdb /mnt/netapp_unmap -o discard It is important to remember that the VMware hypervisor starts UNMAP asynchronously , that is, with a delay. This means that in practice you will most likely never have 1: 1 occupied space, inside the guest OS / on the hypervisor datastor and on the storage system moon.
VVOL
VVOL technology since version vSphere 6.0, VM Hardware version 11 and Windows 2012 GOS with the NTFS file system supports automatic release of space inside the thin moon on the external storage where this virtual machine is located.
Linux GOS- enabled SPC- 4s installed on VM Hardware version 11 on vSphere 6.5 will also be able to return space from inside the virtual machine to a thin storage system.
This significantly improves utilization of the usable space in the SAN infrastructure using Thing Provisioning and Hardware Assistant snapshots.
Read more about tuning ESXi 6.x for ONTAP .
Windows (Bare Metal)
UNMAP command support for this OS family began with Windows Server 2012 with the NTFS file system. To enable automatic UNMAP, use Windows Host Utility 6.0.2 and higher or ONTAP DSM 4.0 and higher. To check if UNMAP is enabled, run
fsutil behavior query disabledeletenotify . The state DisableDeleteNotify = 0 means that UNMAP returns blocks on the go (in-band). If the host has several moons connected with several storage systems, some of which do not support UNMAP, it is recommended to turn it off.Read more about tuning Windows Server for ONTAP .
Linux (Bare Metal)
Linux distros support UNMAP. NetApp officially supports RHEL version 6.2 and higher with the –o discard key of the mount command and the fstrim utility. Read more in the RHEL6 Storage Admin Guide .
Read more about tuning Linux Server for ONTAP .
Zombie
Before ONTAP 9, file deletion operations were generated and so-called Zombie messages.
ONTAP 8 & Zombie Messages
Releasing remote blocks, on the one hand, significantly saves disk space, but on the other hand, if the host requests the storage system to free up more blocks, for example, terabytes of data, your storage system will do this and will not calm down until it is finished, and this is additional activity on the storage system . The release of a terabyte of data will be especially noticeable on disk or hybrid (not All Flash) systems. Therefore, you should be careful about deleting more data in one fell swoop on such systems (terabytes).
If you are in such a situation, consult technical support, you may need to increase the value of wafl_zombie_ack_limit and wafl_zombie_msg_cnt . If you need to delete all data on the moon, then it is better to delete the entire moon on the storage system. All Flash systems, on the other hand, are much less susceptible to such requests, and as a rule, quickly and effortlessly cope with such tasks.
If you are in such a situation, consult technical support, you may need to increase the value of wafl_zombie_ack_limit and wafl_zombie_msg_cnt . If you need to delete all data on the moon, then it is better to delete the entire moon on the storage system. All Flash systems, on the other hand, are much less susceptible to such requests, and as a rule, quickly and effortlessly cope with such tasks.
Deleting files in ONTAP 9 and prioritization
Let me remind you that ONTAP 9 was released in May 2016. In this version of Zombie software, messages are no longer generated to free space, instead the file is simply marked for deletion. In this connection, the logic of the work of freeing space has changed significantly:
- First, the data is deleted in the background.
- secondly, the possibility arose of prioritizing the release of space from those volums that need it more, that is, those who end space.
findings
Implemented UNMAP support, both on the guest OS side, the host, and the NetApp storage side, allows you to more efficiently utilize the space in the SAN environment using Thin Provisioning and as a result, will allow you to more efficiently use the storage space with hardware snapshots. UNMAP support at the hypervisor level, and even more so inside the guest OS, will greatly facilitate the use of these two popular technologies.
I ask to send messages on errors in the text to the LAN . Comments, additions and questions on the article on the contrary, please in the comments.
Source: https://habr.com/ru/post/271959/
All Articles