FabricPool - saving technology for All Flash repositories
This article will be useful in the first public cloud providers, as well as end users, owners of NetApp ONTAP storage systems that have SSD drives.
In this article, I will talk about replication between storage systems, SSD technologies, All Flash storage, where the storage market is moving, object storage systems, FabricPool storage systems, DataFabric vision, snapshots, and ONTAP firmware for storage systems. DSS, DSS, DSS are all systems that are installed, at the customer’s data center to build a classic infrastructure or a private cloud, these are customers who keep their own data center, their storage system and most of their infrastructure runs on their own equipment. What do these repositories, and all of the topics listed, have to do with the cloud and public providers? I will talk about all these technologies that, it would seem, have nothing to do with public cloud providers. But in the course of the story, it will become clear how they all form a trend and help move towards the hybrid cloud: Private + Public and how NetApp will help transfer some of the customer’s data that previously only lived with the customer on his own equipment to the public cloud.
I want for technology you see a picture of the near future of which it consists. Not to see "what a deer is northern", the future must be seen in order to understand which direction to move and develop.
SSD drives are getting cheaper, but for now they are still expensive. Who has computers or laptops SSD? Acceleration is noticeable, isn't it? We have already gone through several stages of development of NAND: SLC, MLC, now the generation of TLC drives, the cost of technology is becoming cheaper with each new development. Many potential new types of carriers promise even more capacity in the future, even more speed, etc. And maybe someday, technology where memory cells do not wear out. SSD even for a laptop is not cheap, but what about the big industrial All Flash Array solutions? How to make SSD cheaper in such solutions? The answer is: use deduplication and data compression technologies to reduce the cost of a terabyte. NetApp has a range of data compression technologies. So, for example, the SQL database was actually really compressed twice, I saw this with one of our customers, while the performance was better than that of other competitors on the test and turning it on / off did not affect the performance. This was in ONTAP 8.3.2 firmware, and now ONTAP 9.2 has already been released with an inline aggregate deduplication . SSD is definitely the future of storage, without exception, all vendors have recognized this. Some vendors refused to develop extremely expensive High-End storage and began to develop cheaper and more efficient AFA.
')
DataFabric is not a single technology, but a whole set of technologies, a vision, a desire for NetApp to move to merge and integrate all its products in as many ways as possible and more data mobility. For example:
→ Read more about DataFabric
StorageGRID is an object storage for data that rarely changes: photos, videos, archives, backups. What is object storage can be found in more detail in the article.
This is a software that can be installed with a cloud provider and is licensed for each terabyte by month: there is a customer who consumes the resource and pays for it — there is money, a license is bought; no money - not bought.
Who worked with VMware and Hyper-V snapshots on high-load virtual machines? Do they work badly? The more snapshots, the worse the virtual machine works, the longer they are deleted and this deletion even more kills the performance of this machine. For NetApp ONTAP, snapshots are a trifle , no performance issues when creating or deleting. Many integrations with backup software such as Veeam or CommVault, etc., allowing you to capture consistent snapshots. VMware vVol allows you to capture hardware-assistant snapshots with a granular storage for each individual virtual machine, which eliminates the problems caused by hypervisor snapshots. Such non-inhibiting snapshots are beneficial for backup and DR. Where it is possible to use hardware-assistant storage snapshots that do not affect performance, you should use them everywhere.
I recall that ONTAP is firmware 9.2 , which is installed either on the FAS / AFF hardware platform or is available as software (SDS) with a virtual machine on your server (ONTAP Select) or in a public cloud (ONTAP Cloud) . A couple of months ago, ONTAP firmware was released, most current customers can now upgrade to 9.2.
In my previous article, I painted a new functionality available in firmware 9.2 .
Replication for DR on FAS or ONTAP Select. Works on the basis of snapshots between two nontaps. So this replication with FabricPool allows you to save not only on the main site. You can put the storage on the backup site with 10-20% of the space from the main one, and again mix everything else into a cold level.
The first thing to say about FabricPool is that, like any tiring technology, it is designed primarily to save storage resources, and this is customer money. If the budget is not limited, then you can stupidly buy the All Flash system and store snapshots on it for 3 years, plus buy one more of the same system on the backup site, build a metro cluster and a third on the backup backup site. But as a rule, there is no money for such an architecture, but there is still interest in the amount of SSD for a product (or some important part of it), but there are no longer any backups, snapshots or reservations. This is where FabricPool comes to the rescue.
FabricPool is a technology that allows you to combine fast and slow storage to create a hybrid pool (aggregate) consisting of SSD and S3 cloud, like how it works in FlashPool caching technology. As in FlashPool, all data first falls on the hot SSD level, and the cold data are then shifted to a slow and cheap level, this saves expensive SSD space.
FabricPool is a technology implemented as part of the DataFabric strategy, which, in turn, should expand the integration of all NetApp products and expand the possibilities of data mobility.
When NetApp makes a new feature, it doesn’t look at others and doesn’t try to add a tick feature. Instead, a technology is released that is really well thought out and will be in demand by consumers. This also applies to the new Tiering feature FabricPool. As the demand for SSDs grows, the number of installations and interest in this technology increases, so NetApp has released the FabricPool functionality that makes it possible to more efficiently use expensive SSD drives.
FabricPool is a cold data offset technology with SSD aggregate (AFF, FAS, ONTAP Cloud systems) to a cold object level - to Amazon S3 cloud or to NetApp StorageGRID object storage system .
Implementing FabricPool to move data to Object Storage will consist of two stages. In 9.2RC, the ability to shift only snapshots will be available. And the next stage will be the possibility of shifting cold data from the active file system of the volyum to the cold level, available from version 9.2GA. So far, FlexFroup, MetroCluster, ONTAP Select, SnapLock, etc. are not supported with FabricPool.
To use the FabricPool functionality on FAS / AFF systems with Amazon S3 tertiary object storage, you must purchase per terabyte licenses that are added to ONTAP, adding a FabricPool license immediately adds 10TB. Then you can add by terabyte.
For the operation of the FabricPool functionality on FAS / AFF storage systems, a license is only necessary if you are using Amazon S3. In the case of NetApp StorageGRID, a license for running FabricPool is not needed.
For ONTAP Cloud systems living in a public cloud, there is also no lossytech licensing from ONTAP Cloud to enable FabricPool - the customer only pays for the actual cold space consumed in the S3 object storage directly to the cloud provider. Currently, FabricPool is only supported on the ONTAP Cloud for Amazon.
As before, SnapMirror replication is used to build a DR site and transfer data between two ONTAP storage systems. Very often, to save finances, a simpler and cheaper storage was bought at the DR site. Now FabricPool allows a slightly different approach to this issue. For example, if an AFF system is used at the main site, then a backup system can be placed on the same or similar, but still AFF system. In All Flash repositories, most controllers do not eat off controllers, but SSD drives themselves. Now, with FabricPool, you can install an AFF system on the backup platform, but with only 10% -20% of the usable space from the main storage, since cold data will go to the cold object level. Allowing thus to save on disks, but not to put controllers that would not pull performance. If there is no room for new hot blocks on the hot level, they will be read directly from the cold object storage.
DSS provides a high-speed, but rather expensive level of data storage. NetApp FabricPool storage owners is a way to save money. And some customers simply according to the laws of their country can not take all or part of their data abroad in Amazon S3. For the cloud provider, FabricPool is a new customer and additional storage service.
FabricPool technology is Tiering technology that allows new and existing customers to shift snapshots (backups) and cold data of productive storage systems using SSD (expensive level) from their data centers to a public cloud to a cold (cheap level). In a public cloud, you need to deploy NetApp StorageGrid software (available as VMware, KVM and Docker images). For cloud-based providers, NetApp StorageGrid is licensed every month by terabyte, which is very convenient: while there are customers for this service, the cloud provider pays for licenses, no - it does not. For owners of NetApp storage systems that will use the StorageGrid license located at the cloud provider, you do not need to buy a license.
The cost will include the following basic costs for the provider:
FabricPool technology allows ONTAP and object storage for cold data to be integrated as part of the DataFabric strategy. For storage owners, FabricPool allows you to more efficiently use the expensive resources of All Flash systems and SSD aggregates, shifting inactive cold data to a cheap level, which in turn makes NetApp All Flash storage systems more affordable for the cost of ownership. Object storage is easily scaled as it grows and theoretically has no space restrictions (current Soft Limit 70PB). For cloud providers, FabricPool is a way to attract new customers who have previously stored productive data only on their equipment.
In this article, I will talk about replication between storage systems, SSD technologies, All Flash storage, where the storage market is moving, object storage systems, FabricPool storage systems, DataFabric vision, snapshots, and ONTAP firmware for storage systems. DSS, DSS, DSS are all systems that are installed, at the customer’s data center to build a classic infrastructure or a private cloud, these are customers who keep their own data center, their storage system and most of their infrastructure runs on their own equipment. What do these repositories, and all of the topics listed, have to do with the cloud and public providers? I will talk about all these technologies that, it would seem, have nothing to do with public cloud providers. But in the course of the story, it will become clear how they all form a trend and help move towards the hybrid cloud: Private + Public and how NetApp will help transfer some of the customer’s data that previously only lived with the customer on his own equipment to the public cloud.
I want for technology you see a picture of the near future of which it consists. Not to see "what a deer is northern", the future must be seen in order to understand which direction to move and develop.
SSD trend
SSD drives are getting cheaper, but for now they are still expensive. Who has computers or laptops SSD? Acceleration is noticeable, isn't it? We have already gone through several stages of development of NAND: SLC, MLC, now the generation of TLC drives, the cost of technology is becoming cheaper with each new development. Many potential new types of carriers promise even more capacity in the future, even more speed, etc. And maybe someday, technology where memory cells do not wear out. SSD even for a laptop is not cheap, but what about the big industrial All Flash Array solutions? How to make SSD cheaper in such solutions? The answer is: use deduplication and data compression technologies to reduce the cost of a terabyte. NetApp has a range of data compression technologies. So, for example, the SQL database was actually really compressed twice, I saw this with one of our customers, while the performance was better than that of other competitors on the test and turning it on / off did not affect the performance. This was in ONTAP 8.3.2 firmware, and now ONTAP 9.2 has already been released with an inline aggregate deduplication . SSD is definitely the future of storage, without exception, all vendors have recognized this. Some vendors refused to develop extremely expensive High-End storage and began to develop cheaper and more efficient AFA.
')
DataFabric
DataFabric is not a single technology, but a whole set of technologies, a vision, a desire for NetApp to move to merge and integrate all its products in as many ways as possible and more data mobility. For example:
- In the new versions of two completely different systems and architectures, such as ONTAP and AltaVault, SnapMirror replication is supported, which was previously supported only between two FAS systems.
- Support for data replication located on the storage system with the hardware platform NetApp E-Series and OS SANtricity to the cloud to expand the backup site there.
- Integration of SolidFire and StorageGRID.
- StorageGRID and AltaVault
- and of course the possibility of FAS <-> ONTAP Cloud / ONTAP Select replication
→ Read more about DataFabric
StorageGRID object storage
StorageGRID is an object storage for data that rarely changes: photos, videos, archives, backups. What is object storage can be found in more detail in the article.
This is a software that can be installed with a cloud provider and is licensed for each terabyte by month: there is a customer who consumes the resource and pays for it — there is money, a license is bought; no money - not bought.
Snapshots
Who worked with VMware and Hyper-V snapshots on high-load virtual machines? Do they work badly? The more snapshots, the worse the virtual machine works, the longer they are deleted and this deletion even more kills the performance of this machine. For NetApp ONTAP, snapshots are a trifle , no performance issues when creating or deleting. Many integrations with backup software such as Veeam or CommVault, etc., allowing you to capture consistent snapshots. VMware vVol allows you to capture hardware-assistant snapshots with a granular storage for each individual virtual machine, which eliminates the problems caused by hypervisor snapshots. Such non-inhibiting snapshots are beneficial for backup and DR. Where it is possible to use hardware-assistant storage snapshots that do not affect performance, you should use them everywhere.
New version of ONTAP 9.2
I recall that ONTAP is firmware 9.2 , which is installed either on the FAS / AFF hardware platform or is available as software (SDS) with a virtual machine on your server (ONTAP Select) or in a public cloud (ONTAP Cloud) . A couple of months ago, ONTAP firmware was released, most current customers can now upgrade to 9.2.
In my previous article, I painted a new functionality available in firmware 9.2 .
Replication
Replication for DR on FAS or ONTAP Select. Works on the basis of snapshots between two nontaps. So this replication with FabricPool allows you to save not only on the main site. You can put the storage on the backup site with 10-20% of the space from the main one, and again mix everything else into a cold level.
Tiering or FabricPool
The first thing to say about FabricPool is that, like any tiring technology, it is designed primarily to save storage resources, and this is customer money. If the budget is not limited, then you can stupidly buy the All Flash system and store snapshots on it for 3 years, plus buy one more of the same system on the backup site, build a metro cluster and a third on the backup backup site. But as a rule, there is no money for such an architecture, but there is still interest in the amount of SSD for a product (or some important part of it), but there are no longer any backups, snapshots or reservations. This is where FabricPool comes to the rescue.
FabricPool is a technology that allows you to combine fast and slow storage to create a hybrid pool (aggregate) consisting of SSD and S3 cloud, like how it works in FlashPool caching technology. As in FlashPool, all data first falls on the hot SSD level, and the cold data are then shifted to a slow and cheap level, this saves expensive SSD space.
FabricPool is a technology implemented as part of the DataFabric strategy, which, in turn, should expand the integration of all NetApp products and expand the possibilities of data mobility.
When NetApp makes a new feature, it doesn’t look at others and doesn’t try to add a tick feature. Instead, a technology is released that is really well thought out and will be in demand by consumers. This also applies to the new Tiering feature FabricPool. As the demand for SSDs grows, the number of installations and interest in this technology increases, so NetApp has released the FabricPool functionality that makes it possible to more efficiently use expensive SSD drives.
FabricPool is a cold data offset technology with SSD aggregate (AFF, FAS, ONTAP Cloud systems) to a cold object level - to Amazon S3 cloud or to NetApp StorageGRID object storage system .
Implementing FabricPool to move data to Object Storage will consist of two stages. In 9.2RC, the ability to shift only snapshots will be available. And the next stage will be the possibility of shifting cold data from the active file system of the volyum to the cold level, available from version 9.2GA. So far, FlexFroup, MetroCluster, ONTAP Select, SnapLock, etc. are not supported with FabricPool.
FabricPool on FAS / AFF and tertiary object cold data storage
To use the FabricPool functionality on FAS / AFF systems with Amazon S3 tertiary object storage, you must purchase per terabyte licenses that are added to ONTAP, adding a FabricPool license immediately adds 10TB. Then you can add by terabyte.
FabricPool on FAS / AFF and StorageGRID
For the operation of the FabricPool functionality on FAS / AFF storage systems, a license is only necessary if you are using Amazon S3. In the case of NetApp StorageGRID, a license for running FabricPool is not needed.
FabricPool and ONTAP Cloud
For ONTAP Cloud systems living in a public cloud, there is also no lossytech licensing from ONTAP Cloud to enable FabricPool - the customer only pays for the actual cold space consumed in the S3 object storage directly to the cloud provider. Currently, FabricPool is only supported on the ONTAP Cloud for Amazon.
FabricPool and Disaster Recovery
As before, SnapMirror replication is used to build a DR site and transfer data between two ONTAP storage systems. Very often, to save finances, a simpler and cheaper storage was bought at the DR site. Now FabricPool allows a slightly different approach to this issue. For example, if an AFF system is used at the main site, then a backup system can be placed on the same or similar, but still AFF system. In All Flash repositories, most controllers do not eat off controllers, but SSD drives themselves. Now, with FabricPool, you can install an AFF system on the backup platform, but with only 10% -20% of the usable space from the main storage, since cold data will go to the cold object level. Allowing thus to save on disks, but not to put controllers that would not pull performance. If there is no room for new hot blocks on the hot level, they will be read directly from the cold object storage.
How can FabricPool be interesting for cloud providers?
DSS provides a high-speed, but rather expensive level of data storage. NetApp FabricPool storage owners is a way to save money. And some customers simply according to the laws of their country can not take all or part of their data abroad in Amazon S3. For the cloud provider, FabricPool is a new customer and additional storage service.
What does the cloud provider need?
FabricPool technology is Tiering technology that allows new and existing customers to shift snapshots (backups) and cold data of productive storage systems using SSD (expensive level) from their data centers to a public cloud to a cold (cheap level). In a public cloud, you need to deploy NetApp StorageGrid software (available as VMware, KVM and Docker images). For cloud-based providers, NetApp StorageGrid is licensed every month by terabyte, which is very convenient: while there are customers for this service, the cloud provider pays for licenses, no - it does not. For owners of NetApp storage systems that will use the StorageGrid license located at the cloud provider, you do not need to buy a license.
Calculate the cost of FabricPool cold data storage service provider
The cost will include the following basic costs for the provider:
- StorageGrid software has system requirements (CPU, Memory), respectively, these are costs for the provider
- The customer will connect via IP to the service provider, respectively, this or just an external white IP address with a forwarding of the https port on the StorageGrid inside the cloud. External white IP and maybe FireWall cost some money; Or is it, again, a white IP for connecting via VPN, according to which the customer will overtake the cold data. Building a VPN also costs money; or is it a dedicated communication channel
- The disk space that will be used to deploy StorageGrid nodes is allocated only once, does not change throughout the life of the node. But the number of nodes can grow over time as the load and the growth of occupied space. The minimum number of 5 StorageGrid nodes (can be virtual machines): 3 nodes (storage node) + 2 administrative nodes (Gateway node & Admin node).
- Monthly - NetApp per terabyte licensing
- And the actual disk space occupied by the storage of the actual data from customers.
findings
FabricPool technology allows ONTAP and object storage for cold data to be integrated as part of the DataFabric strategy. For storage owners, FabricPool allows you to more efficiently use the expensive resources of All Flash systems and SSD aggregates, shifting inactive cold data to a cheap level, which in turn makes NetApp All Flash storage systems more affordable for the cost of ownership. Object storage is easily scaled as it grows and theoretically has no space restrictions (current Soft Limit 70PB). For cloud providers, FabricPool is a way to attract new customers who have previously stored productive data only on their equipment.
Source: https://habr.com/ru/post/330526/
All Articles