NetApp FlexClone: How technology accelerates development
Here I want to briefly describe the traditional architecture of the IT infrastructure to ensure the resiliency of the database and the development scheme using this architecture. And to elaborate on how to simplify the life of testers and developers using FlexClone technology, which use a database or any other large data set needed for testing and development.
The traditional approach is to create a database cluster: main and backup. Further, from time to time this database (as a rule, on the stand-by side) is copied for developers. And the more such a database, the less often it is copied and deleted and the more heavily loaded the storage system from such operations. On the other hand, developers and testers get an obsolete database, it is usually a month or even six months ago. When it comes time to implement the debugged code in the production, it turns out that the database has had time to change a lot: new additional data, tables, fields have appeared, and old ones are probably deleted. This leads to the fact that the debugged code for the old database does not work on the new one, which greatly complicates the development, testing, implementation and operation of the new code.
FlexClone technology allows you to create thin clones, based on NetApp snapshots, which do not affect system performance . Clones can be removed a lot and very often, right in the middle of the working day. Thanks to this simple phenomenon, the life of testers and developers is greatly simplified, and the quality of the code increases. This fits into a popular approach called Copy Data Management (CDM), which aims to use one set of data for several tasks at once. The approach is convenient due to the fact that:
')
- License FlexClone;
- Locate the Prod or Stend-By on the storage with ONTAP firmware (ONTAP Select / ONTAP Cloud / on the NetApp FAS platform) in order to perform thin database cloning using FlexClone technology;
- A product that allows integrating databases with NetApp Hardware-Assistant snapshots to achieve consistency from which clones will be created later:
The products described below perform backups and each of them can create consistent snapshots.
The operations of removing snapshots of a consistent (see the list above), cloning and connecting to the database can be easily automated by providing them as a service for the Test / Dev department using the free Workflow Automation web interface.
CodeEasy is a set of scripts and a methodology for DevOps that uses NetApp technologies such as Snapshot and FlexClone to reduce testing and development time. CodeEasy Toolkit scripts use the NetApp Manageabilitys SDK to automate the steps of preparing the workspace for developers and testers, which allows them to be used without any changes or with minimal settings.
System requirements:
Download the CodeEasy FlexClone Toolkit .
For Windows machines, NetApp PowerShell Toolkit is available, which allows you to create NetApp management scripts.
The NetApp Docker Volume Plugin (nDVP) provides a link between Docker and NetApp storage systems (ONTAP, SolidFire and E-Series) to create and connect permanent storage inside containers. This topic deserves special attention and will be described in one of the following articles .
FlexClone is also very often used in backup environments to test the performance of backups. So a lot of software backup allow you to automate this process. For example, Veeam On-Demand Sandbox integrates with ONTAP and through the use of FlexClone, it allows you to instantly create a clone from a virtual machine of any size without affecting the backup, run the clone and test its performance and recoverability.
Workflow Automation is a free graphical utility available for Windows and Linux that allows you to create sets or bundles of tasks to automate ONTAP management processes. For example, through it you can configure the cloning of volums or moons, transfer them to the test SVM, create new file balls or iGroup, add to it cloned volyums and new initiating hosts from the test platform, raise new LIF interfaces. And after a temporary rupture of the replication relationship, to clone the data, restore it back - and all this with almost one mouse click. WFA also provides the ability to manage through PowerShell and Perl scripts, as well as through the REST API.
The NetApp FlexClone technology also finds application in a virtualization environment, allowing you to quickly deploy clones of virtual machines, thanks to integration with leading global hypervisor vendors. So you can deploy thousands of virtual machines in minutes, without having performance problems (as in the case of VMware Virtual Disk Thing Provisioning, VMware COW Snapshots or Linked Clone).
It is important to note that the FlexClone technology not only significantly speeds up the deployment of large databases or a large amount of any other data, drastically reducing disk space consumption, but also indirectly speeds up their work. Cloning essentially performs a function like deduplication, i.e. reduces the amount of space occupied. The fact is that the NetApp FAS storage system always puts data in the system cache, and the system and SSD caches in turn are Dedup-Aware, i.e. they do not overlap duplicate blocks that are already there, so both caches logically hold much more data than they physically can. This greatly improves performance during storage and especially during moments of Boot-Storm due to the increased hit / read data to / from the cache (a).
In addition to testing / development, subtle cloning can find other applications. For example, sometimes it makes sense to divide access to data into tasks with different priorities. The cloning technology perfectly complements the SnapMirror replication functionality , when the DR site is used not only as a reserve in case of an accident, but also as a development environment (to clone Data Protection Volyum).
An example would be a database that is used as transactional, and monthly or quarterly it is also used to generate reports. Transactions typically generate small, random read / write operations, such operations can be cached quite successfully and should be performed with a low response rate (usually not higher than 10ms). On the other hand, there are reports that generate large sequential read operations, and when the results are saved, there are records (they are usually much smaller, for example 10%). Reports are built once a month, quarter or year, they can be collected for a full day and there is no fundamental difference if they were not collected in a day, but, for example, for three days.
So if you add more reports to transactions, then transactions can suffer greatly, with all that reports if it is late by an hour or day is not important. It turns out a very unfair distribution of resources. This is where FlexClone, QoS and caching policies can come to the rescue:
You can create a thin clone, kill MB / s with it, for example, 3 times less than if reports were run without restrictions, change caching policies and get three times less impact on the disk subsystem, and 3 times more time to build a report (which is usually quite acceptable). And all these processes on the storage system can be automated using the free software NetApp Workflow Automation.
All the same can be applied in the scheme, when the clone is removed from the Stand-By in the database cluster: although it is not loaded with productive transactions directly, it should still be able to keep up with the changes from the main database. And in order not to slow it down, it is again convenient to use FlexClone and QoS.
These tasks are fairly easy to automate using paid or free software .
The FlexClone technology allows not only more optimal use of storage resources (both in terms of performance and space), but also significantly speeds up the business processes of companies, speeding up and simplifying the process of developing and debugging new code.
To download software from NetApp, you may need a login and password.
This may contain links to Habra articles that will be published later.
I ask to send messages on errors in the text to the LAN .
Comments, additions and questions on the article on the contrary, please in the comments.
The traditional approach is to create a database cluster: main and backup. Further, from time to time this database (as a rule, on the stand-by side) is copied for developers. And the more such a database, the less often it is copied and deleted and the more heavily loaded the storage system from such operations. On the other hand, developers and testers get an obsolete database, it is usually a month or even six months ago. When it comes time to implement the debugged code in the production, it turns out that the database has had time to change a lot: new additional data, tables, fields have appeared, and old ones are probably deleted. This leads to the fact that the debugged code for the old database does not work on the new one, which greatly complicates the development, testing, implementation and operation of the new code.
FlexClone is a more efficient way to handle Test / Dev with large databases.
FlexClone technology allows you to create thin clones, based on NetApp snapshots, which do not affect system performance . Clones can be removed a lot and very often, right in the middle of the working day. Thanks to this simple phenomenon, the life of testers and developers is greatly simplified, and the quality of the code increases. This fits into a popular approach called Copy Data Management (CDM), which aims to use one set of data for several tasks at once. The approach is convenient due to the fact that:
')
- QoS can be killed on a thin clone (this does not apply to Space Reclamation when using UNMAP )
- It takes nothing, in time it will only occupy changes, and since it is more readable, there will be few changes
- The FullBackup copying process will not load the disk subsystem.
- Instantly removed, and does not take long hours
- Testers get the most recent data much more often and can better debug their code on actual (rather than outdated data). What is very important to speed up and improve the quality of debugging code.
- It is not difficult to automate using software for removing consistent snapshots.
What is needed for this?
- License FlexClone;
- Locate the Prod or Stend-By on the storage with ONTAP firmware (ONTAP Select / ONTAP Cloud / on the NetApp FAS platform) in order to perform thin database cloning using FlexClone technology;
- A product that allows integrating databases with NetApp Hardware-Assistant snapshots to achieve consistency from which clones will be created later:
Automation and consistency software
The products described below perform backups and each of them can create consistent snapshots.
- SnapCreator is free, but it will need to be modified using a script to be able to freeze the database and remove the snapshot
- NetApp SnapCenter is a paid product that fully automates the cloning process, provides a convenient GUI
- Commvault Simpana is a paid backup product. Everything is the same as SnapCenter, only with frills such as cataloging and archiving to tape.
- Other third-party products (Symantec, Veeam , Syncsort, EMC Networker , HP DataProtector , IBM Tivoli , Acronis , etc.) that can integrate with the database and NetApp ONTAP to remove the Hardware-Assistant Snapshot.
- If you shoot a clone with Stand-By, you can temporarily stop the database (then remove NetApp snapshots to achieve consistency) and run it again, then you will not need software to remove consistent snapshots without writing a script to interact with the database
The operations of removing snapshots of a consistent (see the list above), cloning and connecting to the database can be easily automated by providing them as a service for the Test / Dev department using the free Workflow Automation web interface.
CodeEasy FlexClone Toolkit
CodeEasy is a set of scripts and a methodology for DevOps that uses NetApp technologies such as Snapshot and FlexClone to reduce testing and development time. CodeEasy Toolkit scripts use the NetApp Manageabilitys SDK to automate the steps of preparing the workspace for developers and testers, which allows them to be used without any changes or with minimal settings.
System requirements:
- ONTAP (cDOT) 8.2 or later
- NetApp Manageability Software Development Kit (NMSDK) 5.3.x or later
Download the CodeEasy FlexClone Toolkit .
NetApp-PowerShell Commandlets
For Windows machines, NetApp PowerShell Toolkit is available, which allows you to create NetApp management scripts.
NetApp Docker integration
The NetApp Docker Volume Plugin (nDVP) provides a link between Docker and NetApp storage systems (ONTAP, SolidFire and E-Series) to create and connect permanent storage inside containers. This topic deserves special attention and will be described in one of the following articles .
Backup
FlexClone is also very often used in backup environments to test the performance of backups. So a lot of software backup allow you to automate this process. For example, Veeam On-Demand Sandbox integrates with ONTAP and through the use of FlexClone, it allows you to instantly create a clone from a virtual machine of any size without affecting the backup, run the clone and test its performance and recoverability.
Workflow Automation (WFA)
Workflow Automation is a free graphical utility available for Windows and Linux that allows you to create sets or bundles of tasks to automate ONTAP management processes. For example, through it you can configure the cloning of volums or moons, transfer them to the test SVM, create new file balls or iGroup, add to it cloned volyums and new initiating hosts from the test platform, raise new LIF interfaces. And after a temporary rupture of the replication relationship, to clone the data, restore it back - and all this with almost one mouse click. WFA also provides the ability to manage through PowerShell and Perl scripts, as well as through the REST API.
Server virtualization and VDI
The NetApp FlexClone technology also finds application in a virtualization environment, allowing you to quickly deploy clones of virtual machines, thanks to integration with leading global hypervisor vendors. So you can deploy thousands of virtual machines in minutes, without having performance problems (as in the case of VMware Virtual Disk Thing Provisioning, VMware COW Snapshots or Linked Clone).
Performance
It is important to note that the FlexClone technology not only significantly speeds up the deployment of large databases or a large amount of any other data, drastically reducing disk space consumption, but also indirectly speeds up their work. Cloning essentially performs a function like deduplication, i.e. reduces the amount of space occupied. The fact is that the NetApp FAS storage system always puts data in the system cache, and the system and SSD caches in turn are Dedup-Aware, i.e. they do not overlap duplicate blocks that are already there, so both caches logically hold much more data than they physically can. This greatly improves performance during storage and especially during moments of Boot-Storm due to the increased hit / read data to / from the cache (a).
Additional use cases
In addition to testing / development, subtle cloning can find other applications. For example, sometimes it makes sense to divide access to data into tasks with different priorities. The cloning technology perfectly complements the SnapMirror replication functionality , when the DR site is used not only as a reserve in case of an accident, but also as a development environment (to clone Data Protection Volyum).
Transactions VS Reports
An example would be a database that is used as transactional, and monthly or quarterly it is also used to generate reports. Transactions typically generate small, random read / write operations, such operations can be cached quite successfully and should be performed with a low response rate (usually not higher than 10ms). On the other hand, there are reports that generate large sequential read operations, and when the results are saved, there are records (they are usually much smaller, for example 10%). Reports are built once a month, quarter or year, they can be collected for a full day and there is no fundamental difference if they were not collected in a day, but, for example, for three days.
So if you add more reports to transactions, then transactions can suffer greatly, with all that reports if it is late by an hour or day is not important. It turns out a very unfair distribution of resources. This is where FlexClone, QoS and caching policies can come to the rescue:
You can create a thin clone, kill MB / s with it, for example, 3 times less than if reports were run without restrictions, change caching policies and get three times less impact on the disk subsystem, and 3 times more time to build a report (which is usually quite acceptable). And all these processes on the storage system can be automated using the free software NetApp Workflow Automation.
All the same can be applied in the scheme, when the clone is removed from the Stand-By in the database cluster: although it is not loaded with productive transactions directly, it should still be able to keep up with the changes from the main database. And in order not to slow it down, it is again convenient to use FlexClone and QoS.
These tasks are fairly easy to automate using paid or free software .
findings
The FlexClone technology allows not only more optimal use of storage resources (both in terms of performance and space), but also significantly speeds up the business processes of companies, speeding up and simplifying the process of developing and debugging new code.
To download software from NetApp, you may need a login and password.
This may contain links to Habra articles that will be published later.
I ask to send messages on errors in the text to the LAN .
Comments, additions and questions on the article on the contrary, please in the comments.
Source: https://habr.com/ru/post/280105/
All Articles