How to improve the performance of storage systems in the data center
According to Cisco, by 2019, one user will generate 1.6 GB of data in cloud storage each month. The increase in information is a major factor in the growth of the IT market and, at the same time, a serious challenge for infrastructure. Therefore, in this article we will look at how to increase the performance and efficiency of SSD and HDD in the data center.
/ frame from the video about the Xelent data center that hosts the 1cloud equipment
The performance problems associated with storage are limited by the size of the data blocks, regardless of the storage architecture - whether it is a traditional infrastructure or distributed storage. For example, operations with the same amount of data when working with blocks require different amounts of resources: writing 100 bytes to one block or several will take different times.
')
Another factor is how the system accesses files and how they are stored on disk. Procedures for determining the location of information lead to delays in processing in the case of HDD. At the same time, big data , over petabyte in size, is converted into a huge amount of metadata, which not every file system can handle.
Also on the performance of storage systems affect the physical limitations of technology. It is believed that with increasing productivity, the rates of development of HDD parameters do not follow Moore's law. SSDs are developing predictably: throughput with capacity is growing, as the industry expects.
However, the HDD still has room for maneuver - work is underway to reduce the disk size and increase the recording density, aimed at improving the efficiency of storage devices. For example, only at the beginning of October of this year, WD introduced a new technology of microwave magnetic recording (MAMR), which, according to developers, will allow you to create 40-terabyte HDD.
/ frame from the video about the Xelent data center that hosts the 1cloud equipment
Valdis Filks (Valdis Filks), director of research at Gartner Inc., compares the search for "leakage" of performance in the storage system with the search for treasures. And notes that this requires special tools. Today, the industry offers both embedded systems for evaluating the performance of data storage arrays, as well as third-party storage management software (SRM) and monitoring.
For example, Dell EMC offers built-in storage management and reporting tools. They allow you to generate statistics on the capacity, configuration and performance of the arrays used. Note that such solutions provide and IaaS-providers, if we are talking about tracking the performance of storage in a virtual infrastructure.
You can increase storage performance by properly arranging drives into pools. One option is to combine low-performance disks that store the bulk of the data, and use dynamic memory allocation and deduplication .
To perform these functions, there are a large number of modern solutions. Additionally, they provide data protection and virtual servers. One example is the ClearSky Data solution, which eliminates duplicate data and offers backup and recovery after a disaster.
The idea of ​​the project is as follows: a cache with business-critical information is stored at the client’s site, the “warm” data is placed in the local cloud, and the “cold” data is transferred to the external cloud. The company has developed software that automatically manages the data by moving it between storage points. Algorithms analyze the intensity of use of information and customer requirements.
Another project is a startup Datrium. The idea of ​​the developers is that deduplication is performed twice: on the local flash memory before sending to the long-term storage device, and after it. You can read about other intellectual systems of deduplication and backup in one of our materials.
If the amount of data used by the system is too large for the cache, performance may be significantly reduced. Virtualization of storage helps to solve the problem. Virtualization of mount points in which the storage system communicates with the server and OS allows DRAM to be used to create a shared cache.
Subsequently, it can be used to accelerate application performance and perform buffering. Specialists from StorageSwiss research firm say that this approach shows itself well when working with real-time transaction processing systems.
Upgrading the network helps in cases where performance suffers from deficiencies at the level of the main bus adapter, network switch or cable infrastructure. Cable connections are often overlooked as a factor in data storage efficiency.
Another reason is inefficient traffic management . If the network has limited bandwidth, then part of the channel can be diverted to critical services, “pushing up” the work of secondary (say, e-mail).
To improve the quality of traffic management, experts from the Ashton Metzler & Associates Inc. consulting group. advised to monitor the network and prepare reports to change the data distribution model in a timely manner.
According to Viavi Solutions employees, one of the effective methods of traffic management is a hierarchical Quality of Service (H-QoS) system. It combines network and filtering policies. H-QoS reduces latency without requiring an increase in network bandwidth.
Although SSD and HDD perform the same function, they are based on different technologies. Therefore, these devices require a different approach. For example, in the case of SSD fragmentation does not affect performance. This means that defragmentation does not provide advantages in terms of optimization. Therefore, the PC Magazine Labs team notes that it is sometimes useful to disable this feature for solid-state drives.
Also note that the internal implementation of a number of operations, for example, deleting files, in SSD and HDD is different. And the performance of some standard functions on SSD (for example, formatting) even leads to a deterioration in their performance. And to counter this effect, you should use the TRIM command, which reduces the effect of “ garbage collection ” on an SSD device.
There are situations when TRIM is not supported by the OS. For such cases, special tools have been developed that make it possible to send a command “manually”. These utilities form a list of free blocks and transfer it to the disk as a sequence of TRIM commands. At the same time, there are both universal tools (for example, hdparm ) and specialized software for disks of individual manufacturers ( Intel SSD Toolbox ).
PS Here are some more articles about hardware solutions from our blog:
/ frame from the video about the Xelent data center that hosts the 1cloud equipment
What affects the performance of HDD and SSD in data centers
The performance problems associated with storage are limited by the size of the data blocks, regardless of the storage architecture - whether it is a traditional infrastructure or distributed storage. For example, operations with the same amount of data when working with blocks require different amounts of resources: writing 100 bytes to one block or several will take different times.
')
Another factor is how the system accesses files and how they are stored on disk. Procedures for determining the location of information lead to delays in processing in the case of HDD. At the same time, big data , over petabyte in size, is converted into a huge amount of metadata, which not every file system can handle.
Also on the performance of storage systems affect the physical limitations of technology. It is believed that with increasing productivity, the rates of development of HDD parameters do not follow Moore's law. SSDs are developing predictably: throughput with capacity is growing, as the industry expects.
However, the HDD still has room for maneuver - work is underway to reduce the disk size and increase the recording density, aimed at improving the efficiency of storage devices. For example, only at the beginning of October of this year, WD introduced a new technology of microwave magnetic recording (MAMR), which, according to developers, will allow you to create 40-terabyte HDD.
/ frame from the video about the Xelent data center that hosts the 1cloud equipment
How to improve storage performance
Identify bottlenecks
Valdis Filks (Valdis Filks), director of research at Gartner Inc., compares the search for "leakage" of performance in the storage system with the search for treasures. And notes that this requires special tools. Today, the industry offers both embedded systems for evaluating the performance of data storage arrays, as well as third-party storage management software (SRM) and monitoring.
For example, Dell EMC offers built-in storage management and reporting tools. They allow you to generate statistics on the capacity, configuration and performance of the arrays used. Note that such solutions provide and IaaS-providers, if we are talking about tracking the performance of storage in a virtual infrastructure.
Optimize data
You can increase storage performance by properly arranging drives into pools. One option is to combine low-performance disks that store the bulk of the data, and use dynamic memory allocation and deduplication .
To perform these functions, there are a large number of modern solutions. Additionally, they provide data protection and virtual servers. One example is the ClearSky Data solution, which eliminates duplicate data and offers backup and recovery after a disaster.
The idea of ​​the project is as follows: a cache with business-critical information is stored at the client’s site, the “warm” data is placed in the local cloud, and the “cold” data is transferred to the external cloud. The company has developed software that automatically manages the data by moving it between storage points. Algorithms analyze the intensity of use of information and customer requirements.
Another project is a startup Datrium. The idea of ​​the developers is that deduplication is performed twice: on the local flash memory before sending to the long-term storage device, and after it. You can read about other intellectual systems of deduplication and backup in one of our materials.
Optimize cache
If the amount of data used by the system is too large for the cache, performance may be significantly reduced. Virtualization of storage helps to solve the problem. Virtualization of mount points in which the storage system communicates with the server and OS allows DRAM to be used to create a shared cache.
Subsequently, it can be used to accelerate application performance and perform buffering. Specialists from StorageSwiss research firm say that this approach shows itself well when working with real-time transaction processing systems.
Optimize network performance
Upgrading the network helps in cases where performance suffers from deficiencies at the level of the main bus adapter, network switch or cable infrastructure. Cable connections are often overlooked as a factor in data storage efficiency.
Another reason is inefficient traffic management . If the network has limited bandwidth, then part of the channel can be diverted to critical services, “pushing up” the work of secondary (say, e-mail).
To improve the quality of traffic management, experts from the Ashton Metzler & Associates Inc. consulting group. advised to monitor the network and prepare reports to change the data distribution model in a timely manner.
According to Viavi Solutions employees, one of the effective methods of traffic management is a hierarchical Quality of Service (H-QoS) system. It combines network and filtering policies. H-QoS reduces latency without requiring an increase in network bandwidth.
Use storage device features
Although SSD and HDD perform the same function, they are based on different technologies. Therefore, these devices require a different approach. For example, in the case of SSD fragmentation does not affect performance. This means that defragmentation does not provide advantages in terms of optimization. Therefore, the PC Magazine Labs team notes that it is sometimes useful to disable this feature for solid-state drives.
Also note that the internal implementation of a number of operations, for example, deleting files, in SSD and HDD is different. And the performance of some standard functions on SSD (for example, formatting) even leads to a deterioration in their performance. And to counter this effect, you should use the TRIM command, which reduces the effect of “ garbage collection ” on an SSD device.
There are situations when TRIM is not supported by the OS. For such cases, special tools have been developed that make it possible to send a command “manually”. These utilities form a list of free blocks and transfer it to the disk as a sequence of TRIM commands. At the same time, there are both universal tools (for example, hdparm ) and specialized software for disks of individual manufacturers ( Intel SSD Toolbox ).
PS Here are some more articles about hardware solutions from our blog:
Source: https://habr.com/ru/post/342718/
All Articles