Basic storage protocols: use and prospects

In this article, we will conduct a brief overview of the most popular protocols that are used to build storage area networks (SANs). We also consider the prospects for the development and use of individual protocols, based on publicly available roadmaps of manufacturers.

It is no secret that today, performance problems are moving from the storage area to the SAN area, as the storage systems have already reached huge performance indicators in GB / s and millions of IOps, and the current SAN networks do not allow such data volumes to be pumped through.

General trends

Existing FC 8Gb / s and Ethernet 10Gb / s networks can no longer cope with modern storage systems, even Ethernet 25/50 Gb / s cannot provide acceptable delays when working with the latest storage systems using NVMe disks.
')
Many IT professionals involved in setting up and administering storage infrastructure are wondering about upgrading SAN networks. Why is it so important and necessary today? We formulate several main reasons:

• First of all, the data storage density is constantly growing, the volumes of both HDD drives and flash drives increase. HDDs of 10TB and SSD up to 3.2TB are already available on the market.
• Second, the speed of flash and hybrid storage systems grows in two aspects: throughput increases and latency decreases. For example, the latest models of NVMe-drives have a performance of about 3 GB / s and a delay of 20 μs.
• Third, the need to handle millions of IOps. Already, one NVMe disk is capable of issuing up to 700+ thousand IOps.
• Fourth, the need to increase performance per port connection. If earlier from a single QSFP + port on the InfiniBand protocol, they received a maximum of 56 Gb / s, then already this year there are solutions with a capacity of 200 Gb / s per port.
  

Given all these factors, the key characteristics of access protocols are high throughput and fast response time.

According to existing forecasts, in the two years' perspective (according to Gartner, for 18 months) all-flash arrays will be completed with more productive SSD-drives, and in combination with fast interconnect (for example, NVMe) and new protocols (for example, iWARP) performance all-flash arrays will increase by two orders of magnitude, which will lead to the need to upgrade the SAN.

Protocol Comparison

All SAN networks, whether Ethernet, PCI Express, SAS, NVMeoF, FC, FCoE or InfiniBand, must support the same functionality, namely:

• identify host and storage;
• be able to route traffic;
• divide the network into subnets and isolate traffic in them;
• provide the ability to use multiple paths to storage systems;
• manage the device connection to the network;
• prioritize traffic.

Given the similar functionality implemented, it can be said that the differences between the protocols are reduced to usability, performance parameters and implementation costs. Thus, some protocols have more advanced management and monitoring tools, which makes them easier to implement, administer, and operate. Other protocols have better performance, but are more complex in terms of integration into current SAN networks.

Today, data storage protocols can be divided into two conditional groups:

1. used to connect application servers (FC, FCoE, iSCSI, NFS, SMB);
2. used for connection within the cluster or as an interconnect within the storage system (InfiniBand, NVMe, PCIe).

Fiber Channel (FC)

Fiber Channel is a popular storage protocol that provides low latency and high throughput due to its architectural features. Fiber Channel is not resource demanding and is well suited for transferring large amounts of data, since all FC operations are performed on the HBA side, offloading the CPU.

New versions of the Fiber Channel protocol are backward compatible with previous editions, which opens up good prospects for modernization and scaling. For example, if you implement FC 32GB / s, you can still use FC 8Gb / s and 16GB / s, i.e. You can gradually change the FC switches and FC adapters.

In the near future, FC will be upgraded to 64 Gb / s and 128 Gb / s (now there are switches that support the aggregation of 4 32 Gb / s ports into one 128 Gb / s channel for connecting switches).

Simplicity of setup and ease of administration allowed FC to become one of the most common storage protocols. Most SAN administrators around the world know how it works and what advantages it provides for solving various tasks. At the same time, FC is still more complicated than Ethernet, although it has a large number of management and monitoring tools.

FCoE (Fiber Channel over Ethernet)

The idea of ​​creating FCoE was to consolidate I / O operations and, as a result, to ensure the safe placement of various types of traffic in a single "wire". This idea meant reducing the total cost of ownership of the system (TCO) by reducing the number of cables and adapters, as well as reducing energy consumption. At the same time, the availability and performance of FCoE are not comparable with those of FC, since data transmission requires additional overhead for encapsulation in Ethernet.

In addition, FCoE adds complexity in deploying and administering the entire system, increasing the level of requirements for service personnel and solution support. Despite the hopes of FCoE manufacturers for widespread implementation, this protocol has not yet been able to force out FC and is developing at a very slow pace. The spread of FCoE in the SAN market is currently minimal.



According to the Fiber Channel Industry Association (FCIA) in the FCoE forecast indicators, the speed of the protocol depends on Ethernet implementations.

iSCSI (Internet Small Computer System Interface)

iSCSI is built on the two most commonly used protocols:

• SCSI - protocol for exchanging data blocks between a computer and storage
• IP is a network transport protocol widely used in corporate Ethernet networks.

iSCSI is a low-budget implementation solution. Administration of such installations is very simple, although for fault tolerance it is necessary to build a dedicated network for iSCSI, which brings us closer to a network implementation that is very similar to FC SAN.

It is believed that iSCSI 10Gbps provides the same amount of IOps and bandwidth as the comparable 8Gbps FC, but this is not entirely true. Although iSCSI throughput is higher, its efficiency is lower than that of FC due to additional overhead.

ISCSI performance depends on the existing Ethernet infrastructure (currently the minimum recommended iSCSI network is 10Gbps). In the near future (according to Gartner, 10–12 months), it is worth planning a transition to 25/40 / 50GbE if you need to use high-performance all-flash storage systems.

Ethernet SANs

The iSCSI, NFS, SMB, FCoE protocols use Ethernet networks for data transfer, therefore it is not advisable to use these protocols in common networks. This leads us to the need to deploy dedicated networks for use as a SAN.

Such networks should work in parallel with other major networks and separate user traffic from server data traffic. However, due to network separation, administration complexity and total cost of ownership increase.

The only exception is the use of common networks for object storage, since in this case, ensuring minimum delays and maximum performance is not so critical.

NFS (Network File System)

Network File System (NFS) is a network file access protocol originally developed by Sun Microsystems in 1984. NFS provides clients with transparent access to the files and file system of the server.

NFS is simply configured and administered because used over Ethernet networks. As in other protocols that use Ethernet, speed and delays are entirely dependent on the implementation of the network at the lower level and most often “rest against” the limitations of the 10GbE standard.

NFS is often used as an entry-level protocol for building a SAN network for virtualization. According to Gartner, this trend will continue over the next 10 years.

SMB (Server Message Block)

SMB is a network file sharing protocol that allows computer applications to read and write files, as well as request the services of server programs on a computer network. Developed by Microsoft for implementing the Microsoft Windows Network and File and Printer Sharing. With the increase in data throughput, SMB has become one of the main file access protocols used in storage. In SMB storage systems, it is most often used in 10GbE networks, because of this, its performance is highly dependent on the implementation, settings, and network components used.

Before SMB 3.0, the protocol was mainly used for transferring files on local networks. With the new version that supports SMB-Direct technology (using RDMA), it has been actively used in virtualization clusters based on MS Hyper-V as a protocol for accessing the shared storage pool.

Like NFS, SMB is often used as an entry-level protocol when building a SAN network for virtualization. This trend should also continue in the next decade.

InfiniBand (IB)

InfinBand is a high-speed protocol that provides very high bandwidth and low latency. Used primarily in the industry of high-performance computing (HPC) and as an interconnect when creating high-performance storage.

The most obvious disadvantage of this technology is the difficulty in setting up and administering. As a result, working with IB requires qualified staff.
Among the difficulties of ubiquitous InfinBand are limited tools for monitoring performance and diagnosing problems, as well as not the best compatibility with different operating systems.

Compared to FC and Ethernet, InfiniBand is more “expensive” for implementation and maintenance. In addition, there are only a few read companies that produce equipment and software for working with IB.

NVMe (NVM Express)

NVMe is a fairly new high-performance protocol for accessing solid-state drives connected via the PCI Express bus. The abbreviation “NVM” in the title denotes non-volatile memory (Non-Volatile Memory), which is commonly used in SSDs like NAND flash memory.

The protocol was developed from scratch. The main objectives of its development are low latency and efficient use of high concurrency of solid-state drives. The latter problem is solved by applying a new set of commands and a mechanism for processing I / O queues optimized for working with modern processors.

At the moment, NVMe technology is not yet widespread. Basically, the protocol is used for internal connections in servers and storage. The NVMe specification also allows you to encapsulate it in other protocols, such as Ethernet, FC and InfiniBand, and scale the protocol in larger networks. However, since NVMe uses direct memory access (RDMA), the carrier protocol latency must be very low for the protocol to work properly.

In 2017, NVMe is expected to be actively introduced, as many manufacturers of server platforms are introducing new models with support for dual-port NVMe devices, which will allow designing fault-tolerant storage solutions.

Over the next few years, NVMe is expected to be used as an external gateway, similar to PCIe and InfiniBand. For example, it can be used in small specialized storage networks of dozens of nodes or devices in a homogeneous storage environment. A much wider NVMe will be used as an internal interconnect.

PCIe

PCIe has very low latency and is used primarily in servers to connect expansion cards, including high-performance NVMe disks. Some new products from well-known manufacturers use PCIe as a protocol for connecting servers through small PCIe switches, i.e. PCIe can only be used in small SAN networks.

Only a few storage vendors in the world use PCIe for external connections, which determines the highly specialized application of this protocol.

Since PCIe does not use SCSI and requires its own protocol for data transfer, it can increase throughput by reducing delays, in essence, working at the speed of a pure PCIe-line. Such solutions require the use of proprietary drivers, which makes them difficult to administer and makes it impossible to create heterogeneous infrastructures, as well as build such solutions into existing SAN networks.

Today, the main implementation of the technology used is PCIe 3.x, in which performance is increased by 40% compared to PCIe 2.x.

By the number of lines, PCIe scales from 1GB / s to 16GB / s. In 2017, a new PCIe 4.x standard will be released, which will increase performance by 2 times, i.e. maximum performance will reach 32GB / s.

Object Storage Protocols

Object storage includes a large number of new protocols that operate on top of HTTP, through the API. To date, many new applications are developed taking into account the use of these protocols, this is especially true for backup software.

The performance of these protocols depends heavily on the lower layer, most often it is Ethernet. The main functional application of object storage protocols is working with unstructured data. Because of this, they are used within the data center, as well as widely used to access data and write to cloud storage pools.

Object access protocols require the transfer and storage of large amounts of metadata, which always adds overhead, especially when comparing block access protocols.

Over the past few years, many object access protocols have been developed. The most popular of them are SOAP, S3, OpenStack Swift. In the near future (according to Gartner, for 5 years), these technologies will be of particular interest.

Conclusion

The development of SAN protocols directly depends on the development of applications and load profiles.
Applications and load types like HPC, Big Data analytics and active archives will move SANs towards creating storage systems with very low latency and high bandwidth, as well as supporting shared access using NVMe.

Protocols that will be implemented in the near future (such as 40 / 100GbE), file protocols (such as NFS over RDMA and SMB-Direct) and current block protocols (such as FC 16Gb / s) are already too slow for the next generation of Flash and hybrid storage systems.
The main protocol for the next decade will remain FC, since it created all the necessary infrastructure. Many will upgrade to FC 16GB / s, then FC 32GB / s and newer versions.

Infiniband, PCIe and NVMe will remain protocols for connecting end devices, inter-node connections in clusters or interconnect with low latency. At the same time, there will be small, highly specialized solutions for SAN networks that require minimum delays and maximum throughput. FCoE, iSCSI, NFS, SMB or object protocols will be used primarily as external protocols.

Every year interest in object storage systems grows. This is due to the increase in the amount of unstructured data, the emergence of new tasks for their processing and new requirements for information storage.

Used sources:

• The Fiber Channel Roadmap, FCIA 2016.
• Dennis Martin. Demartek Storage Networking Interface Comparison, Demartek 2016.
• Valdis Filks, Stanley Zaffos. The Future of Storage Protocols, Gartner 2017.