Introduction to the Storage Performance Development Kit

Data storage systems based on solid-state drives are becoming increasingly common in data centers. The current generation of flash-based storage has significant advantages in terms of performance, power consumption and data density in racks compared to disk drives. These advantages will become even more tangible when a new generation of drives appears on the market. Customers introducing current-generation solid-state drives, such as the Intel P3700 NVMe *, have to cope with a complex problem: since bandwidth is much higher than that of disk drives, and the latency is much lower, most of the total transaction time now falls on the storage software.

To assist OEMs and software vendors in integrating such hardware, Intel has created a set of drivers and developed a complete storage architecture. This initiative is called the Storage Performance Development Kit (SPDK) . The goal of the SPDK is to highlight the high efficiency and performance achieved by the combination of Intel technologies in the area of ​​networking, computing and data storage. Intel provides the complete source code for the Linux reference architecture (licensed from Intel) for free. The NVMe user-mode driver is provided as an open source community on the 01.org portal, and the code for other elements of the package will be presented throughout 2016.

How does SPDK work? Extremely high performance is achieved through the use of two main methods: work at the user level and in the polled mode. Let's take a closer look at these two program principles.

First, the operation of our device driver code at the user level means, by definition, that the driver code does not run in the kernel. Failure to interrupt and switch the kernel context allows you to save significant computational resources and spend more cycles on the actual storage of data. Regardless of the complexity of the storage algorithms (deduplication, encryption, compression, or regular block storage), the fewer wasted cycles, the better the performance.
')
Secondly, polled mode drivers (PMD) are constantly waiting for work instead of being sent to work. Imagine such a situation: on Friday evening in the center of the city you are standing on the sidewalk trying to catch a taxi, but cars pass by one by one, because passengers are already sitting in them. It is impossible to predict how much time you will have to wait on the road before there is a free taxi. Such difficulties can be expected by a packet or a data block when trying to “catch a car” in the traditional data storage I / O driver with load distribution based on interrupts. On the other hand, imagine the process of landing in a taxi at a good airport. Taxi drivers are already in place and are waiting for suitable passengers; when a passenger arrives, after a few seconds they serve the car and deliver the passengers and baggage to the required addresses. This is how the polled mode (PMD) drivers work, and so all the components of the SPDK are arranged. Packets and blocks are distributed instantaneously, latency is minimized, thereby reducing delays, increasing throughput, and stabilizing delays (less fluctuations).

The SPDK consists of many components that are interconnected and use common elements of the user level and work in the polled mode. Each of these components was created to overcome a specific performance bottleneck discovered during the creation of the full SPDK architecture. In this case, all developed components can be integrated into other architectures other than SPDK, which gives customers the opportunity to use the technologies used in the SPDK to accelerate the work of their own software. For example, a User Space Network Services Library (UNS) was created to overcome the performance limitations of the Linux TCP / IP set. By creating a user-friendly polled TCP / IP implementation in the SPDK, we achieved much higher IOPS, spending a lot less CPU cycles in sorting and processing TCP / IP packets.



The solution consists of three categories of components: an external network interface, a computing platform, and internal systems.
The external interface consists of the Data Plane Development Kit (DPDK) network adapter driver and user space network services (UNS). DPDK provides a platform for high-speed packet processing at the network adapter level, which allows data to more quickly get from the network adapter to user space. After that, the UNS code is taken for work: it unpacks TCP / IP packets and forms iSCSI commands.

After that, the computing platform receives the contents of the packets and converts the iSCSI commands into block-level SCSI commands. Before sending these commands to the internal drivers in the SPDK, there is an API platform for adding individual functions needed by clients within the SPDK platform (the green box in the figure above). Examples of such functions are caching, deduplication and data compression, encryption, calculations for RAID and distributed storage. Samples of such functions are part of the SPDK, although they should not be mistaken for implementations ready for use in the working environment: they serve only to simulate real use scenarios.

Finally, the data gets to the back-end drivers who interact with physical devices, that is, perform read and write operations. The SPDK includes user-level PMD for various storage media: NVMe devices; universal devices running Linux with traditional disk drives; memory drivers for block addressable memory solutions (for example, RAMDISKS), as well as devices capable of using Intel I / O Acceleration Technology (Crystal Beach DMA). The package of back-end drivers allows you to expand the range of used high-performance storage devices, which increases the versatility of the entire system.

SPDK is not compatible with all storage architectures. Here are a few questions that users can use to determine whether SPDK components fit their architecture.

Does the storage system work on Linux?
SPDK is currently tested and supported only on Linux.

Is storage performance in user mode?
SPDK has been able to improve performance and efficiency by redistributing performance from the network adapter to disk exclusively in user mode.

Does the system architecture allow non-blocking PMD drivers in the thread model?
Since PMD drivers are constantly running in their own threads (instead of going to sleep and releasing the processor when idle), they have strict requirements for the flow model.

Does the Data Plane Development Kit (DPDK) system work with loads consisting of network packets?
DPDK contains a platform for SPDK, so customers using DPDK currently can use SPDK integration.

Does your licensing model allow use of non-redistributable source code?
Some parts of the SPDK are available as open source components with BSD licenses (for example, NVMe and CBDMA user-space drivers). Other fragments are currently licensed under Intel licenses (UNS and iSCSI Target user space), although this, of course, may change. All SPDK source code is free.

Does the development team have the necessary knowledge to independently investigate and troubleshoot?
Intel is not obligated to support this software. Both Intel and the open source software development community working with the SPDK will take commercially reasonable actions to investigate possible errors of un-released software, but Intel will under no circumstances commit to customers for any service or support this software solution.