Sharing with RAIDIX and Apple Xsan. Apple saved for video editors

Efficient storage of large files in various industries becomes a daunting task. So, in the media industry used huge amounts of content. Considering that 4K digital video requires 4 times more free capacity than standard high definition files, a medium-length feature film can take from 2 (in full HD format) to 15 (in 8K) terabytes of data.

Studio specialists operate terabyte-sized files and perform video editing, color correction and final processing of the material with the help of resource-intensive professional applications.

Post-production studios in most cases use Apple equipment and professional software suite, for example, Apple Final Cut Pro, Adobe Premiere Pro, Autodesk Discreet Smoke, DaVinci Resolve and others.
')
One of the best solutions for fast file sharing from various Apple workstations is the Xsan cluster file system, which provides high performance when sharing resources between multiple clients.

Combined with a reliable data storage system, this solution not only provides shared access to materials, but also maximizes throughput, fault tolerance in RAID arrays and flexible architecture scalability. Next, we describe how to build a high-performance storage system based on RAIDIX software and the Xsan file system, we give the elements of infrastructure, technical indicators, recommendations for the equipment, connection and configuration of the system.

Tasks and solution

The constantly growing amount of information and the use of resource-intensive software in post-production implies a growing demand for high-performance storage systems with high data availability. At the same time, modern video data processing processes require that the same files be available simultaneously for several installation stations that perform various tasks.

Other key requirements of media studios include:

1. High speed access. It involves the use of local storage or high-speed access to the network storage system.
2. Distributed access. It involves the use of a file system with simultaneous access to files from multiple workstations.

The simultaneous solution of these polar problems is successfully implemented in the Xsan file system. Apple Xsan is Apple's 64-bit cluster file system for Mac OS X or Mac OS X Server. The system is designed to provide quick access to huge amounts of data. Xsan allows you to share files and volumes up to 16 TB each in a high-speed optical network.

Xsan is characterized by high availability and almost no failures, providing the ability to use a SAN network to consolidate data and replace NAS devices.

Solution Components

Storage System

The single-controller server SM 6048R-E1CR36N with RAIDIX software and FC 16Gb + JBOD 847E1C-R1K28JBOD configuration is used as the storage system. The system supports SATA, SAS, NL-SAS drives. The storage has 4 16 Gb Fiber Channel ports, the total bandwidth reaches 6 GB / s.

Metadata Controller

As a metadata controller (MDC), it is recommended to use the Apple Mac Mini. To ensure high availability, it is preferable to use 2 metadata controllers.

Client workstation

In the role of client stations it is recommended to use Apple Mac Pro server.

File system

Xsan distributed cluster file system allows simultaneous access to video from any workstation. Xsan must be pre-installed and licensed on the Apple Mac Mini.

Switching

Fiber Channel / Thunderbolt. Mac mini and Mac Pro do not support FC adapters, so ATTO ThunderLink FC 2082 for Mac Mini and ATTO ThunderLink FC 2162 for Mac Pro are recommended for connection to the SAN network.



Fiber Channel (FC) switch. When building a solution with a small number of clients (up to 20), the connection between FC adapters is made directly from the storage system to the MDC and clients (without the FC switch). To deploy a SAN network with a large number of clients, at least one FC switch is required.



Most of all components are connected by fiber-optic cable with LC-LC connectors and are connected directly to ATTO FC storage system and workstations adapters, FC Thunderbolt, FC switches.



Important: optical cables connect to FC adapters via SFP transceivers in each port. Most FC adapters come with SFP transceivers, but some switches are supplied without transceivers, which must be considered when purchasing equipment.

The only exception to the optical switching scheme is the connection between the Apple Mac Mini and the Fiber Channel Thunderbolt: it uses a special Thunderbolt cable.

MDC servers and all client workstations must be connected with an ETH channel (network) for metadata.

Solution Architecture

As a controlling component in this architecture, we consider RAIDIX, software for creating universal high-performance storage systems using standard hardware components. Using parallel RAID computing and unique proprietary mathematical algorithms, RAIDIX software meets the high demands of the media industry in terms of performance, fault tolerance and business continuity.

Below is a diagram of the switching components required to build a high-speed and high-available storage system for media studios.



RAIDIX supports single-controller (one node is involved) and dual-controller (Active-Active) system modes. In dual-controller mode, both nodes are active, working simultaneously and have access to a single set of disks. The nodes are hardware-independent components of the storage system, which have their own processors, cache memory, motherboard, and can be clustered.
Below is a diagram of the switching components required to build a high-speed and high-available storage system for media studios.

Recommendations

The following are recommendations for configuring storage systems for high performance and reliability.

ATTO FC Thunderbolt Setup

On the client workstation side, it is recommended to use the latest firmware and driver versions. In most cases, it is recommended to use the default settings for FC Thunderbolt. All settings are set using the ATTO ConfigTool utility, which can be downloaded from the manufacturer's website.

Discs

It is recommended to use NL SAS 7K disks. These disks have the best performance at a cost of 1 GB and provide high density and reliable storage.

Dedicated Ethernet subnet for MDC

Each Apple Mac Pro workstation ships with at least two Ethernet 1G ports built into the motherboard. One of these ports must be connected to an isolated, dedicated for each Xsan client subnet and used for the primary Xsan metadata traffic. The Apple Mac Mini, deployed as a metadata controller, has one Ethernet port. The recommended configuration is to use this port in an isolated network with other clients. This configuration excludes access to the Mac Mini outside the isolated subnet.

One volume (LUN) in one RAID array

In a configuration where sequential throughput is important, it is desirable to minimize the number of disk searches. In this regard, it is recommended to create one volume (LUN) for the entire size of the RAID array. In addition, the Xsan volume should be created from block devices of a single storage system.

RAID Configuration

High speed sequential access and optimal use of disk space is achieved through parallel computing and unique mathematical algorithms. In addition, the solution uses a proprietary RAID-array 7.3i with triple parity distribution. Under checksums, the capacity of three disks is allocated, which results in a higher degree of reliability.

To ensure high throughput, it is necessary to create four initialized RAID 7.3i arrays of 18 disks each (2 arrays — disks on storage nodes; 2 other arrays — JBOD disks). For each RAID array, you should allocate a cache of at least 12 GB of RAM. The remaining 8 disks are recommended to be evenly distributed in the spare pool for 2 disks for each RAID array.

For the Xsan metadata volume, it is recommended to create a separate RAID 10 of two 2.5 ”disks directly connected to the motherboard.

Business results

Xsan allows you to use all the capabilities of a SAN network and configure a data storage server by visualizing the complex network topology in a simple graphical interface. Using this cluster file system, a user can control access to important files, set quotas for each user, set up alerts, consolidate data, scale infrastructure on the fly.

Using Xsan cluster functions, a RAIDIX-based solution allows for maximum channel width in each specific hardware configuration, support for multiple 2K / 4K streams, fast data processing from several Apple Mac Pro workstations, and fault tolerance even if up to 3 drives fail in an array.

Thus, the media customer has the opportunity to reduce the post-production time (in practice by 2–4 times depending on the project), as well as to provide an average of more than 300 days of continuous recording online (before prevention), support for multi-camera video editing and transparent monitoring system.