About storage networks

I decided to write a small article about storage area networks (DSS), this topic is quite interesting, but for some reason it was not disclosed on Habré. I will try to share my personal experience in building and maintaining SAN.

What is it?
A storage area network , or Storage Area Network, is a system consisting of the actual data storage devices — disk or RAID arrays, tape libraries and other data transfer media, and servers connected to it. It is usually used by rather large companies with a developed IT infrastructure for reliable data storage and high-speed access to them.
Simplified, storage is a system that allows servers to distribute reliable, fast, variable-capacity disks from different storage devices to servers.


')
A bit of theory.
The server can be connected to the data storage in several ways.
The first and easiest - DAS, Direct Attached Storage (direct connection), simply put the disks in the server, or the array in the server adapter - and we get a lot of gigabytes of disk space with relatively fast access, and when using a RAID array - sufficient reliability, although reliability spears have been broken for a long time.
However, this use of disk space is not optimal - on one server the place is running out, on the other there is a lot of it. The solution to this problem is NAS, Network Attached Storage (network-attached storage). However, with all the advantages of this solution - flexibility and centralized management - there is one significant drawback - the access speed, not all organizations have a 10 gigabit network implemented. And we come to network storage.

The main difference between SAN and NAS (in addition to the order of letters in abbreviations) is how the connected resources are seen on the server. If the NAS resources are connected to the NFS or SMB protocols , we get a disk connection to the SAN, which we can work with at the level of block I / O operations, which is much faster than the network connection (plus an array controller with a large cache adds speed for many operations).

Using SAN, we combine the advantages of DAS - speed and simplicity, and NAS - flexibility and controllability. Plus we get the ability to scale storage systems as long as there is enough money, in parallel killing several more birds with one stone, which are not immediately visible:

* remove restrictions on the distance to connect SCSI devices, which are usually limited to 12 meters of wire,
* we reduce backup time,
* we can boot from SAN,
* in case of failure of the NAS unload the network,
* we get more I / O speed by optimizing the storage system,
* we get the ability to connect multiple servers to one resource, then we are given the following two birds with one stone:
o make full use of VMWare features - for example, VMotion (virtual machine migration between physical) and others like them,
o we can build fault-tolerant clusters and organize geographically distributed networks.

What does this give?
In addition to mastering the budget for optimizing the storage system, we get, in addition to what I wrote above:

* increased performance, load balancing and high availability of storage systems through multiple access paths to arrays;
* savings on disks by optimizing the location of information;
* accelerated recovery after failures - you can create temporary resources, deploy backup to them and connect servers to them, and recover information without haste, or transfer resources to other servers and calmly deal with the dead iron;
* decrease backup time - due to the high transfer speed, you can back up to the tape library faster, or even take a snapshot (snapshot) from the file system and quietly archive it;
* disk space on demand - when we need - you can always add a pair of shelves to the storage system.
* we reduce the cost of storing a megabyte of information - naturally, there is a certain threshold from which these systems are profitable.
* a reliable place to store mission critical and business critical data (without which the organization can not exist and operate normally).
* Separately, I want to mention VMWare - completely all the features like virtual machine migration from server to server and other goodies are available only on the SAN.

What is it made of?
As I wrote above - the storage system consists of storage devices, transmission media and connected servers. Consider in order:


Data storage systems usually consist of hard disks and controllers, a self-respecting system usually has only 2 - 2 controllers, 2 paths to each disk, 2 interfaces, 2 power supplies, and 2 administrators . Of the most respected manufacturers of systems, HP, IBM, EMC and Hitachi should be mentioned. Here I will quote one EMC representative at the seminar - “HP makes excellent printers. Let her make them! ”I suspect that HP also loves EMC very much. Competition between manufacturers is serious, however, as elsewhere. The effects of competition are sometimes imputed prices per megabyte of storage and compatibility problems and support for competitor standards, especially for older equipment.


Data transmission medium . Usually, SAN is built on optics, which currently gives a speed of 4, in some places 8 gigabits per channel. When building previously used specialized hubs, now more switches, mainly from Qlogic, Brocade, McData and Cisco (the last two on the sites have not seen even once). Cables are used traditional for optical networks - single - mode and multimode , single-mode, more long-range.
Inside it uses FCP - Fiber Channel Protocol , a transport protocol. Typically, classic SCSI runs inside it, and FCP provides addressing and delivery. There is an option with a traditional network and iSCSI connection, but it usually uses (and heavily loads) a local network rather than a dedicated network for data transfer, and requires adapters with iSCSI support, and the speed is slower than in optics.

There is also a buzzword topology, which is found in all SAN textbooks. There are several topologies, the simplest option is point-to-point, connecting two systems together. This is not a DAS, but a spherical horse in vacuum, the simplest version of SAN. Next comes the controlled loop (FC-AL), it works on the “pass on” principle - the transmitter of each device is connected to the next receiver, the devices are closed in a ring. Long chains tend to be initialized for a long time.

Well, the final option - the switched structure (Fabric), it is created using switches. The connection structure is based on the number of ports connected, as well as when building a local network. The basic principle of construction is that all paths and connections are duplicated. This means that each device on the network has at least 2 different paths. Here, too, the word topology is used, in the sense of organizing the scheme of connecting devices and connecting switches. In this case, as a rule, the switches are configured so that the servers do not see anything except the resources assigned to them. This is achieved through the creation of virtual networks and is called zoning, the closest analogy is VLAN . Each device on the network is assigned an analogue of the MAC address on an Ethernet network, it is called WWN - World Wide Name . It is assigned to each interface and each storage resource ( LUN ). Arrays and switches are able to differentiate access to WWN for servers.


Servers are connected to storage via HBA - Host Bus Adapter . By analogy with network cards, there are one-, two-, four-port adapters. The best dog breeders recommend installing 2 adapters per server, this allows both load balancing and reliability.

And then resources are cut on storage systems, they are the same drives (LUN) for each server and space is left in the inventory, everything turns on, system installers prescribe topology, catch glitches in setting up switches and access, everything starts and everyone lives happily ever after *.
I do not specifically touch the different types of ports in the optical network, who needs it - he already knows or reads, who does not need to - just hammer his head. But as usual, if the port type is set incorrectly, nothing will work.

From experience.
Usually, when creating a SAN, arrays are ordered with several types of disks: FC for high-speed applications, and SATA or SAS for not very fast ones. This results in 2 disk groups with different megabyte cost - expensive and fast, and slow and sad cheap. Usually, all databases and other applications with active and fast I / O are hung up on the fast, file resources and everything else are slow on the slow one.

If a SAN is created from scratch, it makes sense to build it on the basis of solutions from a single manufacturer. The fact is that, despite the stated compliance with the standards, there are underwater rakes of equipment compatibility problems, and it’s not a fact that some of the equipment will work with each other without dancing with a tambourine and consulting with manufacturers. Usually, to solve such problems it is easier to call the integrator and give him money, than to communicate with the manufacturers who are turning hands on each other.

If a SAN is created on the basis of the existing infrastructure, everything can be difficult, especially if there are old SCSI arrays and a zoo of old equipment from different manufacturers. In this case, it makes sense to call on the help of the scary beast of the integrator, which will unravel compatibility problems and make a third villa in the Canaries.

Often when creating a storage system, firms do not order the support of the system by the manufacturer. This is usually justified if the company has a staff of competent competent administrators (who have called me a teapot 100 times) and a fair amount of capital, which allows you to purchase spare parts in necessary quantities. However, competent administrators are usually lured away by integrators (he saw it himself), and no money is allocated for the purchase, and after failures, the circus begins with shouts of “Let everyone go!” Instead of calling for support and the arrival of an engineer with a spare part.

Support usually comes down to replacing dead drives and controllers, and also to adding disk shelves and new servers to the system. There is a lot of trouble after the sudden prevention of the system by local specialists, especially after a complete shutdown and disassembly-assembly of the system (and this happens).

About VMWare. As far as I know (virtualization specialists correct me), only VMWare and Hyper-V have the functionality allowing to transfer virtual machines between physical servers “on the fly”. And for its implementation it is required that all servers between which the virtual machine moves are connected to one disk.

About clusters. Similar to the case of VMWare, the fault-tolerant cluster building systems known to me (Sun Cluster, Veritas Cluster Server) require storage connected to all systems.

While I was writing an article, they asked me which RAIDs do drives usually have?
In my practice, it was usually done on either RAID 1 + 0 for each disk shelf with FC disks, leaving 1 spare disk (Hot Spare) and slicing LUNs from this piece for tasks, or making RAID5 from slow disks, again leaving 1 disk to replace. But here the question is difficult, and usually the way of organizing disks in an array is chosen for each situation and is justified. The same EMC, for example, goes even further, and they have additional array configuration for applications that work with it (for example, under OLTP, OLAP). With the rest of the vendors, I did not dig so deeply, but I guess that everyone has a fine tuning.

* before the first serious failure, after it is usually bought support from the manufacturer or supplier of the system.
Since there are no comments in the sandbox, I’ll upload to a personal blog.