Fiber Channel Basics
I continue to broadcast on clarifying the basic ideas about FC SAN. In the comments to the first post I was reproached with the fact that I didn’t dig deep enough. In particular, he said little about FC itself and said nothing about BB credits, IP and multipathing. Multipathing and IP are topics for individual publications, but I’ll continue to talk about FC. Or I will begin how to look.
For a start, a small terminological digression (inspired by a comment to the previous post).
')
By analogy with the OSI network model, Fiber Channel consists of five levels. Each level provides a specific set of functions.
FC-0 - the level of physical interfaces and media. Describes the physical environment: cables, connectors, HBA, transceivers, electrical and optical parameters.
FC-1 Transmission and Encoding Level. It describes how the data will be encoded before transmission and decoded after. At this level, three main functions are defined:
FC-2 - the level of framing and signals. Defines the structure and organization of the transmitted information, as well as control and management of its transmission. Functions performed at this level:
FC-3 - basic services level. The level is laid for new features that may be embedded in the Fiber Channel in the future. At this level, data is encrypted and compressed before it is sent, as well as such things as splitting the data stream (striping) along several paths. But I have not encountered such implementations yet.
FC-4 - protocol display level. Describes the protocols that can use FC as a transport and, in fact, the order of use (mapping of these protocols to the lower levels of FC0-3). For SAN, these can be:
And now more about these and other incomprehensible phrases. In this article, we will consider only the lower three levels, as most significant in the creation and management of the FC SAN infrastructure.
I probably will not give complex tables of cable types, transmitters and their characteristics. Firstly, because it’s inconvenient to insert tables here, secondly, because these tables are everywhere where at least something is written about FC ( Russian Wikipedia , non-Russian Wikipedia ), thirdly (and key), in my opinion , the main thing is to understand the essence, and to find reference data is not a problem.
And the point is that there are two types of fiber: multimode and single mode.
Multimode (Multimode Fiber, MMF) - relatively wide in cross section (50-62.5 microns), designed for short-wave laser beams. "Multimode" means that the light can pass through the channel in different ways - being reflected multiplely from the fiber walls. This makes the cable less sensitive to bending, but reduces the strength and quality of the signal, which limits this type only to small distances - up to 500 m.
Singlemode (Singlemode Fiber, SMF) - a fiber of small diameter (8-10 microns), the signal by which is transmitted by a long-wave laser, the light of which is not visible to the human eye. Here, the light can move in the only way - in a straight line, respectively, the signal is transmitted faster and more accurately, but the equipment to provide such signals is much more expensive, so it is used mainly for communication over long distances (up to 50 km). To bends and in general any distortions single-mode fiber is much more sensitive.
There is a more detailed article about fiber types here .
It should be borne in mind that two cables are used to connect two devices. One is used for transmission, the other is for reception. Therefore it is important to connect them correctly (Tx of one side to Rx of the other).
Separately, I want to mention such a term as dark optics ( dark fiber ). This term does not mean that it is somehow tinted in a special way. These are simply dedicated optical communication lines, as a rule, for communication over long distances (between cities or distant buildings) that are rented and for which use no additional signal amplification equipment (provided by the owner) is required. However, since this is just an optical cable, placed at your complete disposal, as long as you do not let your light signal through it, it remains "dark."
A smooth transition from FC-0 to FC-1 and back provides ASIC - an element of devices such as HBAs, disk arrays and switches.
From Wikipedia :
In Fiber Channel equipment, the ASIC consists of the following functional elements:
ASIC
Transceivers, transceivers or SFPs - in the case of FC switches, these are separate modules needed to connect the cable to the port. They differ in short wave (Short Wave, SW, SX) and long wave (Long Wave, LW, LX). LW transceivers are compatible with multimode and single mode fiber. SW-transceivers - only with multimode. And to those and other cable is connected to the connector LC.
There is also SFP xWDM (Wavelenght Division Multiplexing), designed to transmit data from several sources over long distances in a single light beam. To connect the cable to them, use the SC connector.
The first thing that happens at this level is encoding / decoding information. This is a rather tricky process, during which every 8 bits of incoming information is converted into a 10-bit representation. This is done in order to improve data integrity control, separation of data from service signals and the possibility of recovering the clock signal from the data stream (maintaining a balance of zeros and ones).
This leads to a noticeable reduction in useful bandwidth, because as you can calculate, 20% of the data flow is redundant overhead information. But among other things, a considerable part of this stream can be occupied by service traffic.
However, the good news is that 8/10 coding is used in 1G, 2G, 4G and 8G equipment. In terms of 10G implementations and starting from 16G, coding is carried out on the 64/66 principle, which significantly increases the payload (up to 97% versus 80% in the case of 8/10).
In Russian Wikipedia, this term is translated as “ ordered sets ”. While in my opinion, the word order here should be understood not in the meaning of “order”, but in the meaning of “order, command”.
For a start, it is worth mentioning another term used in the context of FC - transmission word - the minimum portion of data for transmission, equal to 4 bytes. If the transmitted information is smaller in volume, then the transmission word is supplemented with special fill bytes (fill bytes), which are cut out at the receiver.
So, ordered sets are special service transmission words. Divided into three categories:
When a physical connection is established between ports A and B, the following "metabolism" occurs between them:
All data transmitted in the Fiber Channel environment is divided into frames (frames). The frame structure is as follows:
The intervals between the frames are filled with special “filling words” - fill word. As a rule, this is IDLE, although starting with FC 8G, the use of ARB (FF) instead of IDLE has been standardized in order to reduce electrical interference in copper equipment (but by default, switches use IDLE by switches).
Most often, the source tends to transmit much more information to the receiver than 2,112 bytes (the maximum amount of data per frame). In this case, the information is divided into several frames, and the set of these frames is called a sequence. So that something extra is not wedged into the logical sequence of frames during parallel transmission, the header of each frame has the fields SEQ_ID (sequence identifier) and SEQ_CNT (frame number in the sequence).
One or several sequences responsible for some single operation is called exchange. The source and receiver can have several parallel exchanges, but each exchange per unit of time can contain only one sequence. Example exchange: the initiator requests data (sequence 1), the target returns data to the initiator (sequence 2) and then reports the status (sequence 3). Any extraneous frame set cannot insert into this sequence set.
To control this process, the header of each frame includes the OX_ID (Originator Exchange ID — filled by the initiator of the exchange) and RX_ID fields (Responder Exchange ID — filled by the recipient in the response frames, by copying the OX_ID value).
Different applications have different requirements for service level, delivery guarantee, connection duration and bandwidth. Some applications require guaranteed bandwidth during their work (backup). Others have variable activity and do not require constant guaranteed bandwidth, but they need confirmation in receiving each sent packet. To meet these needs and provide flexibility, FC defines the following 6 classes of service.
For this class, a dedicated connection is established that reserves the maximum bandwidth between the two devices. Requires confirmation of receipt. Requires frames to be sent to the receiver in the same order as they came from the source. Due to the fact that it does not allow other devices to use the transmission medium, it is used extremely rarely.
Without a permanent connection, but with delivery confirmation. It does not require matching the order of sent and delivered frames, so that they can pass through the factory in different ways. Less demanding on resources than class 1, but proof of delivery leads to increased bandwidth utilization.
Without a permanent connection and without confirmation of delivery. The most optimal from the point of view of using the resources of the factory, but assumes that the protocols of the upper levels will be able to collect the frames in the necessary order and re-request the transfer of the missing frames. Most commonly used.
It requires constant connection, confirmation and order of frames as well as class 1. The main difference is that it does not reserve all bandwidth, but only part of it. This ensures certain QoS. Suitable for multimedia and enterprise applications requiring guaranteed connection quality.
Not yet fully described and not included in the standard. Previously, a class that does not require a connection, but requires immediate delivery of data as it appears, without buffering on devices.
Option class 1, but multicast. That is, from one port to several sources.
Class F is defined in the FC-SW standard for use in inter-switch connections (Interswitch Link, ISL). This is a service without a permanent connection with delivery failure notifications used to monitor, manage, and configure the factory. The principle is similar to class 2, but it is used for interaction between N-ports (ports of nodes), and class F - for communication of E-ports (inter-switch).
In order to prevent a situation where the sender overloads the recipient with an excessive number of frames so that they begin to be discarded by the recipient, FC uses flow control mechanisms. There are two of them - Buffer-to-Buffer flow control and End-to-End flow control. Their use is governed by the class of service. For example, class 1 uses only the End-to-End mechanism, class 3 uses Buffer-to-Buffer, and class 2 uses both of these mechanisms.
The principle of technology - aloan to every house. Sending any frame must be secured by the availability of a credit for sending.
All frames arriving at the port entrance are placed in a special queue - buffers. The number of these buffers is determined by the physical characteristics of the port. One buffer (place in the queue) corresponds to one loan. Each port has two credit counters:
TX BB_Credit - transfer credit counter. After sending each frame, it decreases by 1. If the value of the counter becomes zero, transmission is impossible. As soon as R_RDY is received from the receiving port, the counter is incremented by 1.
RX BB_Credit - credit reception counter. As soon as the frame is received and placed in the buffer, it is reduced by 1. When the frame is processed or forwarded, the counter is incremented by 1, and R_RDY is sent to the sender. If the value of the counter drops to 0, then in principle, the reception of new frames should be terminated. In practice, due to credit synchronization errors, a situation may arise that the source sent another one or a few frames already after RX BB_credit became zero. This situation is called buffer overflow . In most implementations, the port handles such situations "kindly" - due to backup buffers. Although some equipment in such cases, it may take Link Reset.
From here comes the strong effect of the distance between ports on performance. The higher the distance and the greater the bandwidth, the more frames will be sent (read transfer credits spent) even before the recipient receives at least the first one. The situation is facilitated by the feature of the FC switch architecture. The fact is that the number of buffers is not fixed strictly for each port (except eight mandatory), but is common for all. And in the case of the definition of “distant links” (automatically or manually), the number of buffers allocated by the switch for this port increases. Another plus of shared memory is that you do not need to drive buffers from one port to another inside the switch.
It is implemented by the EE_Credit counter, which defines the maximum frames that the source can send to the receiver without receiving confirmation (Acknowledge, ACK). Unlike BB_Credit, it applies only to frames with data, and the exchange / accounting occurs between the end nodes.
Initially it seemed to me that the article would be two times smaller, but in the course of writing many details surfaced, without which the happiness did not seem complete. A lot of things that I would like to highlight had to be dropped so far - the writing process threatened to become endless. If someone has any comments, suggestions and suggestions for what else it is worth writing, I will be grateful. And thanks to everyone who read to this place.
Were used materials from the following sources:
IBM Redbook "Introduction to SAN and System Networking"
EMC "Network Storage Concepts and Protocols"
Brocade "SAN Fabric Administration Best Practices Guide"
For a start, a small terminological digression (inspired by a comment to the previous post).
Fiber or Fiber ?: Initially, Fiber Channel technology assumed support only for fiber-optic lines (fiber optic). However, when copper support was added, it was decided to keep the name in principle, but to refer to the standard use the British word Fiber. The US Fiber is preserved primarily for fiber optic.IBM Redbook "Introduction to SAN and System Networking"OriginalFiber Channel was originally designed to support fiber optic cabling only. In the case of the use of the English spelling (Fibre) The US English spelling (Fiber) is retained when referring generically to fiber optics and cabling.
')
Start
By analogy with the OSI network model, Fiber Channel consists of five levels. Each level provides a specific set of functions.
FC-0 - the level of physical interfaces and media. Describes the physical environment: cables, connectors, HBA, transceivers, electrical and optical parameters.
FC-1 Transmission and Encoding Level. It describes how the data will be encoded before transmission and decoded after. At this level, three main functions are defined:
FC-2 - the level of framing and signals. Defines the structure and organization of the transmitted information, as well as control and management of its transmission. Functions performed at this level:
- Crop (frame / frame structure definition) .
- Sequence management
- Exchange Management
- Class of Service
- Flow control
FC-3 - basic services level. The level is laid for new features that may be embedded in the Fiber Channel in the future. At this level, data is encrypted and compressed before it is sent, as well as such things as splitting the data stream (striping) along several paths. But I have not encountered such implementations yet.
FC-4 - protocol display level. Describes the protocols that can use FC as a transport and, in fact, the order of use (mapping of these protocols to the lower levels of FC0-3). For SAN, these can be:
- Fiber Channel Protocol for SCSI-3 (SCSI-FCP) - SCSI forwarding
- Fiber Channel Link Encapsulation (FC-LE) - TCP / IP forwarding
And now more about these and other incomprehensible phrases. In this article, we will consider only the lower three levels, as most significant in the creation and management of the FC SAN infrastructure.
FC-0
I probably will not give complex tables of cable types, transmitters and their characteristics. Firstly, because it’s inconvenient to insert tables here, secondly, because these tables are everywhere where at least something is written about FC ( Russian Wikipedia , non-Russian Wikipedia ), thirdly (and key), in my opinion , the main thing is to understand the essence, and to find reference data is not a problem.
And the point is that there are two types of fiber: multimode and single mode.
Multimode (Multimode Fiber, MMF) - relatively wide in cross section (50-62.5 microns), designed for short-wave laser beams. "Multimode" means that the light can pass through the channel in different ways - being reflected multiplely from the fiber walls. This makes the cable less sensitive to bending, but reduces the strength and quality of the signal, which limits this type only to small distances - up to 500 m.
Singlemode (Singlemode Fiber, SMF) - a fiber of small diameter (8-10 microns), the signal by which is transmitted by a long-wave laser, the light of which is not visible to the human eye. Here, the light can move in the only way - in a straight line, respectively, the signal is transmitted faster and more accurately, but the equipment to provide such signals is much more expensive, so it is used mainly for communication over long distances (up to 50 km). To bends and in general any distortions single-mode fiber is much more sensitive.
There is a more detailed article about fiber types here .
It should be borne in mind that two cables are used to connect two devices. One is used for transmission, the other is for reception. Therefore it is important to connect them correctly (Tx of one side to Rx of the other).
Separately, I want to mention such a term as dark optics ( dark fiber ). This term does not mean that it is somehow tinted in a special way. These are simply dedicated optical communication lines, as a rule, for communication over long distances (between cities or distant buildings) that are rented and for which use no additional signal amplification equipment (provided by the owner) is required. However, since this is just an optical cable, placed at your complete disposal, as long as you do not let your light signal through it, it remains "dark."
A smooth transition from FC-0 to FC-1 and back provides ASIC - an element of devices such as HBAs, disk arrays and switches.
From Wikipedia :
ASIC (an abbreviation of the English. Application-specific integrated circuit , "special purpose integrated circuit") - an integrated circuit that is specialized for solving specific problems. Unlike general-purpose integrated circuits, specialized integrated circuits are used in a specific device and perform strictly limited functions that are characteristic only for this device; as a result, functions are performed faster and, ultimately, cheaper. An example of an ASIC may be a chip designed exclusively for controlling a mobile phone, a chip for hardware encoding / decoding audio and video signals (signal processors).
In Fiber Channel equipment, the ASIC consists of the following functional elements:
- Encoder / Decoder - Encodes every 8 bits of transmitted data into a 10-bit representation. And decoding back the received data.
- SERDES (Serializer / Deserializer) - converts a parallel stream of 10-bit chunks of data into a serial stream of 10-bit chunks of data.
- Transceiver - converts electrical pulses into light signals.
ASIC
Transceivers, transceivers or SFPs - in the case of FC switches, these are separate modules needed to connect the cable to the port. They differ in short wave (Short Wave, SW, SX) and long wave (Long Wave, LW, LX). LW transceivers are compatible with multimode and single mode fiber. SW-transceivers - only with multimode. And to those and other cable is connected to the connector LC.
There is also SFP xWDM (Wavelenght Division Multiplexing), designed to transmit data from several sources over long distances in a single light beam. To connect the cable to them, use the SC connector.
FC-1
8/10 and 64/66
The first thing that happens at this level is encoding / decoding information. This is a rather tricky process, during which every 8 bits of incoming information is converted into a 10-bit representation. This is done in order to improve data integrity control, separation of data from service signals and the possibility of recovering the clock signal from the data stream (maintaining a balance of zeros and ones).
This leads to a noticeable reduction in useful bandwidth, because as you can calculate, 20% of the data flow is redundant overhead information. But among other things, a considerable part of this stream can be occupied by service traffic.
However, the good news is that 8/10 coding is used in 1G, 2G, 4G and 8G equipment. In terms of 10G implementations and starting from 16G, coding is carried out on the 64/66 principle, which significantly increases the payload (up to 97% versus 80% in the case of 8/10).
Ordered sets
In Russian Wikipedia, this term is translated as “ ordered sets ”. While in my opinion, the word order here should be understood not in the meaning of “order”, but in the meaning of “order, command”.
For a start, it is worth mentioning another term used in the context of FC - transmission word - the minimum portion of data for transmission, equal to 4 bytes. If the transmitted information is smaller in volume, then the transmission word is supplemented with special fill bytes (fill bytes), which are cut out at the receiver.
So, ordered sets are special service transmission words. Divided into three categories:
- Frame separators (Start-of-Frame, SOF and End-of-Frame, EOF).
- Two basic signals - IDLE (the port is ready to receive or transmit data) and R_RDY (receiver ready - the port freed the buffer to receive the next portion of data)
- Basic sequences (primitive sequences):
- NOS (Not Operational) - port has detected a disconnect / no connection
- OLS (Offline State) - the port initiates connection establishment, or the port received NOS, or the port switches to off-line state
- LR (Link Reset) - initialization of the connection reset. Dispatched in case of receiving OLS or some receive / transmit errors (usually at the Flow Control level). The sending port clears its buffers and their counters.
- LRR (Link Reset Response) - the answer to the LR. The sending port clears its buffers and their counters.
Link initialization
When a physical connection is established between ports A and B, the following "metabolism" occurs between them:
FC-2
Frames (Frames)
All data transmitted in the Fiber Channel environment is divided into frames (frames). The frame structure is as follows:
- SoF - 4 bytes (1 tw) - frame start identifier.
- Header - 24 bytes (6 tw) - header. It contains such information as the source and destination address, frame type (FT-0 - control or FT-1 - data), sequence number and frame sequence number in it, and other service and control information.
- Data - 0-2112 bytes (0-528 tw) - the data itself (for example, SCSI commands).
- CRC - 4 bytes (1 tw) - checksum.
- EoF - 4 bytes (1 tw) - identifier of the end of the frame.
The intervals between the frames are filled with special “filling words” - fill word. As a rule, this is IDLE, although starting with FC 8G, the use of ARB (FF) instead of IDLE has been standardized in order to reduce electrical interference in copper equipment (but by default, switches use IDLE by switches).
Sequences (Sequences)
Most often, the source tends to transmit much more information to the receiver than 2,112 bytes (the maximum amount of data per frame). In this case, the information is divided into several frames, and the set of these frames is called a sequence. So that something extra is not wedged into the logical sequence of frames during parallel transmission, the header of each frame has the fields SEQ_ID (sequence identifier) and SEQ_CNT (frame number in the sequence).
Exchange (Exchange)
One or several sequences responsible for some single operation is called exchange. The source and receiver can have several parallel exchanges, but each exchange per unit of time can contain only one sequence. Example exchange: the initiator requests data (sequence 1), the target returns data to the initiator (sequence 2) and then reports the status (sequence 3). Any extraneous frame set cannot insert into this sequence set.
To control this process, the header of each frame includes the OX_ID (Originator Exchange ID — filled by the initiator of the exchange) and RX_ID fields (Responder Exchange ID — filled by the recipient in the response frames, by copying the OX_ID value).
Classes of Services
Different applications have different requirements for service level, delivery guarantee, connection duration and bandwidth. Some applications require guaranteed bandwidth during their work (backup). Others have variable activity and do not require constant guaranteed bandwidth, but they need confirmation in receiving each sent packet. To meet these needs and provide flexibility, FC defines the following 6 classes of service.
Class 1
For this class, a dedicated connection is established that reserves the maximum bandwidth between the two devices. Requires confirmation of receipt. Requires frames to be sent to the receiver in the same order as they came from the source. Due to the fact that it does not allow other devices to use the transmission medium, it is used extremely rarely.
Class 2
Without a permanent connection, but with delivery confirmation. It does not require matching the order of sent and delivered frames, so that they can pass through the factory in different ways. Less demanding on resources than class 1, but proof of delivery leads to increased bandwidth utilization.
Class 3
Without a permanent connection and without confirmation of delivery. The most optimal from the point of view of using the resources of the factory, but assumes that the protocols of the upper levels will be able to collect the frames in the necessary order and re-request the transfer of the missing frames. Most commonly used.
Class 4
It requires constant connection, confirmation and order of frames as well as class 1. The main difference is that it does not reserve all bandwidth, but only part of it. This ensures certain QoS. Suitable for multimedia and enterprise applications requiring guaranteed connection quality.
Class 5
Not yet fully described and not included in the standard. Previously, a class that does not require a connection, but requires immediate delivery of data as it appears, without buffering on devices.
Class 6
Option class 1, but multicast. That is, from one port to several sources.
Class f
Class F is defined in the FC-SW standard for use in inter-switch connections (Interswitch Link, ISL). This is a service without a permanent connection with delivery failure notifications used to monitor, manage, and configure the factory. The principle is similar to class 2, but it is used for interaction between N-ports (ports of nodes), and class F - for communication of E-ports (inter-switch).
Flow control
In order to prevent a situation where the sender overloads the recipient with an excessive number of frames so that they begin to be discarded by the recipient, FC uses flow control mechanisms. There are two of them - Buffer-to-Buffer flow control and End-to-End flow control. Their use is governed by the class of service. For example, class 1 uses only the End-to-End mechanism, class 3 uses Buffer-to-Buffer, and class 2 uses both of these mechanisms.
Buffer-to-Buffer flow control
The principle of technology - a
All frames arriving at the port entrance are placed in a special queue - buffers. The number of these buffers is determined by the physical characteristics of the port. One buffer (place in the queue) corresponds to one loan. Each port has two credit counters:
TX BB_Credit - transfer credit counter. After sending each frame, it decreases by 1. If the value of the counter becomes zero, transmission is impossible. As soon as R_RDY is received from the receiving port, the counter is incremented by 1.
RX BB_Credit - credit reception counter. As soon as the frame is received and placed in the buffer, it is reduced by 1. When the frame is processed or forwarded, the counter is incremented by 1, and R_RDY is sent to the sender. If the value of the counter drops to 0, then in principle, the reception of new frames should be terminated. In practice, due to credit synchronization errors, a situation may arise that the source sent another one or a few frames already after RX BB_credit became zero. This situation is called buffer overflow . In most implementations, the port handles such situations "kindly" - due to backup buffers. Although some equipment in such cases, it may take Link Reset.
From here comes the strong effect of the distance between ports on performance. The higher the distance and the greater the bandwidth, the more frames will be sent (read transfer credits spent) even before the recipient receives at least the first one. The situation is facilitated by the feature of the FC switch architecture. The fact is that the number of buffers is not fixed strictly for each port (except eight mandatory), but is common for all. And in the case of the definition of “distant links” (automatically or manually), the number of buffers allocated by the switch for this port increases. Another plus of shared memory is that you do not need to drive buffers from one port to another inside the switch.
End-to-End flow control
It is implemented by the EE_Credit counter, which defines the maximum frames that the source can send to the receiver without receiving confirmation (Acknowledge, ACK). Unlike BB_Credit, it applies only to frames with data, and the exchange / accounting occurs between the end nodes.
the end
Initially it seemed to me that the article would be two times smaller, but in the course of writing many details surfaced, without which the happiness did not seem complete. A lot of things that I would like to highlight had to be dropped so far - the writing process threatened to become endless. If someone has any comments, suggestions and suggestions for what else it is worth writing, I will be grateful. And thanks to everyone who read to this place.
Were used materials from the following sources:
IBM Redbook "Introduction to SAN and System Networking"
EMC "Network Storage Concepts and Protocols"
Brocade "SAN Fabric Administration Best Practices Guide"
Source: https://habr.com/ru/post/216369/
All Articles