Using Linux on the SAN network. Little trick
Good day to all.
I do not think that the situation described by me is typical for the majority, but nevertheless I think it will be useful to know. My story about how to protect "from a fool" in the case when you have a SAN network, there are several storage systems (Storage System) that do not support technology to restrict access to certain LUN (Logical Unit Number) and servers based on Linux (in our In this case, all actions will be performed using SLES OS 10SP1).
In 2008 I had to commission the computing infrastructure of a single customer. A small SAN network was built with some storages (about 10 pieces of different classes). Most of these storads were designed to build the file system of a computing cluster, so the problem was not obvious at first and didn’t even think about it.
And the problem is that some infrastructure subsystems have been allocated a stack that does not know how to restrict access to certain LUNs from certain hosts (for each manufacturer it is called Port Masking, LUN MAPPING, etc.), more correctly it can do it, but you need to buy additional. license, and this was not done. And here the problem arose - all the servers that connect to this stack see all its sections. It seems to be nothing terrible in this, but it is necessary to protect the customer from the possibility of formatting someone else's section, because data of each subsystem is very important.
')
Total we have:
Immediately make a reservation for those who do not have strong knowledge in the subject:
Each device in the SAN network has a unique WWN (World Wide Name) address, it is a kind of analog of the MAC address of the network adapter, but, as in the case of Ethernet, the switch does not know what lies inside the packet coming to the switch, so make a restriction on the switch ) it is impossible, you can only limit the scope of Data Storage Systems for certain nodes, which was done.
The circle of finding a solution was limited to the following options:
He disappeared after seeing the information on this module. It contains only the option to scan the device and determines the maximum number of LUNs. So this option disappears.
This option immediately directs us to “carry out land work” in the direction of the udev subsystem. It is she who is responsible in linux for initializing devices, assigning them specialized names and other actions. Simply put, we need to correctly describe the rule for udev, which will create only the devices we need.
Looking ahead, I will say that this decision will limit only the creation of specials. devices (like / dev / sdb, / dev / sdc, etc.), but physically the host will see all the LUNs available to it.
Let's start a detailed study of the topic.
After a quick study of the documentation, as well as the rules that are in the system, you can isolate the following blocks that are interesting to us:
Short summary:
So now we have all the necessary information to write a rule.
Now we proceed directly to writing the rule. We will need udev documentation.
I give the final rule and description of options:
And now the explanation of what's what:
This option tells udev that if the subsystem works with any non-block device, then this rule does not apply.
This is the most important part, in fact, for the sake of this part, and this article was written. The rule defines the behavior: if a device gets through the BUS == "scsi" and it uses the DRIVER == "sd" driver, we also have the manufacturer SYSFS {vendor} == "XXX" and the model name SYSFS {model} == " YYYYY "and LUN of this device corresponds to ID ==" *: *: *: 20 ", then this device should follow all other rules and create specials. device in the / dev directory.
Record ID == "*: *: *: 20" is an entry in the format H: B: T: L, which describes only the LUN identifier, the rest of the fields can be any. This allows us to skip all storage devices that have a given LUN. If it is necessary to expand the list of LUNs, then records are simply added and the identifier we need is recorded in the last field. The main thing is that they should be placed before the following rule.
This rule applies the same conditions as the previous one, only OPTIONS + = “ignore_device, last_rule” tells udev to ignore all further actions with this device. Those. all devices except the specified ones will be ignored.
This rule is a simple end-of-file label. It should be the last in the list.
Now we just have to save this rule and enable it.
We save this rule in the /etc/udev/rules.d folder in the correct format. I used the name "52-ignore-unused-devices.rules". After that, something else needs to be done. If the adapter driver is in the ramdisk, then it should be removed from there and loaded only after the udev subsystem starts; otherwise, this rule will not work, because devices will be created before udev starts.
To do this, you need to make changes to the / etc / sysconfig / kernel file:
Now you can restart the server and make sure that only those devices that we need are created. The rest, although visible in the system, but not specified special. files, and therefore the user can not do anything with them. It will look like this:
As a result, our task was solved elegantly and beautifully. It will also speed up the download, especially if you use the built-in multipath mechanism.
I hope this will be useful for at least one person.
Materials that helped a lot
I do not think that the situation described by me is typical for the majority, but nevertheless I think it will be useful to know. My story about how to protect "from a fool" in the case when you have a SAN network, there are several storage systems (Storage System) that do not support technology to restrict access to certain LUN (Logical Unit Number) and servers based on Linux (in our In this case, all actions will be performed using SLES OS 10SP1).
Prehistory
In 2008 I had to commission the computing infrastructure of a single customer. A small SAN network was built with some storages (about 10 pieces of different classes). Most of these storads were designed to build the file system of a computing cluster, so the problem was not obvious at first and didn’t even think about it.
And the problem is that some infrastructure subsystems have been allocated a stack that does not know how to restrict access to certain LUNs from certain hosts (for each manufacturer it is called Port Masking, LUN MAPPING, etc.), more correctly it can do it, but you need to buy additional. license, and this was not done. And here the problem arose - all the servers that connect to this stack see all its sections. It seems to be nothing terrible in this, but it is necessary to protect the customer from the possibility of formatting someone else's section, because data of each subsystem is very important.
')
Total we have:
- LSI storage system (more precisely, one of its partners)
- Servers based on SLES10 SP1 with a two-headed FC adapter
- A great desire to insure themselves and the customer from mistakes.
Step One, Introductory
Immediately make a reservation for those who do not have strong knowledge in the subject:
Each device in the SAN network has a unique WWN (World Wide Name) address, it is a kind of analog of the MAC address of the network adapter, but, as in the case of Ethernet, the switch does not know what lies inside the packet coming to the switch, so make a restriction on the switch ) it is impossible, you can only limit the scope of Data Storage Systems for certain nodes, which was done.
The circle of finding a solution was limited to the following options:
- Driver availability to use only certain LUNs
- Use OS Tools to Block Unwanted LUNs
First option
He disappeared after seeing the information on this module. It contains only the option to scan the device and determines the maximum number of LUNs. So this option disappears.
Second option
This option immediately directs us to “carry out land work” in the direction of the udev subsystem. It is she who is responsible in linux for initializing devices, assigning them specialized names and other actions. Simply put, we need to correctly describe the rule for udev, which will create only the devices we need.
Looking ahead, I will say that this decision will limit only the creation of specials. devices (like / dev / sdb, / dev / sdc, etc.), but physically the host will see all the LUNs available to it.
Step Two, Study
Let's start a detailed study of the topic.
After a quick study of the documentation, as well as the rules that are in the system, you can isolate the following blocks that are interesting to us:
- The “SUBSYSTEM” block, which defines a subsystem. There are three types of subsystems: “block” (block devices), “char” (character devices) and “pipe” (FIFO devices).
man mknod
We are interested in the “block” subsystem, since Drives belong to this class. - udev can view device attributes. They usually store various data, such as:
- type of manufacturer
- model name
- various hardware data.
To figure out the parameters we need, we will need:
Utility lssci - displays all SCSI devices that are available in the system, as well as the name of the spec. device that is assigned to this SCSI device
Udevinfo utility - shows all the necessary data on the requested device.
Let's start:
First we need to display a list of all devices that are available on this server:# lssci
SCSI ID TYPE VENDOR MODEL REV. device
[5:0:0:10] disk XXX YYYYY 0619 /dev/sdr
[5:0:0:20] disk XXX YYYYY 0619 /dev/sds
[5:0:0:21] disk XXX YYYYY 0619 /dev/sdu
......
[5:0:1:10] disk XXX YYYYY 0619 /dev/sdan
[5:0:1:20] disk XXX YYYYY 0619 /dev/sdao
[5:0:1:21] disk XXX YYYYY 0619 /dev/sdar
The output of this command will be large, so I will give only a part.
So now we know:
SCSI ID - identifier in the format H: B: T: L (Host: Bus: Target: Lun) Host is the ordinal number of the adapter that is responsible for this device, Bus (Channel) is the SCSI channel number on the adapter, Target is the number FC port on the stack, LUN - identifier of the partition allocated on the stack.
VENDOR - the name of the manufacturer storadzh
MODEL - model name
Rev. - FirmWare version in stock
device - special user accessible device
In this operation, we were interested in the "device" and "SCSI ID" fields. The remaining fields are ignored. - The next step is to get the list of storage attributes that we need:
- driver name
- manufacturer name
- model name
For this, we will need the udevinfo utility, as well as the sysfs file system (for kernel versions 2.6), since all the information we need is stored in it.# udevinfo -a -p /sys/block/sds
looking at device '/block/sds':
KERNEL=="sds"
SUBSYSTEM=="block"
SYSFS{stat}==" 151707 2355 3422965 430068 436954 12273 18585533 1098588 0 651944 1528756"
SYSFS{size}=="2129838080"
SYSFS{removable}=="0"
SYSFS{range}=="16"
SYSFS{dev}=="65:32"
looking at device '/devices/pci0000:00/0000:00:04.0/0000:08:00.0/host5/rport-5:0-0/target5:0:0/5:0:0:20':
ID=="5:0:0:20"
BUS=="scsi"
DRIVER=="sd"
SYSFS{ioerr_cnt}=="0x7"
SYSFS{iodone_cnt}=="0xec62d"
SYSFS{iorequest_cnt}=="0xec62d"
SYSFS{iocounterbits}=="32"
SYSFS{retries}=="5"
SYSFS{timeout}=="60"
SYSFS{state}=="running"
SYSFS{rev}=="0619"
SYSFS{model}=="YYYYY "
SYSFS{vendor}=="XXX "
SYSFS{scsi_level}=="6"
SYSFS{type}=="0"
SYSFS{queue_type}=="simple"
SYSFS{queue_depth}=="64"
SYSFS{device_blocked}=="0"
....
looking at device '/devices/pci0000:00/0000:00:04.0/0000:08:00.0':
ID=="0000:08:00.0"
BUS=="pci"
DRIVER=="mptfc"
SYSFS{modalias}=="pci:v00001000d00000646sv00001000sd00001020bc0Csc04i00"
SYSFS{local_cpus}=="00000000,00000000,00000000,0000000f"
SYSFS{irq}=="169"
SYSFS{class}=="0x0c0400"
SYSFS{subsystem_device}=="0x1020"
SYSFS{subsystem_vendor}=="0x1000"
SYSFS{device}=="0x0646"
SYSFS{vendor}=="0x1000"
Request information for each device is not necessary, because The output of "lsscsi" contains duplicate entries. These are the paths to a specific LUN through different devices (for example, 1 FC port on the host and 3 Port on the storage), and since this is the same storage, the parameters we need will be the same for everyone.
Short summary:
- DRIVER == "sd" is the name of the driver that identifies this device. Why “sd”, and not the FC driver “mptfc”, because exactly how the “sd” driver detects the disk device received from the SCSI / FC adapters.
- SYSFS {vendor} == "XXX" is a storage manufacturer. If you pay attention, his name is not just "XXX", but also a certain number of spaces. It is for this reason that the output of lsscsi is not so informative, and in the case of using only “XXX” the rule simply does not work.
- SYSFS {model} == "YYYYY" is the model name. Same reason as in paragraph above.
- ID == "5: 0: 0: 20" is an identifier in the format H: B: T: L. He also needs to be remembered, because we will need it further.
- BUS == "scsi" is a type of bus. We will need to more accurately identify the device.
So now we have all the necessary information to write a rule.
Step three, the office writes!
Now we proceed directly to writing the rule. We will need udev documentation.
I give the final rule and description of options:
# If not a block device (ex. tape) ignore this rule
SUBSYSTEM!="block", GOTO="go_ignore_end"
# Phisical device section
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", ID=="*:*:*:20", GOTO="go_ignore_end"
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", ID=="*:*:*:21", GOTO="go_ignore_end"
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", OPTIONS+="ignore_device,last_rule"
LABEL="go_ignore_end"
And now the explanation of what's what:
SUBSYSTEM!="block", GOTO="go_ignore_end"This option tells udev that if the subsystem works with any non-block device, then this rule does not apply.
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", ID=="*:*:*:20", GOTO="go_ignore_end"
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", ID=="*:*:*:21", GOTO="go_ignore_end"
This is the most important part, in fact, for the sake of this part, and this article was written. The rule defines the behavior: if a device gets through the BUS == "scsi" and it uses the DRIVER == "sd" driver, we also have the manufacturer SYSFS {vendor} == "XXX" and the model name SYSFS {model} == " YYYYY "and LUN of this device corresponds to ID ==" *: *: *: 20 ", then this device should follow all other rules and create specials. device in the / dev directory.
Record ID == "*: *: *: 20" is an entry in the format H: B: T: L, which describes only the LUN identifier, the rest of the fields can be any. This allows us to skip all storage devices that have a given LUN. If it is necessary to expand the list of LUNs, then records are simply added and the identifier we need is recorded in the last field. The main thing is that they should be placed before the following rule.
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", OPTIONS+="ignore_device,last_rule"
This rule applies the same conditions as the previous one, only OPTIONS + = “ignore_device, last_rule” tells udev to ignore all further actions with this device. Those. all devices except the specified ones will be ignored.
LABEL="go_ignore_end"This rule is a simple end-of-file label. It should be the last in the list.
Step four, the last battle
Now we just have to save this rule and enable it.
We save this rule in the /etc/udev/rules.d folder in the correct format. I used the name "52-ignore-unused-devices.rules". After that, something else needs to be done. If the adapter driver is in the ramdisk, then it should be removed from there and loaded only after the udev subsystem starts; otherwise, this rule will not work, because devices will be created before udev starts.
To do this, you need to make changes to the / etc / sysconfig / kernel file:
- Delete in the variable INITRD_MODULES = "...." the device driver FC.
- Add it to the MODULES_LOADED_ON_BOOT = "..." variable
- Regenerate ramdisk with the mkinitrd command.
Now you can restart the server and make sure that only those devices that we need are created. The rest, although visible in the system, but not specified special. files, and therefore the user can not do anything with them. It will look like this:
# lssci
SCSI ID TYPE VENDOR MODEL REV. device
[5:0:0:10] disk XXX YYYYY 0619 -
[5:0:0:20] disk XXX YYYYY 0619 /dev/sds
[5:0:0:21] disk XXX YYYYY 0619 /dev/sdu
......
[5:0:1:10] disk XXX YYYYY 0619 -
[5:0:1:20] disk XXX YYYYY 0619 /dev/sdao
[5:0:1:21] disk XXX YYYYY 0619 /dev/sdar
As a result, our task was solved elegantly and beautifully. It will also speed up the download, especially if you use the built-in multipath mechanism.
I hope this will be useful for at least one person.
Materials that helped a lot
- http://reactivated.net/writing_udev_rules.html
- Existing udev rules
Source: https://habr.com/ru/post/112453/
All Articles