Configure SCST Target on CentOS 7 using FC QLogic adapter

Prehistory

In my infrastructure, among other things, there are three ESXi 5.5 servers connected to the HP MSA1000 SAN (a piece of hardware already around 10 years old) on optics through an HP StorageWorks 2 / 8q switch, using QLogic QLA200 adapters. Of course, the storage is very productive, but also very modest in volume. Since virtualization is now a trend, there is an overwhelming desire to get a really large storage, also connected via block-access optics. I will not explain why the block-based - holivars on this topic began on the Noah's Ark, and this does not apply to the subject of this article. Well, as is always the case, the budget does not even allow you to buy some kind of branded solution, so it remains only to become the hero of the "crazy hands" program.

As a result of a not very long study, the target of SCST was chosen for the following reasons:

• High performance
• Stability
• Can be installed on Centos (our standard for Linux servers)
• QLogic FC Adapter Support
• VAAI support

Next, it was necessary to pick up the hardware. A 2U server with 6 baskets was found in the bins, 4 3TB disks each (the bad ones, two of them are WD Green), QLogic 2460 FC adapter ( SCST supports QLogic 22xx / 23xx / 24xx / 25xx / 26xx adapters ).
Searches for ready-made manuals on the Centos 7 + FC SCST bundle did not lead to anything, so I had to collect information in pieces for quite a long time. As a result, everything turned out and I am satisfied, ready to buy new discs. While I am naively waiting for money, I am writing this howto for the same puzzled people like me. And so let's go!

Action

We carry out the installation and basic configuration of Centos 7 in the Minimal assembly (this is my preference, you can install any assembly). The “basic setting” is different for everyone, so I will not go into details. Do not forget to update:
')

[root@localhost ~]# yum update -y [root@localhost ~]# reboot 

Find out the name of our FC adapter, as well as its RISC ID:

 [root@localhost ~]# dmesg | grep -E 'qla2xxx .* QLogic | Found an ISP' 

We get something like this:

 [ 1.881084] qla2xxx [0000:05:04.0]-001d: Found an ISP2422 irq 24 iobase 0xffffc90001e06000. [ 2.901460] qla2xxx [0000:05:04.0]-00fb:1: QLogic QLA2460 - PCI-X 2.0 Single Channel 4Gb Fibre Channel HBA. 

Where RISC ID is ISP2422 and model is QLogic QLA2460.

Next, you need to download and install the QConvergeConsoleCLI utility and the latest firmware for our adapter from here . Extract the .BIN firmware file, rename it according to the RISC ID (see the table below), and put it in / lib / firmware on the server. Most likely there will already be such a file, so you should simply replace the existing one.

RISC ID - Firmware
ISP 21XX - ql2100_fw.bin
ISP 22XX - ql2200_fw.bin
ISP 2300 - ql2300_fw.bin
ISP 2322 - ql2322_fw.bin
ISP 24XX - ql2400_fw.bin
ISP 25XX - ql2500_fw.bin
ISP 2031 - ql2600_fw.bin
ISP 27XX - ql2700_fw.bin

In my case - ql2400_fw.bin

We update the firmware of our adapter:

 [root@localhost ~]# /opt/QLogic_Corporation/QConvergeConsoleCLI/qaucli -fc -b all /lib/firmware/ql2400_fw.bin 

Downloading the current QLogic SCST driver:

 [root@localhost ~]# yum install git -y [root@localhost ~]# git clone git://git.QLogic.com/scst-qla2xxx.git 

Install the necessary packages for the subsequent build of the kernel, as well as svn:

 [root@localhost ~]#yum install gcc ncurses-devel kernel-devel lsscsi patch subversion bc bison net-tools -y 

Downloading SCST sources:

 [root@localhost ~]# cd / [root@localhost /]# svn co https://svn.code.sf.net/p/scst/svn/trunk scst 

In the SCST directory, make a soft link to the QLogic SCST driver:

 [root@localhost /]# cd scst [root@localhost scst]# ln -s ../scst-qla2xxx/drivers/scsi/qla2xxx qla2x00t_git 

Further, in accordance with the recommendations of the developers of SCST, you should patch the kernel.

We will assemble a new kernel in the environment of a new user:

 [root@localhost /]# useradd builder 

We grant the right to use yum and yum-builddep without a password:

 [root@localhost /]# echo 'builder ALL=(ALL) NOPASSWD: /usr/bin/yum, /usr/bin/yum-builddep' >>/etc/sudoers [root@localhost /]# visudo –sc 

Go to the new user environment:

 [root@centos-test /]# su builder 

Run the script, kindly prepared for us by the developers:

 [builder@localhost /]# ./scst/scripts/rebuild-rhel-kernel-rpm 

Install the new kernel:

 [builder@localhost /]$ su -c 'rpm -ivh --force /home/builder/rpmbuild/RPMS/x86_64/kernel-*.rpm' 

Reboot:

 [builder@localhost /]$ su –c reboot 

Check the current kernel:

 [root@localhost ~]# uname -r 3.10.0-229.el7.centos.scst.x86_64 

If everything is ok, go ahead. Let's unload and add to the black list the default qla2xxx kernel module:

 [root@localhost /]# echo blacklist qla2xxx >/etc/modprobe.d/blacklist-qla2xxx.conf [root@localhost /]# rmmod qla2xxx 

Set the SCST build mode to release:

 [root@localhost /]# cd /scst [root@localhost scst]# make 2release 

Let's assemble the necessary kernel modules:

 [root@localhost scst]# BUILD_2X_MODULE=y CONFIG_SCSI_QLA_FC=y CONFIG_SCSI_QLA2XXX_TARGET=y make all install 

We check that everything went well:

 [root@localhost ~]# ls -l /lib/modules/`uname -r`/extra/qla2* -rw-r--r-- 1 root root 1511833 Sep 10 11:10 /lib/modules/3.10.0-229.11.1.el7.centos.scst.x86_64/extra/qla2x00tgt.ko -rw-r--r-- 1 root root 10699423 Sep 10 11:10 /lib/modules/3.10.0-229.11.1.el7.centos.scst.x86_64/extra/qla2xxx_scst.ko 

You can check the full list of installed modules:

 [root@localhost ~]# ls -l /lib/modules/`uname -r`/extra total 25128 drwxr-xr-x 2 root root 4096 Sep 10 11:10 dev_handlers -rw-r--r-- 1 root root 4309043 Sep 10 11:10 iscsi-scst.ko -rw-r--r-- 1 root root 3100544 Sep 10 11:10 isert-scst.ko -rw-r--r-- 1 root root 1511833 Sep 10 11:10 qla2x00tgt.ko -rw-r--r-- 1 root root 10699423 Sep 10 11:10 qla2xxx_scst.ko -rw-r--r-- 1 root root 5619115 Sep 10 11:10 scst.ko -rw-r--r-- 1 root root 474173 Sep 10 11:10 scst_local.ko 

 [root@localhost ~]# ls -l /lib/modules/`uname -r`/extra/dev_handlers total 3960 -rw-r--r-- 1 root root 305582 Sep 10 11:10 scst_cdrom.ko -rw-r--r-- 1 root root 292948 Sep 10 11:10 scst_changer.ko -rw-r--r-- 1 root root 328092 Sep 10 11:10 scst_disk.ko -rw-r--r-- 1 root root 308934 Sep 10 11:10 scst_modisk.ko -rw-r--r-- 1 root root 292972 Sep 10 11:10 scst_processor.ko -rw-r--r-- 1 root root 292876 Sep 10 11:10 scst_raid.ko -rw-r--r-- 1 root root 312024 Sep 10 11:10 scst_tape.ko -rw-r--r-- 1 root root 704076 Sep 10 11:10 scst_user.ko -rw-r--r-- 1 root root 1199482 Sep 10 11:10 scst_vdisk.ko 

Add them all to the core:

 [root@localhost ~]# for m in scst qla2xxx_scst qla2x00tgt scst_vdisk scst_user scst_disk ...; do modprobe $m; done 

In dmesg, something like this should appear (I give an example from my own version):

 [10753.981044] [21513]: scst: SCST version 3.1.0-pre1 loaded successfully (max mem for commands 709MB, per device 283MB) [10753.981051] [21513]: scst: Enabled features: TRACING [10753.982152] [21519]: scst: Management thread started [10754.024379] qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.05.2.2-SCST.03-kp. [10754.030864] [21527]: qla2x00t: Initializing QLogic Fibre Channel HBA Driver target mode addon version 3.1.0-pre1 [10754.032397] [21527]: qla2x00t: Target mode driver for QLogic 2x00 controller registered successfully [10754.032409] [21527]: scst: Target template qla2x00t registered successfully [10754.038638] [21528]: scst: Virtual device handler vdisk_fileio for type 0 registered successfully [10754.038647] [21528]: scst: Virtual device handler vdisk_blockio for type 0 registered successfully [10754.038653] [21528]: scst: Virtual device handler vdisk_nullio for type 0 registered successfully [10754.038658] [21528]: scst: Virtual device handler vcdrom for type 5 registered successfully [10754.044786] [21529]: scst: Virtual device handler "scst_user" registered successfully [10754.048905] [21530]: scst_user: Cleanup thread started [10754.052682] [21531]: scst: Device handler "dev_disk" for type 0 registered successfully [10754.052691] [21531]: scst: Device handler "dev_disk_perf" for type 0 registered successfully 

Rebuild the ramdisk for initialization (initrd) so that the qla2xxx kernel module is replaced by qla2xxx_scst:

 [root@localhost ~]# mkinitrd -f /boot/initramfs-`uname -r`.img `uname -r` 

Install scstadmin:

 [root@localhost /]# cd scst [root@localhost scst]# make -C scstadmin -s install 

Next you need to create the file /etc/scst.conf, I will give an example of my own:

 HANDLER vdisk_blockio { DEVICE disk1 { filename /dev/sdb } } TARGET_DRIVER qla2x00t { TARGET ::::::: { enabled 1 LUN 0 disk1 } } 

Explanation:

HANDLER vdisk_blockio - the choice of a handler, in my case I wanted block access, if you want file access, then there will be vdisk_fileio.
Filename / dev / sdb - drive for your target
TARGET XX: XX: XX: XX: XX: XX: XX: XX - instead of X, you will enter WWN of your FC adapter
An example of a command with which you can find the WWN:

 cat /sys/class/scsi_host/host*/device/fc_host/host*/node_name 

Apply our configuration file:

 [root@localhost scst]# scstadmin -config /etc/scst.conf Collecting current configuration: done. -> Checking configuration file '/etc/scst.conf ' for errors. -> Done, 0 warnings found. -> Opening device 'disk1' using handler 'vdisk_blockio': done. -> Adding device 'disk1' at LUN 0 to driver/target 'qla2x00t/:::::::': done. -> Enabling driver/target 'qla2x00t/:::::::': done. All done. 

Well, and probably we want the whole business to be launched automatically in case of a target reload.

We include autoloading of the SCST:

 [root@localhost /]# systemctl enable scst.service 

Add kernel modules to autoload:

 [root@localhost /]# vi /etc/modules-load.d/scst_modules.conf scst_disk scst_user scst_vdisk scst libcrc32c crc_t10dif qla2x00tgt 

And finally, we will add the application of the config to rc.local:

 [root@localhost /]# chmod +x /etc/rc.d/rc.local [root@localhost /]# vi /etc/rc.d/rc.local scstadmin –config /etc/scst.conf 

If everything is done correctly, then if after restarting the server run:

 [root@localhost ~]# scstadmin -config /etc/scst.conf 

The following will be released:

 Collecting current configuration: done. -> Checking configuration file '/etc/scst.conf' for errors. -> Done, 0 warnings found. -> Applying configuration. -> Done, 0 change(s) made. All done. 

This completes the target setting. Below are a couple of useful notes.

Important! Do not forget to set up Zoning on your FC switch, if it is used, otherwise the initiators will not see the new target.

Also in the case of ESXi, after setting up the target, as well as in the case of its reboot, all initiating hosts must be rebooted, otherwise the drive will not pick up.

Still such a little performance note. On the Centos 7 virtual machine with fio, I measured 84 iops and avg latency 12 msec to write at iodepth = 1. Which is actually not bad for WD Green.

Update 10/5/2016! I was going to raise the target for my own article, but with the kernel patch, the error occurred: No matching put_page_callback patch found for kernel version.
The development of SCST saved us from having to patch the kernel in the latest version 3.3. So, the section on rebuilding the kernel can be skipped, instead, we simply take the following steps:
After [root @ localhost scst] # ln -s ../scst-qla2xxx/drivers/scsi/qla2xxx qla2x00t_git:
[root @ localhost scst] yum isntall -y bzip2 rpm-build
[root @ localhost scst] make rpm
In the process, an error is likely to come out, which is related to the lack of certain Perl modules, it will be necessary to google it, and right away there will be a solution in the first pages, I just made and forgot to write it down - I confess.
After that, install the RPMs and continue from this point:
[root @ localhost /] # echo blacklist qla2xxx> /etc/modprobe.d/blacklist-qla2xxx.conf