Another do-it-yourself NAS, part 3: the adventures of XXX in the old tower

I wonder if, as a child, he dreamed of shoving a dozen spindle disks into his home PC and imagining himself to be the brave Commander of the Server? Well, after the PCM workshops, space communications hubs, cable harnesses as thick as baobab, data centers of various stripes and other very iron-rich rooms, children's dreams finally come true.
But just sticking a dozen spindles into the “tower” is not enough, they still need to be unrolled, blown up, filled with the file system, allowed access, demarcated rights, and then still monitored for health. We will tell in some detail how to do it comfortably for ourselves. At the same time, we will not sing prayers to the next server case of the famous brand StuporMixer, but we will perform in the genre of rational minimalism with a $ 30 piece of hardware and focus on software and useful trifles. From this post, the reader will learn about where almost half the power of a conventional power supply unit is going to disappear, about the magic of telemetry, about some styles of Unix kungfu, curly differentiation of access rights without an ACL (yes, it happens!) And what the old drives in new servers. Easter eggs and the promised story XXX scattered over the post ;-)

But first, I want to tell you how a part that costs less than $ 1 can “spoil” a $ 100 disk. It's fun, I promise.

DISCLAIMER

The information is provided by AS-IS without any liability for its use by anyone, anywhere and ever. All inadvertently mentioned trademarks are the property of their respective owners. Some of them do not need advertising so much that I make them funny names.


')

HERIAL ATA: new name for old problem

So, you have just spent part of the family (or company) budget for a new storage of indecent capacity, but you didn’t have time to fade the aromas of iron, as the disc “fell down”. Do not rush to panic! If you rarely do the assembly, the seller is very likely and cynical to send you to beat up the thresholds of service centers, and the case may be in the interface cable at the cost of fifty European kopecks. Judge for yourself: out of a dozen SATA cables purchased in different places, I personally had to replace as many as three pieces (which is 30%, by the way). I don’t know if it’s a question of luck, representativeness of the sample, curvature of hands or something else, but the very first themed post of Three key strategies to prevent RAID failure only strengthened my confidence:

According to the study program failures of interconnect devices (including controllers, power supplies, fans and cabling ); and failures in the disk drives themselves. Disk drive failures followed by interconnect failures. It is not necessary to ensure that it can be used in the field of interconnect components.

Even if you do not pull phrases out of context, interface cables are just cheap consumables, so treat them like toilet paper.

If the etymology of the name XERIAL ATA is interesting, then the first letter should be read in Russian. Having identified the following extrasvariety method with the cable, I simply decided to mark it with a cross, which happened to be on the letter S. After reading the resulting word, I even decided to make a post about it.


XERIAL ATA cable

By the way: Crepsondo’s philosophical practices teach you to tag defective items right away — a habit that has saved me time and money more than once. But if you think that revenge is a dish that needs to be served cold, then store defective cables, broken disks, buggy memory and burnt power supplies in one box with a working iron without ticking. I assure you that this will be a very cold and very bitter dish. Perhaps you. And the word XER is read like Xer and can be applied to “person engaged in X”, whatever that means ...

How to detect defective cable

On regular operations, a bad SATA interface cable can be recognized in SMART by the non-zero raw UDMA_CRC_Error_Count attribute value (see RAW VALUE ). This attribute characterizes only the quality of the SATA path, regardless of the state of the magnetic surfaces. Immediately after installing a new disk (but better before initializing the file system), I recommend driving massive (order of GB) read and write operations, looking at the SMART monitor program. If the operations come with errors, and the system log is replete with messages "ATA Status Error", "Retrying command", it is possible that the expensive aroma of the new disk is not to blame, but only the cheap smell of the lace.

UPD:
If replacing the cable does not help, and the error counter is still growing, first try unloading the power supply and repeating the experiment: remove energy-intensive components, disconnect other disks, etc. If everything at once becomes good, change the power supply unit to an expensive and high-quality one, from bad power you only meet a lot of UFOs , just not from our habragalaxy, but from others. They do not land correctly and feed on your data, so do not feed them, let them fly past you.

With your permission, dear readers, I will give some examples of using the dd (1) utility in combination with smartctl (8) . In other systems, you just need to download the necessary utility (see wikislist ) and drive a small, but weighty group of files back and forth with regular copying to properly load the interface. Favorite collection of porn movies fit just right.

Attention: the first command is destructive, it should be executed before the new disk ada1 is populated with the file system!

#dd if=/dev/zero of=/dev/ada1 bs=1M count=1000 #dd if=/dev/ada1 of=/dev/null bs=1M count=1000 #smartctl -x /dev/ada1 ... ID# ATTRIBUTE_NAME FLAG VALUE RAW_VALUE ... 199 UDMA_CRC_Error_Count 0x003e 200 75 

It was noted that it is not always possible to provoke an error by reading alone, therefore an entry is added. In the example, the error counter of the disk ada1 reached 75, and for the replacement of the cable I personally have enough units.

To soothe, you can also run an internal surface test. Inside the modern disk a whole arsenal of diagnostic tools is assembled (and in some instances there is almost a meteorological station hidden away). To get to this whole engineering marvel, just open a simple Swiss knife: launching, checking the progress and obtaining test results are done with the same smartctl (8) utility, in the description you can find other test options (fast, long, selective, etc. .).

An example of running a long test for an ada1 disk, the disk responds with an approximate test end time:

 # smartctl -t long /dev/ada1 ... === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION === Sending command: "Execute SMART Extended self-test routine immediately in off-line mode". Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful. Testing has begun. Please wait 277 minutes for test to complete. Test will complete after Mon Aug 31 23:00:14 2013 Use smartctl -X to abort test. 

Such a test does not use the SATA interface, but checks, roughly speaking, only the state of "pancakes".

Check the progress and results of all ada1 disk tests :

 # smartctl -l selftest /dev/ada1 ... === START OF READ SMART DATA SECTION === SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Self-test routine in progress 90% 123 - 

Here the test passed 10% (90% left) until no errors were detected (dash in LBA_of_first_error ), and at the time of the start of the test, the disk worked for a total of 123 hours. All this is stored in non-volatile memory.

With FreeNAS, you can include such tests in an automated self-diagnostic package and collect scheduled reports (see telemetry examples below). You can make it a rule once a month (or more often) to check reports for early detection of problems. In fact, this is the job of a good Server Commander.

findings

If the problem is really connected, immediately extinguish the server, drain the oil, remove the cable and put a cross on it. Or throw it away to hell grandmother (and better all of the above). With a victorious face, reach out in advance a new cable and put it in place of the discarded one. Although the projectile beats into the same funnel extremely rarely, be sure to check the new cable. Keep your records of errors for each disk serial number: after each cable replacement, record the RAW_VALUE attribute value UDMA_CRC_Error_Count , and if someone knows a universal way to reset only this attribute, write to me in a personal, please.

The magic of telemetry

I am sure everyone will agree that the failed disk in the array should be replaced as soon as possible. If for some it is not obvious, I recommend reading feature articles, at least this one: A failed RAID drive needs immediate attention .
It seems to me that despite the full engineering consensus, the theory and practice differ most strongly in this place: the failed disks continue to sit quietly in the servers, like mice under the brooms of a time bomb. And this concerns not only small offices, which are monitored by young and inexperienced system administrators. I read somewhere the statistics of data loss in large enterprises, and one of the weighty reasons was just an oversight of arrays. The failed disks managed to sit for months in their hot-swap baskets, unnoticed by anyone. Usually it ended with the failure of the second disk, when the volume had already crumbled, and a completely different alarm in the form of an angry Business was triggered. Therefore, I propose to be wiser and tackle this problem without delay.

The most affordable telemetry

Brothers, if you use FreeNAS, turn on at least email notifications , since the Internet is almost everywhere, and now you can surprise your smartphone only with a big giraffe. Do not like smartphones (sometimes it happens) - get yourself, finally, a tablet, the 21st century in the yard. I give an example for Gmail, without port 25:

1. Create a separate system mailbox on Gmail (for free! Do not interfere with personal mail monitoring system health).
2. Set in the settings ( Email Tab ) server name: smtp.gmail.com, port 587, enable TLS mode and SMTP authentication (use the sender's address, username and password from the newly created mailbox - another reason to create a separate one).
3. Add mail collection from the newly created Gmail box on your smartphone; The Gmail Mail app allows you to quickly and conveniently switch between mailboxes.
4. Additionally, work out the unfolding of boring periodicals of regular reports on folders so that the phone “brynkal” only from emergency events.

By the way: since release 9.2, in FreeNAS daily reports have been turned off “about nothing” in order not to distract our precious attention. And I highly recommend everyone to act in the same way: the system should shout “wolves!” Only when they are really going to devour you with your data. And the first to know about the problem is not the user, but the Commander of the Server. This is his main task.

But that's not all, dear readers. The whole point is that the alias root user is now also wrapped in the Gmail mailbox that was created. Therefore, those of you who own the Unix kungfu will now be able to send simple reports by mail using the regular mail (1) command without any frills. For example, this is how I send SMART reports to Cron Jobs :

 #!/bin/bash for i in 0 1 2 3 4 5 6 7 8 9; do smartctl -x /dev/ada$i | mail -s "$HOSTNAME SMART report ada$i `date '+%Y-%m-%d'`" root done 

By the way, the smartctl in this form also delays the self-diagnostic log mentioned above, so that I can, if I wish, evaluate the state of the disks literally on the run without connecting to the server. Agree, it's worth it.

Things pozavovyresteee (like a backup copy of the settings), of course, have to put on base64 and shod in the correct MIME-headers, or at least wrap in uuencode (1) .

How SMART got into a fight with APM

FreeNAS uses the smartd (8) subsystem to monitor disk temperature and critical failures. Additional settings (SMART extra options) are defined according to the format smartd.conf (5) .
An interesting collision occurs when trying to simultaneously use SMART monitoring and the HDD STANDBY function from the disk management section. If you set the polling interval for the Check interval to be shorter than the disc “leaving” time in STANDBY, then SMART with its own polls will always reload the disk countdown timer and it will never “fall asleep”. If you really need to put the disk to sleep, set the SMART polling interval longer than the STANDBY timeout, and the smartd (8) is already trained in the delicate handling of sleeping disks. Professionals, however, do not recommend extinguishing discs, because they start aging on spindle cycles faster than regular sex life of a permanent job.

Memory and processes

FreeNAS version 9 uses collectd to collect and display historical statistics in its graphical interface. You can see, for example, the consumption of RAM per month. As I already wrote in the previous part , statistics accumulates on the RAM disk and once per hour is stored entirely in the / data section on the system flash drive, thanks to which all information survives between reboots. This is one of the differences between the new FreeNAS and NAS4free: the latter’s capabilities in terms of collecting useful statistics are quite rudimentary.

Good old syslog

If possible, I recommend to get yourself somewhere a server under * nix * (you can virtual) and build a tunnel before it to receive a syslog. From the side of FreeNAS there is a trivial Syslog server setting. On the receiving side, you need to remember to open the desired port (usually 514 / udp), it is also good to scatter messages from different hosts on the files. For example, for rsyslog version 5.6, you can add the following structure close to the beginning of rsyslog.conf :

 $ModLoad imudp $UDPServerRun 514 $template FILENAME,"/var/log/%fromhost-ip%/messages.log" *.* ?FILENAME 

This telemetry option can be enhanced by making SMART log changes to all attributes. To do this, just go to the disk management section and add a simple `` -a '' flag in the SMART extra options field.
Do not forget about the rotation of the logs, otherwise they will clog the disk. I think it does not need comments.
Keeping logs separate from the system is generally a very useful concept, and not only in terms of information security, but also for operational decision making. Imagine that the server has suddenly disappeared from the network, or there is no access to it. Holding logs in your cloud, you can even try to understand what happened to it a day, an hour or minutes before the accident even before visiting the site . Sometimes this is a very valuable opportunity.

Cloud storage of reports

In fact, the archive of sent reports to Gmail is practically a DBMS with the IMAP / POP3 access protocol. Store the information there, and when the time comes, the desire and inspiration, take in the hands of perl (PHP, Python, but at least Visual Basic), connect and sculpt parsers and analyzers to your health. Gmail is optional, there are many free mail services.

findings

Ready telemetry is a useful FreeNAS function, and it is necessary to use it at least in the basic version with email. The Commander of the Server , and not Business, should be the first to know about the accident or the background to it .

Kungfu with mirrors

At the same time, I decided to tell a little about the operating experience of FreeNAS with UFS volumes using the GEOM and gmirror (8) machine. Why I abandoned the fashionable ZFS file system is written in the first part of the story.

Lizard style

In the first part of the story, I already wrote that all disk manufacturers can be considered novice programmers : one kilobyte is adopted strictly according to the SI system and equals 1000 bytes. Interestingly, in this case, the physical sector can be, for example, 512 or 4096, but never 500 or 4000 bytes. But this is so, by the way: know your disk, for any file system the size of the physical block is important.
More important for us is that the manufacturer controls random defects on different models differently, relaxing, of course, on budget drives for home computers. I came across four seemingly identical discs ST2000DM001. Two turned out to be exactly 2000398934016 bytes in volume, the other two - about 2 GB less, but this is not enough: there was a difference between the latter in 13Mb. In theory, the lack of 2GB is generally a violation of my rights as a buyer, but I decided to spit and not waste time on disassembling with the seller. You can suffer here because when you try to replace a failed disk with a new one, this very lack of 2GB will prevent you from reconstructing the array. And then either you have to run to the store again after the disk in 3TB, or redo the whole array ...
But let's outsmart the evil forces, slipping the greedy producer and cynical seller from the dead donkey's ears. When creating an array, we leave at the end of the technological reserve, a kind of tail of a lizard, which you can then shoot off if you wish. It is quite a natural desire here to create a “formwork” from the GPT sections of the right size, where the same gmirror (8) is then poured. But what stopped me is that gmirror (8) in FreeNAS works by default only with raw disks, i.e. With GPT partitions, you can in theory run into a problem of support and then sweat on each upgrade. Since this is not a home system, I decided to find a way to limit the size of the volume on raw disks, which I will give here:

 #   zero  "" geom zero load #     ,   2   4: gnop create -s 1996103966720 gzero #       --  ada0,   MIRROR1 gmirror label -v MIRROR1 gzero.nop ada0 #  ,       gmirror forget MIRROR1 gnop destroy gzero.nop geom zero unload #          4096 ,     newfs -U -f 4096 /dev/mirror/MIRROR1 #    ada2    ada0; :   ! gmirror insert MIRROR1 ada2 # ,   gmirror status 

However, the described trick (A) will not allow using the tail and (B) does not compensate for the lack of ERC on models ST2000DM001, ST3000DM001 and ST4000DM001, and therefore they are unsuitable for RAID (read habrahabr.ru/post/92701 ). If you look at Seagate products, then the ST2000VN000, ST3000VN000 and ST4000VN000 models are much better suited for arrays: in addition to ERC , they strictly maintain the volume of 2000398934016 bytes, heat up less on their 5900rpm, and they are only slightly more expensive than desktop brothers .
I do not insist that this is the only option. If the reader prefers GPT or wants to place in the tail, say, swap, then formwork can be made on GPT sections. Examples on the Internet you can find a lot, if you want, I paste it right here.

Cat style

How much I changed the connection of disks to the controller ports, gmirror (8) every time, like a cat, landed on four legs. Therefore, I can say with confidence: gmirror (8) is not afraid of rearranging “raw” disks, but identifies them by metadata. Rearrange as much as you want. If you make partitions on GPT, it should work too. Of the advantages of using gmirror (8) on GPT, I saw only the opportunity to place a swap on the same disks somewhere in the “lizard tail” area.

Turtle style

The bottleneck of our vintage server is, of course, the PCI bus, which gives less than any of the disks on linear reading. From the operating experience, I can confirm the comment of the user AMDmi3 : if you pull at the file system with operations, then the probability of full synchronization after a “bad” shutdown is very high. If you don’t pull at it, you can easily slip through without any synchronization. It should be understood that FreeNAS has UFS background check disabled ( background_fsck="NO" ) and the system start with simultaneous checking of the volume can be very time-consuming. If several mirrors fall on the PCI bus at once with the intention of synchronizing, then ... it is better not to even think what will happen. Everything will be very sad.
Fortunately, my collapsible BAF on the conditions of operation allows you to seal some of the data in the read-only. Temporarily switching the volume to the record, by the way, is not a big deal. FreeNAS will not do this in the GUI, but you can reapply kungfu and change something directly in the local fstab equivalent:

 FreeNAS# sqlite3 /data/freenas-v1.db SQLite version xxxx 20xx-xx-xx xx:xx:xx Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> .mode html sqlite> .headers ON sqlite> select * from storage_mountpoint; 

mp_path	mp_options	mp_volume_id	id
/ mnt / VOL1	rw	one	one
/ mnt / VOL2	rw	2	2
/ mnt / VOL3	rw	3	3

I guess everything is clear further. Changing the table manually - the risk is even greater than using GPT sections. Decide for yourself.

Spider style

The collapsible array has one drawback: it is inconvenient to “share” it over the network. Creating a separate “ball” for each volume is rather clumsy for Business. In addition, FreeNAS only shares storage folders, but it basically does not give system folders: you can, for example, /mnt/VOL1 , but not /mnt or /tmp . I would like to find a way to drive all the volumes into one point, but at the same time not to put one into another: the better the third, if the fourth appears, and the first is planned to be turned off? Inspired by one post , I came up with this method:

1. Each volume must create a master directory, for example, /mnt/VOL1/PUBLIC , /mnt/VOL2/PUBLIC , etc. User folders are not created at the root, under any circumstances.
2. Either on one of the volumes, or on an additional flash drive, an "assembly site" is created, for example, /mnt/SITE , where all the main directories will be listed. It also creates a directory PUBLIC , and below it - empty points VOL1 , VOL2 , VOL3 for mounting for each volume.
3. Using the FreeBSD mount_nullfs (8) tool, start directories of each volume are mounted (injected) on the corresponding points with start scripts. By the way, in the Linux dialect it is simply called: mount --bind .
4. Only the main directories are shared, for example, /mnt/SITE/PUBLIC .
5. In the future, the user’s home folders /mnt/SITE/HOME/%username% can be placed here on the installation site, and inject folders related to specific business processes. So you can build a pretty curly-based system of access rights without suffering from an ACL.

The web of mounted partitions interweaves volumes and their presentation for users.

But in the style of a spider there is one uncomfortable nuance. The fact is that free disk space displayed to customers will report on the state of the installation site , and not volumes. For example, when I built the site on a 1GB flash drive, the free space of all my volumes visible on the network shrunk to one unfortunate gigabyte, and the Windows 7 client immediately refused to write large files there. Anyway, the concept of free disk space on a composite volume is very relative.
Fortunately, Samba itself allows you to get out of this story, and FreeNAS does not interfere. There is such a blow in our kungfu, known as the dfree command . An experienced master can state to the system his own vision of the space in the folder and its consumption by writing a simple script in your favorite language. In the simplest case, you can even take full responsibility for yourself, regardless of the actual situation, reporting ten completely free terabytes:

 #!/bin/sh echo 10737418240 10737418240 

In the FreeNAS system, the location of the script is specified in the Auxiliary Parameters option of the Windows settings (CIFS) Shares , for example:
dfree command = /mnt/SITE/scripts/mydfree

However, this technique is worked out only on CIFS and is adequate exactly to the point where there is really enough free space. When writing to the completed part of the volume, there will, of course, be a failure. A user who is not in possession of the kungfu according to the staffing table will naturally overlook it, and the master dear reader will be guilty of losing the data. Therefore, be careful, combine with other styles, use telemetry. For example, it would be possible to apply the turtle style described above to the conservation of all volumes except the last one, and to give actual information only on it. Choose you, dear Master Yuniksovogo Kungfu!

:

, - Seagate ST3120827AS, . , swap . , , :

 /dev/ufs/T1: CANNOT READ BLK: xxxxxxx /dev/ufs/T1: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY. THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY: ufs: /dev/ufs/T1 (/mnt/T1) File system preen failed, trying fsck -y Can't stat /dev/ufs/T1: No such file or directory Mounting late file systems: mount: /dev/ufs/T1: No such file or directory. Mounting /etc/fstab filesystems failed, startup aborted ERROR: ABORTING BOOT (sending SIGTERM to parent)! init: /bin/sh on /etc/rc terminated abnormally, going to single user mode Enter full pathname of shell or RETURN for /bin/sh: 

, , : «, , ». ENTER, , , , ! , …

findings

, (, ). , UFS .

Χ*

* Χ «»

ERIAL ATA, , — . SATA « » ( backplane ). -, . -, , . . , , .

3.5", - , 5.25". , - RW ( CD-ROM? , , DVD ).


6 3.5"

Hm ( , ) — . , , . - : , , , . Cooler Master STB-3T4-E3-GP , 2013., . Heck. $30 . : , .


, EverCool Armor HDD Cooling Box ( HD-AR ). , Intel Core i7. , P4 … , , .


EverCool Armor HDD Cooling Box (HD-AR)

5.25" 3.5", «» 5.25" - . , , Cooler Master STB-3T4-E3-GP , - . , Noctua NF-R8 ( , ). , , !

, . , HD-AR :





3.5" 2.5" ( )





«»


26dBa



HD-AR Noctua NF-R8. , NF-R8 : 1300rpm ( 10dBa) 800rpm (7dBa). 1800rpm 17dbA.

: NF-R8 : , DC 4..13. NF-R8 NF-R8 PWM , : , 12. , , .

NF-R8


HD-AR

: «» , ( ). 20 , . , .

: «» SATA 50 , eSATA

, , . , . , . , , . , - -! - , . , . . , SATA .


— !

, : « », - . , , . , .

, () . ( staggered spin up ), , backplane ( ). , APM .
2 12 ( ST2000DM001 2.5 , RAID, - , ERC ; habrahabr.ru/post/92701 ). P4 250 . — 6 x 2.0 x 12 = 144. 400 450. .
400: ? 2005 Ӽ-. , , 60% . , , , .




UPD:
, , , . .

, ATX-450PNR , 12: +12V1 14 (, «» SATA), +12V2 16 (4- mini-fit ). 12 360. mini-fit 200 , 450 250. : mini-fit, ( , ).
.

200 ? , 5 16, +12V1 14 «» +12V2 16, «» . , , ( ) , - , : (1) , (2) (3) , . 4- mini-fit, Y- «». - mini-fit «» , .
, - , wirecraft . Ҳ- , mini-fit, Y- ( ).





Ҳ - , 0,01% durability.




Ҳ- , , . , , .


5.25", . , , . . , , SATA. ERIAL ATA . (, ), , SATA. .
, .


Χ

UPD:
, , , . 30%, 200. , . wirecraft , 12 , «». .

, , smartd(8) FreeNAS Email Alert : . , 45°C ( 60°C), , - -25°C. ! , , , . (. )? SSH. , - . 2..3°C . - mbmon(1) , , ( Winbond W83627). ? , — Ẋ, , . , , , - 5.25".
: ( ), ( ). : «», , , , . , , . — , .

findings

• (, ).
• ( staggered spin-up ) « » ( ), APM ( ).
• , .
• UPD: 30% , 200; —

Links

www.freenas.org
doc.freenas.org/index.php/Settings#General_Tab
doc.freenas.org/index.php/Settings#Email_Tab
doc.freenas.org/index.php/Cron_Jobs
doc.freenas.org/index.php/SMART .
doc.freenas.org/index.php/SMART_Tests
doc.freenas.org/index.php/Volumes#Viewing_Volumes
doc.freenas.org/index.php/Volumes#Viewing_Disks
doc.freenas.org/index.php/Windows_%28CIFS%29_Shares
www.freebsd.org/cgi/man.cgi?query=dd&sektion=1
www.freebsd.org/cgi/man.cgi?query=mail&sektion=1
www.freebsd.org/cgi/man.cgi?query=mbmon&manpath=ports&sektion=1
www.freebsd.org/cgi/man.cgi?query=uuencode&sektion=1
www.freebsd.org/cgi/man.cgi?query=smartd.conf&manpath=ports&sektion=5
www.freebsd.org/cgi/man.cgi?gmirror%288%29
www.freebsd.org/cgi/man.cgi?query=mount_nullfs&sektion=8
www.freebsd.org/cgi/man.cgi?query=smartd&manpath=ports&sektion=8
searchstorage.techtarget.com/tip/Failed-RAID-drives-need-immediate-attention
www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html#DFREECOMMAND
forums.nas4free.org/viewtopic.php?f=18&t=1552
linux.die.net/man/8/mount
www.collectd.org
www.rsyslog.com
smartmontools.sourceforge.net/man/smartctl.8.html
www.coolermaster.com/case/case-accessories/4-in-3-device-module
www.evercool.com.tw/products/hd-ar.htm
www.brownbear.ru
www.noctua.at/main.php?show=productview&setlng=en&products_id=9
www.noctua.at/main.php?show=productview&setlng=en&products_id=45
www.fsp-power.ru/product/atx_450pnr
www.fsp-group.com.tw
www.wikipedia.org/wiki/Comparison_of_S.MART_tools
www.wikipedia.org/wiki/Spin-up

NAS :
1: ,
2:
3:
4: