UPD Growing Up With GlusterFS

Hello.

The company in which I work is developing quite dynamically, and the number of servers and users grows accordingly. Resiliency issues are becoming more relevant than before.
Historically, some of the data is in files and connected by clients to backends via NFS. However, I didn't want the central server to be a point of failure.
Rsync options are shallowed immediately. After studying the issue and the experience of other people, as well as some tests of s3fs, which by the way was candidate No. 1 for distributed storage, GlusterFS was chosen.

Without going into details, which you can find by the links on the Main site , the Community will say that such a repository was quite suitable for our purposes. The main feature of GlusterFS is that the client writes to any currently available wizard, and replicates to the inaccessible after he returns to the line.

So proceed to set up. I’ll make a reservation that the old NFS on the servers worked at the same time, they get along quite well on the same machine. We needed this at the time of the transition from NFS.
')
Install the server:

aptitude install glusterfs-server 

We describe the config (/etc/glusterfs/glusterfsd.vol):

 volume posix type storage/posix option directory /mnt/Files end-volume volume locks type features/locks subvolumes posix end-volume volume Files type performance/io-threads option thread-count 8 subvolumes locks end-volume volume server type protocol/server option transport-type tcp option auth.login.Files.allow user #     login+password option auth.login.user.password secret_pass subvolumes Files end-volume 

Duplicate the glusterfs-server package and the config (/etc/glusterfs/glusterfsd.vol) on the second server.

We set the client:

 aptitude install glusterfs-client 

We describe the config for the client (/etc/glusterfs/glusterfs.vol):

 volume server1.com type protocol/client option transport-type tcp option username user option password secret_pass option remote-host 10.132.259.159 # server1.com option remote-subvolume Files end-volume volume server2.com type protocol/client option transport-type tcp option username user option password secret_pass option remote-host 10.184.179.175 # server2.com option remote-subvolume Files end-volume volume replicate type cluster/replicate subvolumes server1.com server2.com end-volume volume writebehind type performance/write-behind option cache-size 1MB subvolumes replicate end-volume volume cache type performance/io-cache option cache-size 512MB subvolumes writebehind end-volume 

Load the fuse module

 modprobe fuse 

try to mount FS

 /usr/sbin/glusterfs -f /etc/glusterfs/glusterfs.vol /mnt/Files df -h /etc/glusterfs/glusterfs.vol 199G 120G 70G 64% /mnt/Files 

Now about the small rake. Gluster-FS is not mounted when rebooting via fstab, because fuse module does not load automatically. You can rebuild the kernel, but we will do it easier.
Add to the end of / etc / initramfs-tools / modules:

 fuse 

We carry out:

 update-initramfs -u -k `uname -r` 

Add to / etc / fstab:

 /etc/glusterfs/glusterfs.vol /mnt/Files glusterfs defaults 0 0 

Reboot and profit.

Well, anticipating questions about the tests, I will give a couple of indicative, Gluster-fs vs NFS vs POHMELFS. File weighing ~ 700MB.

GlusterFS:

 root@domU-12-41-56-0F-34-81:~# time cat /mnt/Files/ubuntu-11.10-desktop-amd64+mac.iso > /dev/null real 0m28.101s user 0m0.000s sys 0m4.700s root@domU-12-41-56-0F-34-81:~# time cat /mnt/Files/ubuntu-11.10-desktop-amd64+mac.iso > /dev/null real 0m20.321s user 0m0.000s sys 0m2.030s root@domU-12-41-56-0F-34-81:~# time cat /mnt/Files/ubuntu-11.10-desktop-amd64+mac.iso > /dev/null real 0m36.444s user 0m0.000s sys 0m1.410s 

POHMELFS (Taken from here POHMELFS: NFS in a new way ):

 stolen@stolen ~/soft/pohmelfs-server.git $ time cat /mnt/pohmel/share/GRTMPVOL_RU_20.10.08.iso >/dev/null real 1m37.150s user 0m0.010s sys 0m1.270s stolen@stolen ~/soft/pohmelfs-server.git $ time cat /mnt/pohmel/share/GRTMPVOL_RU_20.10.08.iso >/dev/null real 0m0.591s user 0m0.003s sys 0m0.257s stolen@stolen ~/soft/pohmelfs-server.git $ time cat /mnt/upload/share/GRTMPVOL_RU_20.10.08.iso >/dev/null real 1m3.719s user 0m0.003s sys 0m0.400s 

NFS:

 root@domU-12-41-56-0F-34-81:run$ time cat /mnt/Files/ubuntu-11.10-desktop-amd64+mac.iso > /dev/null real 0m15.220s user 0m0.008s sys 0m0.423s root@domU-12-41-56-0F-34-81:run$ time cat /mnt/Files/ubuntu-11.10-desktop-amd64+mac.iso > /dev/null real 0m16.139s user 0m0.005s sys 0m0.160s root@domU-12-41-56-0F-34-81:run$ time cat /mnt/Files/ubuntu-11.10-desktop-amd64+mac.iso > /dev/null real 0m18.812s user 0m0.006s sys 0m0.177s 

Tests GlusteFS vs NFS on small files.
For these purposes, I sketched a script that generates 700 files and clogs them with small content:

 #!/bin/bash count=0 while [ $count -lt 700 ] do touch small_files/$count.txt openssl rand -base64 $[$count*42] > /root/small_files/$count.txt count=$[$count+1] done 

GlusterFS copying:

 root@domU-12-31-36-0F-71-81:~# time cp /root/small_files/* /mnt/Files/test_small/ real 0m28.318s user 0m0.010s sys 0m0.160s root@domU-12-31-36-0F-71-81:~# time cp /root/small_files/* /mnt/Files/test_small/ real 0m27.432s user 0m0.000s sys 0m0.170s root@domU-12-31-36-0F-71-81:~# time cp /root/small_files/* /mnt/Files/test_small/ real 0m29.397s user 0m0.010s sys 0m0.150s 

NFS copying:

 root@domU-12-41-39-01-D1-71:~# time cp /root/small_files/* /mnt/Files/test_small/ real 0m15.848s user 0m0.008s sys 0m0.061s root@domU-12-41-39-01-D1-71:~# time cp /root/small_files/* /mnt/Files/test_small/ real 0m12.792s user 0m0.002s sys 0m0.070s root@domU-12-41-39-01-D1-71:~# time cp /root/small_files/* /mnt/Files/test_small/ real 0m13.417s user 0m0.006s sys 0m0.049s 

GlusterFS removal:

 root@domU-12-31-36-0F-71-81:~# time rm /mnt/Files/test_small/* real 0m5.050s user 0m0.060s sys 0m0.000s root@domU-12-31-36-0F-71-81:~# time rm /mnt/Files/test_small/* real 0m7.055s user 0m0.000s sys 0m0.060s root@domU-12-31-36-0F-71-81:~# time rm /mnt/Files/test_small/* real 0m5.300s user 0m0.040s sys 0m0.020s 

NFS removal:

 root@domU-12-41-39-01-D1-71:~# time rm /mnt/Files/test_small/* real 0m4.861s user 0m0.003s sys 0m0.013s root@domU-12-41-39-01-D1-71:~# time rm /mnt/Files/test_small/* real 0m3.618s user 0m0.002s sys 0m0.007s root@domU-12-41-39-01-D1-71:~# time rm /mnt/Files/test_small/* real 0m4.297s user 0m0.003s sys 0m0.024s 

UPD: Added tests on small files for NFS and GlusterFS.