I found several articles on Habré about the basic installation and configuration of GlusterFS, but did not find anything about the types of content distribution that it supports and with which I have been playing for a long time . What will be discussed in this article.


GlusterFS is a very convenient and easy to use and set up distributed file system that works in user space using FUSE technology, i.e. runs on top of the main file system. For installation and configuration we will use the OS Ubuntu 12.04.

We will configure a cluster of two servers and mount (test) on one client. We have the following settings:

IP	Hostname	Short name
192.168.1.100	server1.example.com	server1
192.168.1.101	server2.example.com	server2
192.168.1.102	client1.example.com	client1

1. General settings

First, for convenience, add all the addresses to the hosts on each machine.

root@server1:~# cat /etc/hosts 127.0.0.1 localhost 192.168.1.100 server1.example.com server1 192.168.1.101 server2.example.com server2 192.168.1.102 client1.example.com client1 root@server2:~# cat /etc/hosts 127.0.0.1 localhost 192.168.1.100 server1.example.com server1 192.168.1.101 server2.example.com server2 192.168.1.102 client1.example.com client1 root@client1:~# cat /etc/hosts 127.0.0.1 localhost 192.168.1.100 server1.example.com server1 192.168.1.101 server2.example.com server2 192.168.1.102 client1.example.com client1 

')
If you use a firewall, you need to open TCP ports 111, 24007-24050 on all machines that act as servers (two, in our cases).

2. Configuring the server part (cluster)

On all servers you need to install gluster-server. At the moment, the latest version is 3.5.2. Installation will be done from ppa repositories.

 root@server1:~#apt-get install python-software-properties root@server1:~#add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5 root@server1:~#apt-get update root@server1:~#apt-get install glusterfs-server - root@server2:~#apt-get install python-software-properties root@server2:~#add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5 root@server2:~#apt-get update root@server2:~# apt-get install glusterfs-server - 

Now we connect to one of the servers and create a cluster.

 root@server1:~# gluster peer probe server2.example.com Probe successful 

Check status:

 root@server1:~# gluster peer status Number of Peers: 1 Hostname: server2.example.com Uuid: 0f3aa4f4-f9dd-4cd2-a7a5-126606fbab33 State: Peer in Cluster (Connected) 

As you can see, server2 was added to the cluster and its status is Connected. You can also make sure that on the second server everything is ok.

 root@server2:~# gluster peer status Number of Peers: 1 Hostname: 192.168.1.100 Uuid: ae4e6766-787f-4d0c-8b96-8fc9523ef346 State: Peer in Cluster (Connected) 

GlusterFS supports 5 types of distribution of content combined in volume:

• Distributed
• Replicated
• Striped (divided in parts)
• Distributed Striped (distributed and divided in parts)
• Distributed Replicated (distributed and replicable)

First, we configure each type of content distribution separately and then mount all 5 volums on the client for tests.

2.1 Configuring distributed volume

With this setting, the data will be distributed in a random order between each folder that entered the volume.
Create dist1, dist2 packs on the first server, dist3, dist4 packs on the second server.

 root@server1:~# mkdir /mnt/dist{1,2} root@server2:~# mkdir /mnt/dist{3,4} 

Now you can create and start volume.

 root@server1:~# gluster volume create distributed transport tcp server1:/mnt/dist1 server1:/mnt/dist2 server2:/mnt/dist3 server2:/mnt/dist4 force Creation of volume distributed has been successful. Please start the volume to access data. root@server1:~# gluster volume start distributed Starting volume distributed has been successful 

2.2 Configuring replicated volume

With this setting, the data will be mirrored (like RAID1) copied between each folder (server) that entered volume. Of course, the number of folders (servers) included in the volume must be a multiple of the replica.
Create repl1, repl2 on the first server, repl3, repl4 on the second server.

 root@server1:~# mkdir /mnt/repl{1,2} root@server2:~# mkdir /mnt/repl{3,4} 

Now you can create and start volume.

 root@server1:~# gluster volume create replicated replica 4 transport tcp server1:/mnt/repl1 server2:/mnt/repl3 server1:/mnt/repl2 server2:/mnt/repl4 force Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Do you still want to continue creating the volume? (y/n) y volume create: replicated: success: please start the volume to access data root@server1:~# gluster volume start replicated volume start: replicated: success 

In this case, all files will be replicated to all 4 folders. Those. all folders will contain the same content.

2.3 Configuring striped volume

With this setting, the files will be divided into parts and each piece will be stored in each of the folders included in the volume. This type of content distribution is suitable for storing capacitive data with a very large size (video content in good quality, for example). Of course, the number of folders included in the volume should be equivalent to the value of stripe. Those. if we have 5 servers, the file will be divided into 5 parts and each of its pieces will be stored on all 5 servers.
Let's create pack1 strip1, strip2 on the first server, strip3, strip4 on the second.

 root@server1:~# mkdir /mnt/strip{1,2} root@server2:~# mkdir /mnt/strip{3,4} 

Now you can create and start volume.

 root@server1:~# gluster volume create striped stripe 4 transport tcp server1:/mnt/strip1 server1:/mnt/strip2 server2:/mnt/strip3 server2:/mnt/strip4 force Creation of volume striped has been successful. Please start the volume to access data. root@server1:~# gluster volume start striped Starting volume striped has been successful 

In this case, on each of the 4 servers (folders) there will be one piece of the uploaded file.

2.4 Configuring distributed striped volume

With this setting, the files will be divided into parts and the pieces of the files will be divided into folders and stored in one of them. Of course, the number of folders included in the volume must be a multiple of the stripe value. Those. if we have 8 servers and the stripe value is 4, then the files will be split into 4 parts and stored either on the first 4 servers or on the next 4 servers. In the same way, if we have 4 servers and the stripe is 2, then the files will be split into 2 parts and stored on the first 2 or on the next 2 servers.
Let's create dist-strip1, dist-strip2 packs on the first server, dist-strip3, dist-strip4 on the second.

 root@server1:~# mkdir /mnt/dist-strip{1,2} root@server2:~# mkdir /mnt/dist-strip{3,4} 

Now you can create and start volume. In this case, the order of servers plays a very important role: parts of files will be stored on the first pair of servers (with stripe = 2), or on the first four of servers (with stripe = 4), and the content will be distributed between the first and second pair or four and t .P.

 root@server1:~# gluster volume create distributed-striped stripe 2 transport tcp server1:/mnt/dist-strip1 server2:/mnt/dist-strip3 server1:/mnt/dist-strip2 server2:/mnt/dist-strip4 force Creation of volume distributed-striped has been successful. Please start the volume to access data. root@server1:~# gluster volume start distributed-striped Starting volume distributed-striped has been successful 

2.5 Configuring distributed replicated volume

With this setting, data will be randomly distributed between folders and each folder has its own mirror copy. Of course, the number of folders included in the volume should be a multiple of the replica value. Those. if we have 4 servers and the replica value is 2, then the files will be distributed to the 2 servers in random order, and the 2 remaining servers will store an identical copy of the content of the first two. If we have 8 servers and replica is 4, then we will have one mirror of 4 servers.
Let's create dist-repl1, dist-repl2 packs on the first server, dist-repl3, dist-repl4 on the second server.

 root@server1:~# mkdir /mnt/dist-repl{1,2} root@server2:~# mkdir /mnt/dist-repl{3,4} 

Now you can create and start volume. At the same time, the order of servers plays a very important role: the first pair (foursome) of servers recorded sequentially is one replica. Those. if we have 8 servers and replica is equal to 4, then the first 4 servers will have one identical content, and the second four will have another identical content, etc.

 root@server1:~# gluster volume create distributed-replicated replica 2 transport tcp server1:/mnt/dist-repl1 server2:/mnt/dist-repl3 server1:/mnt/dist-repl2 server2:/mnt/dist-repl4 force Creation of volume distributed-replicated has been successful. Please start the volume to access data. root@server1:~# gluster volume start distributed-replicated Starting volume distributed-replicated has been successful 

Now let's check what volum we created.

 root@server1:~# gluster volume info Volume Name: distributed Type: Distribute Volume ID: 01622619-fd93-4ee1-85ad-ca8cf1d85f7e Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: server1:/mnt/dist1 Brick2: server1:/mnt/dist2 Brick3: server2:/mnt/dist3 Brick4: server2:/mnt/dist4 Volume Name: replicated Type: Replicate Volume ID: 67afcb89-7e5d-4a02-b4ac-0c2de7cd97be Status: Started Number of Bricks: 1 x 4 = 4 Transport-type: tcp Bricks: Brick1: server1:/mnt/repl1 Brick2: server2:/mnt/repl3 Brick3: server1:/mnt/repl2 Brick4: server2:/mnt/repl4 Volume Name: striped Type: Stripe Volume ID: e9ef42bf-8265-4973-85de-4cafd2a68fec Status: Started Number of Bricks: 1 x 4 = 4 Transport-type: tcp Bricks: Brick1: server1:/mnt/strip1 Brick2: server1:/mnt/strip2 Brick3: server2:/mnt/strip3 Brick4: server2:/mnt/strip4 Volume Name: distributed-striped Type: Distributed-Stripe Volume ID: aa70dd67-3ca9-48cb-865b-b10f8ca1ccad Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: server1:/mnt/dist-strip1 Brick2: server2:/mnt/dist-strip3 Brick3: server1:/mnt/dist-strip2 Brick4: server2:/mnt/dist-strip4 Volume Name: distributed-replicated Type: Distributed-Replicate Volume ID: 59a819c4-6e84-4c49-9e90-23daa59d12ee Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: server1:/mnt/dist-repl1 Brick2: server2:/mnt/dist-repl3 Brick3: server1:/mnt/dist-repl2 Brick4: server2:/mnt/dist-repl4 

As you can see, we have 5 volums with different types of content distribution. You can go to the client side.

3. Configure the client part

All clients need to install gluster-client from ppa repositories.

 root@client1:~#apt-get install python-software-properties root@client1:~#add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5 root@client1:~#apt-get update root@client1:~#apt-get install glusterfs-client 

Then you just need to mount the network folder by the name of the created volum-a. In this case, the IP or domain name of the server is not important if we have 10 servers added to one cluster, then on the client you can mount the ball using any of the 10 IP, the name volum-a remains the same.

3.1 Mounting and distributed volume test

Connect to the client, create a folder for the new disk and mount the distributed volume.

 root@client1:~# mkdir /mnt/distrib root@client1:~# mount.glusterfs server1:/distributed /mnt/distrib/ root@client1:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.3G 1.5G 5.5G 21% / udev 236M 4.0K 236M 1% /dev tmpfs 49M 280K 49M 1% /run none 5.0M 0 5.0M 0% /run/lock none 245M 0 245M 0% /run/shm server1:/distributed 30G 6.3G 22G 23% /mnt/distrib 

As you can see, we have successfully mounted a new network drive with a size of 30GB (the total size of the disks of all servers included in the volume). Now create a dozen files.

 root@client1:~# seq 1 10 | while read line; do echo "This is File${line}" > /mnt/distrib/file${line};done root@client1:~# ls /mnt/distrib/ file1 file10 file2 file3 file4 file5 file6 file7 file8 file9 

We look how the content is distributed across servers.

 root@server1:~# ls /mnt/dist[0-9]/ /mnt/dist1/: file10 file3 file4 file9 /mnt/dist2/: file7 root@server2:~# ls /mnt/dist[0-9]/ /mnt/dist3/: file1 file2 /mnt/dist4/: file5 file6 file8 

As you can see, the files are scattered in all four folders that were included in the distributed volume.

3.2 Mounting and replicated volume test

Connect to the client, create a folder for the new disk and mount the replicated volume.

 root@client1:~# mkdir /mnt/replica root@client1:~# mount.glusterfs server1:/replicated /mnt/replica/ root@client1:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.3G 1.5G 5.5G 21% / udev 131M 4.0K 131M 1% /dev tmpfs 28M 280K 28M 1% /run none 5.0M 0 5.0M 0% /run/lock none 140M 0 140M 0% /run/shm server1:/replicated 7.3G 1.9G 5.1G 28% /mnt/replica 

As you can see, we have successfully mounted a new network drive with a size of 7.3GB (the total size of the disks of all servers included in the volume divided by the number of replicas). Now create a dozen files.

 root@client1:~# seq 1 10 | while read line; do echo "This is File${line}" > /mnt/replica/file${line};done root@client1:~# ls /mnt/replica/ file1 file10 file2 file3 file4 file5 file6 file7 file8 file9 

We look how the content is distributed across servers.

 root@server1:~# ls /mnt/repl* /mnt/repl1: file1 file10 file2 file3 file4 file5 file6 file7 file8 file9 /mnt/repl2: file1 file10 file2 file3 file4 file5 file6 file7 file8 file9 root@server2:~# ls /mnt/repl* /mnt/repl3: file1 file10 file2 file3 file4 file5 file6 file7 file8 file9 /mnt/repl4: file1 file10 file2 file3 file4 file5 file6 file7 file8 file9 

As you can see, the files were mirrored into each folder that was included in the replicated volume.

3.3 Mounting and test striped volume

Connect to the client, create a folder for the new disk and mount striped volume.

 root@client1:~# mkdir /mnt/strip root@client1:~# mount.glusterfs server1:/striped /mnt/strip/ root@client1:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.3G 1.5G 5.5G 21% / udev 131M 4.0K 131M 1% /dev tmpfs 28M 280K 28M 1% /run none 5.0M 0 5.0M 0% /run/lock none 140M 0 140M 0% /run/shm server1:/striped 30G 6.3G 22G 23% /mnt/strip 

As you can see, we have successfully mounted a new network drive with a size of 30GB (the total size of the disks of all servers included in the volume). Now create a couple of large files and copy to the folder.

 root@client1:~# dd if=/dev/urandom of=test2.bin bs=30M count=10 root@client1:~# dd if=/dev/urandom of=test1.bin bs=30M count=10 root@client1:~# cp test* /mnt/strip/ root@client1:~# ls -lh /mnt/strip/ total 601M -rw-r--r-- 1 root root 300M Mar 2 14:13 test1.bin -rw-r--r-- 1 root root 300M Mar 2 14:13 test2.bin 

As you can see, we have 2 files in the folder with a size of 300MB. Now let's see how the content is distributed across servers.

 root@server1:~# ls -lh /mnt/strip* /mnt/strip1: total 151M -rw-r--r-- 2 root root 75M Mar 2 14:13 test1.bin -rw-r--r-- 2 root root 75M Mar 2 14:13 test2.bin /mnt/strip2: total 151M -rw-r--r-- 2 root root 75M Mar 2 14:13 test1.bin -rw-r--r-- 2 root root 75M Mar 2 14:13 test2.bin root@server2:~# ls -lh /mnt/strip* /mnt/strip3: total 151M -rw-r--r-- 2 root root 75M Mar 2 14:13 test1.bin -rw-r--r-- 2 root root 75M Mar 2 14:13 test2.bin /mnt/strip4: total 151M -rw-r--r-- 2 root root 75M Mar 2 14:13 test1.bin -rw-r--r-- 2 root root 75M Mar 2 14:13 test2.bin 

As you can see, the files were divided into equal parts of 75 MB and scattered in all four folders, which were included in the striped volume.

3.4 Mounting and test distributed volume

Connect to the client, create a folder for the new disk and mount the distributed volume.

 root@client1:~# mkdir /mnt/distrib-strip root@client1:~# mount.glusterfs server1:/distributed-striped /mnt/distrib-strip/ root@client1:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.3G 1.8G 5.2G 25% / udev 131M 4.0K 131M 1% /dev tmpfs 28M 280K 28M 1% /run none 5.0M 0 5.0M 0% /run/lock none 140M 0 140M 0% /run/shm server1:/distributed-striped 30G 6.9G 21G 26% /mnt/distrib-strip 

As you can see, we have successfully mounted a new network drive with a size of 30GB (the total size of the disks of all servers included in the volume). Now create a couple of large files and copy to the folder.

 root@client1:~# dd if=/dev/urandom of=test2.bin bs=30M count=10 root@client1:~# dd if=/dev/urandom of=test1.bin bs=30M count=10 root@client1:~# cp test* /mnt/distrib-strip/ root@client1:~# ls -lh /mnt/distrib-strip/ total 600M -rw-r--r-- 1 root root 300M Mar 2 14:35 test1.bin -rw-r--r-- 1 root root 300M Mar 2 14:34 test2.bin 

As you can see, we have 2 files in the folder with a size of 300MB. Now let's see how the content is distributed across servers.

 root@server1:~# ls -lh /mnt/dist-strip* /mnt/dist-strip1: total 151M -rw-r--r-- 2 root root 150M Mar 2 14:35 test1.bin /mnt/dist-strip2: total 151M -rw-r--r-- 2 root root 150M Mar 2 14:34 test2.bin root@server2:~# ls -lh /mnt/dist-strip* /mnt/dist-strip3: total 151M -rw-r--r-- 2 root root 150M Mar 2 14:35 test1.bin /mnt/dist-strip4: total 151M -rw-r--r-- 2 root root 150M Mar 2 14:34 test2.bin 

As you can see, the files were scattered in different folders and divided into equal parts of 150 MB.

3.5 Mounting and test distributed replicated volume

Connect to the client, create a folder for the new disk and mount the distributed replicated volume.

 root@client1:~# mkdir /mnt/distrib-repl root@client1:~# mount.glusterfs server1:/distributed-replicated /mnt/distrib-repl/ root@client1:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.3G 1.8G 5.1G 27% / udev 131M 4.0K 131M 1% /dev tmpfs 28M 280K 28M 1% /run none 5.0M 0 5.0M 0% /run/lock none 140M 0 140M 0% /run/shm server1:/distributed-replicated 15G 4.4G 9.5G 32% /mnt/distrib-repl 

As you can see, we have successfully mounted a new network drive with a size of 15GB (the total size of the disks of all servers included in the volume divided by the number of replicas). Now create a dozen files.

 root@client1:~# seq 1 10 | while read line; do echo "This is File${line}" > /mnt/distrib-repl/file${line};done root@client1:~# ls /mnt/distrib-repl/ file1 file10 file2 file3 file4 file5 file6 file7 file8 file9 

We look how the content is distributed across servers.

 root@server1:~# ls /mnt/dist-repl* /mnt/dist-repl1: file10 file3 file4 file7 file9 /mnt/dist-repl2: file1 file2 file5 file6 file8 root@server2:~# ls /mnt/dist-repl* /mnt/dist-repl3: file10 file3 file4 file7 file9 /mnt/dist-repl4: file1 file2 file5 file6 file8 

As you can see, the first server has the same content as the second one scattered across folders.

4. Conclusion

In reality, each folder must be a separately mounted disk (not a root file system). You are now familiar with each type of content distribution that is used in glusterFS.

GlusterFS shows itself well in local networks at basic settings, if you go to data centers distributed across countries, you need to tune the settings both on the server side and when mounted on the client, which will be discussed in the next article.