S3 Storage Balancing with GoBetween + VRRP

Using Ceph to store backups using their S3-compatible RadosGW storage, we came to the conclusion that one radosGW could not cope with the load assigned to it and decided that it would be time to unbalance it with concomitant fault tolerance. As a result, we came to the solution of balancing with the help of GoBetween (a very light L4 balancer, for more details on gobetween.io ), and fault tolerance was organized using VRRP.

There was such a scheme:

1. master node vrrp receives data stream over http (s);
2. gobetween scatters all traffic to itself and the backup vrrp node;
3. radosgw, in turn, write directly to ceph;
4. in the case of the fall of the master node vrrp, backup node takes the entire load on itself until the master rises

Our implementation of this action read below.

Given:
')

1. Ceph cluster (Jewel)
   
   • IP Monitors: 10.0.1.1, 10.0.1.2, 10.0.1.3
2. Two iron servers (CentOS)
   
   • The first server is 10.0.0.1 (let's call it gbt1.example.com)
   • The second server is 10.0.0.2 (gbt2.example.com)
   • The total IP will be 10.0.0.3 (s3.example.com)
3. Domain example.com

Task:

Make balancing failover for S3 storage organized with RadosGW

Stages:

1. Deploy RadosGW on two servers
2. Organize resiliency with VRRP
3. Organize S3 traffic balancing using GoBetween
4. Check

Preparation (on both machines everything is identical)

CentOS 7.4 is installed on the servers, immediately after installing the OS, we will update everything:

# yum -y update 

Install all the software we need for TK (except for ceph itself, because at first only its repository is installed):

 # yum -y install keepalived centos-release-ceph-jewel wget 

At the moment we have not yet installed Ceph, so install it:

 # yum -y install ceph-radosgw 

Immediately configure the firewall, opening the necessary ports and allowing services:

 # firewall-cmd --permanent --add-port=18080/tcp # firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface enp2s0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT # firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface enp2s0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT # firewall-cmd --permanent --add-port=10050/tcp # firewall-cmd --reload 

Turn off SELinux (just in case):

 # sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux # setenforce 0 

Expand RadosGW

Initially, the Ceph cluster has already been raised, I think we will not touch upon the details here, the topic is not of this article, we will immediately move on to setting up radosGW.

The config is given as an example, in your case some parameters may differ:

 # cat /etc/ceph/ceph.conf [global] fsid = 01dea7f3-91f4-48d1-9d44-ba93d4a103c5 mon_host = 10.0.1.1, 10.0.1.2, 10.0.1.3 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 10.0.1.0/24 [client] rbd_cache = true [client.radosgw.gateway] rgw_frontends = civetweb port=18080 rgw_region = example rgw_region_root_pool = .example.rgw.root rgw_zone = example-s3 rgw_zone_root_pool = .example-s3.rgw.root host = s3 keyring = /etc/ceph/client.radosgw.gateway rgw_dns_name = s3.example.com rgw_print_continue = true 

Don't forget to copy the /etc/ceph/client.radosgw.gateway key from any Ceph cluster node
Run radosgw:

 # systemctl start ceph-radosgw@radosgw.gateway 

And add it to the autostart:

 # systemctl enable ceph-radosgw@radosgw.gateway 

Expand VRRP

On the master node (the difference in the state and priority options):

 # cat /etc/keepalived/keepalived.conf global_defs { notification_email { user@example.com } notification_email_from gbt@example.com smtp_server mail.example.com smtp_connect_timeout 30 router_id GBT1 } vrrp_instance VI_1 { state MASTER interface enp2s0 virtual_router_id 33 priority 101 advert_int 1 smtp_alert authentication { auth_type PASS auth_pass 123123123 } virtual_ipaddress { 10.0.0.3 } } 

On the backup node:

 # cat /etc/keepalived/keepalived.conf global_defs { notification_email { user@example.com } notification_email_from gbt@example.com smtp_server mail.example.com smtp_connect_timeout 30 router_id GBT1 } vrrp_instance VI_1 { state BACKUP interface enp2s0 virtual_router_id 33 priority 100 advert_int 1 smtp_alert authentication { auth_type PASS auth_pass 123123123 } virtual_ipaddress { 10.0.0.3 } } 

Restart and add to autostart (both nodes):

 # systemctl restart keepalived # systemctl enable keepalived 

Expand GoBetween

First, download and unpack the gobetween binary:

 # wget https://github.com/yyyar/gobetween/releases/download/0.5.0/gobetween_0.5.0_linux_amd64.tar.gz # tar -xzf gobetween_0.5.0_linux_amd64.tar.gz -C /usr/local/bin/ 

We write gobetween config (for SSL connections we specify the location of the keys). The config on both nodes is the same:

 # cat /etc/gobetween.toml [logging] level = "debug" # "debug" | "info" | "warn" | "error" output = "/var/log/gobetween.log" [api] enabled = true # true | false bind = ":8888" # "host:port" cors = false # cross-origin resource sharing [defaults] max_connections = 0 # Maximum simultaneous connections to the server client_idle_timeout = "0" # Client inactivity duration before forced connection drop backend_idle_timeout = "0" # Backend inactivity duration before forced connection drop backend_connection_timeout = "0" # Backend connection timeout (ignored in udp) [servers] [servers.sample] protocol = "tls" bind = "0.0.0.0:443" balance = "roundrobin" [servers.sample.discovery] kind = "static" static_list = [ "10.0.0.1:18080 weight=1", "10.0.0.2:18080 weight=1" ] [servers.sample.tls] root_ca_cert_path = "/etc/exampleSSC-CA.crt" cert_path = "/etc/s3.example.com.crt" key_path = "/etc/s3.example.com.key" [servers.sample.healthcheck] fails = 1 passes = 1 interval = "2s" timeout="1s" kind = "ping" ping_timeout_duration = "500ms" [servers.sample2] protocol = "tcp" bind = "0.0.0.0:80" balance = "roundrobin" [servers.sample2.discovery] kind = "static" static_list = [ "10.0.0.1:18080 weight=1", "10.0.0.2:18080 weight=1" ] [servers.sample2.healthcheck] fails = 1 passes = 1 interval = "2s" timeout="1s" kind = "ping" ping_timeout_duration = "500ms" 

The gobetween is started by the following command (add to autostart in any way convenient for you):

 # /usr/local/bin/gobetween -c /etc/gobetween.toml 

Check

For verification, you can use any S3 client, for example, such as s3cmd or DragonDisk. The verification option for s3cmd will look like this (taking into account that the s3.example.com is already specified as the server in the config):

 # s3cmd ls 

If you already have at least some bucket there, then his name will be in the exhaust, if there are no buckets, then there will be an empty exhaust.

How it looks now - you can see on the screen below. Statistics per day (graphics in gigabytes per second):

Results

The load has decreased significantly, there are no blunts left and now all backups have time to pack up for the night (before that, at the height of the working day, it could still be collected).

I hope this hautushka will help you in accelerating and reducing the load on the radosgw