One is not a warrior in the field: how to create a failover cluster

It often happens that after the launch of an ambitious Internet project and its successful PR in the media, the company expects a large influx of visitors. Unfortunately, our world is not perfect and it so happens that the site does not cope with such a flow of visitors, called in our circles "habraeffektom", and begins to slow down. Accordingly, the company loses money and reputation. In such cases, programmers usually lay the blame on admins, and admins on programmers. It turns out a vicious circle.



What to do if your application began to slow down? One way is to translate it into a cluster architecture . Unfortunately, there are not many instructions and articles that will tell you how to do this. Therefore, we decided to publish a small example of how you can create a failover cluster based on GlassFish . This process and much more is automated in Jelastic , but if you for some reason cannot switch to cloud hosting, then this article is for you.

Let's start in order:

Training

To create a failover cluster, we need: two machines n1 and n2 for GlassFish servers (one of them will have DAS installed and 2 GF servers); lb1 load balancing machine; db1 machine for DBMS. Make sure all the machines “see” each other (ping).

')

All software will run on Linux. We will use a pre-prepared image of a virtual machine with CentOS. GlassFish uses to manage remote SSH nodes, so we will need an installed SSH server on each of the cluster machines. Also, to work on SSH, we will need a separate account in each OS. I propose to call it a uniform for all machines: glassfish.

# adduser glassfish

# passwd glassfish

GlassFish uses non-standard ports and in order to be able to work with it using the admin console in the browser, you must either set the appropriate rules in the firewall or disable it altogether. I propose to do the last:

# service iptables save

# service iptables stop

# chkconfig iptables off

Well and, of course, we cannot do without JVM.

I warn you right away that you will have to tinker decently, since a large number of settings are required.

Cluster configuration

We will conduct the cluster configuration mainly from the command line using the utility from the delivery of GlassFish asadmin

1. Downloading the GlassFish distribution
   
   wget download.java.net/glassfish/3.1.1/release/glassfish-3.1.1.zip
   
   and unpack it.
2. We start an instance of the administrative domain (domain1):
   
   ./asadmin start-domain domain1
   
   GlassFish by default prohibits remote connections to the administrative domain. And all the cluster nodes (Node Agents), except for the node on which the DAS stands, will not be able to interact with it.
   
   Typical error:
   
   Failed to rendezvous with DAS on n1:4848. Please check if this server is running, that the host and port are correct, and that this server is configured to allow remote access.
   
   To get around this we perform:
   
   ./asadmin enable-secure-admin
   
   and restart the administrative domain.
   
   Create the cluster itself:
   
   ./asadmin create-cluster c1 .
   
   Create two instances of the application server located on the same physical machine:
   
   ./asadmin create-local-instance --cluster c1 i1

./asadmin create-local-instance --cluster c1 i2
   
   
3. Run the cluster:
   
   ./asadmin start-cluster c1
4. Add new instances that are on machine n2 (commands are physically executed on n2):
   
   . /asadmin –host n1 –port 4848 create-local-instance –cluster c1 i3

./asadmin –host n1 –port 4848 create-local-instance –cluster c1 i4
5. In the admin console, we find the newly created instance, change its type to SSH, change the value of the Node Host field to n2. At the same time, we need to specify the username and password for SSH access to the n2 machine. Start the cluster. We see that all instances work.

Load balancing and HA

Now, in fact, everything is ready to put the application in a cluster, but for us this is not enough, because load balancing and HA are not provided.

We need to install and configure a load balancer to make it convenient for users to work with the application:

• addressed only one urla;
• their requests were distributed evenly across all instances of the cluster servers;
• in case of failure of a certain number of instances, users would be able to continue working with the application in the normal mode;
• and did not know about the internal "kitchen" of the cluster.

In general, GlassFish “contains” a balancing plugin for “popular” http servers. The words "contains" and "popular" are not in vain taken in quotes, because This plug-in is not contained in the Open Source version of GF, besides, Oracle iPlanet Web Server, Oracle HTTP Server, Apache HTTP Server, Microsoft IIS are supported. We all know that Apache was once good, but now there are more effective solutions.

Among the candidates: NGINX, HAProxy, lighthttpd.

We will support the domestic manufacturer and choose NGINX , which also more and more captures the market.

1. Install NGINX:
   
   #yum install nginx

#nano /etc/nginx/nginx.conf
2. We configure NGINX to balance round-robbin requests for 4 servers of the same weights with support for sticky sessions (below is a part of the config and the server is started).
   
   upstream backend {

ip_hash;

server 192.168.0.1:28080 max_fails=5 fail_timeout=15s;

server 192.168.0.1:28081 max_fails=5 fail_timeout=5s;

server 192.168.0.2:28082 max_fails=5 fail_timeout=5s;

server 192.168.0.2:28083 max_fails=5 fail_timeout=5s;

}
   
   It is worth noting that this is the easiest config that fixes user sessions to a specific server using the ip_hash method
3. In order for HA to work, it is necessary for hosts n1 and n2 to have a common parent domain. For example, cluster.com. To do this, you need to modify the / etc / hosts files on both machines as follows:
   
   n1:
   
   127.0.0.1 localhost

127.0.1.1 n1.cluster.com

127.0.0.1 n1.cluster.com

192.168.0.2 n2.cluster.com
   
   n2:
   
   127.0.0.1 localhost

127.0.1.1 n2.cluster.com

127.0.0.1 n2.cluster.com

192.168.0.1 n1.cluster.com
4. To test the operation of multicast on each physical machine of the cluster we perform:
   
   ./asadmin validate-multicast
5. In the administrative console DAS GlassFish, you must change the reference n2 to n2.cluster.com
6. In order for GlassFish to replicate Http sessions between cluster instances, it is necessary for the application's web descriptor to contain the appropriate instruction (<distributable /> tag) and set the Avalaibility flag when the application is deployed.

Install and configure the database

In this example, we will use PostgreSQL .

Install the DBMS and create a new user and database.

# yum install postgresql-8.4

# service postgresql initdb

# service postgresql start

In PostgreSQL, only local connections are allowed by default. In order to fix this, you need to correct the configuration file /var/lib/pgsql/data/pg_hba.conf, in which you need to specify the allowed subnets: host all all 192.168.0.0/24 md5



Well that's all! Now we can finally deploy the application (for example, we use jforum).





Frankly speaking, after all these settings, my brain almost “boiled over”. We had to “work hard” to automate the whole process described above in Jelastic and turn it into a few mouse clicks.



You can try what came of it on jelastic.com and share your opinion in the comments to the post.