JBoss 7 Cluster - Load Balancing with Apache
In the last article, we configured the replication of user sessions to all nodes of the JBoss cluster. In itself, such an action does not improve fault tolerance, and in order to use the resulting functionality, a load balancer is required, which will distribute external calls between cluster nodes. To distribute the load between the sites will be Apache web server, in accordance with the recommendations in the documentation for JBoss. All the information in the article is available in various sources, but it is scattered, and I did not come across a single resource where everything would be gathered in one place, with a description of the solution of problems that may arise in practice (I personally had it, so I share the recipes) . The article does not claim to be complete, rather the opposite - the minimum configuration is described. It is intended both for specialists who are interested in specific configurations and settings, and for people far from this topic, but who are interested in what clustering is.
Professionals familiar with clustering principles may skip this section . For clarity, I will consider a specific situation that many people encounter: the existing system, consisting of a single application server, switches to a cluster solution. The best option in such cases is not to try to immediately embrace the immense, but to build a minimal working configuration: two application servers in a cluster, and in front of them a web server as an interface (frontend) to clients.
The scheme of work is as follows: all requests previously sent directly to the http / https JBoss connector should now contact the web server (Apache). Apache is configured to “know” that there are two application servers behind it. When the client first accesses the System, the web server selects one of the application servers ( Node-1 ) and redirects the request to it. A session is created, a cookie is added to it, which is then used by the web server to “paste” all subsequent requests of the same client to the selected application server. Node-1 processing client requests, when creating / changing a session, replicates its state to the other nodes of the cluster, where they hang as dead goods, so far without bringing any benefit.
If session replication is already configured on the JBoss side, then no additional configuration is required, everything that is needed ( AJP connector) is already configured in the default configuration. Requires download and install Apache HTTP Server (installation process is beyond the scope of this article). All configuration is done by editing the conf / httpd.conf file.
If before clustering you used encryption when communicating with the client, communicating with it via the HTTPS protocol, then certificate settings were made on the JBoss side, in the HTTPS connector configuration. When switching to a cluster solution, Apache must manage the certificates.
')
Web server settings in conf / httpd.conf:
In addition to the previous configuration, you need to specify the paths to the certificate and private key that are used by your server to encrypt traffic.
The above configuration of Apache 2.2, starting with version 2.4, should add the modules mod_lbmethod_byrequests.so and mod_slotmem_shm.so, and delete NameVirtualHost ads, they are no longer needed.
After applying the solution described in the article, in principle, you can remove the description of the HTTP / HTTPS connectors as unnecessary from the JBoss settings. now all external calls will come through an AJP connector. This will significantly increase server security, especially for scanning utilities that pay special attention to these protocols.
The proposed clustering option partially solves the problem of load balancing and fault tolerance. For a full configuration, there is not enough clustering of DBMS and duplication of a single entry point - Apache. The configuration of the hardware infrastructure is also very important: duplication of networks and routers, server components, uninterrupted power supply and so on. Nevertheless, this article will help develop the system on the 80/20 principle, i.e. through small costs get a significant increase in performance and fault tolerance. In my practice, I was not able to meet the solutions in which we went further, except clustering of the DBMS occurred, but if I happen to encounter, I will definitely write about it. If your services are highly critical, then the solution I proposed is only the first step.
There is a similar article on Habré, I recommend to get acquainted with it as well. Despite the thematic similarity, my article is noticeably different, and, in my opinion, has every right to exist. I tried to lucidly talk about how and at the expense of which everything works, the contents of the configuration and batch files are simply posted there. In addition, in the article by reference, Apache uses mod_jk for balancing, I have mod_proxy . Stackoverflow writes that the main disadvantage of mod_jk is " Need to build and maintain a separate module ". This is exactly what our system administrators have encountered in practice, who have never been able to install this module on Apache for AIX, so if you have a similar problem, you can try the mod_proxy that comes with Apache and, in my opinion, is easier to configure .
General principles of work
Professionals familiar with clustering principles may skip this section . For clarity, I will consider a specific situation that many people encounter: the existing system, consisting of a single application server, switches to a cluster solution. The best option in such cases is not to try to immediately embrace the immense, but to build a minimal working configuration: two application servers in a cluster, and in front of them a web server as an interface (frontend) to clients.
The scheme of work is as follows: all requests previously sent directly to the http / https JBoss connector should now contact the web server (Apache). Apache is configured to “know” that there are two application servers behind it. When the client first accesses the System, the web server selects one of the application servers ( Node-1 ) and redirects the request to it. A session is created, a cookie is added to it, which is then used by the web server to “paste” all subsequent requests of the same client to the selected application server. Node-1 processing client requests, when creating / changing a session, replicates its state to the other nodes of the cluster, where they hang as dead goods, so far without bringing any benefit.
Read more about Sessions.
In the case of the "fall" of the Node-1 , the next time the client accesses, Apache detects that the server is unavailable, and redirects the request to another node ( Node-2 ). This is where the “dead weight” begins to be used - the session, the state of which is already on all nodes of the cluster. The request is processed by Node-2 , and Apache sticks a session to it, i.e. all subsequent user requests are now immediately sent to the Node-2 .Simplified. A session is an object created on the JBoss side with various attributes. A session has an identifier that is sent to the browser when a session is created, and each time the client accesses the server, the browser is sent back to the server (usually in the form of a cookie). The server by identifier finds the session and processes the request in the context of the data of the found session. The task of replication in a cluster is to transfer session data to other nodes so that when a client accesses it, any other node could, just like Node-1, find a session (and data stored in it) by identifier and process the client's request in its context.
Basic configuration
If session replication is already configured on the JBoss side, then no additional configuration is required, everything that is needed ( AJP connector) is already configured in the default configuration. Requires download and install Apache HTTP Server (installation process is beyond the scope of this article). All configuration is done by editing the conf / httpd.conf file.
- Connecting modules . Uncomment or add the following lines:
LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule headers_module modules/mod_headers.so
The mod_proxy.so, mod_proxy_ajp.so, and mod_proxy_balancer.so modules are required for the load balancer. If the system does not have the concept of "session", then they are enough. I, for example, worked with a system that processed requests via web services, these are single hits that are not related to each other, so the sessions were not created. In our case, there are users and their sessions, so you need to add the mod_headers.so module, which allows you to track session data transmitted by the browser. - Adjust balancing . Add the following lines:
NameVirtualHost 192.168.1.0:80 <VirtualHost 192.168.1.0:80> Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED <Proxy balancer://ajp-cluster> BalancerMember ajp://192.168.1.1:8009 route=node1 BalancerMember ajp://192.168.1.2:8009 route=node2 ProxySet stickysession=ROUTEID </Proxy> <Location /> ProxyPass balancer://ajp-cluster/ </Location> </VirtualHost>
Instead of 192.168.1.0, specify the address of Apache.
In the Proxy section, specify the list of addresses of the AJP-connectors of the cluster nodes, one occurrence of BalancerMember for each node. In the default JBoss 7 configuration, the AJP connector listens on port 8009. The route parameter assigns a unique host name used only by Apache itself, i.e. it is not necessary to match the names specified at the start of JBoss ( the jboss.node.name parameter ). There is no need to change anything else, the cluster is ready to be launched and tested.Configuration DetailsThe " Header add Set-Cookie " directive works in conjunction with the " ProxySet stickysession " directive as follows. When the client first accesses Apache, after the balancer chooses which node it will be redirected, a cookie with the name " ROUTEID " is added to the user's session, the value of which is the identifier of the selected node (node1 / node2 in the configuration above). On subsequent client calls, Apache no longer needs to “think” about which node was previously selected, Cookie explicitly tells the web server to which node the request should be redirected to. The “Header” directive is responsible for the existence of the cookie in the session, and the “ProxySet stickysession” directive for using it when routing to “stick” the session to a specific node. If your server does not work with sessions (for example, as in my example above, only web services), then the Header and ProxySet directives are not needed.JSESSIONIDMany configuration examples suggest using the session identifier JSESSIONID as a node token. It seems logical because This identifier is used in most Java web applications. Example with apache.org :ProxyPass /test balancer://mycluster stickysession=JSESSIONID|jsessionid scolonpathdelim=On <Proxy balancer://mycluster> BalancerMember http://192.168.1.50:80 route=node1 BalancerMember http://192.168.1.51:80 route=node2 </Proxy>
Thus, the main differences from the configuration I described are as follows:- In the configuration described in the article, the node ID and the session ID are in different Cookies, here two values ​​are combined into one Cookie.
- In the configuration described in the article, the node identifier is formed on the balancer side (it is logical: who uses it forms), and the session ID is on the application server side (the same principle: the session identifier is needed by JBoss, not Apache). Here both values ​​are generated on the server side of the application (or the web server on the backend).
SSL Setup (HTTPS)
If before clustering you used encryption when communicating with the client, communicating with it via the HTTPS protocol, then certificate settings were made on the JBoss side, in the HTTPS connector configuration. When switching to a cluster solution, Apache must manage the certificates.
')
General principles
The process is as follows: HTTPS requests from the client are sent to the Apache web server, decrypted and transmitted to the application server in open form via the AJP protocol. The response from the application server is also transmitted to the web server in open form, encrypted there and transmitted to the client via the HTTPS protocol. Thus, between the Apache and JBoss there is a so-called “demilitarized zone”, i.e. an environment closed from external unauthorized access in which trusting relationships are configured between servers that do not require additional protection against unauthorized access at the application level. Only Apache should be accessible from the outside, on the HTTPS port (usually, 443), everything else should be tightly closed.
Web server settings in conf / httpd.conf:
- Intercept port 443 . The Apache web server in the default configuration "listens" only to port 80. In order for it to listen on port 443, add the line:
Listen 443 - Connecting modules . Uncomment or add the following line:
LoadModule ssl_module modules/mod_ssl.so - Adjust balancing . Add the following lines:
NameVirtualHost 192.168.1.0:443 <VirtualHost 192.168.1.0:443> Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED <Proxy balancer://ajp-cluster> BalancerMember ajp://192.168.1.1:8009 route=node1 BalancerMember ajp://192.168.1.2:8009 route=node2 ProxySet stickysession=ROUTEID </Proxy> <Location /> ProxyPass balancer://ajp-cluster/ </Location> ProxyRequests off SSLEngine on SSLCertificateFile c:/Apache2/conf/my.cer SSLCertificateKeyFile c:/Apache2/conf/my.key </VirtualHost>
In addition to the previous configuration, you need to specify the paths to the certificate and private key that are used by your server to encrypt traffic.
The above configuration of Apache 2.2, starting with version 2.4, should add the modules mod_lbmethod_byrequests.so and mod_slotmem_shm.so, and delete NameVirtualHost ads, they are no longer needed.
Problems with certificates and their solution
When setting up encryption on the JBoss side, one file was required containing a key and a certificate, for example, in the PKCS # 12 format. Apache requires splitting into two files, in one file a certificate, in the other - the private key. In addition, Apache has two limitations:
All of the above problems are solved using the utility Openssl . From your key store file and the certificate used in JBoss, you should extract the key and certificate into separate files, while converting it to the correct format and removing the password protection. Here is an example of the key conversion from the PKCS # 12 format (other formats are supported, Google will help you):
ATTENTION! Do not forget to delete the received my.key file from your computer after transferring it to the industrial server, since it is unprotected and may be compromised.
- Only PEM is recognized. You can find out if your key matches this format by opening the key file with a text editor. The content should contain the lines "----- BEGIN PRIVATE KEY -----" and "----- END PRIVATE KEY -----", the length of the lines between them should not exceed 64 characters. The same applies to the certificate , with the strings, respectively, "----- BEGIN CERTIFICATE -----" and "----- END CERTIFICATE -----".
- Building Apache under Windows does not know how to open password-protected keystores.
All of the above problems are solved using the utility Openssl . From your key store file and the certificate used in JBoss, you should extract the key and certificate into separate files, while converting it to the correct format and removing the password protection. Here is an example of the key conversion from the PKCS # 12 format (other formats are supported, Google will help you):
openssl pkcs12 -nocerts -nodes -in C:\key.p12 -out C:\Apache2\conf\my.key openssl pkcs12 -clcerts -nokeys -nodes -in C:\key.p12 -out C:\Apache2\conf\my.cer ATTENTION! Do not forget to delete the received my.key file from your computer after transferring it to the industrial server, since it is unprotected and may be compromised.
Conclusion
After applying the solution described in the article, in principle, you can remove the description of the HTTP / HTTPS connectors as unnecessary from the JBoss settings. now all external calls will come through an AJP connector. This will significantly increase server security, especially for scanning utilities that pay special attention to these protocols.
PS
The proposed clustering option partially solves the problem of load balancing and fault tolerance. For a full configuration, there is not enough clustering of DBMS and duplication of a single entry point - Apache. The configuration of the hardware infrastructure is also very important: duplication of networks and routers, server components, uninterrupted power supply and so on. Nevertheless, this article will help develop the system on the 80/20 principle, i.e. through small costs get a significant increase in performance and fault tolerance. In my practice, I was not able to meet the solutions in which we went further, except clustering of the DBMS occurred, but if I happen to encounter, I will definitely write about it. If your services are highly critical, then the solution I proposed is only the first step.
There is a similar article on Habré, I recommend to get acquainted with it as well. Despite the thematic similarity, my article is noticeably different, and, in my opinion, has every right to exist. I tried to lucidly talk about how and at the expense of which everything works, the contents of the configuration and batch files are simply posted there. In addition, in the article by reference, Apache uses mod_jk for balancing, I have mod_proxy . Stackoverflow writes that the main disadvantage of mod_jk is " Need to build and maintain a separate module ". This is exactly what our system administrators have encountered in practice, who have never been able to install this module on Apache for AIX, so if you have a similar problem, you can try the mod_proxy that comes with Apache and, in my opinion, is easier to configure .
Source: https://habr.com/ru/post/190196/
All Articles