Saturday, July 18, 2009

Setting Apache Load Balancer with a cluster of Tomcat servers

Posted by Kris Gemborys


If you want to configure an apache HTTP server as a load-balancer that forwards requests to a cluster of Tomcat servers and ensure that sticky sessions actually stick you are not alone.If you spent hours reading various articles with incomplete and outdated information and your cluster of servers bahaves not exactly the way you want it to, do not dispair just keep reading.


The one important lesson I have learn is that if you successfully configured your first Tomcat cluster, configuring and testing the next one will take you only minutes. Setting load balancing with sticky sessions is actually trivial if you: a) Pay attention to these slashes when entering Proxy Balancer settings b) Understand that an apache HTTP server follows a specific naming convention for cookies used to route HTTP request to back-end servers


Prerequisites:


1) Fedora Linux - Tested with Fedora 8 - Command to check the Linux version: cat /proc/version


2) Apache HTTP Server - Tested with Apache 2.2.6 bundled with Fedora 8 distribution - Command to check Apache version: httpd -V


3) Tomcat server - Tested with Apache Tomcat 6.0.16 - You can download the latest version of Tomcat from: http://tomcat.apache.org/


Word of Caution:


Before making any significant changes to configuration files you need to remember to make backups!


Cluster Topology


The simple cluster used for configuring load balancer consists with a single Apache HTTP server forwarding traffic to two Tomcat servers. All three components were deployed to a single physical server running Linux. The Apache HTTP server is configured to listen on ports 80 and 443, while the Tomcat servers listens on ports 14180 and 15180.


Figure 1. Simplified Topology Diagram



Figure 2. Apache Load Balance Manager Configuration




Configuring the Apache Load-Balancer


You start configuring load balancing by modifying the apache HTTP server configuration - httpd.conf.


The default location of these file is /etc/httpd/conf/httpd.conf


First check if the following modules are enabled:


LoadModule rewrite_module modules/mod_rewrite.so LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_connect_module modules/mod_proxy_connect.so LoadModule cache_module modules/mod_cache.so


Modules mod_cache and mod_rewrite are optional.


Next you need to enable access to load balancer manager by adding the entries:


<Location /balancer-manager> SetHandler balancer-manager


Order Deny,Allow Deny from all Allow from 192.168.1.30 </Location>


The purpose of Deny and Allow entries is to limit access to specific resources and you need to customize these entries according to your needs.


In order to enable load balancing functionality, you will need to configure the proxy module.


Below configuration entries allow only access from your internal none-routable IP addresses.


<IfModule mod_proxy.c>


<Proxy *> Order deny,allow Deny from all Allow from 192.168.1 </Proxy>


ProxyRequests Off


ProxyPass /balancer-manager ! ProxyPass /server-status ! ProxyPass /server-info !


</IfModule>


If you want to enable access to all users you will need to replace the above <Proxy> settings with:


<Proxy *> Order deny,allow Allow from all </Proxy>


The ProxyPass entry with ! is used to specify exclusions, the urls excluded from being forwarded to beckend servers.


Now it is time to configure our cluster:


NameVirtualHost *:80
<VirtualHost *:80>


ServerName www.example.com
ServerAlias example.com
DocumentRoot "/var/www/html"
ProxyRequests Off


<Proxy *>
Order deny,allow
Deny from all
Allow from 192.168.1
</Proxy>


ProxyPass / balancer://maxcluster/ stickysession=JSESSIONID|jsessionid nofailover=On
ProxyPassReverse / http://127.0.0.1:14180/
ProxyPassReverse / http://127.0.0.1:15180/
<Proxy balancer://maxcluster>
BalancerMember http://127.0.0.1:14180 route=node01
BalancerMember http://127.0.0.1:15180 route=node02
ProxySet lbmethod=byrequests
</Proxy>


</VirtualHost>


The above VirtualHost configuration entry is for the domain www.example.com to load balance requests between two Tomcat servers deployed to the same physical server. The Tomcat server identified as node01 monitors the port 14180 for incoming requests (instead of the default 8080), the second tomcat server identified as node02 monitors port 15180. If Tomcat servers are running on different physical servers and using default ports the balancer configuration entries should look as below:


ProxyPassReverse / http://192.168.1.50:8080/ ProxyPassReverse / http://192.168.1.51:8080/ <Proxy balancer://maxcluster> BalancerMember http://192.168.1.50:8080 route=node01 BalancerMember http://192.168.1.51:8080 route=node02 ProxySet lbmethod=byrequests </Proxy>


When entering the Proxy balancer entries you need to pay extra attention to these slashes!


As you can guess the apache server uses route entries to route HTTP requests to appropriate back-end servers. The apache server expects that the route entry id "node01" is part of the JSESSION cookie identifier. In other words, the naming convention for the tomcat cookie is <SessionID>.<route>. Example of the tomcat session JSESSIONID cookie that follows this convention is: 92AF8327CCC933A562FD6B7EFE8DF9C1.node01.


At this time we are done with apche balancer configuration but you probably still wonder how to add rounting identifiers to you Tomcat session cookie identifiers.


Configuring the Cluster of Tomcat Servers


Now it is about time to start configuring your Tomcat apache servers so the JSESSIONID cookie includes the route.


First we should install two Tomcat servers in separate directories (or different servers). If Tomcat servers are running on the same physical servers you need to update the listener ports in the server.xml configuration files to avoid any port conflicts. You will need to examine server.xml fo any port entries i.e.


<Connector port="14180" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="14143" />


In order to make things easier you may consider updating all default port numbers with a new series of ports i.e use a series of 14000 instead of 8000. Again this is only required if you are running multiple tomcat servers on the same physical server. You can use netstat -an command to verify that there are no port conflicts.


Once you have dealt with port entries, you need to take care of routing:


In the first Tomcat server.xml configuration file search for the Engine entry and modify this entry as follows:


<Engine name="Clustered" defaultHost="localhost" jvmRoute="node01">


Next follow the above steps for the second server


<Engine name="Clustered" defaultHost="localhost" jvmRoute="node02">


Testing the Configuration


First you need to restart all servers; one of the ways to accomplish this is to execute the following commands:


service httpd restart


catalina.sh stop


catalina.sh start


You will also need to disable the default SELinux setting which prevents Apache Server from forwarding HTTP requests:


setsebool -P httpd_can_network_connect=1


You can test your cluster using the Tomcat session example:


http://www.example.com/examples/servlets/servlet/SessionExample


Before running the example you should start monitoring you back-end server acccess logs i.e.:


tail -f catalina.out


Now you need to opern a new browser and enter the SessionExample url. Next keep pressing "Submit Query" button. The Session ID should remain the same and the Last Accessed date should be refreshing. You should also see only one log being updated with new messages. Next close the browser and reopen it. Repeat the above steps and you should see the UI is updated with a new session id and the logs of the corresponding server reporting new entries as you keep pressing "Submit Query" button. You should also notice that the Session Id contains routing information.


Well configuring an apache HTTP server as the load-balncer was not that hard after all, or was it?


References


http://www.markround.com/archives/33-Apache-mod_proxy-balancing-with-PHP-sticky-sessions.html


http://wiki.jboss.org/wiki/UsingMod_proxyWithJBoss


http://httpd.apache.org/docs/2.2/mod/mod_proxy.html


Load Balancing Configuration for beck-end Apache servers:


http://www.howtoforge.com/load_balancing_apache_mod_proxy_balancer


Tomcat Apache Engine Reference:


http://tomcat.apache.org/tomcat-5.5-doc/config/engine.html

No comments:

Post a Comment