Apache 2.x + = Tomcat 4.x + Load Balancing (or Private JVMs)

UBEANS

Technologies =

Apache 2.x + Tomcat 4.x
+ Load Balancing (or Private = JVMs)=20

January 24, 2002 by Pascal=20 Forget
Revised September 25, 2002 by Matt Raible
Original = Article=20 at http://www.ubeans.com/tomcat/.= =20

This article contains step by step instructions for configuring = an=20 Apache 2.x web server which handles static content and delegates = JSP (Java=20 Server Pages) and Servlet requests to two Tomcat 4.x servers using = AJP 13=20 connectors and a load balancing worker.

User Submitted Errata

2002-12-11, Robert Dietrick, = re:=20 Tomcat 4.1.x

Tomcat is now distributed with a =
jk2.properties file (in the conf/=20
subdirectory), which needs to be edited in order to change the port=20
number on which the JK2 connector listens.  Regardless of the port you=20
specify in the connector definition in server.xml, you'll still need to=20
change this property file, or else Tomcat will start the listener on the =

default port (8009).

Here are the changes I made:

In each of the tomcat instances' conf/ directory you'll find a=20
jk.properties file.  Open each one in an editor and uncomment and change =

the line which reads
# channelSocket.port=3D8019

to the following for each tomcat instance
tomcat1:
channelSocket.port=3D11009

tomcat2:
channelSocket.port=3D12009

Introduction

Apache 2.0 is a standards compliant, fast and mature web server = which=20 excels at delivering static content such as static HTML pages and = images.=20 The Tomcat web server is great for serving Java Server Pages and = servlets,=20 but it is not as fast as Apache for delivering static content. =

In order to build a fast, scalable web application, the = requirements=20 call for an Apache server that delegates servicing of JSP and = servlet=20 requests to multiple tomcat servers by using an Apache module, = mod_jk,=20 that performs load balancing with session affinity, also known as = "sticky"=20 sessions.

Session affinity explained. When a client browser requests a = JSP page=20 for the first time, the load balancer redirects the request = received by=20 Apache to one of the two tomcat servers; further requests = originating from=20 the same client session will be automatically forwarded to the = same tomcat=20 server, so that the user's session data is retrieved.

This document describes how I configured Apache 2.x to dispatch = JSP and=20 servlet requests to two Tomcat 4.x instances listening on = different ports.=20 This setup was done on a Linux system. Your mileage may vary.

1. Download the required software

Apache 2.x=20 Binaries (If you plan to use ssl, get the source)=20
Tom= cat=20 4.x Binaries=20
The=20 JK module

2. Compile, Install and Configure Apache

2.1 Install Apache.

Linux: gunzip the *.gz you = downloaded,=20 untar and run install-bindist.sh

For *nux,=20 to install Apache 2.0.42 with mod_sll installed, you will need = to=20 compile from source:=20
I used http://httpd.apach= e.org/docs-2.0/install.html=20 as a reference.

$ lynx http://www.= apache.org/dist/httpd/httpd-2.0.42.tar.gz
$=20 gzip -d httpd-2.0.42.tar.gz
$ tar xvf=20 httpd-2.0.42.tar
$ ./configure = --enable-mods-shared=3Dmost=20 --enable-ssl=3Dshared
$ make
$ = make=20 install
Then download mod_jk-2.0.42.so=20 and put it into your modules directory and rename = it=20 mod_jk.so.

Windows: Execute the downloaded executable and install.=20

2.2 Configure the JK Module in httpd.conf

Edit the Apache server's configuration file = httpd.conf=20 which is located in the /usr/local/apache2/conf = directory.=20

2.2.1 Below "# LoadModule foo_module = modules/mod_foo.so", insert=20 the following lines:

#
# Load mod_jk
#
LoadModule jk_module modules/mod_jk.so

#
# Configure mod_jk
#
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info

NOTE: You will need to change=20 mod_jk.so to mod_jk.dll for = Windows.=20

2.2.2 Below the "DocumentRoot" line, insert the = following two=20 lines:

JkMount /*.jsp loadbalancer
JkMount /servlet/* loadbalancer

2.3 Create the workers.properties file

Now we will create a file called worker.properties, and we will = place=20 it under /usr/local/apache2/conf. The worker.properties file tells = Apache=20 about the various Tomcat servers that are running, and on which = port they=20 are listening.

In my setup, I installed the two Tomcat servers in different=20 directories, on the same machine as Apache. Feel free to put your = Tomcat=20 servers on different machines.

I made the first Tomcat server's AJP13 connector listen on port = 11009=20 instead of the default port which is 8009, and the second one = listens on=20 port 12009.

I have decided to name my tomcat servers tomcat1 and tomcat2. = This is=20 purely my choice.

Create the file exactly like = this:

#
# workers.properties=20
#

# In Unix, we use forward slashes:
ps=3D/

# list the workers by name

worker.list=3Dtomcat1, tomcat2, loadbalancer

# ------------------------
# First tomcat server
# ------------------------
worker.tomcat1.port=3D11009
worker.tomcat1.host=3Dlocalhost
worker.tomcat1.type=3Dajp13

# Specify the size of the open connection cache.
#worker.tomcat1.cachesize

#
# Specifies the load balance factor when used with
# a load balancing worker.
# Note:
#  ----> lbfactor must be > 0
#  ----> Low lbfactor means less work done by the worker.
worker.tomcat1.lbfactor=3D100


# ------------------------
# Second tomcat server
# ------------------------
worker.tomcat2.port=3D12009
worker.tomcat2.host=3Dlocalhost
worker.tomcat2.type=3Dajp13

# Specify the size of the open connection cache.
#worker.tomcat2.cachesize

#
# Specifies the load balance factor when used with
# a load balancing worker.
# Note:
#  ----> lbfactor must be > 0
#  ----> Low lbfactor means less work done by the worker.
worker.tomcat2.lbfactor=3D100


# ------------------------
# Load Balancer worker
# ------------------------

#
# The loadbalancer (type lb) worker performs weighted round-robin
# load balancing with sticky sessions.
# Note:
#  ----> If a worker dies, the load balancer will check its state
#        once in a while. Until then all work is redirected to peer
#        worker.
worker.loadbalancer.type=3Dlb
worker.loadbalancer.balanced_workers=3Dtomcat1, tomcat2

#
# END workers.properties
#

That's it, we're done with Apache. For more information on=20 workers.properties and various settings, see the Workers=20 HowTo.

3. Install and Configure the Tomcat Servers

Now let's suppose that Java 1.4.x is installed under=20 /usr/local/jdk1.4.x/. Create two Tomcat 4.x servers and install = them under=20 /usr/local/:

   tar fvxz jakarta-tomcat-4.x.tar.gz
   mv jakarta-tomcat-4.x /usr/local/tomcat1=20
   cp -R /usr/local/tomcat1 /usr/local/tomcat2

In both /usr/local/tomcat1 and /usr/local/tomcat2, the same = files will=20 be modified. I here by present the modifications made to the files = contained in the /usr/local/tomcat1 directory tree structure. You = should=20 also apply the same changes to the corresponding files located = under the=20 /usr/local/tomcat2 directory tree structure.

3.1 Modify catalina.sh

In my many years of consulting, I have learned not to rely on=20 environment variables which can be unset by ignorant or malicious = people.=20 This is why I explicitely set the JAVA_HOME and CATALINA_HOME = variables=20 directly in the catalina.sh file.

At line 32, before the "# ----- Verify and Set Required = Environment=20 Variables " line, insert the following two lines:

    =
JAVA_HOME=3D/usr/local/jdk1.4 ; export JAVA_HOME
    CATALINA_HOME=3D/usr/local/tomcat1 ; export CATALINA_HOME

(Set CATALINA_HOME to /usr/local/tomcat2 in=20 /usr/local/tomcat2/conf/catalina.sh)

3.2 Modify conf/server.xml

3.2.1 Add a unique jvmRoute to the Catalina engine =

Near line 100, replace:

    <Engine =
name=3D"Standalone" defaultHost=3D"localhost" debug=3D"0">

with:

    <Engine jvmRoute=3D"tomcat1" =
name=3D"Standalone" defaultHost=3D"localhost" debug=3D"0">

For tomcat2, put jvmRoute=3D"tomcat2".

3.2.2 Change the control port

At line 13, replace:

    <Server port=3D"8005"

with:

    <Server port=3D"11005"

For the tomcat2 server, replace port 8005 with 12005. This will = prevent=20 the two servers from conflicting.

3.2.3 Change the AJP13 port

At line 75, in the AJP 13 connector definition, replace: =

    port=3D"8009"

with:

    port=3D"11009"

For the tomcat2 server, replace port 8009 with 12009.

3.2.4 Disable the standalone HTTP port

We don't want or need our tomcat servers to directly respond to = HTTP=20 requests. So we comment out the HttpConnector section between = lines and 58=20 in the server.xml file.

Example:

<!-- Define a non-SSL HTTP/1.1 Connector =
on port 8080 -->
<!--
    <Connector =
className=3D"org.apache.catalina.connector.http.HttpConnector"
               port=3D"8080" minProcessors=3D"5" maxProcessors=3D"75"
               enableLookups=3D"true" redirectPort=3D"8443"
               acceptCount=3D"10" debug=3D"0" =
connectionTimeout=3D"60000"/>
-->   =20

NOTE: If you don't comment = this out, you=20 will need to change the port numbers so they don't conflict = between tomcat=20 instances.

3.2.5 Disable the WARP connector

At line 314, comment out the = <Connector...WarpConnector...> tag.=20

Example:

<Service name=3D"Tomcat-Apache">
<!--
    <Connector =
className=3D"org.apache.catalina.connector.warp.WarpConnector"
     port=3D"8008" minProcessors=3D"5" maxProcessors=3D"75"
     enableLookups=3D"true" appBase=3D"webapps"
     acceptCount=3D"10" debug=3D"0"/>
-->

Do not forget to do the same thing to tomcat2's server.xml = file.

NOTE: You might want to comment out the = entire=20 <Service name=3D"Tomcat-Apache"> element. If so, make sure = and remove=20 the comments within it - XML doesn't like comments within = comments.

3.3 Create test JSP pages (index.jsp)

3.3.1 Create a file named index.jsp and put = it in=20 the /usr/local/tomcat1/webapps/ROOT directory: =

<html>
<body bgcolor=3D"red">
<center>
<%=3D request.getSession().getId() %>
<h1>Tomcat 1</h1>
</body>
</html>

3.3.2 Create a file named index.jsp and put = it in=20 the /usr/local/tomcat2/webapps/ROOT directory: =

<html>
<body bgcolor=3D"blue">
<center>
<%=3D request.getSession().getId() %>
<h1>Tomcat 2</h1>
</body>
</html>

4. Start Tomcat1, Tomcat2 and Apache

    =
/usr/local/tomcat1/bin/startup.sh
    /usr/local/tomcat2/bin/startup.sh
    /usr/local/apache2/bin/apachectl start

5. Test your Installation

Now is the time to test your setup. First, verify that Apache = serves=20 static content.

Click on: http://localhost/. = You should=20 see the default Apache index.html page.

Now test that tomcat (either Tomcat 1 or Tomcat 2) is serving = Java=20 Server Pages.

Click on: http://localhost/index.jsp =

If you get a red page, the page was served by the tomcat1 = server, and=20 if you get a blue page, it was served by the tomcat2 server.

Now test that session affinity - also known as sticky sessions = - works=20 within the load balancer. Hit the reload button of your web = browser=20 several times and verify that the index.jsp page you get is always = received from the same tomcat server.

6. Configuring Private JVMs

If you don't need load-balancing, but you are interested in = configuring=20 Apache/Tomcat for private Tomcat instances, you can add one of the = following near the end of httpd.conf:

6.1 Name-based (1 IP address or NIC). =

NameVirtualHost *

<VirtualHost *>
ServerName localhost1
JkMount /*.jsp tomcat1
JkMount /servlet/* tomcat1
</VirtualHost>

<VirtualHost *>
ServerName localhost2
JkMount /*.jsp tomcat2
JkMount /servlet/* tomcat2
</VirtualHost>

6.2 IP-based (different IP for each site). =

# First Virtual Host.
#
<VirtualHost 192.168.0.1:80>
ServerName localhost
JkMount /*.jsp tomcat1
JkMount /servlet/* tomcat1
</VirtualHost>

# Second Virtual Host.
#
<VirtualHost 192.168.0.2:80>
ServerName localhost2
JkMount /*.jsp tomcat2
JkMount /servlet/* tomcat2
</VirtualHost>

Where the serverNames are fully-qualified host names in a DNS = Server.=20 More information can be found at http://httpd.apache.org= /docs-2.0/vhosts/.=20

NOTE: When using SSL with multiple = Virtual Hosts,=20 you must use an ip-based configuration. This is because SSL = requires you=20 to configure a specific port (443), whereas name-based specifies = all ports=20 (*). You might the following error if you try to mix name-based = virtual=20 hosts with SSL.

[error] VirtualHost _default_:443 -- mixing * ports and = non-*=20 ports with a NameVirtualHost address is not supported, proceeding = with=20 undefined results

Starting Apache and Tomcat on Startup

To start Apache and Tomcat on startup in a *nix environment, = see http://www.r= aibledesigns.com/tomcat/boot-howto.html.=20

On Windows, you can install Tomcat as a service.

Supplemental Information

Question 1:
Why did you choose to use the AJP13 = connector=20 rather than the WARP connector that is recommended?

Answer:
The warp connector is used in conjunction = with=20 mod_webapp, and mod_webapp does not currently support load = balancing.

Also, I found the documentation for the warp connector on the = Jakarta=20 web site to be quite lacking. See: http://jakarta.apache.org/tomcat/tomcat-4.0-doc/config/warp.html=20

I know that the future lies in the warp connector, but in the = meantime,=20 I needed something. The documentation did not explain to me = exactly what=20 benefits I would get from using the Warp connector as opposed to = AJP13.=20

Question 2:
You might specify that creating two = instances of=20 the tomcat installation is not needed as you can share the main = binaries=20 and libs by specifying 2 distinct CATALINA_BASE variables.

True, but in real life the two tomcat servers are usually = located on=20 two different machines. My setup might be overkill for a single = machine=20 setup, but it's easy to tar up the "tomcat2" server and put it on = a second=20 machine; you just have to change "localhost" to the appropriate = machine=20 name in /usr/local/apache2/conf/workers.properties and you're = done.

Question 3:
What does not work and what does work in = load=20 balancing?

Answer:
Load balancing works great.

1. Session affinity works
Which means that when a = client=20 browser is directed to a Tomcat server by the load balancer, then = future=20 queries from that same browser session will always be directed to = the same=20 tomcat server. This is important because sessions that are created = in a=20 specific tomcat server, say "tomcat1", do not exist in the other = server=20 "tomcat2", and thus if the client was directed to another tomcat = server=20 than the one where his session is stored, then all his session = data would=20 be lost.

Some people are working on sessions that will be replicated = across all=20 tomcat servers in the cluster, so I'll just wait for it to become=20 available rather than make a homebrewed distributed session = mechanism.=20

The downside of not having sessions replicated across all the = tomcat=20 servers in the cluster is that if one tomcat server dies, all the = sessions=20 that it contained are lost, which usually makes a lot of unhappy = users.=20

2. Failover works
If one tomcat server dies, the = load=20 balancer then "rebalances" the queries to the remaining tomcat = servers.=20

3. Failback works
When a tomcat server comes back = from the=20 dead, the load balancer automatically starts to send queries to = it. So you=20 can actually add capacity to your cluster on the fly.

4. Weighted load balancing works
In=20 /usr/local/apache2/conf/workers.properties, I assigned a load = balancing=20 factor of 100 to both "tomcat1" and "tomcat2" servers. Then I = changed the=20 lbfactor of "tomcat1" to 101, and I saw that effectively the = "tomcat1"=20 server received more load than the "tomcat2" server, which is = something=20 you want when for example your "tomcat1" server is a faster/newer = machine=20 while your "tomcat2" server is a slower machine which cannnot take = as much=20 load as the other one.

References

For more information, you should read An = Apache=20 Load Balancing Cluster. It talks about mod_jserv, which is now = mod_jk,=20 and it uses JServ instead of Tomcat, but the concepts are still = valid.=20

Conclusion

The list of steps that are required to obtain a scalable web=20 application solution based on Apache 2.x and a group of distibuted = Tomcat=20 servers are well-defined and if you follow the receipe exactly, = you should=20 be able to achieve success.

I hope that this article will be helpful to you. Good Luck. =

Pascal Forget =

These = instructions where=20 tested by Matt = Raible on=20 Windows XP (SP1) / Red Hat Linux 7.3 with J2SE 1.4.1, Apache = 2.0.40 (for=20 Linux, 2.0.42 for Windows), and Tomcat 4.0.5.=20