Thursday, February 24, 2011

Apache & tomcat loadbalancing one single host machine

An obstacle to running multiple instances is each instance would
have to run on a unique port. Enter Apache's Web server. With the use
of mod_jk, we were able to forward requests to specific hosts and contexts
to the respective running instance of Tomcat.

The following software was used for the purposes of this article.
- Tomcat (Recent Version )
- Apache (Recent Version )
- mod_jk (Recent Version )
- sun-java6-jdk sun-java6-jre (Recent Version )

I am certain some of you are wondering why I used the Java software
development kit (SDK) rather than the Java runtime environment (JRE).
The answer is simple: Tomcat requires tools.jar to compile JSP pages,
and tools.jar is provided in the SDK. If you do not wish to use the SDK,
you need to place tools.jar in $CATALINA_HOME\common\lib.
Assumptions
For the purposes of this article, I assume that you already
have Apache, Java and Tomcat installed. With all of the software listed
above installed, except mod_jk, let's set up our environment.

In order for Tomcat to start, you need to set two environment
variables, JAVA_HOME and CATALINA_HOME. JAVA_HOME should point to the
J2sdk installation directory, and CATALINA_HOME should point to the installation
directory for Tomcat. To make life easier, I have placed the
following lines in /etc/bashrc:

export CATALINA_HOME=/usr/share/tomcat5
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export JRE=/jre

Notice that I have set CATALINA_HOME to /usr/share/tomcat5. Although this is
the case, /usr/share/tomcat5 actually is a symlink to
/usr/share/apache-tomcat. Although you can tell which version of
Tomcat you have installed by running $CATALINA_HOME/bin/catalina.sh
version, I prefer to be able to note the version immediately by
looking at the installation directory.

Note: Before We proceed ...
Make sure all the above software installed properly and the module mod_jk has been loaded sucessfully


Multiple Tomcat instances are possible to create with the use of the CATALINA_BASE environment variable. Each instance uses a common binary distribution but uses its own conf, webapps, temp, logs and work directories.
Each instance also has its own JVM and, thereby, its own memory
pool. If you have defined the maximum memory to be 512MB via JAVA_OPTS,
each instance will attempt to allocate a maximum of 512MB.

Let's proceed now to set up these directories. As I mentioned before,
Tomcat is installed in /usr/share/tomcat5. To keep things somewhat organized,
I created the following folders in /usr: /usr/share/tomcat_instance1,
/usr/share/tomcat_instance2 and /usr/share/tomcat_instance3. It probably is more
appropriate, however, to name these folders based on their purposes or
applications. Remember that each of the three folders will contain
conf, webapps, temp and work directories. 

Configuring the First Instance
Tomcat uses a server.xml configuration file to determine the ports, connector engines and various other "server" configuration options. We are going to copy the installed server.xml from $CATALINA_HOME/conf/server.xml to /usr/share/tomcat_instance1/conf/server.xml. While we are at it, we might as well copy $CATALINA_HOME/con/server.xml to /usr/share/tomcat_instance2/conf/server.xml and /usr/share/tomcat_instance3/conf/server.xml as well.

Tomcat also uses a global web.xml file. By global, I mean it is used
for each instance. The web.xml file provides the default configuration
for each Web application running under the given instance. If an option is not defined in the individual Web application, the default web.xml option is used.
We can copy $CATALINA_HOME/conf/web.xml to /usr/share/tomcat_instance1/conf, /usr/share/tomcat_instance2/conf and /usr/share/tomcat_instance3/conf.

Now we must make some edits to server.xml. First, we need to disable
the Coyote connector. To do this, we comment out the Coyote connector
information. This is an XML file, so it uses the same comment syntax as
HTML. After we are done commenting out the the connector, it should look
something like this:

vim /usr/share/tomcat5_instance1/conf/server.xml

Because this is the first running instance, we do not need to modify
any more of this file. For subsequent instances, we are required to
change the shutdown port and the AJP connector port. The AJP connector
port is the port that Apache uses to forward requests.

Next, copy the servlets-examples file provided with the
installation of Tomcat from $CATALINA_HOME/webapps/servlets-examples to
/usr/share/tomcat_instance1/webapps/servlets-examples. Again, copy the sample
application to /usr/share/tomcat_instance2/webapps and
/usr/share/tomcat_instance3/webapps as well. At this point, the set up of
tomcat_instance1 is complete. We now need to set up the second and third
instances before we pull it all together. Configuring the Second Instance
We already copied server.xml from the installation directory. We now
need to make the same edit to /usr/share/tomcat_instance2/conf/server.xml as we
did for the first instance. That is, comment out the Coyote connector exactly as
we did above.

Additional required edits are to change the SHUTDOWN port from 8005 to
8105. We must change the port from 8005 because the first instance
already is using it. You can change the second instance's port to be
any unused port above 1024, but for simplicity and organization's sake,
let's use 8105. Here is the line as it should be in the file:

Now we must change the AJP connector from 8009 to 8109. Again, this is
required because the first instance already is using 8009.

We need to make the same edits to /usr/share/tomcat_instance3/conf/server.xml as we did for the second instance, except we substitute 8205 for 8005 and
8209 for 8009.

Configuring mod_jk
mod_jk uses a file named workers.properties. I recommend placing this file with the rest of your Apache configuration files.

vi /etc/apache2/workers.properties

  workers.tomcat_home=/usr/share/tomcat5
  workers.java_home=/usr/lib/jvm/java-6-sun
  ps=/
 
worker.list=loadbalancer

# tomcat5_instance1
  worker.tomcat5_instance1.port=8009
  worker.tomcat5_instance1.host=localhost
  worker.tomcat5_instance1.type=ajp13
  worker.tomcat5_instance1.lbfactor=1

# tomcat5_instance2
  worker.tomcat5_instance2.port=8109
  worker.tomcat5_instance2.host=localhost
  worker.tomcat5_instance2.type=ajp13
  worker.tomcat5_instance2.lbfactor=1

# load Balancing

worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=tomcat5_instance1, tomcat5_instance2

worker.list is a comma-separated list of worker names. You could have
Tomcat workers defined later in the file that will not be used. Any worker
defined is not used unless the worker is listed in the worker.list value.
Workers are defined in format of worker.NAMEOFWORKER.type, with the value being the type of connector. All of our workers are of type ajp13. In the
above example, we have defined three workers: worker1, worker2 and worker3.

You may have noticed the host portion of the configuration. This can
be used to configure Apache to forward to Tomcat instances on separate
machines. In fact, this is an option that one might choose to employ in
order to make a site more secure. Because Tomcat and Apache both reside
on the same machine, we use localhost.

Each worker also needs to define the port on which the connector is configured
to work. If you remember, earlier we configured instance1 to listen
on port 8009, instance2 to listen on port 8109 and instance3 to listen on port 8209.
Configuring Apache with mod_jk
To get all of this started, we need to tell Apache where to find the
workers.properties file and where to log mod_jk requests. We also need
to specify the format of the log files and the options specific to
mod_jk. I did this by adding the following lines to httpd.conf; I
placed all mod_jk configuration directives just before the virtual
host declarations:

JkWorkersFile "/etc/httpd/conf/workers.properties"
JkLogFile     "/var/logs/www/mod_jk.log"
JkLogLevel  info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions     +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat     "%w %V %T"

The above options tell Apache to use /etc/httpd/conf/workers.properties
for the worker definitions and to use the /var/logs/www/mod_jk.log
log file. If you are experiencing trouble with mod_jk, adjust the
JkLogLevel to "debug" in order to get more verbose messages. JKLogStampFormat
and JkRequestLogFormat define the logging formats. The option
ForwardKeySize instructs mod_jk to forward the SSL key size along with
the request. ForwardURICompat instructs mod_jk to forward the URL to
Tomcat normally. -FowardDirectories instructs mod_jk not to return a
directory listing from Tomcat.
Configuring Apache to Forward
We use domain1, domain2 and domain3 as our virtual hosts. To pull this
off, we also have to edit /etc/hosts to ensure that domain1, domain2
and domain3 are resolved properly. To do this, we make three
VirtualDirectory declarations, each corresponding to a worker defined
in workers.properties. Below is the VirtualHosts section, followed
by an explanation of the options.

Configuring the Instances to Start at Boot Now that we have most of the configuration complete, it is time to set up the instances to start at boot. This partially is done by creating a bash script and placing it in /etc/init.d. I used the start-up script found
here.
cd /etc/rc5.d
ln -s /etc/init.d/tomcat_instance1  S71tomcat_service1
ln -s /etc/init.d/tomcat_instance2 S71tomcat_service2
ln -s /etc/init.d/tomcat_instance3 S71tomcat_service3

#!/bin/sh
    #
    # Startup script for Tomcat

    JAVA_HOME=/usr/lib/jvm/java-6-sun
    export JAVA_HOME
    export env=prod
    export install_site=datacenter
# Entrues added by Gyani & upendra
    export ONE_MOUNT=zstor/one
    export ONE_SHARE_VM_DIR=/opt/vxgrid/data/one/share/vmdir
# entire below addeb by Gyani

    export CATALINA_BASE=/usr/share/tomcat5_instance1
    export CATALINA_HOME=/usr/share/tomcat5

    start_tomcat=/usr/share/tomcat5/bin/startup.sh
    stop_tomcat=/usr/share/tomcat5/bin/shutdown.sh

    start() {
    echo -n "Starting tomcat: "
    #su -c ${start_tomcat} - hxappusr
    ${start_tomcat}
    echo "done."
    }
    stop() {
    echo -n "Shutting down tomcat: "
    ${stop_tomcat}
    echo "done."
    }

    # See how we were called
    case "$1" in
    start)
    start
    ;;
    stop)
    stop
    ;;
    restart)
    stop
    sleep 10
    start
    ;;
    *)
    echo "Usage: $0 {start|stop|restart}"
    esac

    exit 0

Copy the contents above and place them in /etc/init.d/tomcat_instance1, /etc/init.dtomcat_instance2 and
/etc/init.d/tomcat_instance3. Be sure to change CATALINA_BASE to
the appropriate directory for each script. Now, we need to link to
these in /etc/rc5.d. To create the symlinks, issue the following as root:

Testing the Setup
With all of our configuration complete, it now is time to test our
setup. We begin by starting Apache or restarting if it already is
running. Next, bring up the first instance:

/etc/init.d/apache2 restart
/etc/init.d/tomcat_service1 start

Now, open a browser window and go to https://cloud.demo.com/datacenter/.
If all goes well, you should see something similar to Figure 1.
Figure 1. Checking Your Setup
If you do not see a page similar to the above, look in the
log files. Specifically, check /var/logs/httpd/mod_jk.log and
/opt/tomcat_instance1/logs/catalina.out for any errors that may have
occurred.

If everything looks correct, go ahead and start the remaining
two contexts:

/etc/init.d/tomcat_instance2 start
/etc/init.d/tomcat_instance3 start

You can Quota any virtual host, make sure you are using tomcat worker as "loadbalancer"

# Virtual host

       SSLEngine On
       SSLCertificateFile /etc/apache2/ssl/apache.pem
       SSLCertificateKeyFile /etc/apache2/ssl/apache.key
       ServerSignature off
       ServerAdmin gyani.pillala@hexagrid.com
       ServerName cloud.demo.com
       DocumentRoot /var/www/cloud/
      
               Options FollowSymLinks
               AllowOverride None
      
      
               Options Indexes FollowSymLinks MultiViews
               AllowOverride None
               Order allow,deny
               allow from all
      
    Alias /plugin /var/www/plugin
      
               Order allow,deny
               Allow from all
      
      
              JkMount /datacenter/*    loadbalancer
              JkMount /datacenter      loadbalancer
              JkUnMount /datacenter/*.css loadbalancer
              JkUnMount /datacenter/*.jpg loadbalancer
              JkUnMount /datacenter/*.gif loadbalancer
              JkUnMount /datacenter/*.png loadbalancer
              JkUnMount /datacenter/*.jpeg loadbalancer
      
       ErrorLog /var/log/apache2/cloud-error.log

Posted by at
Labels: ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)