Tomcat Clustering and a pile of problems

I've been trying to familiarize myself with Tomcat's clustering capabilities today. At the end of it I succeeded but the experience was way less than pleasant. But let's start from the beginning...

At first all seemed really nice: Sunday afternoon, not much to do, some thoughts around playing with Tomcat for the sake of finding out what it's capable of.. Basically the idyll. So I though wouldn't it be nice if I learned how to setup a Tomcat cluster? I already done some horrible things to Tomcat in the past (Single Sing On for instance) but I've never tried clustering. How about that? And so I started what anybody would do - to read the configuration files. No - I'm kidding :) I started to Google for some clustering HOWTO to see if I'm alone in what I'm trying to accomplish. Fortunately it turned out to be a pretty well described topic with some ready-to-copy-and-paste examples. Highly encouraged by what I found myself copying parts of what I found in that miserable HOWTO into configuration of my 2 Tomcats that are supposed to form "the cluster". After about 2 minutes of frantic alt-tabbing, CTRL+C/CTRL+V-ing I arrived at the glamorous point where I was able to start at least one of the Tomcat's to see if the configuration even allows to the server to start. Well, it started, all right. But it didn't work.

Oct 28, 2012 5:14:09 PM org.apache.catalina.ha.deploy.FarmWarDeployer start
SEVERE: FarmWarDeployer can only work as host cluster subelement!

Oh what the heck is this? I though.Wasn't the example from original documentation? I was always banking on those to be really of good quality and now this dissapointment? There's a number of things wrong with the example (and as it turned out with the original server.xml as well). But let's start from the beginning.

server.xml and it's comments

There is a dark place in server.xml that shows how to configure a cluster in Tomcat.

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!-- <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/> -->

So I removed the comment in two instances but it turned out not to be what I expected. Well it did something (not sure what exactly) but it wasn't all that inspiring. There is however a reference to further documentation. I followed that and then the frantic copy/paste activity started. All was quite nice up until I started Tomcat and saw the SEVERE I mentioned before. Back to Google. And then some. Finally I arrived at this discussion where I learned that the <Cluster> element is supposed to be nested within the <Host> element and not where it was suggested by the original server.xml file. Go figure. Anyways, I moved the section down to the Host and this time

Oct 28, 2012 5:01:34 PM org.apache.catalina.ha.deploy.FarmWarDeployer start
INFO: Cluster FarmWarDeployer started.

Great Scott! It will work now! And so I copied my beloved distributable, servlet-api-3.0-enabled hello.war into the /tmp/war-listen folder hoping to see Tomcat spring to work, install the application and make my day. But nothing happened! The example in Tomcat documentation specifies watchEnable="false" by default. What was going through the mind of that someone who wrote it is beyond my understanding of the universe. Anyways, after I changed it to true finally something started to happen.

Since Tomcat ain't for the faint-hearted I was presented with a puzzle:

Oct 28, 2012 5:29:36 PM org.apache.catalina.ha.deploy.FarmWarDeployer fileModified
INFO: Installing webapp [/hello] from [/tmp/war-deploy1/hello.war]
Oct 28, 2012 5:29:36 PM org.apache.catalina.ha.deploy.FarmWarDeployer remove
INFO: Cluster wide remove of web app [/hello]

What the heck? Cluster wide what? I just installed this application, didn't I? I mean thought it will be enough to copy the file in the proper folder and that should be it, should it not?

As it turns out it does work quite nicely if you specify the deployDir to where Tomcat expects to auto deploy applications (by default webapps, or rather ${catalina.base}/webapps). After I changed that, copied the Cluster configuration to the second Tomcat, started both and copied the hello.war application to /tmp/war-listen both Tomcats have received the proper nourishment and started talking my language.

Summary

There's a number of lessons learned from this encounter. First and foremost Tomcat is just awesome. It's so unbelievably small yet so feature-packed. It just makes my day. While debugging Tomcat configuration issues it really helps to start it (at least under Linux) not by ./startup.sh but ./catalina.sh run. This way you'll have the console to view the logs and to terminate it all in case you're not happy with what you see. And last but not least: Google is your friend. I mean it. It is the one thing all this could not be possible without in terms of knowledge sharing.

Have a nice day!