Monday, April 19, 2010

Restarting Pentaho Automatically - the quick and dirty way

I look after a Pentaho 2 deployment which manifested a pattern of freezing after about a week. That appears to be a weakness in the code where an object grows bigger and bigger and is not garbage collected, leading the JVM to hang while attempting a full GC.

Not having the expertise to profile the memory usage on the pentaho system, I decided to have the application server restart every few days. The first issue I faced is that the provided stop script - stop-pentaho.sh returns immediately after being invoked and does not guarantee that the application has been stopped.

Compiling jsvc (available in tomcat's bin directory as jsvc.tar.gz) enabled me to have a guaranteed way of stopping tomcat. The following script shows the contents of a script I named stop-tc.sh. This script accepts the tomcat HTTP listen port as the sole argument.


#stop tomcat whose port provided at command line
tmp_file=/tmp/tc-stop.pid

#use lsof to get PID, write to temp file and call jsvc
lsof -a -i TCP:$1 -c java | grep LISTEN | awk '{print $2}' > $tmp_file && /usr/local/bin/jsvc -stop -pidfile $tmp_file org.apache.catalina.startup.Bootstrap


I thereafter turned my attention to the hypersonic database that holds pentaho admin information. There should be a command to shutdown the database, but I decided to kill the process and delete the lock files. The complete restart script, which references the one above, is given below.


#! /bin/bash

export JAVA_HOME=/usr/lib/jvm/java-6-sun
#stop tomcat
/storage/kuali-scripts/stop-tc.sh 8286
#ensure tomcat quits - the file involved is created by stop-tc.sh above
kill -9 `cat /tmp/tc-stop.pid`

#kill hypersonic
kill -9 `lsof -a -i TCP:9001 -c java | grep LISTEN | awk '{print $2}'`

#delete hypersonic log files
rm /storage/financials/pentaho2/biserver-ce/data/hsqldb/quartz.lck
rm /storage/financials/pentaho2/biserver-ce/data/hsqldb/hibernate.lck
rm /storage/financials/pentaho2/biserver-ce/data/hsqldb/sampledata.lck

#start pentaho
/storage/financials/pentaho2/biserver-ce/start-pentaho.sh &> /dev/null