Making ProxyPassMatch work with Location in apache2

I have recently created a configuration where apache2 proxies requests to pentaho running on a tomcat backend. The first problem I faced with the ProxyPassMatch directive is to do with the way the url pointing to the backend is written. Specifically, using a url that includes a non standard port like 8080 may result in a ProxyPass Unable to parse URL error. I fortunately found a workaround for that bug on the apache bug list.

When ProxyPassMatch is enclosed in Location - the first argument - which is a regex pattern, is left out. It is used as an argument to the Location directive instead.

The url provided to the enclosed ProxyPassMatch should be combined with the expected result from regex pattern. This is done using $n where n is count of parenthesized matches starting at number 1.
If the parenthesized matche(s) are not provided to the url argument of ProxyPassMatch, the whole matching string will be concatenated with that url and this may result in an unintended path.

Example

<Location ~ ^/(pentaho.*)$>
    Order Allow,Deny
    Allow from All
    ProxyPassMatch ajp://127.0.0.1:8329/$1
</Location>

Restarting Pentaho Automatically - the quick and dirty way

I look after a Pentaho 2 deployment which manifested a pattern of freezing after about a week. That appears to be a weakness in the code where an object grows bigger and bigger and is not garbage collected, leading the JVM to hang while attempting a full GC.

Not having the expertise to profile the memory usage on the pentaho system, I decided to have the application server restart every few days. The first issue I faced is that the provided stop script - stop-pentaho.sh returns immediately after being invoked and does not guarantee that the application has been stopped.

Compiling jsvc (available in tomcat's bin directory as jsvc.tar.gz) enabled me to have a guaranteed way of stopping tomcat. The following script shows the contents of a script I named stop-tc.sh. This script accepts the tomcat HTTP listen port as the sole argument.

#stop tomcat whose port provided at command line
tmp_file=/tmp/tc-stop.pid

#use lsof to get PID, write to temp file and call jsvc
lsof -a -i TCP:$1 -c java | grep LISTEN | awk '{print $2}' > $tmp_file && /usr/local/bin/jsvc -stop -pidfile $tmp_file org.apache.catalina.startup.Bootstrap

I thereafter turned my attention to the hypersonic database that holds pentaho admin information. There should be a command to shutdown the database, but I decided to kill the process and delete the lock files. The complete restart script, which references the one above, is given below.

#! /bin/bash

export JAVA_HOME=/usr/lib/jvm/java-6-sun
#stop tomcat
/storage/kuali-scripts/stop-tc.sh 8286
#ensure tomcat quits - the file involved is created by stop-tc.sh above
kill -9 `cat /tmp/tc-stop.pid`

#kill hypersonic
kill -9 `lsof -a -i TCP:9001 -c java | grep LISTEN | awk '{print $2}'`

#delete hypersonic log files
rm /storage/financials/pentaho2/biserver-ce/data/hsqldb/quartz.lck
rm /storage/financials/pentaho2/biserver-ce/data/hsqldb/hibernate.lck
rm /storage/financials/pentaho2/biserver-ce/data/hsqldb/sampledata.lck

#start pentaho
/storage/financials/pentaho2/biserver-ce/start-pentaho.sh &> /dev/null

Matching Numeric Ranges with Ruby

regular-expressions.info have dedicated a page to how this can be accomplished using regular expressions alone. This approach is limited in the ranges that can be matched and the patterns add to the complexity of the expression needed to match desired lines.

Realizing this shortcoming of a pure regex approach, I decided to follow a programmatic one with ruby. Here is a command that will display Java VM (run with -verbose:gc) Garbage Cleaning output with a pause time of between 2 and 5 seconds.

cat ./logs/catalina.out | ruby -e 'STDIN.each {|line| puts line if line =~ /\[GC [0-9]+K->[0-9]+K\([0-9]+K\), ([0-9]+\.[0-9]+) secs\]/ and (2..5).include?($1.to_f)}'

Ruby receives a standard input a line at a time and checks it against the regular expression. If a line matches, the part of the pattern enclosed by un-escaped parentheses is made available in the variable $1. That variable is converted to float and compared to the given range. The line is then printed to standard output if the value falls within the range.

Password-less SSH Login

A google search will turn up several good step by step procedures to accomplish this. I realized after a few futile attempts that the file that is mentioned as authorized_keys in many places should actually be authorized_keys2 as described here.

I think the difference in file naming may have occurred due to changes in the OpenSSH tool set and a review of the change logs may be needed to confirm at what version the change was introduced.

Resizing an LVM /usr partition in OpenSUSE 11.1

The yast2 tool makes resizing lvm partitions a very straightforward matter. The only requirement is to unmount the partition being resized. To achieve an unmount of /usr one has to boot to single user mode since there are many process which have open files on that partition. Running lsof /usr | wc -l as root on one instance turned up 2588 files!

It turns out that after running umount /usr successfully in single user mode, yast2 disk fails since there are some files that yast2 needs in the unmounted /usr partition. This left me only with the option of using the console tools. Below is the sequence of commands that I run to add 5G to my /usr partition.

modprobe dm_mod
lvextend -L +5G /dev/mapper/system-usr
shutdown -r 0
resize2fs /dev/mapper/system-usr

The first command to load the device mapper kernel module was necessary since the module was not loaded automatically in my case. I was only able to resize the file system after a reboot because I could not figure out a way to mount the partition in single user mode but I believe it should be possible.

Generating a Tomcat PID file on the fly

I needed to use jsvc to stop tomcat processes, something that required a PID file containing the Process ID of the tomcat being stopped. My first alternative was to use jsvc to start Tomcat and specify the PID file to generate. Looking at the myriad of command line options that jsvc needed, I decided to stick to the provided startup.sh.

This left me with the option of generating the PID file after Tomcat had been started normally. For this, I used the lsof command of which I found an excellent reference. This is the command I used:

lsof -t -l -a -i TCP:$1 -a -c java > $tmp_file && /usr/local/bin/jsvc -stop -pidfile $tmp_file org.apache.catalina.startup.Bootstrap

The line above consists of two commands, one where lsof writes Tomcats PID into a file and the other where jsvc uses that file to shutdown Tomcat. Here is a break down of the options to lsof.

-t output the PID only
-l show ports that are in listening mode
-a and (this enables a boolean relation to be constructed between options)
-i TCP:$1 show a process with an open TCP port held in the shell input variable
-a and
-c java show processes with the name java

Pitfalls of stopping a running script in Oracle SQL Developer

When Oracle SQL Developer is executing a long running script it usually displays a progress bar with a 'cancel' button like the one on the right displayed next to it.

Recently, I erroneously launched a script and hurried to press the said button. Sure enough, the progress bar disappeared and the cancel button was greyed out. Still, I went ahead and pressed the roll back transaction several times.

It was a great shock to me that the 'cancelled' transaction was never stopped, and the data had been posted to the database. The roll back has obviously not worked as well, possibly because SQL Developer considered the script as 'cancelled'.

My two cents on this is that if a transaction that does not commit automatically is started erroneously, let it complete successfully, then roll it back.