Send As SMS

Wednesday, March 29, 2006

On PostgreSQL, JDO and the shaving of Yaks

Many years ago I performed a couple of mountain-top packet radio experiments. The aim was simply to establish an Internet connection from a very remote area. I did this on two successive years in Kosciuszko National Park, the first time atop Mount Twynam (with Jimbo, Leif and, I think, Kathy Katz), the second time atop the (aptly named!) Paralyser (with Jimbo). Getting the required equipment (~60Kg of gear in two packs, plus an antenna and mast) to the summits took some doing. The second assent was particularly trying because, on five or six occasions, we saw what appeared to be the summit but which, upon reaching it, turned out not to be the summit. It was not until we spotted a trig station on one of these "summits" that we knew that we were approaching the real thing.

I couldn't help but be reminded of this experience when I first encountered the story of Tantalus (origin of the word tantalise), and again more recently when I encountered the term Yak Shaving, although perhaps the latter term is best used when the activities are of various different types and where each step is undertaken to meet a requirement arising from the previous step. Without ever having heard the term, it was pretty obvious to me that one should avoid such things, and so generally I have. It is only over the last several days that I've had a recent experience worthy of this latter term.

I have, for a couple of years, built and maintained an application in Java which uses JDO as its persistence mechanism. The runtime tool on which it depends is Versant's Open Access (Versant purchased it from Hemtech who wrote it initially as JDO Genie). The version that I'm using is quite old but, with the exception of a slow memory leak and some irritating bugs that can be worked around, entirely adequate. Unfortunately, new changes to the application involve tickling the memory leak far more frequently and, as a result, the memory leak is starting to become a concern. I therefore decided to spend some time on resolving it.

In the time that I've used this tool, not only has it changed hands, but the purchasing vendor (Versant) has opened the source; they published it sometime last year under the Eclipse Public License and, at about the same time, donated some of the design-time code to the Eclipse foundation for inclusion in Eclipse. This is actually not a bad thing for me; being able to get my hands on the code provides a path for diagnosing, working around and fixing bugs. Inconveniently, when I heard sometime last year that this had happened, I neglected to retain a copy of the code. Subsequently, Versant has apparently decided to abandon it; during a recent website reorganisation, http://versant.com/opensource/ simply disappeared and, apparently, Versant has no intention of ever re-instating it. Worse, the Eclipse Foundation were only interested in the design-time code (and I'm only interested in the runtime code), so the code releases from Eclipse do not include anything of use to me. As a result, I found myself in the uncomfortable situation of not having access to any way to tackle my problem short of shifting to some other JDO implementation entirely. This is a near-certain long-term approach, but I wasn't quite ready to start down that path. So:

- I asked around, obtained copies of various versions of the opened source, created a SourceForge project to host it lest others have the same "where did it go?" experience that I had, struggled through my first encounter with SourceForge's awful File Release System (by odd coincidence, a SourceForge employee who's on a closed mailing list that I'm also on asked yesterday for comments on the FRS and the bug tracking system because he's about to do some work to improve them; I was in a position to provide detailed feedback, so some good has come of this!) and then returned my attention to trying out the newer code.

- Several small changes were required just to get the code to compile and a deployed version of the code running, but I persisted with these. Then, the real fun began.

- It turns out that, at some stage, Versant switched to using updateable cursors. These are a good idea and provide a worthwhile performance improvement in certain situations, but the particular stack of PostgreSQL 7.4.7, some version of the JDBC driver and OpenAccess 4.0.* yielded an error about this being an unsupported feature.

- Extensive poking around at PostgreSQL turned up nothing clear other than a lot of older mailing list postings about updateable cursors being unsupported or experimental.

- It did seem a little odd to me that Versant would have made this change in such a way as to break interoperability with a supported database so, on a whim, this morning I upgraded to PostgreSQL 8.1 (which meant allowing bits of Debian Sid into my test machine, however...), upgraded the associated JDBC driver and made some other tweaks.

- To my considerable joy, the complaint about unsupported updateable cursors went away, but a new problem materialised. Somewhere in the PostgreSQL+JDBC+OpenAccess stack that we're curently using is a problem with booleans. By the time that they are exposed to Java code, they've been remapped to a Character having the values 't', 'f' or null. I've lived with this inconvenience for a long time and even have a utility class to take care of dealing with it sensibly. Unfortunately the PostgreSQL-8.1+JDBC+OpenAccess-4.0.* stack treats this situation as an error, meaning that I can't even load many of my existing persistent objects because they have Character fields that correspond to BOOLEAN columns in the database.

To deal with this latest problem would require numerous source changes plus a means of dealing with the introduction of bits of Sid into a stable Sarge environment. I have not, however, seen my trig station as yet; that is, I have no way of knowing how many additional problems remain and can't discover them other than by working through the ones that are immediately in front of me.

This then is about the point at which I am declaring the discovery of a yak and pursuing some other means of dealing with the memory leak.

Java privates, aren't

I've just discovered something that's been lurking in Sun's Java implementation for many years: runtime checking of access specifiers is turned off for applications (i.e. not applets). According to the Closed, will not be fixed bug 4030988, this was done because:
Some day the offending sort of 1.0 code (non-applets which were compiled -O) will be rare in the field, and we can make "java -verify" the default, with "java -noverify" a compatibility option.
I don't know exactly when 1.1 was released (this code name history suggets that it was more than eight years ago) and I don't recall that the compiler had any way of determining/specifying whether code that you were compiling was going to end up in an applet or not (and setting aside entirely that the -verify option is undocumented), but it seems to me that this "explanation" is more an excuse for a bad decision that someone with some clout didn't feel like revisiting: we don't need no stinking type safety!

Wednesday, March 22, 2006

Customising docbook-xsl output

Docbook-xsl includes some reasonably presentable HTML renderings of docbook sources, but I wanted to place some additional navigation links in the header of the resulting document. The manual helpfully points out that that a series of templates called user.{header|footer}.{navigation|content} are available for this purpose, but fails to explain the need to create something called a customization layer in order to do this. The start of the HTML customisation chapter does refer to these layers, but does so in a slightly roundabout way (instead of "you need to create a layer to use the options in this chapter" it advises you to consider whether you really need one before writing one; knowing neither what nor how, I moved on). A post to the docbook-apps mailing list was answered in under an hour (wow!) by Jörg Möbius who pointed out that a customization layer was required, and even provided a sample. For future reference:
<?xml version='1.0'?>

<xsl:stylesheet xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:import href="/usr/share/xml/docbook/stylesheet/nwalsh/html/chunk.xsl">
<xsl:template name="user.header.navigation">
<div>STUFF</div>
</xsl:template>
</xsl:stylesheet>
(The hard-coded path is reasonable for use on Debian and similar installations of docbook-xsl.)

Monday, March 20, 2006

More NHS

I wrote a while ago about the peculiar discussion that I had when calling the local practice for test results (an update on that: the situation was indeed resolved by talking to a doctor who really could access information that the test results person could not). Today I called to make a followup appointment but, apparently, my doctor is on leave this week. No big problem, let's make an appointment for the day of his return:
  • Receptionist: "Oh no sir, with the new appointment system, you can only make an appointment for the same day, or the next day. You'll have to call back..."
It's magnificent.

Friday, March 17, 2006

As K&R would say...

... or might have, had they not been busy devising a language:

sed -e '/^[when][coders]/!d' -e '/^...[discover].$/d' -e '/^..[real].[code]$/!d' /usr/share/dict/words

(Spotted in Martin f. Krafft's .sig)

Wednesday, March 15, 2006

Grafting in SVN

SVN is a pretty good tool on the whole, but sometimes too much thinking is required.

I had a branch created a while back that needed to use work that I'd merged into the trunk since creating the branch. If I was feeling particularly brave I'd try merging the changes back into my branch and hoping that the merge algorithm would work it all out correctly. I'm not yet that brave.

What I did decide to do was to commit all changes to my branch, delete it, rebranch from the current trunk and then merge the previous changes, confidently assuming that the semantics of SVN's merge are such that this would be a reliable operation and that conflicts would be reported immediately (rather than at eventual merge time, had I chosen to attempt to merge the concurrent changes back into the existing branch).

The assumption was correct, but there were some pitfalls. The final commands were
svn copy -q http://server/repos/trunk http://server/repos/branch -m "comment"

svn co -q http://server/repos/branch

cd branch

svn merge -r 4028:4109 http://server/repos/branch@4028

The magic revolves around those two numbers (4028, 4109) and, in particular, remembering to append @4028. I forgot the latter for the first 30 minutes of struggling with this. I did on some previous occasion get my head around the reason for this appending, but the details escape me at this moment. So:
  • 4028 was the revision at which the branch was previously created using "svn copy". This is most easily discovered with "svn log --stop-on-copy" before deleting and re-creating the branch (because stop-on-copy will then be useful), but can readily be discovered through browsing a much longer log afterwards.
  • 4109 was the revision of the last commit to the branch before the delete (deletion occurred at 4110).
That's all there was to it. Next time...

Friday, March 10, 2006

Four years in the UK

So, as of about 9:30 last night, it's four years since I arrived here.

How time flies!