On PostgreSQL, JDO and the shaving of Yaks
Many years ago I performed a couple of mountain-top packet radio experiments. The aim was simply to establish an Internet connection from a very remote area. I did this on two successive years in Kosciuszko National Park, the first time atop Mount Twynam (with Jimbo, Leif and, I think, Kathy Katz), the second time atop the (aptly named!) Paralyser (with Jimbo). Getting the required equipment (~60Kg of gear in two packs, plus an antenna and mast) to the summits took some doing. The second assent was particularly trying because, on five or six occasions, we saw what appeared to be the summit but which, upon reaching it, turned out not to be the summit. It was not until we spotted a trig station on one of these "summits" that we knew that we were approaching the real thing.
I couldn't help but be reminded of this experience when I first encountered the story of Tantalus (origin of the word tantalise), and again more recently when I encountered the term Yak Shaving, although perhaps the latter term is best used when the activities are of various different types and where each step is undertaken to meet a requirement arising from the previous step. Without ever having heard the term, it was pretty obvious to me that one should avoid such things, and so generally I have. It is only over the last several days that I've had a recent experience worthy of this latter term.
I have, for a couple of years, built and maintained an application in Java which uses JDO as its persistence mechanism. The runtime tool on which it depends is Versant's Open Access (Versant purchased it from Hemtech who wrote it initially as JDO Genie). The version that I'm using is quite old but, with the exception of a slow memory leak and some irritating bugs that can be worked around, entirely adequate. Unfortunately, new changes to the application involve tickling the memory leak far more frequently and, as a result, the memory leak is starting to become a concern. I therefore decided to spend some time on resolving it.
In the time that I've used this tool, not only has it changed hands, but the purchasing vendor (Versant) has opened the source; they published it sometime last year under the Eclipse Public License and, at about the same time, donated some of the design-time code to the Eclipse foundation for inclusion in Eclipse. This is actually not a bad thing for me; being able to get my hands on the code provides a path for diagnosing, working around and fixing bugs. Inconveniently, when I heard sometime last year that this had happened, I neglected to retain a copy of the code. Subsequently, Versant has apparently decided to abandon it; during a recent website reorganisation, http://versant.com/opensource/ simply disappeared and, apparently, Versant has no intention of ever re-instating it. Worse, the Eclipse Foundation were only interested in the design-time code (and I'm only interested in the runtime code), so the code releases from Eclipse do not include anything of use to me. As a result, I found myself in the uncomfortable situation of not having access to any way to tackle my problem short of shifting to some other JDO implementation entirely. This is a near-certain long-term approach, but I wasn't quite ready to start down that path. So:
- I asked around, obtained copies of various versions of the opened source, created a SourceForge project to host it lest others have the same "where did it go?" experience that I had, struggled through my first encounter with SourceForge's awful File Release System (by odd coincidence, a SourceForge employee who's on a closed mailing list that I'm also on asked yesterday for comments on the FRS and the bug tracking system because he's about to do some work to improve them; I was in a position to provide detailed feedback, so some good has come of this!) and then returned my attention to trying out the newer code.
- Several small changes were required just to get the code to compile and a deployed version of the code running, but I persisted with these. Then, the real fun began.
- It turns out that, at some stage, Versant switched to using updateable cursors. These are a good idea and provide a worthwhile performance improvement in certain situations, but the particular stack of PostgreSQL 7.4.7, some version of the JDBC driver and OpenAccess 4.0.* yielded an error about this being an unsupported feature.
- Extensive poking around at PostgreSQL turned up nothing clear other than a lot of older mailing list postings about updateable cursors being unsupported or experimental.
- It did seem a little odd to me that Versant would have made this change in such a way as to break interoperability with a supported database so, on a whim, this morning I upgraded to PostgreSQL 8.1 (which meant allowing bits of Debian Sid into my test machine, however...), upgraded the associated JDBC driver and made some other tweaks.
- To my considerable joy, the complaint about unsupported updateable cursors went away, but a new problem materialised. Somewhere in the PostgreSQL+JDBC+OpenAccess stack that we're curently using is a problem with booleans. By the time that they are exposed to Java code, they've been remapped to a Character having the values 't', 'f' or null. I've lived with this inconvenience for a long time and even have a utility class to take care of dealing with it sensibly. Unfortunately the PostgreSQL-8.1+JDBC+OpenAccess-4.0.* stack treats this situation as an error, meaning that I can't even load many of my existing persistent objects because they have Character fields that correspond to BOOLEAN columns in the database.
To deal with this latest problem would require numerous source changes plus a means of dealing with the introduction of bits of Sid into a stable Sarge environment. I have not, however, seen my trig station as yet; that is, I have no way of knowing how many additional problems remain and can't discover them other than by working through the ones that are immediately in front of me.
This then is about the point at which I am declaring the discovery of a yak and pursuing some other means of dealing with the memory leak.
I couldn't help but be reminded of this experience when I first encountered the story of Tantalus (origin of the word tantalise), and again more recently when I encountered the term Yak Shaving, although perhaps the latter term is best used when the activities are of various different types and where each step is undertaken to meet a requirement arising from the previous step. Without ever having heard the term, it was pretty obvious to me that one should avoid such things, and so generally I have. It is only over the last several days that I've had a recent experience worthy of this latter term.
I have, for a couple of years, built and maintained an application in Java which uses JDO as its persistence mechanism. The runtime tool on which it depends is Versant's Open Access (Versant purchased it from Hemtech who wrote it initially as JDO Genie). The version that I'm using is quite old but, with the exception of a slow memory leak and some irritating bugs that can be worked around, entirely adequate. Unfortunately, new changes to the application involve tickling the memory leak far more frequently and, as a result, the memory leak is starting to become a concern. I therefore decided to spend some time on resolving it.
In the time that I've used this tool, not only has it changed hands, but the purchasing vendor (Versant) has opened the source; they published it sometime last year under the Eclipse Public License and, at about the same time, donated some of the design-time code to the Eclipse foundation for inclusion in Eclipse. This is actually not a bad thing for me; being able to get my hands on the code provides a path for diagnosing, working around and fixing bugs. Inconveniently, when I heard sometime last year that this had happened, I neglected to retain a copy of the code. Subsequently, Versant has apparently decided to abandon it; during a recent website reorganisation, http://versant.com/opensource/ simply disappeared and, apparently, Versant has no intention of ever re-instating it. Worse, the Eclipse Foundation were only interested in the design-time code (and I'm only interested in the runtime code), so the code releases from Eclipse do not include anything of use to me. As a result, I found myself in the uncomfortable situation of not having access to any way to tackle my problem short of shifting to some other JDO implementation entirely. This is a near-certain long-term approach, but I wasn't quite ready to start down that path. So:
- I asked around, obtained copies of various versions of the opened source, created a SourceForge project to host it lest others have the same "where did it go?" experience that I had, struggled through my first encounter with SourceForge's awful File Release System (by odd coincidence, a SourceForge employee who's on a closed mailing list that I'm also on asked yesterday for comments on the FRS and the bug tracking system because he's about to do some work to improve them; I was in a position to provide detailed feedback, so some good has come of this!) and then returned my attention to trying out the newer code.
- Several small changes were required just to get the code to compile and a deployed version of the code running, but I persisted with these. Then, the real fun began.
- It turns out that, at some stage, Versant switched to using updateable cursors. These are a good idea and provide a worthwhile performance improvement in certain situations, but the particular stack of PostgreSQL 7.4.7, some version of the JDBC driver and OpenAccess 4.0.* yielded an error about this being an unsupported feature.
- Extensive poking around at PostgreSQL turned up nothing clear other than a lot of older mailing list postings about updateable cursors being unsupported or experimental.
- It did seem a little odd to me that Versant would have made this change in such a way as to break interoperability with a supported database so, on a whim, this morning I upgraded to PostgreSQL 8.1 (which meant allowing bits of Debian Sid into my test machine, however...), upgraded the associated JDBC driver and made some other tweaks.
- To my considerable joy, the complaint about unsupported updateable cursors went away, but a new problem materialised. Somewhere in the PostgreSQL+JDBC+OpenAccess stack that we're curently using is a problem with booleans. By the time that they are exposed to Java code, they've been remapped to a Character having the values 't', 'f' or null. I've lived with this inconvenience for a long time and even have a utility class to take care of dealing with it sensibly. Unfortunately the PostgreSQL-8.1+JDBC+OpenAccess-4.0.* stack treats this situation as an error, meaning that I can't even load many of my existing persistent objects because they have Character fields that correspond to BOOLEAN columns in the database.
To deal with this latest problem would require numerous source changes plus a means of dealing with the introduction of bits of Sid into a stable Sarge environment. I have not, however, seen my trig station as yet; that is, I have no way of knowing how many additional problems remain and can't discover them other than by working through the ones that are immediately in front of me.
This then is about the point at which I am declaring the discovery of a yak and pursuing some other means of dealing with the memory leak.
<< Home