Persistent JRDF

After thinking about it many times I've started to integrate a persistent store for JRDF. Now I know Kowari/Mulgara would be the obvious choice but I decided to try something different - Apache Derby.

A recent post on the Sesame developers list about persistent blank node maps got me into action as well as all this talk about XA2 of course (JRDF's modified RIO parser in Kowari/Mulgara uses a persistent StringToLongMap).

One of the reasons to use Derby is because it has an XAResourceManager. Though I don't expect it to scale or be as fast as most stores (even if the table size is supposed to be unlimited).

I couldn't find an easy way to create a DiskHashtable (although the TestDiskHashtable gave some clues). Here's how I managed to get a persistent DiskHashtable going called derbyDB (it may not be quite right of course):
String driverStr = "org.apache.derby.jdbc.EmbeddedDriver";
EmbeddedDriver driver = (EmbeddedDriver) Class.forName(driverStr).newInstance();
final EmbedConnection30 connection = (EmbedConnection30) DriverManager.getConnection(
"jdbc:derby:derbyDB;create=true");
final LanguageConnectionContext languageConnectionContext = connection.getLanguageConnection();
languageConnectionContext.setRunTimeStatisticsMode(true);
TransactionController controller = languageConnectionContext.getTransactionExecute();
ContextService service = ContextService.getFactory();
service.setCurrentContextManager(languageConnectionContext.getContextManager());
DiskHashtable diskHashtable = new DiskHashtable(controller, TEMPLATE, INDEXES, true, true);


I have some ideas that it might be possible to take the relational RDF operations and put them into Derby (or vice-versa). The datatype support would be nice to leverage (especially the XML datatype for example). It's very preliminary at the moment and I may ditch it in the future.

Announcing Andrae

lca: Andrae Muys on RDF "On the first day of linux.conf.au, I ran into Andrae Muys. He hacks Java and RDF for clients who want semantic web hackery done. I have to admit that early Semantic Web hype put me off: it sounded too much like 1970s AI hype. Andrae was interesting, though, and completely free of the wide-eyed uncritical enthusiasm that characterized a lot of my early RDF engagement."

"Andrae runs the Mulgara project, a Java RDF store. His goal is to be able to deal with 1E13 statements (aka tuples, facts, assertions) in three years. It'll do 1E9 right now, next stop is 1E11. He refers to this goal as "3 Ts: three trillion triples". A consortium is forming around Mulgara to make this happen: if it coalesces, Andrae will be the coder to make it happen."

"The next version of Mulgara, 1.3, will ship in February and have this relational mapping in it. A quick Google search shows a lot of RDF-relational mappings going on, but the list of other mappings he had impressed me: Lucene, RSS, mbox, ID3."

"I think it's time I looked again at the world of RDF. They may yet be doing interesting things. I said as much to Andrae and he replied, "I am an engineer. In the early days it was scientists and logicians in RDF. Now the engineers have arrived, and we just want it to work and to scale." Bold claim! If you have a favourite RDF package or practice, let me know in the comments."

Andrae also announced the paper presented at linux.conf.au. Among other things it references David Wood's paper presented in 2004 "Scaling the Kowari Metastore" ("Makepeace" is a good search term).