Sesame Native Store

I'm very impressed at the moment with OpenRDF's native store as others have been in the past. One of the best things is how easy it was to work into the existing JRDF code.

As I've said before I've been searching for an on disk solution for loading and simple processing of RDF/XML. In the experiments I've been doing OpenRDF's btree index is much faster than any other solution (again not unexpected based on previous tests). The nodepool/string pool or ValueStore though is a bit slower than both Bdb and Db4o.

Loading 100,000 triples on my MacBook Pro 2GHz takes 37 secs with pure Sesame, 27 with the Sesame index and Db4o value store, 35 with Bdb value store and ehCache is still going (> 5 minutes). A million takes around 5 minutes with Sesame index and Db4o nodepool (about 3,400 triples/second) and 3 minutes with a Sesame index and memory nodepool (about 5500 triples/second).

There's lots of cleanup to go and there's no caching or anything clever going on at the moment, as I'm trying to hit deadlines. 0.5.2 is going to be a lot faster than 0.5.1 for this stuff.

Update: I've done some testing on some fairly low-end servers (PowerEdge SC440, Xeon 1.86GHz, 2GB RAM) and the results are quite impressive. With 100,000 triples averaging around 11,000 triples/second and 10 million averaging 9,451 triples/second.

Update 2: JRDF 0.5.2 is out. This is a fairly minor release for end user functionality but meets the desired goal of creating, reading and writing lots of RDF/XML quickly. Just to give some more figures: Bdb/Sesame/db4o (SortedDiskJRDFFactory) is 30% faster for adds and 10% slower for writing out RDF/XML than Bdb/Sesame (SortedBdbJRDFFactory). Both have roughly the same performance for finds. I removed ehcache as it was too slow compared to the other approaches.

The State of the Semantic Web

TripCom define their project as: "We will improve the ideas of Tuple Space computing by adding semantics by use of a graph-based data-model to rely on Triples. The Triple Space serves as a persistent publication system for semantically linked information in semantically clustered subspaces. We will develop a scalable and linkable Triple Space storage, based on improving and combining current RDF Stores and Tuple Space infrastructures."

I came across their "State of the art and Requirements Analysis" it mentions YARS, Jena, Sesame, 3store, Kowari, JRDF, Edutella and Oracle. They seem especially keen on YARSQL and its support for provenance. They also seem interested in extending SPARQL to support subqueries and inserting and deleting statements.

Semantic Web News

• Paul announced Mulgara 1.0 which has JRDF 0.3.4.3 (hopefully I'll get a chance to upgrade it to 0.4 now it's Java 1.5). Danny picked up on this and JRDF SPARQL GUI releases.


• TopBraid Composer integrates D2RQ. Featuring Google maps integration and RDF Graph visualization. The documentation has more information on it's use of Jena and Pellet and allows ontology development using the Eclipse framework.


• making sesame2 SPARQL protocol conformant Some thoughts on SPARQL support in Sesame and comments on other SPARQL implementations.


• Not really related to the Semantic Web but via the AI3 blog, Data Visualization at Warp Speed, a brilliant presentation on improvements in health and economic development through the last century through visualization and frantic explanation.
  

Searching for Links

• Re: Implementations using "compositional semantics" Sesame looking at using relational algebra for implementing SPARQL. This maybe similar to the relational RDF approach. More in the Sesame CVS repository and JRDF Subversion repository.


• DARQ - Federated Queries with SPARQL "It provides transparent query access to multiple, distributed SPARQL endpoints as if querying a single RDF graph. DARQ enables the applications to see a single query interface, leaving the details of federation to the query engine."


• Why the World is ready for the Semantic Web "Current web applications that could be cited as Semantic Web proofs of concept are…" Google, RSS, Tagging and Mash-ups. Somewhat a response to, "The 7 Flaws of the Semantic Web".


• OpenRDF "I don't know the people at Aduna, but frankly if sesame is the best they can produce, it's sad. Looking at the code, it's very clear sesame will run slow and scale like crap. The code isn't as convoluted and crappy as Jena2, but it's not what I consider good code either. The design and implementation is flawed all over and will likely need to be thrown out once people try to use it. So far the semantic web community has managed to produce only crap and doesn't know how to implement a scalable, and fast rule engine."


• The really easy way... "People form, and break...their own social circles, in their own social ways...not Pre-fabricated with all sorts of SPARQLy knobs, and levers, and release valves...Simple tools that make life just a little bit easier at a time is all the world both wants AND needs."

Why have millions of triples? When you can have billions.

Big news, literally, with BigOWLIM. "BigOWLIM is a high-performance semantic repository, implemented in Java and packaged as a Storage and Inference Layer (SAIL) for the Sesame RDF database. BigOWLIM uses the TRREE engine to perform RDFS, OWL DLP, and OWL Horst reasoning, based on forward-chaining of entailment rules. The most expressive language supported is a combination of limited OWL Lite and unconstrained RDFS. BigOWLIM can manage billions of explicit statements on server hardware. A principle limitation of BigOWLIM is the relatively slow delete operation. The upload, reasoning, and the query evaluation proceed fast even against huge ontologies and datasets."

"BigOWLIM successfully passed the threshold of 1 billion (10^9) statements of OWL/RDF – it loaded an 8000-university dataset of the LUBM benchmark and answered the evaluation queries correctly. Evaluation setup and statistics:

BigOWLIM successfully passed the threshold of 1 billion (10^9) statements of OWL/RDF –
it loaded an 8000-university dataset of the LUBM benchmark and answered the evaluation queries correctly. Evaluation setup and statistics:

• Hardware: 2 x Opteron 270, 16GB of RAM, RAID 10; assembly cost < 5000 EURO
  
• OS: Suse 10.0 Linux, x86_64, Kernel 2.6.13-15-smp; 64-bit JDK 1.5 -Xmx12000m
  
• Loading, inference, and storage took 69 hours and 51 min
  
• LUBM(8000,0) contains 1.06 billions of explicit statements
  
  
  
  • The "inferred closure" contains about 786M statements
    
  • BigOWLIM had to manage over 1.85 billions of statements in total
    
  • These figures indicate that, if used as a plain-RDF repository, BigOWLIM can easily handle around 2 billion statements on the same hardware setup
    
  
  
• 92GB RDF/XML files; 95 GB binary storage files
  
• Average Speed: 4 538 statements/sec."

While not available for download, a scaled down version, OWLIM is.

Hot on the heals of recent postings, Scalability of the Semantic Web and performance, triplestores, and going round in circles...

Lazy Links

* MKSearch Beta 1 Released - includes web crawler, HTML metadata extractor, and RDF storage using Sesame. Also, MG4J (Managing Gigabytes for Java).
* Open Source Java Application Management: BlueGlue and MyJavaPack. A little different to Ivy (also interesting IvyCruise).
* Concurrency JSR-166 Interest Site includes interesting posts like Java Memory Model versus dotnet Memory Model mentions the forthcoming problems with Java code when multi-core systems are rolled out. Also, Concurrent Skip List Map (coming to Java 6).

JRDF 0.3.4.1 Soon

So the JRDF 0.3.4 was replaced last night due to a problem in the Ant build script creating it with source instead of classes - so much for QA.

Following that little discovery a few other problems related to RDF/XML parsing have been fixed: Does not resolve relative URIs correctly and Stacked xml:base directives with relative URIs not processed. The first is due to the difference between the way Java's URI classs resolves and the way its defined in RDF/XML. In section 5.2 of RFC2396 says: "...any characters after the last (right-most) slash character, if any, are excluded." and the RDF/XML specification says: "These specifications do not specify an algorithm for resolving a fragment identifier alone...The empty string is transformed into an RDF URI reference by substituting the in-scope base URI."

The second is a fix from Sesame's RIO.

JRDF 0.3.4

JRDF Comes with the port of Sesame's RIO parser to JRDF, the start of a SPARQL parser and remote querying, bug fixes and other small changes. This is the last version for Java 1.4. The next version will use some of the new 5.0 features like Generics for RDF collections (seems to make sense) and the much touted concurrency libraries.

Keh-nig-it

* "The SEMANTIC Knight always triumphs! Have at you! Come on, then. I have an battalion of KR theorists on my side". Via Too Close To Home
* Google Sponsored Semantic Web API abstraction project "While a lot of semantic web APIs are available (Jena, Sesame, Kowari etc..) especially for the java language, there is no standard set of interfaces and wrappers so that middle ware RDF toolkits or higher level RDF based API can be built regardless of the underlying api/srdf storage. The project will certainly not start from scratch, but rather from earlier discussions and code to build upon (See jrdf, classes in the Simile projects etc..)."
* Tetris Shelving.

Using the Semantic Web to Handle Petascale Datasets

Scientific Data Management in the Coming Decade "Increasingly, the datasets are so large, and the application programs are so complex, that it is much more economical to move the end-user’s programs to the data and only communicate questions and answers rather than moving the source data and its applications to the user‘s local system."

"If the data is to be analyzed by generic tools, the tools need to “understand” the data. You cannot just present a bundle-of-bytes to a tool and expect the tool to intuit where the data values are and what they mean. The tool will want to know the metadata."

"In addition to standardization, computer-usable ontologies will help build the Semantic Web: applications will be semantically compatible beyond the mere syntactic compatibility that current-generation of Web services offer with type matching interfaces. However, it will take some time before high-performance general-purpose ontology engines will be available and integrated with data analysis tools...The XML integration in modern Database Management Systems (DBMS) opens the door for existing standards like RDF and OWL."

This would be something that I would disagree with - existing DBMS are not flexible enough to handle things like RDF. And OWL is even more crucially effected where you need the indexing available from something like Kowari. Other systems like Sesame and Jena are both adding tree structures to store their RDF. Some of that is maintenance and setup but also performance.

"As file systems grow to petabyte-scale archives with billions of files, the science community must create a synthesis of database systems and file systems. At a minimum, the file hierarchy will be replaced with a database that catalogs the attributes and lineage of each file. Set-oriented file processing will make file names increasingly irrelevant – analysis will be applied to “all data with these attributes” rather than working on a list of file/directory names or name patterns."

This is very similar to the TMex V2 architecture that I worked on.

Also Distributed Computing Economics "Computing economics are changing. Today there is rough price parity between (1) one database access, (2) ten bytes of network traffic, (3) 100,000 instructions, (4) 10 bytes of disk storage, and (5) a megabyte of disk bandwidth.".

Via Peta-scale Data Centers - a Trio of Great Jim Gray Papers (from: Kevin Schofield’s Weblog).

Java + SW = Jena

How Big Is Your Store? "The repository will end up holding metadata about more than 16 million articles (plus their associated authors, affiliations, publications, etc) and as you'd imagine thats going to end up exploding into a large number of triples."

"I'd be interested to hear about how big a store people have worked with, including which APIs, etc they've been using.

To give a bit more context, as we're mainly a Java shop I've begun by considering any store that can be plugged into Jena. So the Jena persistence model support would be my baseline, with Kowari being another candidate. I see that RDFStore may be adding Jena support so we may explore that too."

16 million triples shouldn't be a problem for many stores.

I find that Jena does impose some limitations on scalability in that it tends to want to use in memory Models for a lot of things. It also tends to over use iterators. It's also a terribly complicated thing for something, RDF, that is quite simple.

I've been recently thinking that JRDF, Sesame's RIO, Kowari store and SOFA for OWL inferencing would make a small, scalable solution (around 5MB). Is anyone interested in that though?