An Efficient Link Store

• Another web of data store that produces a subset of RDF/XML, Astoria, is from an unlikely source, Microsoft. Instead of a proprietary Semantic Web, Danny sees it as going to town with URIs and REST.


• Silverlight was the other surprising Microsoft development, nothing beats running code - except maybe browser-based dynamic code 2000 times faster. Applets are cool again.


• Some ideas for static triple indexing "Most mature triplestores also index a 4th query element ‘graph’ or ‘context’. I intend to support this query type without expanding the index by using a trick: In my triples format the fact that the subjects are auto-generated and local to the graph means I can choose them to be sequential and effectively re-use them as graph indexes..."


• Plugged In/Invisible Worlds/Tucana/Northrop/TKS/TMex/Kowari/Mulgara podcast (links to the Talis page).


• PAGE a distributed triple store using DHT and YARS (the original). It does seem to miss the DELIS work on P2P RDF which scaled up to 64 nodes.


• Haskell and the Faith of Programming Languages Phillip Wadler gives a rather brilliant talk on programming languages. Covers Haskell, Java generics, combining different typed languages (weak, strong, very strong) as well as monads and Links.

MPTStore

Presentation Summary on MPTStore. A summary of an interesting approach by the Fedora guys to storing lots of triples, fast.

The real motivation behind experimenting with a new triplestore, however, was the NSDL use case. The National Science Digital Library5 (NSDL) is a moderately large repository (4.7 million objects, 250 million triples) with a lot of write activity (driven by periodic OAI harvests; primarily mixed ingests and datastream modifications). The NSDL data model also includes existential/referential integrity constraints that must be enforced. Querying the RI to determine correct repository state proved to be difficult: Kowari is aggressively buffering triple, sometimes on the order of seconds, before writing them to disk. Flushing the buffer after every write is also computationally expensive (hence the drive to use buffers in the first place).

Based on this observation, their solution, called “Mapped Predicate Tables,” creates a table for every predicate in the triplestore. This has several advantages: a low computational cost for triple adds and deletes, queries for known predicates are fast, complex queries benefit from the relatively mature RDBMS planner having finer-granularity statistics and query plans, and flexible data partitioning to help address scalability. This solution comes with several disadvantages, however: one needs to manage predicate to table mapping, complex queries crossing many predicates require more effort to formulate, and with a naive approach simple unbound queries scale linearly with the number of predicates.

They achieved basically the same performance with either asynchronous or synchronous modification.

The project is available on Sourceforge, including slides and javadoc (which has a similar design to JRDF except no blank nodes).

Announcing Andrae

lca: Andrae Muys on RDF "On the first day of linux.conf.au, I ran into Andrae Muys. He hacks Java and RDF for clients who want semantic web hackery done. I have to admit that early Semantic Web hype put me off: it sounded too much like 1970s AI hype. Andrae was interesting, though, and completely free of the wide-eyed uncritical enthusiasm that characterized a lot of my early RDF engagement."

"Andrae runs the Mulgara project, a Java RDF store. His goal is to be able to deal with 1E13 statements (aka tuples, facts, assertions) in three years. It'll do 1E9 right now, next stop is 1E11. He refers to this goal as "3 Ts: three trillion triples". A consortium is forming around Mulgara to make this happen: if it coalesces, Andrae will be the coder to make it happen."

"The next version of Mulgara, 1.3, will ship in February and have this relational mapping in it. A quick Google search shows a lot of RDF-relational mappings going on, but the list of other mappings he had impressed me: Lucene, RSS, mbox, ID3."

"I think it's time I looked again at the world of RDF. They may yet be doing interesting things. I said as much to Andrae and he replied, "I am an engineer. In the early days it was scientists and logicians in RDF. Now the engineers have arrived, and we just want it to work and to scale." Bold claim! If you have a favourite RDF package or practice, let me know in the comments."

Andrae also announced the paper presented at linux.conf.au. Among other things it references David Wood's paper presented in 2004 "Scaling the Kowari Metastore" ("Makepeace" is a good search term).

B, b, b boca gives you an enterprise ready RDF store

IBM Semantic Layered Research Platform announced the release of Boca: "Boca is designed to make it possible to build multi-user, distributed RDF applications" Supports named graphs, security, replication, revision history and JMS notification of changes.

The users guide lists some other interesting features such as a client stack with "a fair amount of compatibility with HP's Jena API" and text indexing using Lucene. Based on the configuration it looks like it requires DB2 or Apache Derby as well as Java 5.

Via, IBM SLRP Release.

The State of the Semantic Web

TripCom define their project as: "We will improve the ideas of Tuple Space computing by adding semantics by use of a graph-based data-model to rely on Triples. The Triple Space serves as a persistent publication system for semantically linked information in semantically clustered subspaces. We will develop a scalable and linkable Triple Space storage, based on improving and combining current RDF Stores and Tuple Space infrastructures."

I came across their "State of the art and Requirements Analysis" it mentions YARS, Jena, Sesame, 3store, Kowari, JRDF, Edutella and Oracle. They seem especially keen on YARSQL and its support for provenance. They also seem interested in extending SPARQL to support subqueries and inserting and deleting statements.

Why have millions of triples? When you can have billions.

Big news, literally, with BigOWLIM. "BigOWLIM is a high-performance semantic repository, implemented in Java and packaged as a Storage and Inference Layer (SAIL) for the Sesame RDF database. BigOWLIM uses the TRREE engine to perform RDFS, OWL DLP, and OWL Horst reasoning, based on forward-chaining of entailment rules. The most expressive language supported is a combination of limited OWL Lite and unconstrained RDFS. BigOWLIM can manage billions of explicit statements on server hardware. A principle limitation of BigOWLIM is the relatively slow delete operation. The upload, reasoning, and the query evaluation proceed fast even against huge ontologies and datasets."

"BigOWLIM successfully passed the threshold of 1 billion (10^9) statements of OWL/RDF – it loaded an 8000-university dataset of the LUBM benchmark and answered the evaluation queries correctly. Evaluation setup and statistics:

BigOWLIM successfully passed the threshold of 1 billion (10^9) statements of OWL/RDF –
it loaded an 8000-university dataset of the LUBM benchmark and answered the evaluation queries correctly. Evaluation setup and statistics:

• Hardware: 2 x Opteron 270, 16GB of RAM, RAID 10; assembly cost < 5000 EURO
  
• OS: Suse 10.0 Linux, x86_64, Kernel 2.6.13-15-smp; 64-bit JDK 1.5 -Xmx12000m
  
• Loading, inference, and storage took 69 hours and 51 min
  
• LUBM(8000,0) contains 1.06 billions of explicit statements
  
  
  
  • The "inferred closure" contains about 786M statements
    
  • BigOWLIM had to manage over 1.85 billions of statements in total
    
  • These figures indicate that, if used as a plain-RDF repository, BigOWLIM can easily handle around 2 billion statements on the same hardware setup
    
  
  
• 92GB RDF/XML files; 95 GB binary storage files
  
• Average Speed: 4 538 statements/sec."

While not available for download, a scaled down version, OWLIM is.

Hot on the heals of recent postings, Scalability of the Semantic Web and performance, triplestores, and going round in circles...

RDF2GO

RDF2GO is an abstraction over triple (and quad) stores. It allows developers to program against rdf2go interfaces and choose or change the implementation later easily.

A common API for Jena and YARS. Includes a comparison with JRDF.

YARS

I thought I'd put a link in as well into my recent Graph and Triple Store to YARS. Lots of good stuff in there (small, HTTP interface, etc). Interestingly, they use JDBM which doesn't use NIO. I evaluated JDBM for blank node tracking but found that it was too slow for Kowari.