Some Nice Programming with Jenabeans

Writing out SIOC triples using Jena + Jenabean "Jenabean’s model connected programming model makes this easy, using interfaces that declare each of the vocabularies as a set of methods." The code example shows how you can piece together (using a fluent interface) different vocabularies:
Thing thing = new Thing(m);
thing.at(uri1).
as(DCTerms.class).
title("Creating connections between discussion clouds with SIOC").
created("2006-09-07T09:33:30Z").
isa(Sioc.Post.class).
has_container(thing.at(uri2)).
...

Mistakes I've Made a Few

So the ghosts of implementations past have come to haunt me in recent weeks mostly around Kowari/Tucana (now Mulgara).

Firstly, there was exclude and NAF in iTQL:

> It was introduced to provide
> a limited form of negation, and one that interacts poorly with the
> open-world assumption. We also now have minus, which is well
> defined, corresponds closely to our intuitive understanding of the
> operation, and is (I am told) what was actually required.
>
> If my memory is correct we should probably at least deprecate, if
> not remove exclude entirely from mulgara.
>
> Do we agree that exclude should be removed?

Well *I* agree anyway. As you say, it doesn't do what anyone thinks
it does.

> If it should be removed, when should this occur?

Yesterday.

And now adding the Jena API to Kowari cost customers:

Another possible reason is Tucana/Kowari/Mulgara’s Jena support - originally put in to provide a migration path for companies looking to move on from research projects to scalable infrastructure - which as Jena is the defacto semweb tool of choice, people used to evaluate Kowari’s scalability. Jena’s lack of scaling hurt us several times, I can remember lots of frantic calls as some company wrote us off because of our Jena API.

I'm current still working in this area (maybe somewhat surprisingly) and Jena still dominates (all of the tools I'm currently looking at are Jena based). And I still haven't seen a Jena implementation that scales (see page 10). Maybe the decision to open source Kowari cost another round of funding too. Maybe this is why Garlick or Radar Network's triple stores are still behind closed doors.

The State of the Semantic Web

TripCom define their project as: "We will improve the ideas of Tuple Space computing by adding semantics by use of a graph-based data-model to rely on Triples. The Triple Space serves as a persistent publication system for semantically linked information in semantically clustered subspaces. We will develop a scalable and linkable Triple Space storage, based on improving and combining current RDF Stores and Tuple Space infrastructures."

I came across their "State of the art and Requirements Analysis" it mentions YARS, Jena, Sesame, 3store, Kowari, JRDF, Edutella and Oracle. They seem especially keen on YARSQL and its support for provenance. They also seem interested in extending SPARQL to support subqueries and inserting and deleting statements.

Precedence in SPARQL

A recent, "bug report" on the Jena mailing list was sent to me. Basically, it's saying that ARQ doesn't execute SPARQL the way that was expected and JRDF was being held up as correct.

This has to do with the "Nested OPTIONALs" problem, as described by Richard Cyganiak. It basically boils down to the fact that ARQ interprets queries in a left to right, top to bottom fashion; which is the way the SPARQL standard defines it. For example, { <pattern 1> opt { <pattern 2> opt { <pattern 3> } } } should be executed as: take the results of <pattern 1> opt <pattern 2> and then take the results and optionally match <pattern 3>.

This is contradictory to the expression: 6 / (3 - 2), where you perform what's most deeply nested first. To me SPARQL is saying the correct answer is 0 (2 - 2) rather than 6 (6 / 1).

For JRDF to meet SPARQL's requirements I'd either have to reorder the query or change to another compiler compiler, as SableCC is only an LALR parser. Again, not currently a problem for my current JRDF work but if I really want to stamp it SPARQL, when there are test cases that will fail, it will have to be fixed.

Update: This still seems an ongoing issue with the DAWG, "Issues with evaluating optional: Commutativity of AND".

JRDF SPARQL Performance

I did some performance gathering of JRDF's implementation of SPARQL using some of the FOAF data from Mindswap.

Average for Query 1, 100,000 triples:
* Jena (ARQ 0.9.2) - 14685 ms
* JRDF and JRDF using Tuple Subsumption - 3652 ms

There is only one UNION implementation in JRDF.

Average for Query 2, 100,000 triples:
* Jena (ARQ 0.9.2) - 22872 ms
* JRDF - 6615 ms
* JRDF using Tuple Subsumption - 3750 ms

Average for Query 3, 100,000 triples:
* Jena (ARQ 0.9.2) - 15306 ms
* JRDF - 8019 ms
* JRDF using Tuple Subsumption - 4780 ms

The point is not that it was faster that Jena (although yay!), it's that the relational optimisations had a positive effect on querying speed. The current downloadable version of the JRDF SPARQL GUI (0.2) has very slow versions of hash code and equals methods for performance sensitive classes like AttributeValuePair. It still showed the benefit of the optimisations but made it 10 times slower to answer the queries. The modified version of these classes is only available in the JRDF subversion repository.

Semantic Web News

• Paul announced Mulgara 1.0 which has JRDF 0.3.4.3 (hopefully I'll get a chance to upgrade it to 0.4 now it's Java 1.5). Danny picked up on this and JRDF SPARQL GUI releases.


• TopBraid Composer integrates D2RQ. Featuring Google maps integration and RDF Graph visualization. The documentation has more information on it's use of Jena and Pellet and allows ontology development using the Eclipse framework.


• making sesame2 SPARQL protocol conformant Some thoughts on SPARQL support in Sesame and comments on other SPARQL implementations.


• Not really related to the Semantic Web but via the AI3 blog, Data Visualization at Warp Speed, a brilliant presentation on improvements in health and economic development through the last century through visualization and frantic explanation.
  

Three More SW Applications

1. FOAF to hCard via SPARQL "So I used the scutterplan from the FOAFBulletinBoard, going 2 steps, producing 17382 statements, containing I think it was 2035 foaf:Person nodes...To the results I applied sparql2hcard.xsl using xsltproc, script was sparql-hcard.sh (I do hope this was the latest SPARQL XML results format...), producing these hCards." XLST, querying - sounds like descriptors.


2. Timeline "...a DHTML-based AJAXy widget for visualizing time-based events. It is like Google Maps for time-based information."


3. iris: open your eyes to the future of the desktop "Iris is a java open source source integrated desktop environment including email, calendar, file browser, web browser, chat, and a data mining toolkit (clustering, indexing, and lots more)." Uses Jena.

Searching for Links

• Re: Implementations using "compositional semantics" Sesame looking at using relational algebra for implementing SPARQL. This maybe similar to the relational RDF approach. More in the Sesame CVS repository and JRDF Subversion repository.


• DARQ - Federated Queries with SPARQL "It provides transparent query access to multiple, distributed SPARQL endpoints as if querying a single RDF graph. DARQ enables the applications to see a single query interface, leaving the details of federation to the query engine."


• Why the World is ready for the Semantic Web "Current web applications that could be cited as Semantic Web proofs of concept are…" Google, RSS, Tagging and Mash-ups. Somewhat a response to, "The 7 Flaws of the Semantic Web".


• OpenRDF "I don't know the people at Aduna, but frankly if sesame is the best they can produce, it's sad. Looking at the code, it's very clear sesame will run slow and scale like crap. The code isn't as convoluted and crappy as Jena2, but it's not what I consider good code either. The design and implementation is flawed all over and will likely need to be thrown out once people try to use it. So far the semantic web community has managed to produce only crap and doesn't know how to implement a scalable, and fast rule engine."


• The really easy way... "People form, and break...their own social circles, in their own social ways...not Pre-fabricated with all sorts of SPARQLy knobs, and levers, and release valves...Simple tools that make life just a little bit easier at a time is all the world both wants AND needs."

RDF2GO

RDF2GO is an abstraction over triple (and quad) stores. It allows developers to program against rdf2go interfaces and choose or change the implementation later easily.

A common API for Jena and YARS. Includes a comparison with JRDF.

New Kowari Soon

Kowari v1.1 Pre-release Feature Peek "The v1.1 release of Kowari will include the following features:

o The addition of Paul Gearon's Krule (pronounced "cruel") rules engine. Paul has fast and scalable RDFS support implementated (with the exception of XSD data typing) and is working on implementing more rules for an OWL subset.

o Rewriting of ConstraintExpressions and ModelExpressions to collapse Resolver-specific constraints that should be evaluated together into compound constraints (query rewriting) and permitting Resolvers to advise query engine of the evaluation order of constraints to ensure binding of required variables (query annotation). These features were added by Andrae Muys and Simon Raboczi and will form the basis for a cleaner query language syntax and cleaner integration of external Resolvers, such as GIS systems.

o The removal of Jena support. Kowari v1.0 implemented a Jena storage backend and included an RDQL query language API. This has caused difficulties in maintenance and support as the two projects have gone in different directions. Kowari will remove (not deprecate) support for Jena as of the v1.1 release.

o Support for compiling Kowari under Java 1.5. The Kowari team will be moving to Java version 1.5 as the canonical build version, replacing Java 1.4.2."

Via, Kowari v1.1 Pre-release Feature Peek.

Also, new MindSwap Kowari Library part of PhotoStuff.

Keh-nig-it

* "The SEMANTIC Knight always triumphs! Have at you! Come on, then. I have an battalion of KR theorists on my side". Via Too Close To Home
* Google Sponsored Semantic Web API abstraction project "While a lot of semantic web APIs are available (Jena, Sesame, Kowari etc..) especially for the java language, there is no standard set of interfaces and wrappers so that middle ware RDF toolkits or higher level RDF based API can be built regardless of the underlying api/srdf storage. The project will certainly not start from scratch, but rather from earlier discussions and code to build upon (See jrdf, classes in the Simile projects etc..)."
* Tetris Shelving.

Using the Semantic Web to Handle Petascale Datasets

Scientific Data Management in the Coming Decade "Increasingly, the datasets are so large, and the application programs are so complex, that it is much more economical to move the end-user’s programs to the data and only communicate questions and answers rather than moving the source data and its applications to the user‘s local system."

"If the data is to be analyzed by generic tools, the tools need to “understand” the data. You cannot just present a bundle-of-bytes to a tool and expect the tool to intuit where the data values are and what they mean. The tool will want to know the metadata."

"In addition to standardization, computer-usable ontologies will help build the Semantic Web: applications will be semantically compatible beyond the mere syntactic compatibility that current-generation of Web services offer with type matching interfaces. However, it will take some time before high-performance general-purpose ontology engines will be available and integrated with data analysis tools...The XML integration in modern Database Management Systems (DBMS) opens the door for existing standards like RDF and OWL."

This would be something that I would disagree with - existing DBMS are not flexible enough to handle things like RDF. And OWL is even more crucially effected where you need the indexing available from something like Kowari. Other systems like Sesame and Jena are both adding tree structures to store their RDF. Some of that is maintenance and setup but also performance.

"As file systems grow to petabyte-scale archives with billions of files, the science community must create a synthesis of database systems and file systems. At a minimum, the file hierarchy will be replaced with a database that catalogs the attributes and lineage of each file. Set-oriented file processing will make file names increasingly irrelevant – analysis will be applied to “all data with these attributes” rather than working on a list of file/directory names or name patterns."

This is very similar to the TMex V2 architecture that I worked on.

Also Distributed Computing Economics "Computing economics are changing. Today there is rough price parity between (1) one database access, (2) ten bytes of network traffic, (3) 100,000 instructions, (4) 10 bytes of disk storage, and (5) a megabyte of disk bandwidth.".

Via Peta-scale Data Centers - a Trio of Great Jim Gray Papers (from: Kevin Schofield’s Weblog).

New SOFA

SOFA (Simple Ontology Framework API) New home and new release available for download. Updated support for Jena 2, new methods on OntologyModel and ThingModel discarded (will have to change this for Kowari).

AudioMan meets Kowari

AudioMan and Kowari "The only problem is the ID3 parser implementation, I guess it chokes on my iTunes-tagged files. I’m not aware of any Java ID3 lib that gets this right, let’s hope Ryan’s jid3rL can solve this."

"About AudioMan(ager), I’m considering if I should use Kowari. I always liked the idea of storing the AudioManager data as triples, but Kowari looks big and complex to me, at least for the moment. However, Kowari would nicely fit into Durham. I’d imagine the existing MP3 content handler could be modified to use the jid3rL parser, and additional handlers be created for XPSF, M3U, Ogg, etc.

For now AudioManager will stick with the Jena API (as I was already using it and Kowari provides some Jena interfaces anyway), and I’ll get working on MusicBrainz integration."

Crisis

In the same week Tucana goes under we get probably the busiest week on the Kowari mailing list and 45 downloads for Pre-release 2. Considering three quarters of these is the 50MB version, there's either a lot of good bandwidth out there or great patience.

Before the holidays there was great progress being made. We have been attempting to make the "10,000 triple/second challenge" and had succeeded (at least the first 1 million triples, it degraded to a few thousand a second after 200 million). This is on the same 1.6 Mhz Opteron system that we use for all our tests.

Andrae was working on a large refactoring of the transaction operations so that any JRDF, Jena, or iTQL (or anything new) could be put within a transaction or use an existing one. They were already in a transaction but they couldn't tell whether they already had the write phase for example. From what I can remember Simon was working on the problems associated with using file and HTTP protocols in the FROM (external resolvers). David M was working on speeding up deleting triples as well as loading speed. Paul had been working on SOFA and OWL. Robert had started looking at JDO, EJB and mapping relational databases to RDF.

Our commercial focus had been on speed and scalablilty. With Tucana gone some of this focus is probably going to change. It takes a lot of time, effort and resources to keep everything changing and to continue to perform at a commercial level. Between 1.0 and 1.1 there practically isn't a portion of code that wasn't modified - more often than not it was completely changed. This kind of development, on multiple fronts, is something that probably can't occur in the future.

So the future is probably smaller, simpler, with an eye on standards compliance. I hope some things like multiple writes, phase holding, pluggable datatypes, inferencing Hotspot, SPARQL support etc. will be developed but I doubt many of these things will see the light of day now. I'm not discounting them entirely, but there are many things that had to occur in parallel that need a team of developers. A lot of the new features depended on the multiple writers feature. Multiple writers is not an easy feature to describe but suffice to say it's more like Lucene than a normal relational database.

So I think that means improving RDF, RDFS and OWL support. We made some good improvements in Pre-Release 2 with better datatype support. David M added the functionality to allow literal and URI prefix matching - someone just has to add it into the query layer. Combining this with matching nodes based on type (literal, URI or bnode) and trans/walk queries and they're a very powerful combination of features. In the background there's also been a focus on languages (I know we were talking about 3066bis support). Paul and I were looking at inference models (seems to have similarities with pseudo models and datalog vs tableau). Paul's Masters will probably push some of this into reality.

So all in all there does seem to be some good opportunities in the future. I know that Tucana had customers who were paying lots of money for TKS and I know that Kowari did help in the risk assessment. So my intentions, at least in the short term, is to see that good support occurs, bugs get fixed and the like. Although there isn't anyone left to do TKS releases.

I don't really understand why this happened (or what's going on) but to quote Marge Simpson: "One person can make a difference. But most of the time they probably shouldn't."

Visual Browser

Visual Browser is a Java application that can visualise the data in RDF scheme. The main principle of the visualisation is that:

* the triple (resource, resource, resource) is represented by two nodes connected by an edge
* the triple (resource, resource, literal) is represented by a hint (small window appearing on mouse over the subject node)

Visual Browser uses the Jena framework to obtain the data, since the RDF scheme can be saved in different forms (a single XML file or a relational database).

The visualisation engine is derived from TouchGraph LLC."

Only for Java 1.5.

Jena 2.2 beta 1

"This release is primarily a maintenance release. The "beta" designation
means that not all work on Jena 2.2 is completed but also it is a
recognition that a pre-release to the growing Jena user community is
needed before a full release in order to smooth the transition between
versions."

"This release was built with Java 1.5.0 with source and target set to
level 1.4. It has been tested on WindowsXP, Linux and MacOS with a
mixture of Java 1.4.1, 1.4.2, 1.5.0."

Some good stuff from the release notes:
* "Reifier::getHiddenTriples() and getReificationTriples() REMOVED and replaced by iterator..."
* "added new API operation Model::removeAll() which removes "all" statements from a model"
* "a new memory-based Graph, SmallGraphMem, has been introduced."

Available from Sourceforge to download.

Also released was NG4J 0.3.

Provenance to Unify Browsing

A Framework for Unified Information Browsing via RDFX "This Unified Information Service Browser (UISB) project is investigating the means of joining heterogeneous information from multiple sources into a combined and navigable structure...Attempting to unify disparate data sources illustrates that hierarchical and relational data models can place undesirable constraints on the stored data; however, the same data can be stored without constraint in a dynamic graph, which also provides significant potential for enhanced traversal and discovery capabilities."

"The solution that was adopted was to include provenance information for every statement that is added to the graph. Thereafter if a graph, or a statement or a resource is removed from the store, it can be done without the risk of losing information. In addition to identifying where a statement originated, the provenance objects can store metadata describing (among other things) when it was discovered, and when it should expire, therefore enabling the automated management of expired information, or potentially providing an event mechanism to instigate the refresh of the data. The benefit of this solution, is that it does not rely on the graph implementation to ensure that graph merging and un-merging happens as expected, however the added overhead of four extra statements for tracking provenance may negatively affect performance and increase the size of the data store. This will depend on the implementation of the graph store since some stores (e.g. Jena 6.4) can reify statements more efficiently."

SWAP Highlights

From the Semantic Web Applications and Perspectives program:
* Peer-to-Peer Semantic Coordination which considers "...the problem of coordinating hierarchical classifications". A paper on the algorithm listed, CtxMatch, is "Improving CtxMatch by means of grammatical and ontological knowledge – in order to handle attributes".
* RDFGrowth and the Dbin Project. Another paper was published here. Dbin is "..a p2p semantic web algorithm designed so that each single peer is only allowed to cause a minimal, bounded, remote computational burden".
* Semantic Web Tool Evaluation "We have tested the following systems: Jena, Sesame, RdfSuite, Triple, RdfStore, 4Suite, RdfDB....RDF(S) is unable to express basic representation needs typical in simple ontologies...few tools are unable to correctly handle RDF(S) ontologies...no semantic web tool is able to correctly handle even OWL-Lite...the only tool able to correctly handle OWL-Lite ontologies is the description logic inference engine Racer." I'm not sure quite how correct these statements are, maybe with respect to the projects listed, I know we intend for Kowari to support RDFS and OWL-Lite.
* Querying the Semantic Web: a new approach Also talks about using CtxMatch, "...we introduce in the paper a first set of semantic parameters we consider relevant: (i) the type of relation, i.e., the semantic relation that holds between a concept of the source schemas and concepts in the target schemas (e.g., equivalence, greater or lesser generality); (ii) the ontological distance, namely the distance between a source and a target concept with respect to some reference ontology; and (iii) the lexical distance, that is the distance between the formulation of semantically related concepts in different schemas."
* A Framework for Unified Information Browsing Which discusses the RDFX Project.

Kowari 1.0.5 Released

From the release notes:
* iTQL now supports the having clause, exclude constraints, repeated variables in a where constraint, improved subqueries and empty select clauses:
o The having clause allows restrictions to be placed on aggregate functions, such as count.
o exclude imposes a logically opposite match on the graph to normal constraints.
o Repeated variables in the where clause allow, for example, the ability to find statements with the same subject and object values.
o Subqueries now support the use of trans, walk and exclude.
o An empty select clause returns true if the items in the where clause exist in the given graph.
* The existing string pool was rewritten to allow support to add new hard-coded datatypes. The caching was also moved closer to the string pool implementation, increasing load speed by up to 50%.
* Complete rewrite of the Jena support layer. Multiple Jena models/graphs can be created and accessed independently on multiple machines. Fastpath support allows RDQL queries to make use of Kowari's native query handling. A client/server API is added allowing client access to a Jena Model or Graph backed by Kowari. There are also bugs fixes and further enhancements to performance, such as reading RDF files.
* Enhanced JRDF support including a client/server interface, similar to Jena's. A new OWL API, called SOFA is also available. This can operate on any JRDF compliant implementation. Currently this can be in-memory or Kowari.
* N3 file support for importing and exporting of models.
* Improved RMI support including the ability to load or save data from a client to a remote server or from a remote server to a client.

It's been a long time between releases. A Kowari 1.1 preview release should be along next, followed by a Kowari 1.0.6 release.