Many things have changed since I started JRDF in 2003. It feels like JRDF has come to a natural conclusion.
Some of them are things I've failed to do very well: get contributors, implement different file format parsers, find enough time to refactor existing bits, etc. I'm also not that interested in Java anymore (as was becoming increasingly obvious as it did have Scala in there at one point and has some Groovy DSL code in there).
The most recent change I've seen is that JSON has achieved some of what RDF was trying to do and I see it more and more in the way people use it to expose their data in a RESTful way. The tooling is less onerous and the ease of use is higher even if what you get is much less.
Also, external factors like W3C's official RDF API (for Java and Javascript) is largely the same thing but with official backing.
I've enjoyed developing it and meeting and talking to other people in other groups (especially Jena and Sesame). And of course, none of this would've happened if it wasn't for a lot of other people: Paul Gearon, Simon Raboczi, David Wood, David Makepeace, Tom Adams, Yuan Fang-Li, Robert Turner, Brad Clow, Guido Governatori, Jane Hunter, Imran Khan and Abdul Alabri and the other guys and girls Tucana/Plugged In Software/UQ.
Showing posts with label jrdf. Show all posts
Showing posts with label jrdf. Show all posts
Sunday, May 08, 2011
Tuesday, December 29, 2009
Syntax Hell
As anyone who has applied a more functional style to Java will realize, the Java syntax really gets in the way. I was using my usual guinea pig, JRDF, with "with" so you don't have to iterate but apply a function and so you don't have to close a ClosableIterator. The typical code to print out the graph is:
Using "ClosableIterators.with" this becomes:
ClosableIterable<Triple> triples = graph.find(ANY_TRIPLE);
try {
for (Triple triple : triples) {
System.out.println("Graph: " + triple);
}
} finally {
triples.iterator().close();
}
try {
for (Triple triple : triples) {
System.out.println("Graph: " + triple);
}
} finally {
triples.iterator().close();
}
Using "ClosableIterators.with" this becomes:
with(graph.find(ANY_TRIPLE), new Function<Void, ClosableIterable<Triple>>() {
public Void apply(ClosableIterable<Triple> object) {
for (Triple triple : object) {
System.out.println("Graph: " + triple);
}
return null;
}
});
It's typically one line less but that's not that much of an improvement.
public Void apply(ClosableIterable<Triple> object) {
for (Triple triple : object) {
System.out.println("Graph: " + triple);
}
return null;
}
});
Saturday, November 21, 2009
JRDF 0.5.6 and testing
JRDF 0.5.6 has JSON SPARQL support and many things got a good refactoring including the MergeJoin (which now looks more like typical text book examples - for better or worse). The upgrade to Java BDB version 4 has improved disk usage and performance mostly around temporary results from finds and the like. The next version will include named graph support.
I'm considering this being the last version to support Java 5.
This release was also about learning things like Hamcrest, JUnit 4.7, various mock extensions (Unitils, Powermock and Spring automocking (which didn't make it due to not being able to mix and match runners). This seems to be a design flaw that I've been encountering with JUnit - you can't mix and match features from various runners. Even Powermock and JUnit's Rules (for exceptions anyway) was problematic. The answer was to go back to the inner class block version.
I'm considering this being the last version to support Java 5.
This release was also about learning things like Hamcrest, JUnit 4.7, various mock extensions (Unitils, Powermock and Spring automocking (which didn't make it due to not being able to mix and match runners). This seems to be a design flaw that I've been encountering with JUnit - you can't mix and match features from various runners. Even Powermock and JUnit's Rules (for exceptions anyway) was problematic. The answer was to go back to the inner class block version.
Sunday, July 12, 2009
JRDF 0.5.5.5
It's been a long while between updates but it's finally here. There's been some general concessions made to long standing "features" in JRDF namely relational semantics and checked exceptions - both are gone. Yuan-Fang added merge-join support which improved join performance (by up to 8 times). There's Groovy support and a nasty memory leak fixed. It's in the usual place. The next version won't be so far away with some further SPARQL query improvements including perhaps some of the newly proposed features.
Thursday, December 04, 2008
Getting Groovy with JRDF
In an effort to speed up and improve the test coverage in JRDF I've started writing some of the tests in Groovy. It's been a good experience so far - so much so that I'm probably not going back to writing tests in Java in the future.
One of the things I wanted to try was a RdfBuilder, this is similar to Groovy's NodeBuilder.
There are a couple of things that make it a bit tricky. When parsing or debugging builders I haven't yet found a way to find the methods/properties available even using MetaClass. And of course, when the magic goes wrong it's a bit harder to debug Groovy versus java.
It certainly smartens up the creation of triples, for example (bits from the NTriples test case):
Using the builder results in a file that's smaller than the test case file. You could remove some duplication by creating a method that takes in a number and the object and generates "eg:resource$number" "eg:property" "$object" but doing that may actually make it harder to read.
If you stick to only using URIs you can do things like:
I expect that JRDF will only be more Groovy friendly in the future.
One of the things I wanted to try was a RdfBuilder, this is similar to Groovy's NodeBuilder.
There are a couple of things that make it a bit tricky. When parsing or debugging builders I haven't yet found a way to find the methods/properties available even using MetaClass. And of course, when the magic goes wrong it's a bit harder to debug Groovy versus java.
It certainly smartens up the creation of triples, for example (bits from the NTriples test case):
def rdf = new RdfBuilder(graph)
rdf.with {
namespace("eg", "http://example.org/")
namespace("rdfs", "http://www.w3.org/2000/01/rdf-schema#")
"eg:resource1" "eg:property":"eg:resource2"
"_:anon" "eg:property":"eg:resource2"
"eg:resource1" "eg:property":"_:anon"
(3..6).each {
"eg:resource$it" "eg:property":"eg:resource2"
}
"eg:resource7" "eg:property":'"simple literal"'
"eg:resource17" ("eg:property":['"\\u20AC"',
'"\\uD800\\uDC00"', '"\\uD84C\\uDFB4"', '"\\uDBFF\\uDFFF"'])
"eg:resource24" "eg:property":'"<a></a>"^^rdfs:XMLLiteral'
"eg:resource31" "eg:property": '"chat"@en'
}
The first two lines defines the two namespaces used. The third line shows the general use of RDF and Groovy. It works out well, an RDF predicate and object maps to an attribute and value in Groovy. The next two lines show how you refer to the same blank node across two statements. And the following lines show using ranges and creating different types of literals. The third last line creates 4 triples with the same subject and predicate but with different objects. Using the builder results in a file that's smaller than the test case file. You could remove some duplication by creating a method that takes in a number and the object and generates "eg:resource$number" "eg:property" "$object" but doing that may actually make it harder to read.
If you stick to only using URIs you can do things like:
rdf.with {
urn.foo6 {
urn.bar {
urn.baz1
urn.baz2
}
}
}
Which produces two triples: "urn:foo6, urn:bar, urn:baz1" and "urn:foo6, urn:bar, urn:baz2".I expect that JRDF will only be more Groovy friendly in the future.
Wednesday, October 22, 2008
Time for REST
The best part of the latest release of JRDF is not code that we've written but Restlet. Restlet provided an excellent abstraction for providing RESTful web services and allows deployment without a container (the distributed and local server are both about ~6MB in size).
This has allowed us to quickly develop a web service that answers SPARQL queries that can be from one or many machines. We're using it to query the results from MapReduce tasks in our Hadoop cluster but it could probably be used as a general way to query other SPARQL services.
It's still early days and there's a lot that still needs to be added (and re-added) such as limiting result sizes, being able to choose which format to return and better JSON support. A lot of it has been pushed out in time to show a front-end at SSWS.
It would be good to have a Javascript client that submits the SPARQL queries and handles the ordering and such rather than having the distributed query server hang onto all results until returning.
Download it here.
This has allowed us to quickly develop a web service that answers SPARQL queries that can be from one or many machines. We're using it to query the results from MapReduce tasks in our Hadoop cluster but it could probably be used as a general way to query other SPARQL services.
It's still early days and there's a lot that still needs to be added (and re-added) such as limiting result sizes, being able to choose which format to return and better JSON support. A lot of it has been pushed out in time to show a front-end at SSWS.
It would be good to have a Javascript client that submits the SPARQL queries and handles the ordering and such rather than having the distributed query server hang onto all results until returning.
Download it here.
Monday, August 11, 2008
JRDF is very crap (part 2)
It didn't take as long as expected which means it's probably wrong. For non-filtered queries it's three times faster and for certain FILTER queries (with equals) it's 47 times faster (from 284 to 6 seconds). At least it's now in the same order of magnitude as most tools and it's a a tiny bit faster than some (although adding more features will probably slow it down again).
What changed:
* AttributeValuePair has been removed and replaced with maps (as discussed previously).
* Seeing as though maps were used so much hashCode and equals were optimized. As I've found before (I think), isAssignableFrom is slower than try/catch for equals (depending on your usage of course).
* Queries go through unsorted and uncopied rather than standard graph finds. I'd forgotten about how much effort had gone into allowing remove and automatic sorting on iterators.
* A very simple optimizer (it's really only simplifying the FILTER constraints at the moment) was added. Tree manipulation was painful - I resorted to mutating in place operations.
* Better designed. It's a bit hard to qualify this except what was there was truly awful - objects being created in constructors and passing itself in. The nice thing about IoC is it's quite easy to see when you're not using objects at the same architectural level.
Update: For download.
What changed:
* AttributeValuePair has been removed and replaced with maps (as discussed previously).
* Seeing as though maps were used so much hashCode and equals were optimized. As I've found before (I think), isAssignableFrom is slower than try/catch for equals (depending on your usage of course).
* Queries go through unsorted and uncopied rather than standard graph finds. I'd forgotten about how much effort had gone into allowing remove and automatic sorting on iterators.
* A very simple optimizer (it's really only simplifying the FILTER constraints at the moment) was added. Tree manipulation was painful - I resorted to mutating in place operations.
* Better designed. It's a bit hard to qualify this except what was there was truly awful - objects being created in constructors and passing itself in. The nice thing about IoC is it's quite easy to see when you're not using objects at the same architectural level.
Update: For download.
Thursday, August 07, 2008
JRDF is very crap (part 1)
I've been spending some time looking at the querying part of JRDF. And it's quite bad. How bad? Well I've been profiling it and noticing that a lot of time was spent comparing attribute value pairs. An attribute in JRDF consists of a name (variable or position in a triple) and type (position in a triple or literal, URI Reference or blank node). Comparisons are done during most operations (like joins) and they are done on sorted attribute values. This is incredibly dumb. What's much better is to have a map of attributes to values. No sorting required and O(1) lookup - hurrah. The code around the comparisons also got a lot simpler and is obviously better. I think there's at least one other case of this at a different level and potentially room for about an order of magntitude speed up over the current release. Test queries are already 2-3 times faster.
The main reason for this though, is that currently the FILTER in JRDF runs about 10 times more slowly than a query using triple matching (this isn't a complexity measurement - it's based on a rather small set of triples). So a query with "?a <some:value> 'foo'" is much slower than "?a <some:value> ?b . FILTER(str(?b) = 'foo')". The queries aren't the same but their performance shouldn't be that much slower. However, in order to get to a stage of improving FILTER's performance the code has to be refactored - hopefully simpler and faster.
FILTER is nicely functionaly - it seems a shame to implement it in Java - it's eye poppingly bad at the moment. I was thinking functional Java but instead of taking the gateway drug I was think of just going to the hard stuff. FILTER is operated by creating different operations within a relation - which allows you to put ANDed FILTERs vertically across a relation (columns) and ORed FILTERs horizontally (rows). I don't know if anyone else implements it this way - it might be another bad idea over time.
The main reason for this though, is that currently the FILTER in JRDF runs about 10 times more slowly than a query using triple matching (this isn't a complexity measurement - it's based on a rather small set of triples). So a query with "?a <some:value> 'foo'" is much slower than "?a <some:value> ?b . FILTER(str(?b) = 'foo')". The queries aren't the same but their performance shouldn't be that much slower. However, in order to get to a stage of improving FILTER's performance the code has to be refactored - hopefully simpler and faster.
FILTER is nicely functionaly - it seems a shame to implement it in Java - it's eye poppingly bad at the moment. I was thinking functional Java but instead of taking the gateway drug I was think of just going to the hard stuff. FILTER is operated by creating different operations within a relation - which allows you to put ANDed FILTERs vertically across a relation (columns) and ORed FILTERs horizontally (rows). I don't know if anyone else implements it this way - it might be another bad idea over time.
Friday, July 04, 2008
JRDF 0.5.5.1 Released
Just a quick note about a new version of JRDF. It's been a short time between releases but it still contains one significant advance over the previous one and that's persistent graphs. It's still in the early stages but it's basic enough for simple use cases. It also contains text serialization (based on NTriples) that is useful for moving RDF molcules around nodes in a cluster (for example). A lot of this code is fairly much "spike" code and I expect that another release will be released after we exercise these new features more (and write some tests/rewrite the code).
Update: 0.5.5.2 is now available fixing many bugs and introducing FILTER support.
Update: 0.5.5.2 is now available fixing many bugs and introducing FILTER support.
Thursday, June 26, 2008
JRDF 0.5.5
The main difference in version 0.5.5 from the previous one is the inclusion of a RDF molecule store. Both in memory and disk based versions are supported and can be queried just like a normal triple store. This is also the first version that has been renamed URQL instead of SPARQL for the query evaluation. The SPARQL grammar is the same but it does not support the weird outliers that SPARQL has for empty graph patterns but follows relational (and other) algebras. There's also the usual bug fixes and other features.
Update: Due to a couple of bugs found in 0.5.5 there will be a 0.5.5.1 version released soon.
Update: Due to a couple of bugs found in 0.5.5 there will be a 0.5.5.1 version released soon.
Tuesday, February 05, 2008
50% Less Code or Your Money Back
Less that two weeks ago 0.5.3 of JRDF is released and now it's 0.5.4's turn.
This is mainly driven by making the fix to the bug in the btree (or Sesame's version) available.
It does however mean that the changes to the Resource API are made available more quickly. I remember looking at Jena's Resource and thinking it was bloated, confusing and rather poorly thought out from an efficiency point of view (holding onto all Graph and associated objects) but now I finally understand the positive effects it has on the code and I like it. JRDF's Resource sits on top of an RDF graph and automatically performs conversion from Java objects like URIs (which become URIReferences) and insert or removes them from the graph. It too has many methods that do the same thing but with different types. It's better than the recent changes to TripleFactory as it allows blank nodes too. JRDF's Graph implementation is no longer the heavy weight object it once was, it references the indexes and nodepool but no longer has any real control over them (this meant removing serialization).
Code becomes a lot smaller too, for example (the create call is URI.create):
Creates the triples:
It used to be something like:
This is mainly driven by making the fix to the bug in the btree (or Sesame's version) available.
It does however mean that the changes to the Resource API are made available more quickly. I remember looking at Jena's Resource and thinking it was bloated, confusing and rather poorly thought out from an efficiency point of view (holding onto all Graph and associated objects) but now I finally understand the positive effects it has on the code and I like it. JRDF's Resource sits on top of an RDF graph and automatically performs conversion from Java objects like URIs (which become URIReferences) and insert or removes them from the graph. It too has many methods that do the same thing but with different types. It's better than the recent changes to TripleFactory as it allows blank nodes too. JRDF's Graph implementation is no longer the heavy weight object it once was, it references the indexes and nodepool but no longer has any real control over them (this meant removing serialization).
Code becomes a lot smaller too, for example (the create call is URI.create):
Resource supplier = elementFactory.createResource();
supplier.addValue(create("urn:supplier"), "S1");
supplier.addValue(create("urn:name"), create("urn:Smith"));
supplier.addValue(create("urn:status"), 20);
supplier.addValue(create("urn:city"), "London", XSD.STRING);
Creates the triples:
_:1 urn:sno "sno"
_:1 urn:name urn:Smith
_:1 urn:status "20"^^xsd:int
_:1 urn:city "London"^^xsd:string
It used to be something like:
Resource supplier = elementFactory.createResource();
URIReference supplierPred = elementFactory.
createURIReference(create("urn:supplier"));
graph.add(supplier, supplierPred, "S1");
...
Wednesday, January 23, 2008
Nodes by Type
JRDF 0.5.3 released. This has a couple of bug fixes and features added. The find with no constraints (find(ANY_SUBJECT_NODE, ANY_PREDICATE_NODE, ANY_OBJECT_NODE)) has always been fast with the on disk version but not if it was constrained in anyway (a fixed subject for instance). This was because the wrong method on the btree was being called. This is a big improvement in performance obviously.
A cleanup on the Graph interface also occurred. There were all these methods like findUniquePredicates, getBlankNodes and getResources. There is now findNodes(NodeType) where NodeType is any of the positional (SPO) or value (Resource, URIRef, bnode, Literal) node types. This was able to be done because JRDF has a NodeTypePool where all nodes are in maps by their type. It is missing some types being supported but that should be done next release.
The other thing that was very annoying and now fixed was that a Literal or URI Reference had to be added via the GraphElementFactory before you could add it to the Graph. This only makes sense for blank nodes - other node types can just be re-localized (get their internal node id) each time. Handy for the amount of graph copying we have done lately.
There were some other boring changes like a bug fix to the RDF/XML parser and TripleImpl now being null proof.
Much more to do - haven't had a chance to cleanup the Molecule implementation or add the persistence support.
A cleanup on the Graph interface also occurred. There were all these methods like findUniquePredicates, getBlankNodes and getResources. There is now findNodes(NodeType) where NodeType is any of the positional (SPO) or value (Resource, URIRef, bnode, Literal) node types. This was able to be done because JRDF has a NodeTypePool where all nodes are in maps by their type. It is missing some types being supported but that should be done next release.
The other thing that was very annoying and now fixed was that a Literal or URI Reference had to be added via the GraphElementFactory before you could add it to the Graph. This only makes sense for blank nodes - other node types can just be re-localized (get their internal node id) each time. Handy for the amount of graph copying we have done lately.
There were some other boring changes like a bug fix to the RDF/XML parser and TripleImpl now being null proof.
Much more to do - haven't had a chance to cleanup the Molecule implementation or add the persistence support.
Thursday, November 15, 2007
Sesame Native Store
I'm very impressed at the moment with OpenRDF's native store as others have been in the past. One of the best things is how easy it was to work into the existing JRDF code.
As I've said before I've been searching for an on disk solution for loading and simple processing of RDF/XML. In the experiments I've been doing OpenRDF's btree index is much faster than any other solution (again not unexpected based on previous tests). The nodepool/string pool or ValueStore though is a bit slower than both Bdb and Db4o.
Loading 100,000 triples on my MacBook Pro 2GHz takes 37 secs with pure Sesame, 27 with the Sesame index and Db4o value store, 35 with Bdb value store and ehCache is still going (> 5 minutes). A million takes around 5 minutes with Sesame index and Db4o nodepool (about 3,400 triples/second) and 3 minutes with a Sesame index and memory nodepool (about 5500 triples/second).
There's lots of cleanup to go and there's no caching or anything clever going on at the moment, as I'm trying to hit deadlines. 0.5.2 is going to be a lot faster than 0.5.1 for this stuff.
Update: I've done some testing on some fairly low-end servers (PowerEdge SC440, Xeon 1.86GHz, 2GB RAM) and the results are quite impressive. With 100,000 triples averaging around 11,000 triples/second and 10 million averaging 9,451 triples/second.
Update 2: JRDF 0.5.2 is out. This is a fairly minor release for end user functionality but meets the desired goal of creating, reading and writing lots of RDF/XML quickly. Just to give some more figures: Bdb/Sesame/db4o (SortedDiskJRDFFactory) is 30% faster for adds and 10% slower for writing out RDF/XML than Bdb/Sesame (SortedBdbJRDFFactory). Both have roughly the same performance for finds. I removed ehcache as it was too slow compared to the other approaches.
As I've said before I've been searching for an on disk solution for loading and simple processing of RDF/XML. In the experiments I've been doing OpenRDF's btree index is much faster than any other solution (again not unexpected based on previous tests). The nodepool/string pool or ValueStore though is a bit slower than both Bdb and Db4o.
Loading 100,000 triples on my MacBook Pro 2GHz takes 37 secs with pure Sesame, 27 with the Sesame index and Db4o value store, 35 with Bdb value store and ehCache is still going (> 5 minutes). A million takes around 5 minutes with Sesame index and Db4o nodepool (about 3,400 triples/second) and 3 minutes with a Sesame index and memory nodepool (about 5500 triples/second).
There's lots of cleanup to go and there's no caching or anything clever going on at the moment, as I'm trying to hit deadlines. 0.5.2 is going to be a lot faster than 0.5.1 for this stuff.
Update: I've done some testing on some fairly low-end servers (PowerEdge SC440, Xeon 1.86GHz, 2GB RAM) and the results are quite impressive. With 100,000 triples averaging around 11,000 triples/second and 10 million averaging 9,451 triples/second.
Update 2: JRDF 0.5.2 is out. This is a fairly minor release for end user functionality but meets the desired goal of creating, reading and writing lots of RDF/XML quickly. Just to give some more figures: Bdb/Sesame/db4o (SortedDiskJRDFFactory) is 30% faster for adds and 10% slower for writing out RDF/XML than Bdb/Sesame (SortedBdbJRDFFactory). Both have roughly the same performance for finds. I removed ehcache as it was too slow compared to the other approaches.
Friday, November 09, 2007
JRDF 0.5.1 Released
This release is mainly a bug release. There are improvements and fixes to the Resource API, datatype support and persistence. Another persistence library has been added, db4o, which has some different characteristics compared to the BDB implementation. However, it's generally a little slower that BDB. The persistence offered is currently only useful for processing large RDF files in environments with low memory requirements.
Also, the bug fixes made to One JAR have been integrated, so JRDF no longer has its own version.
Available here.
Also, the bug fixes made to One JAR have been integrated, so JRDF no longer has its own version.
Available here.
Sunday, August 26, 2007
JRDF 0.5.0
As mentioned this version of JRDF supports a Resource interface, datatype support, persistence (via Berkeley DB), and Java 6. It does run under Java 5 but the RDF/XML writer now uses the StAX API, so it requires Woodstox or some other JSR 173 implementation.
A few things in the works such as an Elmo like API, Globalized nodes, and MSG support.
Sourceforge download.
A few things in the works such as an Elmo like API, Globalized nodes, and MSG support.
Sourceforge download.
Sunday, July 08, 2007
JRDF Development
I've started adding datatype support to JRDF. Nothing too flash at the moment, mainly to support the RDF/XML test cases and prompted by a similar piece of work that was completed by the people at KP Lab. They also implemented an Elmo like API (with lazy collections) and added a Resource interface (which adds methods on top of both Blank Nodes and URI References and is quite similar to Jena's Resource interface) which will be integrated soon.
An interesting aspect to the datatype support is of course semantically equal types like integer and long. In the current version of JRDF they wouldn't return equal. In Jena there is "sameValueAs" that returns true if they are semantically equal. In JRDF, I originally decided to use Comparable - but then the use in ordered maps may get confusing (although it would be similar to BigDecimal but it's the exception rather than the rule). Using Comparable in this way would also be different to the implementation of datatypes in Kowari/Mulgara. So I've kept the old behaviour and added an equivalent comparison operator that is a copy of Comparable but takes in semantic similarities.
The other thing that's been cleared up is the containers and collections - a lot of (90%) was redundant due to Generics and didn't need to be there anymore. However, I did come across, "The pseudo-typedef antipattern", which pretty much says that creating a StringList (that extends ArrayList) is an anti-pattern. In JRDF collections and containers are basically Lists or Maps restricted to object nodes. Which seems to fit the anti-pattern. However, for me it does add different behaviour over the standard collections and restricting it on type does seem to make sense. I'm willing to be convinced though.
I'm also aware of some of the more weird RDF/XML edge cases that don't look like they were ever implemented correctly in JRDF or Kowari/Mulgara (like turning rdf:li into numbers even if outside of collections). I don't think it's a big use case and no one has ever raised it as a bug as far as I know.
Update: It's been pointed out to me again that implementing comparable isn't flexible enough - I should've known better of course as it's been pointed out to me many times in the past and I've used it in the past the equivComparator is a bit dumb.
An interesting aspect to the datatype support is of course semantically equal types like integer and long. In the current version of JRDF they wouldn't return equal. In Jena there is "sameValueAs" that returns true if they are semantically equal. In JRDF, I originally decided to use Comparable - but then the use in ordered maps may get confusing (although it would be similar to BigDecimal but it's the exception rather than the rule). Using Comparable in this way would also be different to the implementation of datatypes in Kowari/Mulgara. So I've kept the old behaviour and added an equivalent comparison operator that is a copy of Comparable but takes in semantic similarities.
The other thing that's been cleared up is the containers and collections - a lot of (90%) was redundant due to Generics and didn't need to be there anymore. However, I did come across, "The pseudo-typedef antipattern", which pretty much says that creating a StringList (that extends ArrayList
I'm also aware of some of the more weird RDF/XML edge cases that don't look like they were ever implemented correctly in JRDF or Kowari/Mulgara (like turning rdf:li into numbers even if outside of collections). I don't think it's a big use case and no one has ever raised it as a bug as far as I know.
Update: It's been pointed out to me again that implementing comparable isn't flexible enough - I should've known better of course as it's been pointed out to me many times in the past and I've used it in the past the equivComparator is a bit dumb.
Tuesday, June 05, 2007
JRDF 0.4.1.1 Released
JRDF 0.4.1.1. A bug fix release for an error in the wiring. It was wiring up the join code rather than the union code when it was suppose to do union operations. Both Derby and Hadoop were removed for this release (which reduces the GUI jar by 2 MB).
Tuesday, February 27, 2007
JRDF 0.4.1 and GUI 0.3.1 Released
JRDF 0.4.1 and the GUI are now available.
The main things changes from the last release:
The main things changes from the last release:
- Initial SPARQL support that's been available in the JRDF GUI.
- RDF/XML writer.
- Push and pull RDF NTriples parser. In the future an RDF pull parser will also implement the RDFEventReader interface (which is similar to the StAX XMLEventReader).
- Allows sorted results from a graph find() to be returned.
Monday, January 22, 2007
Persistent JRDF
After thinking about it many times I've started to integrate a persistent store for JRDF. Now I know Kowari/Mulgara would be the obvious choice but I decided to try something different - Apache Derby.
A recent post on the Sesame developers list about persistent blank node maps got me into action as well as all this talk about XA2 of course (JRDF's modified RIO parser in Kowari/Mulgara uses a persistent StringToLongMap).
One of the reasons to use Derby is because it has an XAResourceManager. Though I don't expect it to scale or be as fast as most stores (even if the table size is supposed to be unlimited).
I couldn't find an easy way to create a DiskHashtable (although the TestDiskHashtable gave some clues). Here's how I managed to get a persistent DiskHashtable going called derbyDB (it may not be quite right of course):
I have some ideas that it might be possible to take the relational RDF operations and put them into Derby (or vice-versa). The datatype support would be nice to leverage (especially the XML datatype for example). It's very preliminary at the moment and I may ditch it in the future.
A recent post on the Sesame developers list about persistent blank node maps got me into action as well as all this talk about XA2 of course (JRDF's modified RIO parser in Kowari/Mulgara uses a persistent StringToLongMap).
One of the reasons to use Derby is because it has an XAResourceManager. Though I don't expect it to scale or be as fast as most stores (even if the table size is supposed to be unlimited).
I couldn't find an easy way to create a DiskHashtable (although the TestDiskHashtable gave some clues). Here's how I managed to get a persistent DiskHashtable going called derbyDB (it may not be quite right of course):
String driverStr = "org.apache.derby.jdbc.EmbeddedDriver";
EmbeddedDriver driver = (EmbeddedDriver) Class.forName(driverStr).newInstance();
final EmbedConnection30 connection = (EmbedConnection30) DriverManager.getConnection(
"jdbc:derby:derbyDB;create=true");
final LanguageConnectionContext languageConnectionContext = connection.getLanguageConnection();
languageConnectionContext.setRunTimeStatisticsMode(true);
TransactionController controller = languageConnectionContext.getTransactionExecute();
ContextService service = ContextService.getFactory();
service.setCurrentContextManager(languageConnectionContext.getContextManager());
DiskHashtable diskHashtable = new DiskHashtable(controller, TEMPLATE, INDEXES, true, true);
I have some ideas that it might be possible to take the relational RDF operations and put them into Derby (or vice-versa). The datatype support would be nice to leverage (especially the XML datatype for example). It's very preliminary at the moment and I may ditch it in the future.
Monday, December 11, 2006
The Difference is Antijoin
I made a mistake about making a mistake. I got caught up with the fact that SPARQL's diff is not the same as relational difference but that's not how left outerjoin is defined. So I'm correct that relational set difference is not compatbile with SPARQL's difference as defined in "The SPARQL Algebra" but wrong that this means the definitions for left outer join are not equivalent.
So antijoin is:
( R1 difference (project(R1)(R1 join R2)) )
So using the previous example:
{{ (?x = 2, ?y = 3) }} \ {{ (?y = 3) }} is {} in SPARQL.
Now, as I said using plain difference it's:
{{ (?x = 2, ?y = 3) }} - {{ (?y = 3) }} is {{ (?x = 2, ?y = 3) }} in relational algebra.
But using antijoin the right hand side is really:
This is why JRDF looked like it was producing the right result - it was doing the same thing - it's just the operations were further decomposed. I'm sure I've done this before but for some reason I was certain I was wrong the other day. I was told that antijoin and SPARQL's diff were equivalent when it was changed but I had to question it.
There still remains an outstanding issue around null joins but maybe that's not needed either. I know that the NULLs in "SPARQL RULES!" are required for Logic Programs but it doesn't need to be there for relational algebra as long as you have the null accepting/untyped join.
So SPARQL in JRDF looks fine again.
So antijoin is:
( R1 difference (project(R1)(R1 join R2)) )
So using the previous example:
{{ (?x = 2, ?y = 3) }} \ {{ (?y = 3) }} is {} in SPARQL.
Now, as I said using plain difference it's:
{{ (?x = 2, ?y = 3) }} - {{ (?y = 3) }} is {{ (?x = 2, ?y = 3) }} in relational algebra.
But using antijoin the right hand side is really:
- project ({{ (?x = 2, ?y = 3) }}) ({{ (?x = 2, ?y = 3) }} join {{ (?y = 3) }})
- project ({{ (?x = 2, ?y = 3) }}) ({{ (?x = 2, ?y = 3) }})
- {{ (?x = 2, ?y = 3) }}. Which matches the left hand side.
This is why JRDF looked like it was producing the right result - it was doing the same thing - it's just the operations were further decomposed. I'm sure I've done this before but for some reason I was certain I was wrong the other day. I was told that antijoin and SPARQL's diff were equivalent when it was changed but I had to question it.
There still remains an outstanding issue around null joins but maybe that's not needed either. I know that the NULLs in "SPARQL RULES!" are required for Logic Programs but it doesn't need to be there for relational algebra as long as you have the null accepting/untyped join.
So SPARQL in JRDF looks fine again.
Subscribe to:
Posts (Atom)