Showing posts with label kowari. Show all posts
Showing posts with label kowari. Show all posts

Tuesday, June 26, 2007

The History of Trans and Walk

Paul recently started talking about the history of trans and walk in Kowari/Mulgara. I did a bit of searching in old Tucana emails and feature requests.

I also vaguely remember trying to create an abstract class for walk and the base transitive classes (like the Exhaustive and Direct transitive closure classes). And one of the feature requests also mentions the idea of "set depth" to reduce the recursion on unbounded traversal which would also be handy for the shortest path between two nodes.

Tuesday, April 24, 2007

Mistakes I've Made a Few

So the ghosts of implementations past have come to haunt me in recent weeks mostly around Kowari/Tucana (now Mulgara).

Firstly, there was exclude and NAF in iTQL:

> It was introduced to provide
> a limited form of negation, and one that interacts poorly with the
> open-world assumption. We also now have minus, which is well
> defined, corresponds closely to our intuitive understanding of the
> operation, and is (I am told) what was actually required.
>
> If my memory is correct we should probably at least deprecate, if
> not remove exclude entirely from mulgara.
>
> Do we agree that exclude should be removed?

Well *I* agree anyway. As you say, it doesn't do what anyone thinks
it does.

> If it should be removed, when should this occur?

Yesterday.


And now adding the Jena API to Kowari cost customers:
Another possible reason is Tucana/Kowari/Mulgara’s Jena support - originally put in to provide a migration path for companies looking to move on from research projects to scalable infrastructure - which as Jena is the defacto semweb tool of choice, people used to evaluate Kowari’s scalability. Jena’s lack of scaling hurt us several times, I can remember lots of frantic calls as some company wrote us off because of our Jena API.


I'm current still working in this area (maybe somewhat surprisingly) and Jena still dominates (all of the tools I'm currently looking at are Jena based). And I still haven't seen a Jena implementation that scales (see page 10). Maybe the decision to open source Kowari cost another round of funding too. Maybe this is why Garlick or Radar Network's triple stores are still behind closed doors.

Monday, January 22, 2007

Persistent JRDF

After thinking about it many times I've started to integrate a persistent store for JRDF. Now I know Kowari/Mulgara would be the obvious choice but I decided to try something different - Apache Derby.

A recent post on the Sesame developers list about persistent blank node maps got me into action as well as all this talk about XA2 of course (JRDF's modified RIO parser in Kowari/Mulgara uses a persistent StringToLongMap).

One of the reasons to use Derby is because it has an XAResourceManager. Though I don't expect it to scale or be as fast as most stores (even if the table size is supposed to be unlimited).

I couldn't find an easy way to create a DiskHashtable (although the TestDiskHashtable gave some clues). Here's how I managed to get a persistent DiskHashtable going called derbyDB (it may not be quite right of course):
String driverStr = "org.apache.derby.jdbc.EmbeddedDriver";
EmbeddedDriver driver = (EmbeddedDriver) Class.forName(driverStr).newInstance();
final EmbedConnection30 connection = (EmbedConnection30) DriverManager.getConnection(
"jdbc:derby:derbyDB;create=true");
final LanguageConnectionContext languageConnectionContext = connection.getLanguageConnection();
languageConnectionContext.setRunTimeStatisticsMode(true);
TransactionController controller = languageConnectionContext.getTransactionExecute();
ContextService service = ContextService.getFactory();
service.setCurrentContextManager(languageConnectionContext.getContextManager());
DiskHashtable diskHashtable = new DiskHashtable(controller, TEMPLATE, INDEXES, true, true);


I have some ideas that it might be possible to take the relational RDF operations and put them into Derby (or vice-versa). The datatype support would be nice to leverage (especially the XML datatype for example). It's very preliminary at the moment and I may ditch it in the future.

Tuesday, January 16, 2007

Announcing Andrae

lca: Andrae Muys on RDF "On the first day of linux.conf.au, I ran into Andrae Muys. He hacks Java and RDF for clients who want semantic web hackery done. I have to admit that early Semantic Web hype put me off: it sounded too much like 1970s AI hype. Andrae was interesting, though, and completely free of the wide-eyed uncritical enthusiasm that characterized a lot of my early RDF engagement."

"Andrae runs the Mulgara project, a Java RDF store. His goal is to be able to deal with 1E13 statements (aka tuples, facts, assertions) in three years. It'll do 1E9 right now, next stop is 1E11. He refers to this goal as "3 Ts: three trillion triples". A consortium is forming around Mulgara to make this happen: if it coalesces, Andrae will be the coder to make it happen."

"The next version of Mulgara, 1.3, will ship in February and have this relational mapping in it. A quick Google search shows a lot of RDF-relational mappings going on, but the list of other mappings he had impressed me: Lucene, RSS, mbox, ID3."

"I think it's time I looked again at the world of RDF. They may yet be doing interesting things. I said as much to Andrae and he replied, "I am an engineer. In the early days it was scientists and logicians in RDF. Now the engineers have arrived, and we just want it to work and to scale." Bold claim! If you have a favourite RDF package or practice, let me know in the comments."

Andrae also announced the paper presented at linux.conf.au. Among other things it references David Wood's paper presented in 2004 "Scaling the Kowari Metastore" ("Makepeace" is a good search term).

Tuesday, October 17, 2006

The State of the Semantic Web

TripCom define their project as: "We will improve the ideas of Tuple Space computing by adding semantics by use of a graph-based data-model to rely on Triples. The Triple Space serves as a persistent publication system for semantically linked information in semantically clustered subspaces. We will develop a scalable and linkable Triple Space storage, based on improving and combining current RDF Stores and Tuple Space infrastructures."

I came across their "State of the art and Requirements Analysis" it mentions YARS, Jena, Sesame, 3store, Kowari, JRDF, Edutella and Oracle. They seem especially keen on YARSQL and its support for provenance. They also seem interested in extending SPARQL to support subqueries and inserting and deleting statements.

Friday, July 07, 2006

The Future is a little Brighter

David links to a posting he forwarded from the Kowari developers list (original post here) from Amit Kapoor about the future of Kowari: "The Topaz Foundation (http://www.topazproject.org) is very pleased to forward the email, from Michael H. Wallach (Senior Counsel, Northrop Grumman) to Richard Fontana...We trust that this letter will end any confusion with respect to the future status of Kowari, which has been secured, and that the community will now be able to focus on making Kowari one of the most vibrant open source projects."

"Northrop Grumman respects the rights that users of open source Kowari software receive under the MPL. Northrop Grumman intends that open source Kowari software, licensed under the MPL, is and will remain free, open source software.

Moreover, Northrop Grumman has no objection to the continued appropriate use of the "Kowari" name by developers participating in the Kowari open source project."

Sunday, February 19, 2006

Debt

Design Debt "Design debt can be hard to spot. If you see these problems, your project is already in serious trouble:

* Your team describes the software as "kludgey" or "hacked-together."
* The team often encounters unexpected problems.
* Each change results in a cascade of new defects.
* Defects thought fixed frequently reappear.

To avoid the above level of debt, make debt elimination a priority:

* Reduce schedule pressure.
* Establish a culture of design quality.
* Utilize refactoring, continuous design, and other code-quality practices.
* Make debt reduction a continuous process, not a one-time activity."

When I was working on Kowari/TKS I remember Simon mentioning his idea of the three stages of software:
* Stage 1 - New easy to understand, easy to add new features, classes, components, and frameworks are largely self contained. It's easy to estimate the time it will take to make changes.
* Stage 2 - Difficult to understand, changes have a larger effect that you expect - breaking tests that you don't expect, classes are generally a little too coupled. Generally a reasonably good programmer can achieve meeting deadlines as long as they make "pragmatic decisions" not to fixup all the code.
* Stage 3 - The code is basically impossible to understand as single classes - the voodoo stage. You change something and it breaks something else that appears to be completely unrelated. Making predictable changes is no longer possible.

The obvious answer is to rewrite but this is just what several people have suggested not to do. For example, in one of the Joel on Software articles, "Things You Should Never Do, Part I" he mentions: "It's harder to read code than to write it...This is why code reuse is so hard...When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work."

A possible way to solve this is to prevent it from occurring. The focus on TDD is often on the testing part of it but it's the design part and the idea to explain intent and reduce the time reading code and increase the time writing it.

In, "Test-Driven Development Isn’t Testing": "Once you start looking at your test cases as a description of the software design, they start to look different. Pay attention to the names of your tests. Instead of writing tests named after a method on an object you're testing, try using the test name to capture the intended consequences...There's at least as much code in the test case as in the actual class being tested.

I guess this makes it sort of "design and test," but in the end we still haven't ensured our object is bug-free."

Tuesday, January 10, 2006

Northrop Grumman Killing Kowari?

Resignation from Kowari due to Northrop Grumman Letter "It is with sincere regret that I resigned as an administrator and developer of the Kowari Metastore."

"Northrop Grumman's position seems to be that they "purchased all rights associated with the Kowari software", a position not reconcilable with their continued release of the software under the Mozilla Public License, version 1.1."

So it would seem that Northrop, for whatever reason, dislikes Kowari's existence.

To get David and then Tucana (whose IP was eventually sold to Northrop) to start and continue with an open source version of TKS (the closed source version of the RDF database) was my little hobby horse (I fought fairly long and hard you could say). The idea was to get the Semantic Web bootstrapped and to create a value network around Semantic Web technology (I got this originally from "A critical look at object-orientation"). It is deeply disappointing for me to see it (the value network and Kowari) threatened in this way.

Like any OS project it is the ecosystem that is built around Kowari that makes it work. With so many contributions, just how much of the total code does Northrop actually own the copyright to? By accepting changes it isn't following the dual licencing model used by the owners of Berkley DB. To me Kowari was always using open source as a development and distribution model. Already, we have some contributors threatening to pull out their code (more at, "Disturbing News From Kowari").

Having administrators leave due to Northrop's actions is a good way to kill the project. Hopefully, Northrop's bullying will get a lot of interest - it's good to see, for instance, that this has been picked by aggregators like Topix's Aerospace and Defense.

The other posting David W linked to: Is Northrup Grumman Smushing Kowari?, "But I’d like to know what the status of Kowari is going to be, open source or not, because there are clients for whom Kowari might be the right choice, and we won’t bet on a dog that Northrup Grumman seems determined to publicly beat to death.

Relying on Kowari is now not prudent, given our obligation to do our best for our clients; but it’s also bad for Semantic Web uptake in the US federal government, and that’s something Northrup Grumman should think very carefully about."

Update: Paul and Andrae have also added their thoughts.

Update 2: David has published the letter from Northrop's lawyers. Danny posts, "...clearly if the current Kowari is MPL’d they can’t stop other developers working on the system."

Friday, December 23, 2005

More on Disjunction

I think this is just going to happen time and time again, "RDF non-sense": "The W3C Working Group members argue SPARQL is a mixed mode language that does support OR, though they are calling it an "optional union". Frankly, I see no point in renaming something people understand to mean one thing. It gives me the impression the W3C want to be thought police and enforce a certain way of thinking.

On the practical side, many analysts happen to like OR disjunctions and would complain loudly. Of course, there are plenty of cases where users abuse the power and write deeply nested disjunctions. That is not a valid reason in my mind to avoid disjunction. It saves the user time and allows them to write simpler rules using disjunction. The W3C seems love RDF and wants the world to love it. Unfortunately, the current specification is a complete piece of junk. I hope RDF dies a quick and public death."

I declare...backward chaining suits me fine! "My two favorite declarative tools right now are Pellet and Prova, both of which are open source java SemWeb tools that are highly compatible with Jena, which recently got a bump to 2.3 with fairly complete SPARQL support. Pellet is an implementation of OWL-DL and some related description logic facilities by the Mindswap guys in Maryland, who have absorbed some of the l33t Kowari/Tucana guys, too (Tucana was recently picked up by Northrop, BTW)."

"Prova is a prolog-variant built on top of Mandarax. It is a very effective and fun medium for scripting of high-level relationships and operations. The integration of prolog unification, java types, java methods, and java exceptions is done very nicely, and yields fine code economy. There are some rough edges in the docs, but we are helping to get these worked out in the pretty soon."

I guess that means David W is l33t! :-)

Sunday, October 09, 2005

JRDF 0.3.4.1 Out

Now available for download.

Some thoughts related to this:
* To ensure valid refactorings and bug fixes it seems sensible to go back and do some things properly. It means that the next highest priority will have to be writing an NTriples parser to validate the RDF Test cases. The SableCC grammar for NTriples is already done.
* While this is a good start, it's not nearly fine grained enough to actually write the code and have fast running unit tests. Tom has done some of this test driving a parser with his SPARQL work although I suspect I may do it a little differently (using EasyMock).
* Graph equality and isomorphism on graphs is a pain - which is to say that it's complicated with blank nodes (see the test cases). In "Matching RDF Graphs it lists possible ways to do mappings, including a blank node may map to a labelled node. I know that in Kowari loading the sample FOAF files twice into the same graph adds the blank nodes twice. While this seems like a mistake it is the fastest way to load a graph. Maybe implementing different loading modes (no duplicate checking, blank node to blank node and blank node to labelled node) and signing grahs are two ways to help this.

Wednesday, August 31, 2005

New Kowari Soon

Kowari v1.1 Pre-release Feature Peek "The v1.1 release of Kowari will include the following features:

o The addition of Paul Gearon's Krule (pronounced "cruel") rules engine. Paul has fast and scalable RDFS support implementated (with the exception of XSD data typing) and is working on implementing more rules for an OWL subset.

o Rewriting of ConstraintExpressions and ModelExpressions to collapse Resolver-specific constraints that should be evaluated together into compound constraints (query rewriting) and permitting Resolvers to advise query engine of the evaluation order of constraints to ensure binding of required variables (query annotation). These features were added by Andrae Muys and Simon Raboczi and will form the basis for a cleaner query language syntax and cleaner integration of external Resolvers, such as GIS systems.

o The removal of Jena support. Kowari v1.0 implemented a Jena storage backend and included an RDQL query language API. This has caused difficulties in maintenance and support as the two projects have gone in different directions. Kowari will remove (not deprecate) support for Jena as of the v1.1 release.

o Support for compiling Kowari under Java 1.5. The Kowari team will be moving to Java version 1.5 as the canonical build version, replacing Java 1.4.2."

Via, Kowari v1.1 Pre-release Feature Peek.

Also, new MindSwap Kowari Library part of PhotoStuff.

Sunday, July 10, 2005

Querying for Hierachies

Storing Hierarchical Data in a Database "Whether you want to build your own forum, publish the messages from a mailing list on your Website, or write your own cms: there will be a moment that you’ll want to store hierarchical data in a database. And, unless you’re using a XML-like database, tables aren’t hierarchical; they’re just a flat list. You’ll have to find a way to translate the hierarchy in a flat file."

"If you want to display the tree using a table with left and right values, you’ll first have to identify the nodes that you want to retrieve. For example, if you want the ‘Fruit’ subtree, you’ll have to select only the nodes with a left value between 2 and 11. In SQL, that would be:

SELECT * FROM tree WHERE lft BETWEEN 2 AND 11;"

It struct me how complicated either of these solutions are.

In iTQL it would be using walk:
"select $subject
...
where walk($subject <rdfs:subClassOf> <food:fruit>
and $subject <rdfs:subClassOf> $object);

Or in Sesame's SeRQL (concrete example here):
"SELECT DISTINCT _fruit
FROM _fruit serql:directSubClassOf <food:fruit>"

They aren't quite the same query - Sesame is inferring new statements and walk is not. In designing iTQL we were always deciding between magical predicates or functions (like walk) and sometimes slavishly sticking to triple patterns. Combining the walk and trans operations using a predicate, like in SeRQL, seems like a good approach to take.

Also, interesting in a fairly recent Sesame release are the ANY and ALL, EXISTS and MINUS features.

Tuesday, June 28, 2005

Northrop Buys Tucana and Continues Kowari

Northrop Grumman Buys the Tucana Knowledge Server "Stunningly for a company their size, Northrop has not only agreed to support Kowari but rushed to do so. I certainly didn't expect a US federal systems integrator to "get" Open Source Software, but times have clearly changed. Their senior managers have made a legitimate effort to figure out the licensing and how to make it work within their business model. I have confidence that we can figure out a way to make it work for both the Kowari community and Northrop Grumman."

A Grumman employee? Or a Thoughtworker with inside knowledge? Building an Agile Enterprise "I have confidence that we can figure out a way to make it work for both the Kowari community and Northrop Grumman."

Update: Paul's doing the handover

Tuesday, June 14, 2005

JRDF for Learning

In the past I used another project to practically use trendy new things like patterns, XML, Swing, etc. Similarly, I'm going to use JRDF for the same purpose. Kowari is a bit too big for things like going over to Java 1.5, IoC, mocking (real unit tests), lock free alogirthms (including B-Trees) and a few other things that I want to try. I'm not sure it's possible to have a system that doesn't have transactions but it would be interesting to find out. So basically this is just to let people know to expect some changes in JRDF.

Practically, it might mean an RDF/XML based pull parser, persistent JRDF, and more interesting APIs. I'm convinced that developing web services is too expensive and may implode under its own weight - so maybe something based on netKernel or a REST based framework would be a good idea. At the moment I'm just using it to see how much I can get out of IntelliJ.

Friday, June 03, 2005

Keh-nig-it

* "The SEMANTIC Knight always triumphs! Have at you! Come on, then. I have an battalion of KR theorists on my side". Via Too Close To Home
* Google Sponsored Semantic Web API abstraction project "While a lot of semantic web APIs are available (Jena, Sesame, Kowari etc..) especially for the java language, there is no standard set of interfaces and wrappers so that middle ware RDF toolkits or higher level RDF based API can be built regardless of the underlying api/srdf storage. The project will certainly not start from scratch, but rather from earlier discussions and code to build upon (See jrdf, classes in the Simile projects etc..)."
* Tetris Shelving.

Wednesday, March 09, 2005

Using the Semantic Web to Handle Petascale Datasets

Scientific Data Management in the Coming Decade "Increasingly, the datasets are so large, and the application programs are so complex, that it is much more economical to move the end-user’s programs to the data and only communicate questions and answers rather than moving the source data and its applications to the user‘s local system."

"If the data is to be analyzed by generic tools, the tools need to “understand” the data. You cannot just present a bundle-of-bytes to a tool and expect the tool to intuit where the data values are and what they mean. The tool will want to know the metadata."

"In addition to standardization, computer-usable ontologies will help build the Semantic Web: applications will be semantically compatible beyond the mere syntactic compatibility that current-generation of Web services offer with type matching interfaces. However, it will take some time before high-performance general-purpose ontology engines will be available and integrated with data analysis tools...The XML integration in modern Database Management Systems (DBMS) opens the door for existing standards like RDF and OWL."

This would be something that I would disagree with - existing DBMS are not flexible enough to handle things like RDF. And OWL is even more crucially effected where you need the indexing available from something like Kowari. Other systems like Sesame and Jena are both adding tree structures to store their RDF. Some of that is maintenance and setup but also performance.

"As file systems grow to petabyte-scale archives with billions of files, the science community must create a synthesis of database systems and file systems. At a minimum, the file hierarchy will be replaced with a database that catalogs the attributes and lineage of each file. Set-oriented file processing will make file names increasingly irrelevant – analysis will be applied to “all data with these attributes” rather than working on a list of file/directory names or name patterns."

This is very similar to the TMex V2 architecture that I worked on.

Also Distributed Computing Economics "Computing economics are changing. Today there is rough price parity between (1) one database access, (2) ten bytes of network traffic, (3) 100,000 instructions, (4) 10 bytes of disk storage, and (5) a megabyte of disk bandwidth.".

Via Peta-scale Data Centers - a Trio of Great Jim Gray Papers (from: Kevin Schofield’s Weblog).

Tuesday, March 08, 2005

Kowari Changes since Pre-Release 2

Here's what's been committed so far (further updates will be put here):
* Transaction Timeout configurable. SR
* Client JRDF modifications - client side API now handles blank nodes, currently memory bound. AN
* KModel modifications - modified with changes to client JRDF and added RMI versions of API. Have to finish blank node client side changes. AN
* MP3 Content handler - changed to use different library. AN
* JRDF's RDF/XML parser (based on RIO) passes all tests. AN/DM
* UUID changes for blank node generation in JRDF and Kowari. RT/AN
* Fulltext acceptance tests now work without making changes to the OS environment. AN
* xsd:dateTime now millisecond accurate - also now using thread safe version to parser/format dates. ES/AN
* Refactoring of Tuples layer. More to come to remove duplication, reduce coupling and possibly adding features. AN

Wednesday, March 02, 2005

Fedora 2.0 and Kowari

Fedora 2.0 Release Notes "Fedora 2.0 includes significant new features and improvements including the introduction of the Fedora Object XML (FOXML) schema as the new internal storage format for objects, introduction of the Resource Index that provides enhanced search capability, introduction of a Batch Modify utility, upgrades of all third party libraries, performance enhancements, and a number of bug fixes."

Kowari is mentioned in "Fedora: An Architecture for Complex Objects and their Relationships": "Fedora expresses relationships by defining a base relationship ontology using RDFS [17] and provides a slot in the digital object abstraction for RDF expression of relationships based on this ontology. Assertions from other ontologies may also be included along with the base Fedora relationships. All relationships are reflected in a native RDF triple-store using Kowari [44]. The query interface to this triple-store is exposed as a web service, providing a rich information foundation for external services."

Looks like there's some Trippi work too.

Chris Wilper and Edwin Shin both have contributed back to Kowari too. Most recently, with changes to the DateTime datetypes in the post-Kowari Pre-Release 2.

Friday, February 18, 2005

Comparing YARS and Kowari

Cold, Mastering OWL, Scalability, Indexing "There are two differences in their implementation to our own. The first is that they used B-trees rather than the AVL trees that Kowari uses. That is not a significant difference, and I blogged about this several times before. The main advantage of the AVL tree is that they are cheaper to write to the structure than B-trees are, though a little slower to read. They also pointed out that they wanted to use an existing library for the index, and while B-tree libraries are common, we never found a completely working AVL tree library (deletions were always buggy).

The second difference is a set of special statements which count the number of triples in parts of each of the indexes. This is certainly novel, but not an approach I would use. I believe that counting is still very efficient in Kowari (O(log(n)), and the space overhead they incurred would seem prohibitive. More importantly, writing is always the slowest operation, and their system would incur a large writing penalty for using this scheme."

Saturday, January 29, 2005

New SOFA

SOFA (Simple Ontology Framework API) New home and new release available for download. Updated support for Jena 2, new methods on OntologyModel and ThingModel discarded (will have to change this for Kowari).