A New SQL

ETech '07 Summary - Part 2 - MegaData.

Here's the thing, we need a new kind of data store, a new kind of SQL, something that does for storing and querying large amounts of data what SQL did for normalized data.

Sure you can store a lot of data in a relational database, but when I say large, I mean really large; a billion or more records. I know we need this because I keep seeing people build it.

All this talk about making SPARQL behave like SQL maybe for nothing if people realize that's not what they need after all.

The back of the envelope scalability for an RDF store would be potentially 100s of billions of statements.

The key requirements highlighted are: distributed, joinless (no referential integrity at the store level), denormalized and transactionless.

I was aware of this because a comment linked to one of my previous posts about Kowari scalability (which I must of snuck through at some stage). Kowari got up to 10,000 triples/second later on its life.

Really Dynamic Framework

Semantic Rails, Semantic Django: Pushing RDF into MVC "...what Rails-Django would look like if instead of SQL and RDBMS... What if the "M" in MVC were composed of RDF, SPARQL, and a triplestore like Kowari?

Sure, Kowari is probably slower than MySQL, and you probably know SQL a lot better than SPARQL, but RDF is a schemaless data representation thingie. You can start with as little or as much schema as you want or need, and you can use the full expressive power of OWL (which is significantly more powerful than SQL's DDL) when you need it."

Four points are raised in "Semantic MVC".

nodel - HOWTO "Nodel is a kind of application generator. It is a halfway house between the ontomatic, which isn't yet stable enough in implementation or customisable enough in interface, and the nodedb, which was very hardcoded and application specific."

Related to: RDF, the ultimate agile database, Another Agile Database User and Scripting the Semantic Web.

Agile Databases Again

Let them eat layer cake: flexibility versus clarity in data "Recently I've had some experience at the non-planetary scale of trading off between the extreme flexibility of technology like RDF versus a domain model that a person coming after could reasonably be expected to understand...It turns out that RDF is surprisingly cheap stuff to generate. The downside was that for purposes of communicating intent, ongoing maintenance and adding functionality against the collected data, RDF is not very pleasant to work with, at least not compared to SQL, Objects or XML. This is especially so at the presentation layer. It's also a different paradigm, and by using it you're technologically committed to yet another data model, directed graphs, along the usual suspects - objects, markup and relations. The cost of introducing a new model should not be underestimated. As a result RDF has been useful but not as cheap to manipulate as one would like."

Also, "This is another area where RDF falls down. Yes, there is Sparql and before that other SQL like languages, but again you're left iterating over raw RDF graph result sets, which is not always ideal."

As linked to previously, SPARQL results can be either a sub-graph or variable bindings.

"Arguably we could have used an RDF store such as Kowari, the in built persistence mappings of Jena, or even XQuery, along with the RDF interchange. The reality is there's only so much new technology you can apply in one go without taking on too much risk, especially in a short time frame, whereas we had a good idea of what we were getting into with a relational store."

Are triplestores good databases? " The Sparql language may not quite be finished but it’s certainly comparable to SQL...Overall this would suggest that RDF stores are potentially good DBs, on many points potentially much better than regular RDBMSs (or XML DBs) because of the more flexible model. But for this to be practicable it assumes the performance can be brought to a comparable level as RDBMSs, which if it hasn’t already been done would I think would only be a small matter of programming."

I've been fairly negative about SPARQL in the past because it's lack of counting and sorting and it also doesn't make sense to me to have DISTINCT - everything should be distinct, it's a graph.

There's also some comments about Kowari using Lucene for the database and not having transactions which is just wrong. Kowari uses NIO and AVL Trees for its store and has since the beginning. It's much faster than JDBM (which we looked at to store our blank node map and discarded because of speed).

Another Agile Database User

Re: Non SemWeb uses of RDF "Of course this could also be done using a big relational database. A big benefit of RDF we've found is that you can dump the data together first, and then join it up with heuristics (e.g. switch port has same MAC entry as server NIC etc..). I suspect that this is an order-of-magnitude time saving compared with doing static schema design up front."

RDF, the ultimate agile database

Perspective on XML: Be humble, not imperial "The revolt against imperial modeling of code has already taken shape in the form of languages and agile methods. Agile programming emphasizes highly iterative development in close collaboration with the eventual users of the product. Even more important, it stresses the inevitability of change and evolution. In effect, agile developers pride themselves on being able to rapidly accommodate change."

"The same revolution is in the offing for data modeling. There have been some developments in agile databases which, literally, adapt the ideas of agile programming to the design of (usually relational) databases, but there is also progress occurring in semi-structured databases and, in particular, XML."

Is XML Zen in opposition to "strong" data modeling? "One thing I seem to share with so many of my colleagues in the XML world is a wary attitude towards traditional data modeling practices. It's an attitude that has also informed my thinking in related articles pondering data supermodels, coupling of distributed systems, OO encapsulation, and the like.

Some of us see XML as a bit of a refuge from established schools of data modeling. OO and Unified Process in my case, E-R and other relational based modeling in others'. Some just came from document-centric backgrounds where such extremely rationalized data modeling was not the mainstay. In my case, interest in XML was part of a general interest in data modeling as a vehicle for human expression rather than for robotic simulation of the real world."

As I've said before, XML is not relational enough and RDF is relational or rather "RDF provides a relational data model of the Web".