Friday, September 17, 2004

Drops to make an Ocean

many little models "Like many others (e.g. lost boy and more news) I’m wondering about the scalability of RDF data stores. In particular, we have many very small models (under 25 statements for most models, the largest is unlikely to exceed 100 statements) which we don’t really need super fast access to, but store more for reflection. While at current we only have twelve thousand of these models, we are expecting to collect much closer to 100,000 each term, and want to keep data around for at least the last couple of years (si lets say a million or so models). The 12,000 or so has already kicked our database to its knees (MySQL, Jena 2.1, an old P3 700Mhz with minimal ram), which we didn’t really expect."

1 comment:

Anonymous said...

If you cut some corners, it is quite possible to create a store that works well with millions of small models... See http://www.isb-sib.ch/~ejain/rdf/store.html.