Build Broken Images

One of the things that we encourage where I work is to visually and aurally notify you when a build has broken. I did some images a while back for this purpose.

They are:

• Lemmings (oh no)


• Lemmings (blocker - from original sprites)


• Watchman (Rorschach's Journal)


• Swedish Chef (borked build)

Using SVNKit from within Groovy using SvnAnt

To load in SVN Ant you use (this assumes it's in the classpath):
ant.typedef(resource: "org/tigris/subversion/svnant/svnantlib.xml")

This then allows you to use the runSvn closures:
runSvn {
"export"(srcUrl: "", destPath: "")
"import"(url: "", path: "", message: "")
"copy"(srcUrl: "", destUrl: "", message: "")
"delete"(url: "", message: "")
"checkout"(url: "", destPath: "")
"propset"(path: "", name: "", value: "")
}
SvnAnt documentation has more details.

Adding Active Directory properties to ApacheDS

One of the things I have to do during my day is replicate Active Directory in ApacheDS. The most simple properties are member and memberOf. The following uses an LDIF file to add these properties. I've reused the OID for memberOf but I think the adUser OID is from a tutorial or something so it may clash.


dn: m-oid=1.2.840.113556.1.5.8, ou=objectClasses, cn=other, ou=schema
objectclass: metaObjectClass
objectclass: metaTop
objectclass: top
m-oid: 1.2.840.113556.1.5.8
m-name: group
m-must: cn
m-may: member

dn: m-oid=1.2.840.113556.1.2.102,ou=attributeTypes,cn=other,ou=schema
m-usage: USER_APPLICATIONS
m-equality: distinguishedNameMatch
objectClass: metaAttributeType
objectClass: metaTop
objectClass: top
m-name: memberOf
m-oid: 1.2.840.113556.1.2.102
m-obsolete: FALSE
m-noUserModification: FALSE
m-syntax: 1.3.6.1.4.1.1466.115.121.1.12

dn: m-oid=1.3.6.1.4.1.18060.0.4.3.3.1, ou=objectClasses, cn=other, ou=schema
objectclass: metaObjectClass
objectclass: metaTop
objectclass: top
m-oid: 1.3.6.1.4.1.18060.0.4.3.3.1
m-name: adUser
m-supObjectClass: inetOrgPerson
m-must: cn
m-may: memberOf


What's a Dual?

Erik Meijer has been giving a talk at YOW (in Melbourne) and San Francisco at QCON about Key Value Stores being the dual of SQL.

The obvious question is what is a dual? And like most really good questions you already know the answer. Set operations AND and OR are dual. Which means that if A OR F = A is true, by replacing AND with OR and T with F (or as Erik said switching the arrows), you get A AND T = A. Likewise, A OR T = T becomes A AND F = F.

This is the bit where Maths almost becomes magical. Why should these properties hold? Why doesn't this work for multiplication and addition?

In his talk, Erik quoted Benjamin Black, that the noSQL movement took away the relational model and gave nothing back. His talk also seems to be about the beginnings of a noSQL/coSQL algebra and a project to write a noSQL database within Microsoft (noSQL Server perhaps?). He's not the only one thinking about these kinds of things obviously.

He also mentioned, closure as an important property of numbers and other things. That is you plug 1 + 1 in, you get 2 and you can plug 2 into + 1 and get 3 and so on - the answer of one operation can be used for the next. You can't do that with SQL or SPARQL, which in my mind is a big fail. Erik agrees - and suggested that noSQL (or coSQL) should support this property. He's not the only one to think that noSQL should support the Unix philosophy (which is not just closure but is an important part of it). And he's certainly not going to be the last.

He described these properties in the same way as I discovered them in RDF, which was through graphs. The object model in LINQ has a graph of objects - objects which don't have a globally identifiable name - much like blank nodes. The cool thing about LINQ, well one of the many cool things, is that it supports closure - you query your object graph, you get back another object graph and can continue to plug away at it as you narrow your query. This certainly makes a lot of sense from a user querying data perspective (among others) - where you start with all students, you ask for all male students, then you ask for all male students with a grade point average about 5 or whatever. Narrower and narrower is one typical way people find the answer they're looking for.

The other thing he mentions is that SQL is a nightmare due to normalization, the complexity of queries, null semantics and computing over them and the difference between objects and tables. Erik showed that objects and relations are dual.

This is the bit where I introduce monads and say it's like pizza delivery or something. Screw that. It's just good in that if your operations follow laws then they can be mapped to other things and inherit any work done to make them good - like optimisation or scaling. In Erik's talk that meant finding out cool things like noSQL being dual to SQL and LINQ can be done as MapReduce (see this slide of Erik's). This works because LINQ operations are associative and MapReduce operations are also associative. It's not just a good idea it's a monadic law. If you can, read 10 pages on category theory (I'm only up to page 8 though).

Obviously, not everything is peaches. The noSQL/coSQL people say don't do joins - they don't scale - you can't self join on the web for example. In a MapReduce/noSQL system you either end up querying multiple times or you end up processing the data again. I'd suggest that someone will write a library that does multiple queries and joins over noSQL databases so that you don't have to do it yourself - maybe locally and in memory at first, and then with disk based structures for scalability later - I guess this is where LINQ would fit in.

There were lots of dualing properties in the relational vs noSQL/coSQL talk. I only remember a few SQL vs noSQL/coSQL: closed world vs open world, identity by value vs identity by reference (or properties), and (I might be wrong) transactional vs non-transactional.

So with our arrows in hand, ignoring how cool the world will be without JOINs and UNIONs and our mighty knowledge of set theory, what would JOIN and UNION look like if they were a dual and if such a thing existed as 0 and U for relations. U JOIN R = R becomes 0 UNION R = R and likewise U UNION R = U becomes 0 JOIN R = 0 (which matches my existing prejudices).

Just to put the cherry on this pig of a blog post (to mix metaphors), objects and functions look like duals too (given that people claim there's no meaning behind what an object is and what a dual is I feel safe anyway):

But from another perspective, the apply "method" of a closure can be used as a low-level method dispatch mechanism, so closures can be, and are, used to implement very effective objects with multiple methods...With closures seen as a building block with which to implement objects, it's clear that objects are a poor man's closures...If your language only has these restricted closures, and you're forced to build an object system on top of them, it's clear that closures are a poor man's objects.

Erik via Eugene Wigner said there's an unreasonable effectiveness of mathematics and I have to agree. Monads, duals, properties for LINQ, noSQL, relational databases, SPARQL, and so on. Very unreasonable and very cool.

WebDAV No More

This is reasonably old news. And has been obvious pretty much since WebDAV began. But it's interesting to listen to Greg Stein talk about WebDAV, it's history and the reason why it was put into Subversion. At the moment if you want to go over HTTP for Subversion there's not any other option.

In the next version of Subversion, 1.7, there's going to be a new, not your father's, Subverison over HTTP. It basically means stop pretending to be WebDAV and fix it's problems. Perhaps the most interesting of which is that versions will have public URIs rather than going around and around using PROPFINDs. The WebDAV methods remain but it does look like the sprinkle it with some proprietary headers like X-SVN-Version-Name (which seems like a sensible way to send "the version I last knew about" information without the WebDAV tomfoolery).

As well, there's some other Git like features (svn patch to support Git's format). While I probably still favour DCVS for OS projects, Subversion still makes sense a lot of times.

Quality without a Name

Wikipedia on fail fast including a PDF highlighting it as a way to write software.

Searching for this kind of work comes across some nice hybrid articles: Fail fast (for XP), James Shore on the Quality without a Name, also came across Patterns for Scalability, Availability and Stability (including Java 7's ParallelArray).

Java Lamba

No time to explain:

• Exception transparency in Java


• Defender Methods


• Lone throws
  

Functional Mainstream

From reading, "Functional Programming Doesn't Work (and what to do about it)" - I found this post along similar lines, "Functional Programming Went Mainstream Years Ago":

You can view a Perl (or Python or Ruby or Lua or Rebol) program as a series of small functional programs connected by a lightweight imperative program...0% of the impressive magic from early functional languages has been rolled into mainstream languages. That last 10%, well, it's not clear that anyone is really wanting it or that the benefits are actually there. Purity has some advantages, but it's so convenient and useful to directly modify a dictionary in Python. Fold and map are beautiful, but they work just as well in the guise of a foreach loop.

Syntax Hell

As anyone who has applied a more functional style to Java will realize, the Java syntax really gets in the way. I was using my usual guinea pig, JRDF, with "with" so you don't have to iterate but apply a function and so you don't have to close a ClosableIterator. The typical code to print out the graph is:

ClosableIterable<Triple> triples = graph.find(ANY_TRIPLE);
try {
   for (Triple triple : triples) {
     System.out.println("Graph: " + triple);
   }
 } finally {
   triples.iterator().close();
 }

Using "ClosableIterators.with" this becomes:

with(graph.find(ANY_TRIPLE), new Function<Void, ClosableIterable<Triple>>() {
   public Void apply(ClosableIterable<Triple> object) {
     for (Triple triple : object) {
       System.out.println("Graph: " + triple);
     }
     return null;
   }
});

It's typically one line less but that's not that much of an improvement.

JRDF 0.5.6 and testing

JRDF 0.5.6 has JSON SPARQL support and many things got a good refactoring including the MergeJoin (which now looks more like typical text book examples - for better or worse). The upgrade to Java BDB version 4 has improved disk usage and performance mostly around temporary results from finds and the like. The next version will include named graph support.

I'm considering this being the last version to support Java 5.

This release was also about learning things like Hamcrest, JUnit 4.7, various mock extensions (Unitils, Powermock and Spring automocking (which didn't make it due to not being able to mix and match runners). This seems to be a design flaw that I've been encountering with JUnit - you can't mix and match features from various runners. Even Powermock and JUnit's Rules (for exceptions anyway) was problematic. The answer was to go back to the inner class block version.

Simpler SPARQL Results

SPARQL Results have two different, distinct and unnecessary ways of doings query results - one for SPARQL SELECT and one for SPARQL ASK (the whole SELECT vs ASK is worth pursuing too).

The section of a SPARQL SELECT XML result looks like:

...
  <results>
    <result>
      <binding name="x"> ... </binding>
      <binding name="hpage"> ... </binding>
    </result>
...

An ASK result looks like:

...
  <boolean>true</boolean>
...

The with the value inside the boolean either true or false (or possibly fred - depending on how you parse it - it would be nice to have an XSD). One set of results live in a results/result/binding and the other in a boolean. A simpler way to do it is just to use the absence or presence of a binding within the results/result.

For true:

  <results>
    <result>
      <binding/>
    </result>
  </results>

For false:

...
  <results>
    <result/>
  </results>
...

The JSON looks pretty much the same:

{
...
  bindings : [{}] // true
  bindings : []   // false
}

No parsing of true or false or having special cases for code producing or consuming SPARQL different kinds of answers.

A Dangerous Adventure

One of the many messages left by leaders of the world on a silicon disk was this: "May the high courage and the technical genius which made this achievement possible be so used in the future that mankind will live in a universe in which peace, self expression, and the chance of a dangerous adventure are available to all."

Via, Debate over future of US space program.

See and Feel the iPhone

A recent patent backs up an idea I had about a haptic interface for the iPhone (it's not a new idea even for the iPhone):

The proposed solution is the adoption of "haptic" display technologies which allow for some tactile feedback from touch screen displays. Apple proposes including a grid of piezoelectronic actuators that can be activated on command. By fluctuating the frequency of these actuators, the user will "feel" different surfaces as their finger moves across it.

JRDF 0.5.5.5

It's been a long while between updates but it's finally here. There's been some general concessions made to long standing "features" in JRDF namely relational semantics and checked exceptions - both are gone. Yuan-Fang added merge-join support which improved join performance (by up to 8 times). There's Groovy support and a nasty memory leak fixed. It's in the usual place. The next version won't be so far away with some further SPARQL query improvements including perhaps some of the newly proposed features.

Happiness by Empiricism (and no numbers or God)

Daniel Everett: Endangered Languages and Lost Knowledge. There are some quite interesting observations made by Daniel about the Piraha:

...so English has how many verb forms, well it has about 5, sing, sang, sung, singing, sings...Spanish or Portuguese might have 40 different verb forms, well Piraha like many American Indian languages has a very complex verbal system. So Piraha has 16 different suffix that can go with the end of a verb, that gives 2 to the 16th power possible verb forms and that is a lot. That is more than 40. And of those things, 3 suffixes are very important and those tell you how you got your evidence. So every verb has to have on it the source of the evidence, did you hear about it, did you see it with your own eyes, or did you deduce it from the local evidence. So if I say did John go fishing? They can say John went fishing "heai" which means I heard that he did, or they can say John went fishing “sibiga” and that means I deduced that he did, or they can say John went fishing “ha” and that means I saw he went. In some respects they are the ultimate empiricist...

They actually demanded evidence for what I believe and I realized, I could not give it as well as they wanted me to give it. So, this changed my profoundly, but I remember telling them about Jesus one time and they said “So Dan, is Jesus is he brown like us or is he white like you? “I do not know I haven’t seen him.” “What did your dad say? Because your dad must have seen him.” “No, he never saw him.” “Oh what did your friends say who saw him?” “No I do not know anybody who saw him.” “Why are you telling us about him then? Why would you talk about something you do not have evidence for?” But of course we do that all the time.

This is coupled with the fact that they don't have the concept of one, instead there are only relative terms like little amount. They also mention that the Rosetta Project is available on DVD.

Linking Resources without HTML, XHTML, Atom...

In HTML you can link to alternative versions of your HTML using the link tag:
<link rel="alternate" type="application/rss+xml" title="RSS Feed"
href="report.rss" />

There is a way to do linking just using HTTP through the LINK header (as seen in POWDER. It looks very similar but is more powerful, from the IETF draf document:

Link: <http://example.org/>; rel="index start";
rel="http://example.net/relation/other" rev=copyright


Defines 4 links for http://example.org/, three outbound (index, start and http://example.net/relation/other) and one inbound (copyright).

More information: HTTP Status Report and LinkHeader.

Trams in Brisbane and Per Capita Carbon Emissions

China expert raps luxurious Aussie life

Prof Pan was unimpressed with Australia's environmental standards, saying public transport seemed poor and the buildings and street lighting were not energy efficient.

He labelled as "insufficient" Australia's pledge to cut greenhouse emissions by five to 15 per cent by 2020.

The Chinese experts called for a global climate pact that would involve each country being allowed to emit a certain amount, based on their populations.

This is ominous for Australia because it has very high per capita emissions, whereas China has fairly low per capita emissions.

Australian climate adviser Ross Garnaut backed the per capita push in a video address to the conference, saying it was fair.

This was at the Australia-China Climate Change Forum.

Although the biggest mistake is that while there is hope that there is clean coal serious investment in other technologies doesn't exist, as highlighted by the recent Clean Coal Air Freshener advert from This is Reality.

As for public transport, Brisbane used to have trams including a line along Milton Road. There was fire in Paddington where Paddington Central is now and the trams were replaced by buses. The tram e-petition got a massive 600 signatures which puts it well behind things like a petition against recycling water.

MapReduce vs SQL Databases

A Comparison of Approaches to Large-Scale Data Analysis: MapReduce vs. DBMS Benchmarks

...we present the results of running the benchmark on a 100-node cluster to execute each task. We tested the publicly available open-source version of MapReduce, Hadoop [1], against two parallel SQL DBMSs, Vertica [3] and a second system from a major relational vendor.

First, as we demonstrate in Section 4, at 100 nodes the two parallel DBMSs range from a factor of 3.1 to 6.5 faster than MapReduce on a variety of analytic tasks. While MR may indeed be capable of scaling up to 1000s of nodes, the superior efficiency of modern DBMSs alleviates the need to use such massive hardware on datasets in the range of 1–2PB (1000 nodes with 2TB of disk/node has a total disk capacity of 2PB). For example, eBay’s Teradata configuration uses just 72 nodes (two quad-core CPUs, 32GB RAM, 104 300GB disks per node) to manage approximately 2.4PB of relational data. As another example, Fox Interactive Media’s warehouse is implemented using a 40-node Greenplum DBMS. Each node is a Sun X4500 machine with two dual-core CPUs, 48 500GB disks, and 16 GB RAM (1PB total disk space) [7]. Since few data sets in the world even approach a petabyte in size, it is not at all clear how many MR users really need 1,000 nodes.

In section 3.1 there's some points made about the advantages of databases over MR in relation to data integrity, "...a MR framework and its underlying distributed storage system has no knowledge of these rules, and thus allows input data to be easily corrupted with bad data. By again separating such constraints from the application and enforcing them automatically by the run time system, as is done by all SQL DBMSs, the integrity of the data is enforced without additional work on the programmer’s behalf."

They mention that "all DBMSs require that data conform to a well-defined schema, whereas MR permits data to be in any arbitrary format. Other differences also include how each system provides indexing and compression optimizations, programming models, the way in which data is distributed, and query execution strategies."

If you strip it away they are talking about text processing versus indexed data structures (and other parts of a DBMS).

For loading, "Without using either block-or record-level compression, Hadoop clearly outperforms both DBMS-X and Vertica since each node is simply copying each datafile from the local disk into the local HDFS instance and then distributing two replicas to other nodes in the cluster." The obvious difference to me would be that the SQL databases are creating "...a hash partitioned across all nodes on the salient attribute for that particular table, and then sorted and indexed on different attributes..."

For text processing, they note that the main problems with Hadoop are the start-up costs (10-25 seconds before all Map tasks start) and during the Reduce phase the cost of combining many small files. When you are comparing a fully indexed system versus text processing then you would expect the indexed system to be faster. Compression was also considered an advantage in the systems like Vertica's over Hadoop - where it actually reduced performance. It depends on the work being done whether the overhead of compression is worth the overhead so obviously - it's not explained why compression was a negative for Hadoop.

They also talk about the problems in setting up and configuring the parallel databases over Hadoop, which is not an insignificant difference when you are scaling to 100s and 1000s of nodes.

In the summary they talk about 25 years of database development and the advantages of B-Trees and column stores. It begs the question, then why wasn't a similar system used on the Hadoop infrastructure? MR is really more like distributed processing not an indexed, querying system.

If you took away the distributed layer what they are doing is comparing something like grep (a really bad implementation of grep) with Lucene or MySQL. Would anyone be surprised with the results then? A better comparison would've been comparing it against HBase or other distributed, indexed, data stores like Hive or Cloudbase.

Update: There's a good followup on the hadoop list by Jonathan Gray "Hadoop is not suited for random access, joins, dealing with subsets of
your data; ie. it is not a relational database! It's designed to
distribute a full scan of a large dataset, placing tasks on the same nodes
as the data its processing. The emphasis is on task scheduling, fault
tolerance, and very large datasets, low-latency has not been a priority.
There are no "indexes" to speak of, it's completely orthogonal to what it
does, so of course there is an enormous disparity in cases where that
makes sense. Yes, B-Tree indexes are a wonderful breakthrough in data
technology". He suggested Pig, Hive and Cascading would be more suitable for comparison.

TDD for Ontologies

As it was said to me recently, very often something goes from being mysterious and magical to obvious (I'm looking at you not-so-magical-anymore Grails wiring). So it goes with test driving ontologies (or at least it did for me). One of the standard methodologies for ontology development is creating competency questions for an ontology to successfully answer. This generally means gathering requirements, use cases, organising the data and ontology, testing them and maintenance. Doesn't that sound an awful lot like waterfall? It would appear obvious, that just like code, ontologies could be developed using test driven design (neh development) instead. This should have the same benefits as agile code - like being more adaptable to change.

The closest paper I've come to that talks about this idea is, "Unit tests for ontologies" (and there's also bachelor thesis, Unit Testing for Ontologies). Unfortunately, it talks about adding tests during the maintenance of software. Luckily, it talks about using them as a way to initially build the ontology and surely that would be a way to drive out an ontology. Good stuff. I could imagine that certain good patterns of ontology design like making them small, self consistent, fast, intuitive, etc. would all come naturally out of applying TDD techniques.

As an aside the unit testing paper also mentions the usefulness of autoepistemic operators (K and A). You can state that for a given instance there must exist a certain property, for example, a country must have a capital city.

One Way to Describe What I Did Today

I've thought a bit about how best to describe what I did today. I spent a whole day, knowingly, adding more technical debt rather than two days removing existing debt. Now writing bad code is not linear and it's hard to judge but from what I can tell there is now twice as much debt as yesterday. And this is seen as progress.