More News

One Platform

2012-01-26T08:18:00.002+10:00

A couple of things have struck me about the iPad and iBook Author. If you want to read some background John Gruber has a good summary. It may well come down to whether being focused on one thing is wrong.

Firstly, Steve Jobs is quoted in his biography saying he'd give away textbooks. This is a pretty big bargaining chip when Apple was talking to the textbook publishers: go with us or we'll give your product away for free. How does this differ from Bill Gates saying they'd cut off Netscape's oxygen supply?

The other thing is it reminds me of is Bill Gates' developer virtuous cycle. This is where developers write applications for a particular platform, users pick the platform with the most applications which then feeds back to developers supporting that platform. In the past, developers have had to make a single choice as to which platform they wanted to support in order to succeed. It continues to happen with Android and iPhone. Jeff Raikes has given a good example in the early days of Microsoft, "The Principle of Agility", he says:

I suspect many of you or in the audience might if I ask you the question of, "What was Microsoft's first spreadsheet?" You might think Excel was our spreadsheet. But in fact, we had a product called Multiplan...Our strategy was to be able to have our application products run on all of those computing platforms because at that time there were literally hundreds of different personal computers.

And on January 20th, 1983 I realized, I think Bill Gates also realized we had the wrong strategy. Any guesses to what happened on January 20th, 1983? Lotus, it was the shipment of Lotus 1-2-3. How many personal computers did Lotus run on in January of 1983? One, and exactly one. And it was a big winner. So what we learned was when it came to customer behavior. It wasn't whether you had a product that run on all of those computing platforms. What really mattered to the customer was, did you have the best application product on the computer that they own. And Lotus 1-2-3 was the best spreadsheet. In fact, it was the beginning of a "formula for success in applications". That I defined around that time called, "To win big, you have to make the right bet on the winning platform."

So what's the principle? The principle is agility. If you're going to be successful as an entrepreneur what you have to do is you have to learn. You have to respond. You have to learn some more. You have to respond some more. And that kind of agility is very important. If we had stayed on our old strategy, we would not be in the applications business today. In fact, one of the great ironies of that whole episode is that in the late '80s or early '90s our competitors, WordPerfect, Lotus. What they really should have been doing was betting on Windows. But instead they were betting on and WordPerfect was the best example. Betting on, putting WordPerfect on the mainframe, on minicomputers. In fact, they went to the lowest common denominator software strategy which we switched out of in the 1983 timeframe. So, for my key principle is, make sure that you learn and respond. Show that kind of agility.

This is echoed in one of the recent exchanges (about an hour into MacBreak Weekly 283) where Alex Lindsay talks about how important it is to him to make education interesting and how he's not going to wait for standards, he just wants to produce the best. Leo Laporte responds by saying how important it is for an open standard to prevail in order to prevent every child in America having to own an iPad in order to be educated or informed.

You have to wonder if developers have reached a level of sophistication that allows them to use a cross platform solution or whether that will ever happen. I think that it's inevitable that a more open platform will succeed but I'm not sure whether multiple platforms can succeed - we shall see.

If you want to here more there are many interesting conversations around including: Hypercritical 51, MacBreak Weekly 283 and This Week in Tech 337.

Pretending We All Don't Know

2012-01-12T07:30:00.003+10:00

Some amazing writing and performance by Mike Daisy (mp3):

He just walked up to the Foxconn plant and wanted to see if anyone wanted to talk to him:

I wouldn't talk to me...she runs right over to the very first worker...and in short order we cannot keep up...the line just gets longer and longer...everyone wants to talk...it's like they were coming to work everyday thinking, "You know it'd be great? It'd be so great if somebody who uses all this crap we make, everyday all day long, it'd be so great, if one of those people came and asked us what was going on because we would have stories for them...

I haven't gotten all the way through but he has a bit about talking to a girl that cleaned the glass on the assembly line:

You'd think someone would notice this, you know? I'm telling you that I don't know Mandarin, I don't speak Cantonese...I don't know fuck all about Chinese culture but I do know that in my first two hours on my first day at that gate I met workers who were 14 years old, 13 years old, 12. Do you really think that Apple doesn't know? In a company obsessed with the details. With the aluminium being milled just so, with the glass being fitted perfectly into the case. Do you really think it's credible that they don't know? Or are they just doing what we're all just doing, do they just see what they want to see?

It seems absolutely credible that they do know.

Jesus Says Share Files

2012-01-04T07:39:00.000+10:00

A famous story, Jesus takes some fish and loaves (accounts differ - although maybe he did it more than once, setting up the "Jesus's Food Multiplier" stall every second and fourth Saturday of the month) and feeds some people (again accounts differ and they don't count women and children as people - let's just skirt around that entire issue shall we).

Everyone was impressed - even the disciples that were fisherman who had deep ties with the community. They didn't say, "Hey, Jesus you've just destroyed our business model, you can't go around feeding thousands of people per fish. One person, one fish - that's the way it has always been and that's the way it should always be."

Partitioning Graphs in Hadoop

2012-01-03T14:15:00.002+10:00

A recent article at Linked In called "Recap: Improving Hadoop Performance by (up to) 1000x" had a
section called "Drill Bit #2: graph processing" mentioning the problem of partitioning the triples of RDF graphs amongst different nodes.

According to "Scalable SPARQL Querying of Large RDF Graphs" they use MapReduce jobs to create indexes where triples such as s,p,o and o,p',o' are on the same compute node. The idea of using MapReduce to create better indexing is not a new one - but it's good to see the same approach being used to process RDF rather than actually using MapReduce jobs to do the querying. It's similar to what I did with RDF molecules and creating a level of granularity between graphs and nodes as well as things like Nutch.

Why the Cloud isn't the Internet

2011-12-31T16:59:00.000+10:00

I think people are just starting to realize what some of the cloud vendors are providing and their drawbacks. Steve Jobs is quoted in his biography describing the intent behind iCloud:

We need to be the company that manages your relationship with the cloud - streams your videos and music from the cloud, stores your pictures and information, and maybe even your medical data...over the next few years, the hub is going to move from the computer into the cloud...So we wrote all these apps - iPhoto, iMove, iTunes - and tied in our devices, like the iPod and iPhone and iPad...We can provide all the syncing you need, and that way we can lock in the customer.

The Mac has always been different to Windows. One of those differences Windows users' notice is that you switch between applications in OS X compared to documents (or windows) in Windows. The Apple cloud maintains that pattern by syncing between applications rather than documents (or individual files). This approach confuses a lot of people.

This is different to how most Mac users currently sync their files with Dropbox. iCloud has ended up following its Mac heritage whereas Dropbox sticks to file syncing. Matthew writes:

The difference between Dropbox and iCloud synchronization is that Dropbox is theoretically just a file system...If you have a document that you edit on your iPad and sync with Dropbox you can edit that same file, using a different application, on your PC...The iCloud experience is completely different. The only way to edit a document across platforms or devices is to use a version of the application for each device. Not a compatible application...it may actually make me change the desktop application that I use purely based on iCloud support.

If you want to read more about Dropbox and Apple there's a really good article in Forbes which details how Steve Jobs personally made an offer to buy Dropbox.

The edges of iCloud - the integration points to applications and the operating system - it's incomplete even if you buy into the idea of applications over documents.

For example, on iOS devices there is a Notes application but on OS X these notes are in a tab in the Mail application. This seems like a weird and non-standard place to put it - if you are going to sync by application you'd think it should be the same application across platforms.

In iCloud for Windows, Windows users get more choice than OS X users. Mail, Contacts and Calendar integrate with Outlook but you can choose your application for Bookmarks (IE or Safari) and Photo Stream.

Even within applications Apple haven't quite gotten syncing right with iCloud yet either including the new rules around where files are stored and what is automatically removed or backed up.

The cloud is about vendor lock as much as any other platform, like application servers or databases, but with the extra problem that your data is tied to the vendor's application, cloud and user base. A stickier solution.

Some, like Google and Facebook, offer export services, but these almost don't matter, because you get an almost useless hunk of data, lose the ability to run the applications and you can't access users on their network (who may well have been collaborators).

With the Internet, the Web and open source you still have the possibility to use your data with applications shared by many people across different networks.

Global Code Retreat 2012

2011-12-31T16:56:00.000+10:00

I went to the local Global Code Retreat held on the 3rd of December. Overall, it was an amazing event - very well hosted and attended. The basic structure of the day was 6 or so 45 minute sessions trying to implement something with a different person each time. At the end of the 45 minutes, no matter how far you had got, you deleted your solution.

The problem was "The Game of Life". I'm pretty familiar with this problem having come across "Conway's Game of Life" early on in a magazine like Compute! or Byte.

However, if you walked away with a really awesome solution to "The Game of Life" you probably missed the point - most of the things that were being taught were hidden.

The solution was really beside the point. One of the main reasons is to repeat solving the problem from scratch based on an idea called kata (movements practiced by yourself or in pairs). This is something that I had come across in "The Pragmatic Programmer" which at the time reminded me of the time I had spent with projects at home - reimplementing the same thing over and over again.

Steve Yegge mentions the same thing in his article "Practicing Programming". He mentions that even as you program in your day job you may not actually be practicing programming. Repetition in solving the same problems seems to be about keeping the problem fixed and then changing how you approach it and freeing you from any time constraints. Most programming jobs involve solving the solution once (or if you're lucky doing a proof of concept and then implementing it again).

The first time around it was awful. I didn't know what I was doing, my environment was a little bit shaky, we couldn't agree on a language and I spent a lot of the time just setting it up.

It made me become aware that for the first time, practically ever, my personal computer had diverged from my work computer. Not in the "normal" Windows at work, Linux and OS X at home - but what I do at home and at work have diverged to the point where I'm learning stuff in many directions and there's almost no overlap between the two.

The second time was much better. There was less discussion on languages to use, how to approach the problem, how do you test drive it, whose computer to use and so on. There was still discussion but we both shared a bit more context this time which made the discussion flow. A big difference to the first time.

The third time around changed the format a little to where you couldn't talk to the person but you could only express requirements through tests. So this sorted out the people who were testing from those who weren't. But it also seemed to reduce the clutter around what needed to be done. Tests are much less ambiguous compared to talking through requirements and so once you setup a rhythm of tests it became much easier. Also, the whole room was very quiet. You could imagine that a team doing silent TDD and pair programming wouldn't be the noisiest group in the room (for once).

Each round thereafter changed the programming requirements: no loops, methods no more than 3 lines, and no if statements.

What did I learn? Heaps.

I ended up doing Ruby quite a bit and mostly the solution came out at about 30 lines of production code and 30 lines of tests and you could pretty much do it in the time allocated. I also did solutions C# and Haskell. The Haskell solution came out at about 30 lines total - both tests and production code - and met every constraint (no loops, small functions, no if statements).

Doing the same problem over and over again is surprisingly effective and nothing replaces sitting with a person to learn a new language or to be exposed to a variety of solutions. One of the tricks - and you find this frequently with pair programming - you have to be very good at communication - both by saying what you're doing but also getting the other person to explain themselves.

I also learnt:

Even with something as well defined and familiar as "The Game of Life", the solutions were varied and some of the requirements (based on the rules on Wikipedia) were redundant.
My brain is very weak compared to how well Google search works.
Between each new attempt you tend to reflect on each previous solution and see the negatives and positives.
By continually starting a new project setup time was greatly reduced - dependencies that get in the way were slowly reduced - editors, libraries, searching the web, etc.
That it's good to throw code away. It frees you up by allowing you to try different approaches or learn something new (like a different language).
Think before you hack.
A functional approach seemed to be where the answers were converging - meeting all the programming constraints that were given.

A Review of QI Live

2011-11-03T06:52:00.001+10:00

When Douglas Adams visited Brisbane in 2000 (possibly 1999) I had a friend sign a copy of "Starship Titanic" - I was too busy at work to see him myself. I have always been a little disappointed that I didn't take the hour to see him. When Stephen Fry announced "QI Live" on Twitter I made sure I wasn't going to miss out. It was only in Melbourne and Perth (at the time) and bugger it if I was going to Perth, so even though I didn't live in Melbourne, I got some.

As I walked into the theatre the place names of France that sound funny appeared on the screen. I remembered them from a previous episode of the TV series. It didn't matter, they'd just played the theme for "Pinky and the Brain" and were playing Tom Lehrer's "The Elements". I was in a good mood and I was happy to be there.

My wife thought she saw the producer, "What John Lloyd?". Surely not, but soon after, an usher told us to stop taking photos. It was a little annoying because I was mainly taking photos of the theatre - I'd never been to the Queen Victoria Theatre before. Then a prissy voice said that Mr Fry's man servant would like you all to turn off your phones - so I did.

The only bit I now remember of Stephen's opening was him telling a joke about a chicken going to a library. It was told well, I guess, but too long for me because I'd heard it before and spent most of the joke remembering my grandmother telling me it about 20 years before. Maybe that was what he was going for. What was worse, it lead to the fact that a frog in California is the only species in the world to go ribbit. Another recycled fact and I was getting a little bit annoyed.

Colin Lane was the first panelist and pretty forgettable. He and Andrew played tennis with the "Nobody Knows" paddles which was okay the first time. Denise Scott didn't really seem to get the format but she had some good anecdotes - such as being recognised as looking like that person on TV but that she couldn't possibly be that person. Andrew Denton was by far and away the star - he seemingly, confused Stephen by saying that, "If nobody knows, why isn't he on the panel?" It made the whole thing almost worth it. Except that Stephen then went on to spend most of the evening calling him stupid.

The first question was about koalas having fingerprints that are indistinguishable from human fingerprints and that maybe they were doing all the robbing in Australia (which they said has the highest burglary rate in the world). This fact doesn't seem to be true now although it was true in 2001 which is probably closer to when it was written.

Some of the other content that was from previous episodes included: kangaroos not farting and having 3 vaginas, when does the sun set (a video), Beatles' HELP album cover, the most popular song being the default Nokia ring tone and slavery not being illegal in the UK until recently. This is just me guessing but Andrew Denton knew every answer. The members of the audience shouting out certainly did. But then so did Alan and so did I. Well, I might not have known every question. Alan answered over the top of the question for 100 points so I didn't hear what it was.

I might be wrong, but one of the ways the show works is that Alan doesn't know the answer to every question. Sometimes some of the guests do (John Sessions and Rory McGrath) but the point is: Alan is the kind of guy the show is supposed to be educating - he's the audience - he's coming along with us for the ride and a laugh. He probably did reflect some of the audience that night too though, he looked a bit bored and a bit uncomfortable.

Given the amount of recycled content I don't think they would've done this show in the UK but apparently it's okay to do this in Australia.

At the end of the show, when Stephen was awarding the points, he said either the people in the audience were smart or that they had downloaded episodes not shown in Australia from BitTorrent. This is what made me mad. All episodes had been shown on the ABC from series A (you can Google the series number and ABC to find them on iView like Series 4). And some of the facts were also available in the QI books (like Animal Ignorance). Stephen seemed to be saying that we (whoever we are - the producers of the show?) reserve the right to charge you for content that we don't think you should've seen and we're a bit surprised you have because you've really cheated yourselves out of $200 a seat.

I'm still mad. QI has a philosophy about facts and curiosity. Recycling facts, that are now wrong, goes against that philosophy. The original idea for the show seems to have been going against accepting common knowledge and laziness. The show seemed to scream, "Look it up!" or "Why do you think that?". Recycling content was pure laziness. Stephen and Alan knew that the content was recycled and it ruined the show or at least the idea behind the show.

In the end, it's hard to know if you should regret the things you don't do versus the things you do.

As an aside, some interesting facts (sticking with the Australian theme): all living marsupials are from South America not Australia (from Wikipedia) and echidnas and platypus can make custard (as they produce both eggs and milk).

Building a Network Over Transactions

2011-11-02T09:10:00.003+10:00

The new meaning of customer value:a systemic perspective analyses providing value to customers from a systems perspective.

I had a thought, a little while ago, that Google is probably one of the first companies where the users and content providers are basically the same people and that they make money, through adverts, between connecting these two together using search.

It's probably not a new thought but at the time I started to draw a diagram of how it all works. I happened across this diagram (on the left) in this paper (page 4).

A perverse example is when you search for something and the first hit is your own blog. You're now both the producer and consumer of the same content - with adverts sandwiched in the middle - hmm value sandwich.

In the paper they use Google and Apple as examples:

"Google has indeed realized the usability of systemic value-creation principles in building its offering. In contrast to Apple, it uses the value network to generate the revenues. Google provides free, easy-to-use tools for customers to use on the internet, the aim being to generate “eyeballs” for the ads of the advertising customers. In collecting these “eye balls” it has or it creates a product for every internet activity that attracts lots of traffic. From the firm's perspective, the offering elements are integrated to provide the audience for the ads, information being gathered in order to better scope the ads or just to make the customers happy and to promoteother products."

There's an old idea, for the Web anyway, of building a network of customers above extracting value out of each transaction. Over-valuing the creation of the network lead to the whole dot com bubble and I have been thinking about how business models have progressed since then.

One horizontal and one vertical monitor for Ubuntu 11.04

2011-09-05T13:09:00.004+10:00

I recently had problems configuring Ubuntu with dual screens using an NVidia card - the first screen is horizontal and the second screen is vertical - they are both Dell U2711 monitors. The idea is to be able to have Firebug running on the screen without obscuring the web page.

This should be a simple thing. But from what I can tell NVidia's TwinView driver doesn't support different monitor rotations (under Linux). But having X Windows it should still be easy using dual X Screens. It should just be a matter of going to NVidia X Server Settings, selecting "Separate X Screen" and then selecting "Enable Xinerama". Unfortunately for me, this caused general weirdness where the first screen was mostly black and the second screen was displayed horizontally.

The way I fixed the problem was to disable Compiz. The easiest way I found to disable Compiz was to log in using the "Ubuntu Classic (No effects)" session.

Then it was just a matter of enabling multiple XServers and Xinerama and enabling rotation (RandRRotation). Here are the bits in my xorg.conf to rotate my second monitor to the left:

Section "Monitor" 
    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor1" 
    VendorName     "Unknown" 
    ModelName      "DELL U2711" 
    HorizSync       29.0 - 113.0
    VertRefresh     49.0 - 86.0
    Option         "RandRRotation" "on" 
    Option         "DPMS" 
EndSection

Section "Screen" 
    Identifier     "Screen1" 
    Device         "Device1" 
    Monitor        "Monitor1" 
    DefaultDepth    24
    Option         "TwinView" "0" 
    Option         "metamodes" "DFP-2: nvidia-auto-select +0+0" 
    Option         "Rotate" "left"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

When Kworkers Don't

2011-08-08T14:20:00.007+10:00

I recently had the problem where kworker threads were taking up 100% on my Ubuntu box.

Various threads seem to suggest the problem lies with interrupts around PCI or power saving features. The thread with the answer that worked for me, "HELP !!! Zombie attack ... (kworker)"

To turn both of them off use:


noapic acpi=off

I found I only needed to turn off acpi though:


acpi=off

Put that in your grub configuration (sudo vi /etc/default/grub) and restart.

I hate not knowing why though.

End of JRDF

2011-05-08T09:14:00.006+10:00

Many things have changed since I started JRDF in 2003. It feels like JRDF has come to a natural conclusion.

Some of them are things I've failed to do very well: get contributors, implement different file format parsers, find enough time to refactor existing bits, etc. I'm also not that interested in Java anymore (as was becoming increasingly obvious as it did have Scala in there at one point and has some Groovy DSL code in there).

The most recent change I've seen is that JSON has achieved some of what RDF was trying to do and I see it more and more in the way people use it to expose their data in a RESTful way. The tooling is less onerous and the ease of use is higher even if what you get is much less.

Also, external factors like W3C's official RDF API (for Java and Javascript) is largely the same thing but with official backing.

I've enjoyed developing it and meeting and talking to other people in other groups (especially Jena and Sesame). And of course, none of this would've happened if it wasn't for a lot of other people: Paul Gearon, Simon Raboczi, David Wood, David Makepeace, Tom Adams, Yuan Fang-Li, Robert Turner, Brad Clow, Guido Governatori, Jane Hunter, Imran Khan and Abdul Alabri and the other guys and girls Tucana/Plugged In Software/UQ.

Linkage

2011-02-28T07:28:00.004+10:00

Elastic Search another key/value store that allows multiple servers to cluster Lucene and uses a simple JSON API.

Katta a project for using Hadoop and Lucene together.

ElephantDB which uses MapReduce to produce indexes for key/values.

How SimpleGeo Built a Scalable Geospatial Database with Apache Cassandra using Geohash (to flatten a 2d space into 1d) and Cassandra to provide a GIS database for exact match, range and proximity searching. Claims that rebalancing isn't necessary

Google's Percolator paper about providing incremental updates to search indexes instead of using MapReduce.

End of IPv4 or is it: IPocalypse and Vint Cerf and Geoff Huston at the recent Linx Conf Australia

Efficient SPARQL joins, ontologies as games, IOS Semantic Web Journal

TDD as a way of getting things done, Testing Javascript with Jasmine and The fallacy of trading quality in software.

Brisbane Functional Users Group videos

Build Broken Images

2011-01-07T05:45:00.004+10:00

One of the things that we encourage where I work is to visually and aurally notify you when a build has broken. I did some images a while back for this purpose.

They are:

Lemmings (oh no)

Lemmings (blocker - from original sprites)

Watchman (Rorschach's Journal)

Swedish Chef (borked build)

Using SVNKit from within Groovy using SvnAnt

2010-12-13T14:52:00.003+10:00

To load in SVN Ant you use (this assumes it's in the classpath):
ant.typedef(resource: "org/tigris/subversion/svnant/svnantlib.xml")

This then allows you to use the runSvn closures:

runSvn {
  "export"(srcUrl: "", destPath: "")
  "import"(url: "", path: "", message: "")
  "copy"(srcUrl: "", destUrl: "", message: "")
  "delete"(url: "", message: "")
  "checkout"(url: "", destPath: "")
  "propset"(path: "", name: "", value: "")
}

SvnAnt documentation has more details.

Adding Active Directory properties to ApacheDS

2010-12-13T14:49:00.003+10:00

One of the things I have to do during my day is replicate Active Directory in ApacheDS. The most simple properties are member and memberOf. The following uses an LDIF file to add these properties. I've reused the OID for memberOf but I think the adUser OID is from a tutorial or something so it may clash.


dn: m-oid=1.2.840.113556.1.5.8, ou=objectClasses, cn=other, ou=schema
objectclass: metaObjectClass
objectclass: metaTop
objectclass: top
m-oid: 1.2.840.113556.1.5.8
m-name: group
m-must: cn
m-may: member

dn: m-oid=1.2.840.113556.1.2.102,ou=attributeTypes,cn=other,ou=schema
m-usage: USER_APPLICATIONS
m-equality: distinguishedNameMatch
objectClass: metaAttributeType
objectClass: metaTop
objectClass: top
m-name: memberOf
m-oid: 1.2.840.113556.1.2.102
m-obsolete: FALSE
m-noUserModification: FALSE
m-syntax: 1.3.6.1.4.1.1466.115.121.1.12

dn: m-oid=1.3.6.1.4.1.18060.0.4.3.3.1, ou=objectClasses, cn=other, ou=schema
objectclass: metaObjectClass
objectclass: metaTop
objectclass: top
m-oid: 1.3.6.1.4.1.18060.0.4.3.3.1
m-name: adUser
m-supObjectClass: inetOrgPerson
m-must: cn
m-may: memberOf

What's a Dual?

2010-12-13T04:42:00.021+10:00

Erik Meijer has been giving a talk at YOW (in Melbourne) and San Francisco at QCON about Key Value Stores being the dual of SQL.

The obvious question is what is a dual? And like most really good questions you already know the answer. Set operations AND and OR are dual. Which means that if A OR F = A is true, by replacing AND with OR and T with F (or as Erik said switching the arrows), you get A AND T = A. Likewise, A OR T = T becomes A AND F = F.

This is the bit where Maths almost becomes magical. Why should these properties hold? Why doesn't this work for multiplication and addition?

In his talk, Erik quoted Benjamin Black, that the noSQL movement took away the relational model and gave nothing back. His talk also seems to be about the beginnings of a noSQL/coSQL algebra and a project to write a noSQL database within Microsoft (noSQL Server perhaps?). He's not the only one thinking about these kinds of things obviously.

He also mentioned, closure as an important property of numbers and other things. That is you plug 1 + 1 in, you get 2 and you can plug 2 into + 1 and get 3 and so on - the answer of one operation can be used for the next. You can't do that with SQL or SPARQL, which in my mind is a big fail. Erik agrees - and suggested that noSQL (or coSQL) should support this property. He's not the only one to think that noSQL should support the Unix philosophy (which is not just closure but is an important part of it). And he's certainly not going to be the last.

He described these properties in the same way as I discovered them in RDF, which was through graphs. The object model in LINQ has a graph of objects - objects which don't have a globally identifiable name - much like blank nodes. The cool thing about LINQ, well one of the many cool things, is that it supports closure - you query your object graph, you get back another object graph and can continue to plug away at it as you narrow your query. This certainly makes a lot of sense from a user querying data perspective (among others) - where you start with all students, you ask for all male students, then you ask for all male students with a grade point average about 5 or whatever. Narrower and narrower is one typical way people find the answer they're looking for.

The other thing he mentions is that SQL is a nightmare due to normalization, the complexity of queries, null semantics and computing over them and the difference between objects and tables. Erik showed that objects and relations are dual.

This is the bit where I introduce monads and say it's like pizza delivery or something. Screw that. It's just good in that if your operations follow laws then they can be mapped to other things and inherit any work done to make them good - like optimisation or scaling. In Erik's talk that meant finding out cool things like noSQL being dual to SQL and LINQ can be done as MapReduce (see this slide of Erik's). This works because LINQ operations are associative and MapReduce operations are also associative. It's not just a good idea it's a monadic law. If you can, read 10 pages on category theory (I'm only up to page 8 though).

Obviously, not everything is peaches. The noSQL/coSQL people say don't do joins - they don't scale - you can't self join on the web for example. In a MapReduce/noSQL system you either end up querying multiple times or you end up processing the data again. I'd suggest that someone will write a library that does multiple queries and joins over noSQL databases so that you don't have to do it yourself - maybe locally and in memory at first, and then with disk based structures for scalability later - I guess this is where LINQ would fit in.

There were lots of dualing properties in the relational vs noSQL/coSQL talk. I only remember a few SQL vs noSQL/coSQL: closed world vs open world, identity by value vs identity by reference (or properties), and (I might be wrong) transactional vs non-transactional.

So with our arrows in hand, ignoring how cool the world will be without JOINs and UNIONs and our mighty knowledge of set theory, what would JOIN and UNION look like if they were a dual and if such a thing existed as 0 and U for relations. U JOIN R = R becomes 0 UNION R = R and likewise U UNION R = U becomes 0 JOIN R = 0 (which matches my existing prejudices).

Just to put the cherry on this pig of a blog post (to mix metaphors), objects and functions look like duals too (given that people claim there's no meaning behind what an object is and what a dual is I feel safe anyway):

But from another perspective, the apply "method" of a closure can be used as a low-level method dispatch mechanism, so closures can be, and are, used to implement very effective objects with multiple methods...With closures seen as a building block with which to implement objects, it's clear that objects are a poor man's closures...If your language only has these restricted closures, and you're forced to build an object system on top of them, it's clear that closures are a poor man's objects.

Erik via Eugene Wigner said there's an unreasonable effectiveness of mathematics and I have to agree. Monads, duals, properties for LINQ, noSQL, relational databases, SPARQL, and so on. Very unreasonable and very cool.

WebDAV No More

2010-09-12T15:38:00.003+10:00

This is reasonably old news. And has been obvious pretty much since WebDAV began. But it's interesting to listen to Greg Stein talk about WebDAV, it's history and the reason why it was put into Subversion. At the moment if you want to go over HTTP for Subversion there's not any other option.

In the next version of Subversion, 1.7, there's going to be a new, not your father's, Subverison over HTTP. It basically means stop pretending to be WebDAV and fix it's problems. Perhaps the most interesting of which is that versions will have public URIs rather than going around and around using PROPFINDs. The WebDAV methods remain but it does look like the sprinkle it with some proprietary headers like X-SVN-Version-Name (which seems like a sensible way to send "the version I last knew about" information without the WebDAV tomfoolery).

As well, there's some other Git like features (svn patch to support Git's format). While I probably still favour DCVS for OS projects, Subversion still makes sense a lot of times.

Quality without a Name

2010-06-28T07:50:00.003+10:00

Wikipedia on fail fast including a PDF highlighting it as a way to write software.

Searching for this kind of work comes across some nice hybrid articles: Fail fast (for XP), James Shore on the Quality without a Name, also came across Patterns for Scalability, Availability and Stability (including Java 7's ParallelArray).

Java Lamba

2010-06-28T07:19:00.001+10:00

No time to explain:

Exception transparency in Java

Defender Methods

Lone throws

Functional Mainstream

2010-01-17T11:41:00.003+10:00

From reading, "Functional Programming Doesn't Work (and what to do about it)" - I found this post along similar lines, "Functional Programming Went Mainstream Years Ago":

You can view a Perl (or Python or Ruby or Lua or Rebol) program as a series of small functional programs connected by a lightweight imperative program...0% of the impressive magic from early functional languages has been rolled into mainstream languages. That last 10%, well, it's not clear that anyone is really wanting it or that the benefits are actually there. Purity has some advantages, but it's so convenient and useful to directly modify a dictionary in Python. Fold and map are beautiful, but they work just as well in the guise of a foreach loop.

Syntax Hell

2009-12-29T23:43:00.008+10:00

As anyone who has applied a more functional style to Java will realize, the Java syntax really gets in the way. I was using my usual guinea pig, JRDF, with "with" so you don't have to iterate but apply a function and so you don't have to close a ClosableIterator. The typical code to print out the graph is:


ClosableIterable<Triple> triples = graph.find(ANY_TRIPLE);
try {
    for (Triple triple : triples) {
        System.out.println("Graph: " + triple);
    }
} finally {
  triples.iterator().close();
}

Using "ClosableIterators.with" this becomes:


with(graph.find(ANY_TRIPLE), new Function<Void, ClosableIterable<Triple>>() {
     public Void apply(ClosableIterable<Triple> object) {
             for (Triple triple : object) {
                     System.out.println("Graph: " + triple);
             }
             return null;
     }
});

It's typically one line less but that's not that much of an improvement.

JRDF 0.5.6 and testing

2009-11-21T05:49:00.005+10:00

JRDF 0.5.6 has JSON SPARQL support and many things got a good refactoring including the MergeJoin (which now looks more like typical text book examples - for better or worse). The upgrade to Java BDB version 4 has improved disk usage and performance mostly around temporary results from finds and the like. The next version will include named graph support.

I'm considering this being the last version to support Java 5.

This release was also about learning things like Hamcrest, JUnit 4.7, various mock extensions (Unitils, Powermock and Spring automocking (which didn't make it due to not being able to mix and match runners). This seems to be a design flaw that I've been encountering with JUnit - you can't mix and match features from various runners. Even Powermock and JUnit's Rules (for exceptions anyway) was problematic. The answer was to go back to the inner class block version.

Simpler SPARQL Results

2009-08-12T14:55:00.003+10:00

SPARQL Results have two different, distinct and unnecessary ways of doings query results - one for SPARQL SELECT and one for SPARQL ASK (the whole SELECT vs ASK is worth pursuing too).

The section of a SPARQL SELECT XML result looks like:


...
  <results>
    <result>
      <binding name="x"> ... </binding>
      <binding name="hpage"> ... </binding>
    </result>
...

An ASK result looks like:


...
  <boolean>true</boolean>
...

The with the value inside the boolean either true or false (or possibly fred - depending on how you parse it - it would be nice to have an XSD). One set of results live in a results/result/binding and the other in a boolean. A simpler way to do it is just to use the absence or presence of a binding within the results/result.

For true:


  <results>
    <result>
      <binding/>
    </result>
  </results>

For false:


...
  <results>
    <result/>
  </results>
...

The JSON looks pretty much the same:


{
...
  bindings : [{}] // true
  bindings : []   // false
}

No parsing of true or false or having special cases for code producing or consuming SPARQL different kinds of answers.

A Dangerous Adventure

2009-07-26T07:52:00.003+10:00

One of the many messages left by leaders of the world on a silicon disk was this: "May the high courage and the technical genius which made this achievement possible be so used in the future that mankind will live in a universe in which peace, self expression, and the chance of a dangerous adventure are available to all."

Via, Debate over future of US space program.

See and Feel the iPhone

2009-07-12T23:49:00.004+10:00

A recent patent backs up an idea I had about a haptic interface for the iPhone (it's not a new idea even for the iPhone):

The proposed solution is the adoption of "haptic" display technologies which allow for some tactile feedback from touch screen displays. Apple proposes including a grid of piezoelectronic actuators that can be activated on command. By fluctuating the frequency of these actuators, the user will "feel" different surfaces as their finger moves across it.

JRDF 0.5.5.5

2009-07-12T23:37:00.006+10:00

It's been a long while between updates but it's finally here. There's been some general concessions made to long standing "features" in JRDF namely relational semantics and checked exceptions - both are gone. Yuan-Fang added merge-join support which improved join performance (by up to 8 times). There's Groovy support and a nasty memory leak fixed. It's in the usual place. The next version won't be so far away with some further SPARQL query improvements including perhaps some of the newly proposed features.

Happiness by Empiricism (and no numbers or God)

2009-04-27T22:16:00.002+10:00

Daniel Everett: Endangered Languages and Lost Knowledge. There are some quite interesting observations made by Daniel about the Piraha:

...so English has how many verb forms, well it has about 5, sing, sang, sung, singing, sings...Spanish or Portuguese might have 40 different verb forms, well Piraha like many American Indian languages has a very complex verbal system. So Piraha has 16 different suffix that can go with the end of a verb, that gives 2 to the 16th power possible verb forms and that is a lot. That is more than 40. And of those things, 3 suffixes are very important and those tell you how you got your evidence. So every verb has to have on it the source of the evidence, did you hear about it, did you see it with your own eyes, or did you deduce it from the local evidence. So if I say did John go fishing? They can say John went fishing "heai" which means I heard that he did, or they can say John went fishing “sibiga” and that means I deduced that he did, or they can say John went fishing “ha” and that means I saw he went. In some respects they are the ultimate empiricist...

They actually demanded evidence for what I believe and I realized, I could not give it as well as they wanted me to give it. So, this changed my profoundly, but I remember telling them about Jesus one time and they said “So Dan, is Jesus is he brown like us or is he white like you? “I do not know I haven’t seen him.” “What did your dad say? Because your dad must have seen him.” “No, he never saw him.” “Oh what did your friends say who saw him?” “No I do not know anybody who saw him.” “Why are you telling us about him then? Why would you talk about something you do not have evidence for?” But of course we do that all the time.

This is coupled with the fact that they don't have the concept of one, instead there are only relative terms like little amount. They also mention that the Rosetta Project is available on DVD.

Linking Resources without HTML, XHTML, Atom...

2009-04-23T05:28:00.006+10:00

In HTML you can link to alternative versions of your HTML using the link tag:
<link rel="alternate" type="application/rss+xml" title="RSS Feed"
href="report.rss" />

There is a way to do linking just using HTTP through the LINK header (as seen in POWDER. It looks very similar but is more powerful, from the IETF draf document:


Link: <http://example.org/>; rel="index start"; 
rel="http://example.net/relation/other" rev=copyright

Defines 4 links for http://example.org/, three outbound (index, start and http://example.net/relation/other) and one inbound (copyright).

More information: HTTP Status Report and LinkHeader.

Trams in Brisbane and Per Capita Carbon Emissions

2009-04-16T06:40:00.007+10:00

China expert raps luxurious Aussie life

Prof Pan was unimpressed with Australia's environmental standards, saying public transport seemed poor and the buildings and street lighting were not energy efficient.

He labelled as "insufficient" Australia's pledge to cut greenhouse emissions by five to 15 per cent by 2020.

The Chinese experts called for a global climate pact that would involve each country being allowed to emit a certain amount, based on their populations.

This is ominous for Australia because it has very high per capita emissions, whereas China has fairly low per capita emissions.

Australian climate adviser Ross Garnaut backed the per capita push in a video address to the conference, saying it was fair.

This was at the Australia-China Climate Change Forum.

Although the biggest mistake is that while there is hope that there is clean coal serious investment in other technologies doesn't exist, as highlighted by the recent Clean Coal Air Freshener advert from This is Reality.

As for public transport, Brisbane used to have trams including a line along Milton Road. There was fire in Paddington where Paddington Central is now and the trams were replaced by buses. The tram e-petition got a massive 600 signatures which puts it well behind things like a petition against recycling water.

MapReduce vs SQL Databases

2009-04-15T05:40:00.016+10:00

A Comparison of Approaches to Large-Scale Data Analysis: MapReduce vs. DBMS Benchmarks

...we present the results of running the benchmark on a 100-node cluster to execute each task. We tested the publicly available open-source version of MapReduce, Hadoop [1], against two parallel SQL DBMSs, Vertica [3] and a second system from a major relational vendor.

First, as we demonstrate in Section 4, at 100 nodes the two parallel DBMSs range from a factor of 3.1 to 6.5 faster than MapReduce on a variety of analytic tasks. While MR may indeed be capable of scaling up to 1000s of nodes, the superior efﬁciency of modern DBMSs alleviates the need to use such massive hardware on datasets in the range of 1–2PB (1000 nodes with 2TB of disk/node has a total disk capacity of 2PB). For example, eBay’s Teradata conﬁguration uses just 72 nodes (two quad-core CPUs, 32GB RAM, 104 300GB disks per node) to manage approximately 2.4PB of relational data. As another example, Fox Interactive Media’s warehouse is implemented using a 40-node Greenplum DBMS. Each node is a Sun X4500 machine with two dual-core CPUs, 48 500GB disks, and 16 GB RAM (1PB total disk space) [7]. Since few data sets in the world even approach a petabyte in size, it is not at all clear how many MR users really need 1,000 nodes.

In section 3.1 there's some points made about the advantages of databases over MR in relation to data integrity, "...a MR framework and its underlying distributed storage system has no knowledge of these rules, and thus allows input data to be easily corrupted with bad data. By again separating such constraints from the application and enforcing them automatically by the run time system, as is done by all SQL DBMSs, the integrity of the data is enforced without additional work on the programmer’s behalf."

They mention that "all DBMSs require that data conform to a well-deﬁned schema, whereas MR permits data to be in any arbitrary format. Other differences also include how each system provides indexing and compression optimizations, programming models, the way in which data is distributed, and query execution strategies."

If you strip it away they are talking about text processing versus indexed data structures (and other parts of a DBMS).

For loading, "Without using either block-or record-level compression, Hadoop clearly outperforms both DBMS-X and Vertica since each node is simply copying each datafile from the local disk into the local HDFS instance and then distributing two replicas to other nodes in the cluster." The obvious difference to me would be that the SQL databases are creating "...a hash partitioned across all nodes on the salient attribute for that particular table, and then sorted and indexed on different attributes..."

For text processing, they note that the main problems with Hadoop are the start-up costs (10-25 seconds before all Map tasks start) and during the Reduce phase the cost of combining many small files. When you are comparing a fully indexed system versus text processing then you would expect the indexed system to be faster. Compression was also considered an advantage in the systems like Vertica's over Hadoop - where it actually reduced performance. It depends on the work being done whether the overhead of compression is worth the overhead so obviously - it's not explained why compression was a negative for Hadoop.

They also talk about the problems in setting up and configuring the parallel databases over Hadoop, which is not an insignificant difference when you are scaling to 100s and 1000s of nodes.

In the summary they talk about 25 years of database development and the advantages of B-Trees and column stores. It begs the question, then why wasn't a similar system used on the Hadoop infrastructure? MR is really more like distributed processing not an indexed, querying system.

If you took away the distributed layer what they are doing is comparing something like grep (a really bad implementation of grep) with Lucene or MySQL. Would anyone be surprised with the results then? A better comparison would've been comparing it against HBase or other distributed, indexed, data stores like Hive or Cloudbase.

Update: There's a good followup on the hadoop list by Jonathan Gray "Hadoop is not suited for random access, joins, dealing with subsets of
your data; ie. it is not a relational database! It's designed to
distribute a full scan of a large dataset, placing tasks on the same nodes
as the data its processing. The emphasis is on task scheduling, fault
tolerance, and very large datasets, low-latency has not been a priority.
There are no "indexes" to speak of, it's completely orthogonal to what it
does, so of course there is an enormous disparity in cases where that
makes sense. Yes, B-Tree indexes are a wonderful breakthrough in data
technology". He suggested Pig, Hive and Cascading would be more suitable for comparison.

TDD for Ontologies

2009-04-13T21:49:00.007+10:00

As it was said to me recently, very often something goes from being mysterious and magical to obvious (I'm looking at you not-so-magical-anymore Grails wiring). So it goes with test driving ontologies (or at least it did for me). One of the standard methodologies for ontology development is creating competency questions for an ontology to successfully answer. This generally means gathering requirements, use cases, organising the data and ontology, testing them and maintenance. Doesn't that sound an awful lot like waterfall? It would appear obvious, that just like code, ontologies could be developed using test driven design (neh development) instead. This should have the same benefits as agile code - like being more adaptable to change.

The closest paper I've come to that talks about this idea is, "Unit tests for ontologies" (and there's also bachelor thesis, Unit Testing for Ontologies). Unfortunately, it talks about adding tests during the maintenance of software. Luckily, it talks about using them as a way to initially build the ontology and surely that would be a way to drive out an ontology. Good stuff. I could imagine that certain good patterns of ontology design like making them small, self consistent, fast, intuitive, etc. would all come naturally out of applying TDD techniques.

As an aside the unit testing paper also mentions the usefulness of autoepistemic operators (K and A). You can state that for a given instance there must exist a certain property, for example, a country must have a capital city.

One Way to Describe What I Did Today

2009-03-31T21:39:00.004+10:00

I've thought a bit about how best to describe what I did today. I spent a whole day, knowingly, adding more technical debt rather than two days removing existing debt. Now writing bad code is not linear and it's hard to judge but from what I can tell there is now twice as much debt as yesterday. And this is seen as progress.

Things Get Dark

2009-03-20T07:38:00.003+10:00

The Crisis of Credit Visualized. It was quite good except maybe the explanation of leverage was a bit extreme, with 10,000 turning into 990,000.

Haptic Technology for the iPhone

2009-03-19T20:59:00.004+10:00

One of the questions that often occurs is how could you improve on the iPhone without changing the form factor too much. One of the things that occurred to me was mentioned was a keyboard. But it would be nice to have a keyboard that worked as well as the current one does (well better) so that it could work in both portrait and landscape mode. Is that crazy talk?

Well the idea would be to use electroheological fluid to create a keyboard dynamically. I came across some discussion on how to use it for braille and it should work equally well for creating a QWERTY keyboard - although power usage maybe a problem. For more technical discussion (and I haven't really read it except for the practical use in virtual reality gloves), Haptic Interfaces Using
Electroheological Fluids.

You Should Read This

2009-03-12T15:34:00.000+10:00

Tony's letter to the Medical Board of Queensland.

Some Nice Programming with Jenabeans

2009-03-12T13:46:00.002+10:00

Writing out SIOC triples using Jena + Jenabean "Jenabean’s model connected programming model makes this easy, using interfaces that declare each of the vocabularies as a set of methods." The code example shows how you can piece together (using a fluent interface) different vocabularies:

Thing thing = new Thing(m);
thing.at(uri1).
as(DCTerms.class).
title("Creating connections between discussion clouds with SIOC").
created("2006-09-07T09:33:30Z").
isa(Sioc.Post.class).
has_container(thing.at(uri2)).
...

CloudBurst

2009-03-12T10:19:00.003+10:00

This is software that uses MapReduce to handle massive amounts of sequence data and reassemble it. It includes an explanation of the algorithmic side of things. I wonder how it compares with using de Bruijn graphs.

With the Passage of Time

2009-03-11T18:17:00.003+10:00

Who would've thought that RoboCop was overly optimistic.

First, it was assumed that people would still be living there. Second, that they'd still be able to make things like robots. And third, that cars in America would get 8 mpg and not 1-3.5 mpg.

URLs for 10 minutes

2009-03-03T13:39:00.006+10:00

Protect Your Site With URL Rewriting seems a little bit of a mad suggestion. They are basically suggesting that you change your application's URIs every 10 minutes to prevent XSS and XSRF (Cross Site Request Forging).

We could mitigate much of the risk of these vulnerabilities by frequently changing our URLs—not once every 200 years but once every 10 minutes. Attackers would no longer be able to exploit application vulnerabilities by mass e-mailing poisoned hyperlinks because the links would be broken and invalid by the time the messages reached their intended victims. With all due respect to Sir Tim, while "cool" URIs may not change, secure ones certainly do.

The negatives seem vast and I wonder what this is really trying to solve, as they say at the end:

An automatically expiring URL can still be exploited by an attacker with access to a Web server of his own. Instead of sending out malicious hyperlinks that point directly to the vulnerable page, he can send out hyperlinks that point to his own site. When his site gets a hit from one of the phished e-mails, it can contact a landing page on the vulnerable site to obtain a valid time stamp and then redirect the user accordingly.

If you are a REST advocate then maybe a quick read over, The Resource-Oriented Architecture in Action, may soothe you (from the excellent book, RESTful Web Services).

A Quick Survey of Bio-Ontologies

2009-03-03T10:38:00.007+10:00

The other day I was trying to find a paper that talks about the need for ontologies in biology that dated sometime around the early 90s - way before OWL and the Semantic Web. I couldn't find the paper I was thinking about, but here are some others that are pretty good that seem to follow at least the same themes:

"Ontologies for molecular biology": "Molecular biology has a communication problem. There are many databases using their own labels and categories for storing data objects and some using identical labels and categories but with a different meaning."

"Ontological Foundations for Biology Knowledge Models": This one was good because it talked about processes and transformations which is where the rules and inferencing stuff comes in.

"Toward Principles for the Design of Ontologies Used for Knowledge Sharing": While not specifically about biology, this is probably the most cited paper and it's what I often think about when you're explaining ontologies and the process of improvement that occurs when you make one.

"Bio-ontologies: current trends and future directions": This covers the process and the parts that make a good ontology on the web. I guess the key for the use of the Web for ontologies is a means to share knowledge. It also has a good history of ontologies going back to the 1600s.

And this one is one of the original GO papers, "Gene Ontology: tool for the unification of biology".

Plastic Ocean

2009-03-03T09:58:00.006+10:00

"Sailing the Great Pacific Garbage Patch" has some interesting points including that there's no such thing as a free range fish (because the oceans are so polluted) and that the island of plastic is more a concentration of plastic - which sounds far worse as far as the environment is concerned.

Adding Some Herbs to Open Databases

2009-03-03T09:41:00.002+10:00

"Harnessing the Crowd to Make Better Drugs: Merck’s Friend Nails Down $5M to Propel New Open Source Era"

Friend, 54, is leaving his high-profile job as Merck’s senior vice president of cancer research, after having nailed down $5 million in anonymous donations to pursue this vision at a nonprofit organization getting started in Seattle called Sage.

Sage is built on the premise that vast networks of genes get perturbed, or thrown off-kilter, in complex diseases like cancer, diabetes, and obesity. Scientists can’t just pick one faulty gene or protein and make a magic bullet to shut it down. But what if researchers around the world capturing genomic profiles on patients could get all of their data to talk to each other through a free, open database? A researcher in Seattle looking at how all 35,000 genes in breast cancer patients are dialed on or off at a certain stage of illness might be able to make critical comparisons by stacking results up against a deeper and broader data pool that integrates clinical, genetic, and other molecular data from peers in, say, San Francisco, New Haven, CT, or anywhere else.

Some big names have signed on for the early incubating phase. Besides the full-time efforts of Friend and Schadt, the Sage board includes Nobel Laureate Lee Hartwell of the Fred Hutchinson Cancer Research Center; Paul Ramsey dean of the School of Medicine at the University of Washington; Richard Lifton, the chairman of genetics at Yale University; and Hans Wigzell, director emeritus of Sweden’s Karolinska Institute. For insight into how to apply lessons from the open-source computing world, the board has brought on John Wilbanks, the vice president of science at the San Francisco-based Creative Commons.

As with any far-out vision, plenty of things can derail it along the way. What if researchers use different gene analysis machines, from Affymetrix, Illumina, or Applied Biosystems? How will Sage reconcile differences in how experiments are designed by different scientists? How will researchers be enticed to let go of their precious data, currently stored on password-protected hard drives and servers? How will Sage manage the intellectual property that arises from the database? Why would companies want to participate and run the risk of putting valuable proprietary data out in public? How will this get financed?

Some of these things Friend can answer, and some still need to be worked out. Software is already making it possible to manage differences between the various instruments scientists use, and deal with the differences in experimental design, Friend says.

LOL (List of Links)

2009-02-12T11:15:00.031+10:00

Hamcrest integrated in JUnit 4.4 and above (JMock like expectations). Also allows Popper theories (the link in the release notes is available through archive.org). which are similar to TestNG's data points. There are also some examples: here, here, here and here (using Groovy).

Window Licker an interesting approach to Swing testing. See the Calculator example and XP Day 2008 presentation.

The bushfires are retribution for the damned. Apparently, God waited 5 months (or maybe he was too busy appearing in toast) to smite those Victorians down. The most annoying part of this is the fuzzy thinking. Instead of considering that it might be poor fuel management practices, bad advice on whether to stay or leave or global warming it's actually about abortion. If you can speak of a real Christian attitude, the people of North Queensland donating their flood relief money straight to the Victorians seems more along the right lines.

Haskell book to read and buy. Luckily in that order.

SemWeb project VoiD a simple vocabulary for linking and describing different data sets.

A couple of ways of fixing the billion dollar mistake using Java Rebel or maybe Maybe Monad (using Java 5 for loops even). Speaking of which: Monoid fingertree (trees in Haskell) and JQuery is a Monad (apparently that makes it slow). I also realized that Void is a valid type in Java Generics which is helpful in the dreaded Visitor pattern.

Bill Moyers on Gaza and Dr Izzeldine Abuelaish sharing his grief on Israeli television. More information is available from ABC's Foreign Correspondent story, "The Doctor and his Daughters" because the wikipedia page has been deleted.

The Poppendieck's on Lean development as to balance Martin Fowler on Scrum.

Straight Skeleton as a way to generate a skeleton from a polygon.

RESTEasy JBoss' REST framework.

Microsoft's terrible vengence, Songsmith. If you love music don't click here.

And What's up for Java in 2009?

Metadata for Fork Sake

2008-12-12T09:35:00.009+10:00

Once in a while, I get excited and frustrated about the state of metadata - particularly in file systems. Previously, I've written about rumors of OS X doing metadata better and the wonders of BeOS. A resource fork is something that Macs have had for ages, although it has appeared in other systems before and since:

The concept of a resource manager for graphics objects, to save memory, originated in the OOZE package on the Alto in Smalltalk-76...Although the Windows NT NTFS can support forks (and so can be a file server for Mac files), the native feature providing that support, called an alternate data stream, has never been used extensively...Early versions of the BeOS implemented a database within the filesystem...Though not strictly a resource fork, AmigaOS stores meta data in files known as .info files...NeXT operating systems NeXTSTEP and OPENSTEP, and its successor, Mac OS X, and other systems like RISC OS implemented another solution. Under these systems the resources are left in an original format, for instance, pictures are included as complete TIFF files instead of being encoded into some sort of container.

This links to the Grand Unified Model 1 and Grand Unified Model 2 which are also good for a few quotes:

Almost every piece of data in the Macintosh ended up being touched by the Grand Unified Model. Even transient data, data being cut and pasted within and between applications, did not escape. The Scrap Manager labeled each piece of data on the clipboard with a resource type. In another Mac innovation, multiple pieces of data, each of a different type, could be stored on the clipboard simultaneously, so that applications could have a choice of representation of the same data (for example, storing both plain and styled text). And since this data could easily be stored on disk in a resource file, we were able to provide cutting and pasting of relatively large chunks of data by writing a temporary file called the Clipboard.

Since resource objects were typed, indicating their internal data format, and had ID's or names, it seemed that files should be able to be typed in the same way. There should be no difference between the formats of an independent TEXT file, stored as a standalone file, and a TEXT resource, stored with other objects in a resource file. So I decided we should give files the same four-byte type as resources, known as the type code. Of course, the user should not have to know anything about the file's type; that was the computer's job. So Larry Kenyon made space in the directory entry for each file for the type code, and the Mac would maintain the name as a completely independent piece of information.

Restlet Talk

2008-12-09T09:22:00.005+10:00

I spoke last night at the Java Users Group about Restlet. It's a basic introduction to both Restlet and trying to link data across web sites. I wasn't very happy with the example - it was basically stolen from a Rails introduction. At least I could answer the question about why you would allow your data to be searched (to sell adverts on your recipe web site). I think it went down okay, mainly because most Java developers are used to large frameworks and complicated APIs in order to do what Restlet does (so it's impressive), the Rails developers knew some of the concepts already and while most are wary of RDF, SPARQL, OWL and the Semantic Web stack it was a fairly incremental addition to achieve something reasonably powerful.

Getting Groovy with JRDF

2008-12-04T08:51:00.020+10:00

In an effort to speed up and improve the test coverage in JRDF I've started writing some of the tests in Groovy. It's been a good experience so far - so much so that I'm probably not going back to writing tests in Java in the future.

One of the things I wanted to try was a RdfBuilder, this is similar to Groovy's NodeBuilder.

There are a couple of things that make it a bit tricky. When parsing or debugging builders I haven't yet found a way to find the methods/properties available even using MetaClass. And of course, when the magic goes wrong it's a bit harder to debug Groovy versus java.

It certainly smartens up the creation of triples, for example (bits from the NTriples test case):


def rdf = new RdfBuilder(graph)
rdf.with {
  namespace("eg", "http://example.org/")
  namespace("rdfs", "http://www.w3.org/2000/01/rdf-schema#")
  "eg:resource1" "eg:property":"eg:resource2"
  "_:anon" "eg:property":"eg:resource2"
  "eg:resource1" "eg:property":"_:anon"
  (3..6).each {
    "eg:resource$it" "eg:property":"eg:resource2"
  }
  "eg:resource7" "eg:property":'"simple literal"'
  "eg:resource17" ("eg:property":['"\\u20AC"', 
    '"\\uD800\\uDC00"', '"\\uD84C\\uDFB4"', '"\\uDBFF\\uDFFF"'])
  "eg:resource24" "eg:property":'"<a></a>"^^rdfs:XMLLiteral'
  "eg:resource31" "eg:property": '"chat"@en'
}

The first two lines defines the two namespaces used. The third line shows the general use of RDF and Groovy. It works out well, an RDF predicate and object maps to an attribute and value in Groovy. The next two lines show how you refer to the same blank node across two statements. And the following lines show using ranges and creating different types of literals. The third last line creates 4 triples with the same subject and predicate but with different objects.

Using the builder results in a file that's smaller than the test case file. You could remove some duplication by creating a method that takes in a number and the object and generates "eg:resource$number" "eg:property" "$object" but doing that may actually make it harder to read.

If you stick to only using URIs you can do things like:


rdf.with {
  urn.foo6 {
    urn.bar {
      urn.baz1
      urn.baz2
    }
  }
}

Which produces two triples: "urn:foo6, urn:bar, urn:baz1" and "urn:foo6, urn:bar, urn:baz2".

I expect that JRDF will only be more Groovy friendly in the future.

Turing in Life

2008-11-27T14:28:00.001+10:00

Reading the Wikipedia entry on Conway's Game of Life answers the question of why it was developed "Conway was interested in a problem presented in the 1940s by renowned mathematician John von Neumann, who tried to find a hypothetical machine that could build copies of itself and succeeded when he found a mathematical model for such a machine with very complicated rules on a rectangular grid." It's interesting that von Neumann's idea of self replication actually predates the discovery of DNA by a few years.

So I asked Google the question and someone has implemented a Turing machine inside Coway's Game of Life; way back in 2000. A book called "Collision Based Computing" and a applet called LogiCell (which uses Conway's Game of Life to do simple calculations) is available here.

Indexing for Efficient SPARQL

2008-11-12T08:41:00.004+10:00

Another interesting way of indexing triples: A role-free approach to indexing large RDF data dets in secondary memory for efficient SPARQL evaluation "We propose a simple Three-way Triple Tree (TripleT) secondary-memory indexing technique to facilitate efficient SPARQL query evaluation on such data sets. The novelty of TripleT is that (1) the index is built over the atoms occurring in the data set, rather than at a coarser granularity, such as whole triples occurring in the data set; and (2) the atoms are indexed regardless of the roles (i.e., subjects, predicates, or objects) they play in the triples of the data set. We show through extensive empirical evaluation that TripleT exhibits multiple orders of magnitude improvement over the state of the art on RDF indexing, in terms of both storage and query processing costs."

While looking around at arXiv I did a quick search and found two more interesting papers that seems related to a previous discussion on how the Semantic Web needs it's own programming language or I would say at least a way to process the web of data, both by Marko A. Rodriguez: "The RDF Virtual Machine" and "A Distributed Process Infrastructure for a Distributed Data Structure".

While you were away...

2008-11-11T07:43:00.004+10:00

Now that I'm currently looking around for jobs, I came across a presentation on some of the work the the easyDoc project did at Suncorp, "Technical Lessons Learned Turning the Agile Dials to Eleven". It includes automating getter/setter testing, hibernate, and immutability. It's good to see the sophistication continued to increase after I left to reach quite a high level (like automatic triangulation and doing molecule level testing).

Cats Stealing Dogs' Jobs

2008-10-30T14:14:00.003+10:00

Who knew that people from Japan read my blog. Tama the station master (actually in true Japanese style "super station master") is doing almost the exact thing I mentioned. Except they added merchandising - those damn cats.

A Billion Triples in your Pocket

2008-10-30T03:57:00.016+10:00

At the Billion Triples Challenge this afternoon Cathrin Weiss showed off i-MoCo. It's a demonstration not only of a Semantic Web application with an iPhone using a stupid number of triples, but also of their indexing technique Hexastore which does all 6 indexes of RDF's subject, predicate and object in order to improve querying (they are ordered which allows you to do merge sorts). Actually, this made me think that the next steps in RDF triple stores will be indexes that are optimised for SPARQL operations and OWL inferences. Indexes for transitive closure perhaps? The data is regular and the storage is available to index triples that improve querying performance.

That wasn't the only impressive demo today. For me it's a toss up between another iPhone SemWeb demo, DBPedia mobile and SemaPlorer. DBPedia stood out because it was the only one that allowed you to write data to the Semantic Web rather than just read the carefully prepared triples. For a similar reason I thought SemaPlorer was good because they tried to do more by making it more generic like integrating flickr. But they were all excellent, all of them showing what you get with a billion or more triples and inferencing.

That combined with the guys at Freebase making all of their data available as RDF and it was a big day for the Semantic Web.

Update: I noticed that John Giannadrea's talk when he mentioned the three aspects to Freebase he went from the bottom up - probably reading too much into it.

Update: Also caught an interview with John Francis the guy who stopped talking, started walking, stopped taking mechanical transport and tried to change the world.

Update: ISWC 2008 awards have been published.

Time for REST

2008-10-22T21:57:00.002+10:00

The best part of the latest release of JRDF is not code that we've written but Restlet. Restlet provided an excellent abstraction for providing RESTful web services and allows deployment without a container (the distributed and local server are both about ~6MB in size).

This has allowed us to quickly develop a web service that answers SPARQL queries that can be from one or many machines. We're using it to query the results from MapReduce tasks in our Hadoop cluster but it could probably be used as a general way to query other SPARQL services.

It's still early days and there's a lot that still needs to be added (and re-added) such as limiting result sizes, being able to choose which format to return and better JSON support. A lot of it has been pushed out in time to show a front-end at SSWS.

It would be good to have a Javascript client that submits the SPARQL queries and handles the ordering and such rather than having the distributed query server hang onto all results until returning.

Download it here.

President of Real America

2008-10-22T09:01:00.003+10:00

Yes, Virginians you may not be in the real part, "Jon Stewart Clarifies Palin Remarks, Expands To 'F%ck All Y'All'". America seems to have gone completely mental. The second video covers the current lunacy including the pro and anti American members of congress and the endorsement of Colin Powell, a black militant backing Obama, what a surprise.

Case Classes

2008-10-21T07:03:00.000+10:00

Reading "Is Scala Not “Functional Enough”? linked to a post I've read before "Are Scala's case classes a failed experiment?" but hadn't blogged about it. In Java I simply avoid using switching based on type (for example, if foo.getClass() else if bar.getClass()) I really like case classes in Scala especially for language processing.

5TB Processing

2008-10-10T09:36:00.005+10:00

Scaling Hadoop to 4000 nodes at Yahoo! "The 4000-node cluster throughput was 7 times better than 500’s for writes and 3.6 times better for reads even though the bigger cluster carried more (4 v/s 2 tasks) per node load than the smaller one." They did some performance monitoring of the cluster and achieved sorting 6TB of data in 37 minutes.

Update: Ex-Google and Yahoo employees create start-up (called Cloudera) and Hadoop Primer by Sun (PDF).

Update: Hadoop users group meeting with IBM talking about different join techniques and Cloudbase project which is an SQL abstraction over log files (tutorial PDF).

Update: Microsoft's first project is Hadoop.

Monks Practicing Evolution

2008-10-08T07:43:00.000+10:00

Before the printing press allowed exact copies of texts, such as Bibles and other works, scribes would copy manuscripts by hand. These copies were imperfect and these mistakes would then be replicated as other scribes made further copies. The implication is that the church was practicing evolution before science had even discovered it. Darwin could've just popped down to his local monastery or church instead of cruising around the world.

Scientists have used phylogentic software to look at these texts in order to discovery the original document sources (creating a book of life if you will). Like evolution in the natural world, the mutations aren't random and you get errors such as recombination, lateral transfer, deletions, and even convergent evolution. There are some interesting relationships that can be determined, such as certain areas of text are more likely to have mistakes in them than others.

What's cool about a theory, such as evolution, is that it can be applied to many different areas such as natural languages, behavioural patterns, archaeological artifacts, and written works such as chain letters and medieval manuscripts.

More information, Manuscript evolution and Phylogenetics of artificial manuscripts.

Coffee Inspired

2008-10-02T07:59:00.005+10:00

The Global graphs in JRDF was inspired by the work done in MSG (minimum self-contained graphs, published in the RDFSync paper) and RDF Molecules. The former links to an implementation of DBin (P2P Semantic Web client) and there's also GVS (Graph Versioning System).
It's a trap (cloud computing) It's a fairly typical Stallman statement - not wrong but not aware of the compromises people make. It is obvious that one of the reasons vendors are excited about cloud computing is because it is a chance for them to try and own your data (or at least make switching too hard). But you do have to do more than put data in the cloud - you do have to have executable services there. There are open source cloud computing infrastructure that you can run-up on your own servers like Hadoop or CouchDB. And it's not just back to mainframes and renting CPU time out by the hour it really is different to what has gone before.

Live the Cloud Life lists the cloud computing applications in categories such as email, documents, data, music, photo editing and storing and browser synchronisation. Some categories are missing like RSS reading and calendaring.

Apple drops NDA - the outrage worked.

The main eResearch 2008 conference is over and some of the papers are available.

Speaking of which, all development, and that definitely includes software, in whatever organisation (universities, governments, banks, etc) should have failure as an option. One sign of a truly stuffed culture is to never have a project fail.

Muradora a refactoring of Fedora to allow pluggable authentication and enable metadata editing.

Acer Aspire One links: Dual Monitor Support,
Installing Firefox 3 on Acer Aspire One Linux and
Updated repositories. It's a shame that OpenOffice doesn't support presenters view yet.

More Data

2008-09-30T09:20:00.000+10:00

Neurocommons have released their integrated RDF datasets. It is composed of different modules or bundles including MeSH, Medline, OBO and others.

Who Holds the Power?

2008-09-24T08:45:00.039+10:00

On Monday I went and saw John Wilbanks talk on "Publishing in Today`s Environment" (I can't seem to find a decent URL but a good overview of some of the topics discussed is "The Open Access Interviews: John Wilbanks"). I didn't take many notes but a few things have stuck with me:

Libraries providing a role as repositories of data (both public and private).

The idea of "free as in puppy" in relation to digital curation - a great metaphor. When someone gives you a puppy it is initially free but the upkeep of it is anything but.

The Queensland Government has its own Creative Commons initiative and the Australian Government is following suite.

There's a power struggle occurring between researchers and publishers to make data, papers and the like freely available. The power used to be on the side of publishers but as the producers band together (by country, university, faculty and so on) the power is going back to them.

Creative Commons are continuing to fight, in cunning ways, to get back to decent copyright law.

The kind of behavior that publishers have exhibited appears to be on the way out and seems to be going the other way. This is where databases interoperate with each other, papers can link to the original data, and tools, data and papers are all part of an integrated experience. The linking and integration of papers and other artifacts seems to be a one way process, it's hard to imagine a developer, scientist, arts graduate or anyone else spending huge amounts of money and time finally wresting control from one bunch of people only to tie it to another.

This leads to the T-Mobile G1 announcement. The announcement was very underwhelming - guys speaking for the first 10-15 minutes saying what a wonderful job they had done and the guy talking about how good an experience the phone was at playing Pacman. The more intersting part is the open platform. Much like the publishing area the ability to have open access will be a massive differentiator. It reminded me a bit about the discussion on how AOL, Prodigy and the other closed networks quickly died when faced with the open web:

By contrast, the proprietary networks of CompuServe, AOL, Prodigy, and Minitel were out beating the bushes for content, arranging to provide it through the straightforward economic model of being paid by people who would spend connect time browsing it. If anything, we would expect the proprietary networks to offer more, and for a while they did. But they also had a natural desire to act as gatekeepers—to validate anything appearing on their network, to cut individual deals for revenue sharing with their content providers, and to keep their customers from affecting the network’s technology. These tendencies meant that their rates of growth and differentiation were slow.

The closed versus open network is not quite the whole picture with respect to the iPhone versus Android. While there is competition between iTunes versus Amazon, Streetview versus normal Google Maps and there will be other content battles, I don't think that's the source of the real innovation. I think it's probably the technical innovation that the online providers couldn't match that was decisive and will be for phones:

The software driving these communities was stagnant: subscribers who were both interested in the communities’ content and technically minded had few outlets through which to contribute technical improvements to the way the communities were built. Instead, any improvements were orchestrated centrally. As the initial offerings of the proprietary networks plateaued, the Internet saw developments in technology that in turn led to developments in content and ultimately in social and economic interaction: the Web and Web sites, online shopping, peer-to-peer networking, wikis, and blogs.

Is developing on an iPhone going to lead to more technical innovation over Android? Does the ability to have open source code on Android beat the Apple NDAs?

Apple will probably recognize that it's the developers that will ultimately have the power but like publishing it depends on the actions of both parties. At the moment both of these new phone platforms are a little limited - the ability for people to innovate on iPhones using iTunes, Mail is out but it doesn't seem terribly better for Android (being DRM free is a good start though). If history is any guide, it would seem to favor open development over closed.

Update: Similar more succinct explanation.

Update: No Pragamatic book due to NDA - the outrage!

For Silver

2008-09-19T09:09:00.000+10:00

So now I can finally run Silverlight on Chrome on Windows, all the ducks are aligned.

Make Every Web Site a Semantic Web Site

2008-09-16T08:59:00.001+10:00

Back in March this was news but I completely missed it. Dapper has a rather nice way of turning web sites into data - XML, JSON, etc but it also includes a semanify service. It includes using existing namespaces (FOAF, GSS, Creative Commons, Media RSS and Dublin Core) supported by Yahoo's search engine.

This is covered in more depth in Semantify Hacks - Creating a your own RDF schema using Dapper:

So now, building a Dapp means you also built your own RDF compatible schema, that you can use wherever by just pointing to the webservice:

http://www.dapper.net/websiteServices/dapp-scheme.php?dappName=MYDAPP

The given example is MSN's search engine which you can see in all its RDF/XML glory.

ReadWriteWeb has step-by-step instructions.

Into Thick Air

2008-09-08T09:44:00.012+10:00

Chrome, JavaScript, and Flash: Two (Mostly) Opposing Views the second comment closely follows a reason update on CounterNotions, Google Chrome: Bad news for Adobe.

But a full-fledged browser. One that behaves, however, as a platform to host applications best tied to cloud computing with built-in local persistence for offline computing. Sure, in its current form Chrome can’t compete with Silverlight or Flex/AIR for what Adobe calls “expressiveness,” meme-speak for rich graphics, animations, integrated video and other visual UI goodies.

Chrome may shut it off for good. It’s possible that various open source Chrome technologies could melt into Safari and Firefox. But –– whether as a stand-alone product or a progenitor of fast, powerful and expressive browsers –– Chrome signals to anybody but the diehard Microsoft constituents that the browser itself, not a proprietary plug-in or a separate runtime, is the future of RIAs. With its huge ecosystem, Microsoft can live with that. At least until its enterprise monopoly seriously erodes. But Adobe cannot.

In a world where the online pie is divided among the .NET army of Microsoft, the browser-gang of Apple+Mozilla+Google, and the lone Adobe, it’s not difficult to predict whose share will shrink into insignificance. If the exclusion of Flash from the iPhone wasn’t a wake-up call for Adobe, Chrome should certainly be one.

Most of the commentary is focused on the browser within an operating system angle. Although one of the easter eggs is a familiar screensaver. I think it's more helpful to concentrate on the fact that these browsers are getting rich enough to remove the applications embedded within browsers. There is already a lot of functionality developed or being developed such as SVG and storage (part of HTML 5). Chrome ships with Gears though and Webkit, to see how HTML5 and Gears relates see Aaron Boodman's talk on implementing HTML5 in Gears. They create two namespaces one for implementing the standard APIs and one for non-standard APIs - it seems like it has quite a solid development process behind it. Is there really a lot of reason left to support these proprietary applications within applications?

More MapReduce Groovy

2008-09-08T09:41:00.001+10:00

A very good post on Cascading (covered previously), GOODBYE MAPREDUCE, HELLO CASCADING

Cascading’s logical model abstracts away MapReduce into a convenient tuples, pipes, and taps model. Data is represented as “Tuples”, a named list of objects. For example, I can have a tuple (”url”, “stats”), where “url” is a Hadoop “Text” object and “stats” is my own “UrlStats” complex object, containing methods for getting “numberOfHits” and “averageTimeSpent”. Tuples are kept together in “streams”, and all tuples in a stream have the exact same fields.

An operation on a stream of tuples is called a “Pipe”. There are a few kinds of pipes, each encompassing a category of transformations on a tuple stream. For instance, the “Each” pipe will apply a custom function to each individual tuple. The “GroupBy” pipe will group tuples together by a set of fields, and the “Every” pipe will apply an “aggregator function” to all tuples in a group at once.

One of the most powerful features of Cascading is the ability to fork and merge pipes together.

Once you have constructed your operations into a “pipe assembly”, you then tell Cascading how to retrieve and persist the data using an abstraction called “Tap”. “Taps” know how to convert stored data into Tuples and vice versa, and have complete control over how and where the data is stored. Cascading has a lot of built-in taps - using SequenceFiles and Text formats via HDFS are two examples. If you want to store data in your own format, you can define your own Tap. We have done this here at Rapleaf and it has worked seamlessly.

Senior Re-Searcher

2008-09-08T09:35:00.001+10:00

Someone who continues to search (re-search) Google until he finds what is required. Senior in that he remembers a time before Google.

Stats for Nerds

2008-09-03T16:56:00.002+10:00

I read "Is google chrome under / partially reporting RAM usage ?" which seems to be the typical problem with Microsoft's task manager. A better way to check out Chrome vs others is to hit Shift-Escape, click on "Stats for Nerds" and it has a comparison there.

Antimetabole

2008-09-02T11:01:00.001+10:00

Bill Clinton had an interesting one recently in his Democratic convention speech: "People the world over have always been more impressed by the power of our example than by the example of our power."

Thesis and Antithesis

2008-09-01T21:29:00.015+10:00

David Anderson on Agile. He introduces his talk by discussing some very broad ideas, including how agility has been applied to other areas. He talks about how the agile manifesto seems to have become more about belief and superstition than a scientific model. It's also rarely clicked through - to see the principles behind the manifesto.

The agile community has found it is better to develop in a failure tolerant environment. This was a reaction to the focus on more and more accurate estimates which lead to analysis patterns (or antipatterns). They are antipatterns because there was never enough detail in the analysis to provide accurate enough estimates and to deal with the unexpected.

Through his talk he listed a bunch of thought bubbles on software development:
* Value is providing functionality fastest.
* Knowledge work is perishable.
* Perfect is the enemy of good enough.
* Develop a high trust, high social capital.
* Have a highly collaborative culture.
* Reflect and adjust.
* Sustainable pace.
* Craftmanship.
* Value is contextual, context is temporal.
* Waste over scale.

Traditional industries have relied on bigger batch sizes to provide better economies of scale. With software development the transaction costs causes large batch sizes to work against you. Smaller batches allow you release more business value more quickly and reduce waste (waste over scale).

He introduces kanban which has ideas such as: pull, flow, and regulate work in progress. It does not use time boxed iterations and little or no planning or estimations (at least from a traditional agile approach). It still has constant improvement and delivery. It drops the idea of a generalist approach to labor which is seen by some as waterfall in disguise. The reason this comes about is because at an enterprise scale it's not feasible to hire a lot of experts who are excellent generalists - the labor pool does not exist. This leads to a tension with typical lean principals of reducing waste by having generalists.

He sees two main changes coming in the future: software factories (software product lines - which includes DSLs) and cloud services (deploying web services in the cloud). Architecture and modeling will come back in fashion as a value chain allows incentives for delivering common behavior. He's talking about a 100 fold improvement in productivity through using these ideas.

Finally, he talks about how CMM/CMMI is actually quite a good indication of the ability for software projects to succeed and the industry as a whole. He mentions a report on how agile and CMMI can work together, the idea being that not only is organizational maturity a good idea but it actually means that agile methods can be implemented better (it's called "CMMI or Agile: Why Not Embrace Both?!" - I only found a few references, for example Agile+CMMI Panel @ SEPG).

Via.

Still have Richard P. Gabriel's talk to go through too.

Answering the Right Question

2008-08-28T07:13:00.003+10:00

From a reddit comment on Ubiquity:

It's a natural-language interface to the entire web. And not a lame "find me the page at http://xxxxx", but "find the page, filter it of information, extract and cross-reference this data with multiple other data-sets and deliver the results to me in a format and medium of my choosing".

It's not really naturally language anymore than "Sam and Max" was natural language but the point of the filtering, finding, and integrating seems spot on to me - that's answering the right question.

It's a bit like Linus talking about how Subversion sucks (about half way through is a good place to start) - they made branching cheap - who cares.

It's this whole Unix pipes for the web thing I guess.

Jesus Camp

2008-08-27T15:35:00.009+10:00

This seems like quite an amazing depiction of American fundamentalists. It seems as if bigotry and fundamentalism is being turned into a competition with the Middle East. If you don't like to be scared about American evangelicals look away.

Harry Potter is the Devil - JESUS CAMP - the one that started it all - kinda weird but interesting.

Bill Maher on Jesus Camp - talks about this (from last year).

Jesus Camp Bush Insanity (or Child Abuse). Your call. - laying on of hands to a cardboard George Bush, the mixture of religion and politics - very scary.

Oprah Denies Christ - she tries to reason that there must be more than one way to go to heaven - but of course her audience members don't allow her to get away with that one. Scarier.

CBC Interview with 'Jesus Camp' Filmmakers - scariest.

Finally a Reason for HTML Email

2008-08-27T15:33:00.002+10:00

Ubiquity for Firefox. I've been a little hestitant recently to put the stuff I've been finding interesting. But this has to go up - it's a really great example of user driven mashups and what this Semantic Web thing should be all about.

Update: More information at "If You Want To Create a Mashup, Just Ask Your Browser. Mozilla Labs Launches Ubiquity." and the Mozilla Labs blog.

MapReduce Groovy

2008-08-22T11:57:00.001+10:00

Cascading "The processing API lets the developer quickly assemble complex distributed processes without having to "think" in MapReduce. And to efficiently schedule them based on their dependencies and other available meta-data."

Also, HBase now supports transactions (using Optimistic Concurrency Control).

Via “Beyond Relational Databases”.

The Trouble with Mini-Wheats

2008-08-22T09:32:00.001+10:00

Australian Mini-Wheats are apparently the only Mini-Wheats with triticale, oats, barley, wheat, and rye. Why is that interesting? Triticale or quadrotriticale (cause we're in the future) was in "The Trouble with Tribbles". How many Star Trek references do you eat?

Information Poo

2008-08-21T04:36:00.008+10:00

Inspiron 910 hopefully will come out on Friday. Inspiron means cheap in Dell marketing? I guess it's better than Dell E. There's something wrong with looking forward to a Dell. I mean there is no Dell Insider or Inspiron Rumors.

SearchMonkey Tutorials. SearchMonkey uses a format similar to RDF called DataRSS. The FAQ describes the parts of the Semantic Web that work with the parts of SearchMonkey - no reasoning just the triples.

Introduction to High-Level Programming With Scala. A compiled language that is finally catching up with Ruby's duck typing.

Dr Horrible Ringtones.

Freebase and Parallax talks about the excellent demo making the rounds. It's available as an open source project. Who knew George Bush has had more than one assassination attempt made on him? That's the power of the Semantic Web.

The goodness of Google apps talks about another happy corporate customer (except for Google Code - integrating GitHub does sound good) and mentions Guido's code review tool part of rietveld a Google App Engine application.

ccREL: The Creative Commons Rights Expression Language. Using RDFa so you don't repeat yourself.

SCOPE, Automatic Hadoop Optimization and Cassandra (Facebook's BigTable) all via Greg. Cassandra is a Google Code project and includes some good documentation.

OWL ED papers including an ontology versioning system and OWL constraints.

One Machine

2008-08-14T11:29:00.013+10:00

Kevin Kelly predicting the next 5,000 days of the web. The specifications for the current web, the one machine, is interesting. It uses 5% of the electricty on Earth and holds and processes the equivalent of one human brain (1 HB). If technology continues growing at the current rate, 6 billion HB will be achieved somewhere between 2020 and 2040, outstripping the current human population (and hopefully the population then as well). He also mentions that every bit will be a web bit, that all bits go through the web, and you see that happening with traditionally non-web data usage like word processing or mobile phone traffic (actually internet would probably be more accurate). This coincides with the convergence of digital and atomic "worlds", rather than moving from one to the other (like in "The Matrix") he suggests that it'll be integrated and that we are the extension of the web rather than the other way around.

The second half of the talk is about the Semantic Web (found from here, here and here). He describes his own definition of the Semantic Web by showing how it fits into what has gone before. The stages of networking includes: connecting by site from one computer to another, connecting by page and linking between them and the last two stages (I didn't see a clear distinction between the two) is by data, idea or item.

His web site also mentions that computers are beating some of the best players at Go (the article is quite bad - teraflops as a measure of storage - this one is better).

Real Developers don't use Ruby

2008-08-12T11:30:00.002+10:00

Hadoop: When grownups do open source. It's quite an amusing read - especially the part about the word count example on 9,000 blogs, the digg at Twitter, Starfish being practically useless (using MySQL and no Reduce phase) and the bit about understanding something being harder than writing a Ruby version of it.

Twitter decided they would be cute and trendy. They wrote their code in Ruby: the official state language of the hipster-developer nation. Doug Cutting, on the other hand, decided he would get xxxx done, and wrote Hadoop in Java. Starling was hidden away in some corner and forgotten (it's hosted at RubyForge...). Hadoop lives prominently at the Apache Software Foundation. Starling is a re-hash of an existing Java Enterprise API called JMS that has several open source implementations. Hadoop is an implementation of Google's MapReduce, a system that publicly only existed on paper. Hadoop has the added benefit of actually working.

Ahh the joys of installing Visual Studio - enough time to install IntelliJ, run it up, and catch up on news.

Pigeon Programming

2008-08-12T11:08:00.002+10:00

Intellij 8 continues with "all you need is a space bar and meta combinations to program" functionality: "Pressing Ctrl-Shift-Space twice allows you to find values of the expected types which are "two steps away" (can be retrieved through a chained method call)." It seems to be supporting what I'd consider a code smell.

Burnator

2008-08-12T10:08:00.000+10:00

Ack, using ISOs on Windows XP fits in that time period where I think I'll do it often enough to remember what software to use but ends up being long enough that I don't. So the two I've used the most is InfraRecord and IsoRecorder (a secret).

JRDF is very crap (part 2)

2008-08-11T16:08:00.005+10:00

It didn't take as long as expected which means it's probably wrong. For non-filtered queries it's three times faster and for certain FILTER queries (with equals) it's 47 times faster (from 284 to 6 seconds). At least it's now in the same order of magnitude as most tools and it's a a tiny bit faster than some (although adding more features will probably slow it down again).

What changed:
* AttributeValuePair has been removed and replaced with maps (as discussed previously).
* Seeing as though maps were used so much hashCode and equals were optimized. As I've found before (I think), isAssignableFrom is slower than try/catch for equals (depending on your usage of course).
* Queries go through unsorted and uncopied rather than standard graph finds. I'd forgotten about how much effort had gone into allowing remove and automatic sorting on iterators.
* A very simple optimizer (it's really only simplifying the FILTER constraints at the moment) was added. Tree manipulation was painful - I resorted to mutating in place operations.
* Better designed. It's a bit hard to qualify this except what was there was truly awful - objects being created in constructors and passing itself in. The nice thing about IoC is it's quite easy to see when you're not using objects at the same architectural level.

Update: For download.

JRDF is very crap (part 1)

2008-08-07T09:08:00.006+10:00

I've been spending some time looking at the querying part of JRDF. And it's quite bad. How bad? Well I've been profiling it and noticing that a lot of time was spent comparing attribute value pairs. An attribute in JRDF consists of a name (variable or position in a triple) and type (position in a triple or literal, URI Reference or blank node). Comparisons are done during most operations (like joins) and they are done on sorted attribute values. This is incredibly dumb. What's much better is to have a map of attributes to values. No sorting required and O(1) lookup - hurrah. The code around the comparisons also got a lot simpler and is obviously better. I think there's at least one other case of this at a different level and potentially room for about an order of magntitude speed up over the current release. Test queries are already 2-3 times faster.

The main reason for this though, is that currently the FILTER in JRDF runs about 10 times more slowly than a query using triple matching (this isn't a complexity measurement - it's based on a rather small set of triples). So a query with "?a <some:value> 'foo'" is much slower than "?a <some:value> ?b . FILTER(str(?b) = 'foo')". The queries aren't the same but their performance shouldn't be that much slower. However, in order to get to a stage of improving FILTER's performance the code has to be refactored - hopefully simpler and faster.

FILTER is nicely functionaly - it seems a shame to implement it in Java - it's eye poppingly bad at the moment. I was thinking functional Java but instead of taking the gateway drug I was think of just going to the hard stuff. FILTER is operated by creating different operations within a relation - which allows you to put ANDed FILTERs vertically across a relation (columns) and ORed FILTERs horizontally (rows). I don't know if anyone else implements it this way - it might be another bad idea over time.

Mice Spiders

2008-08-02T07:52:00.001+10:00

Interview with Simon Pegg on Spaced. The UK DVDs are still the only source of the show (and ABC2) for Australians until October.

YADS and RDF Molecules

2008-07-29T10:32:00.002+10:00

BNodes Out! discusses how any usefully scalable system doesn't use blank nodes. What is interesting is the comment on YADS (Yet Another DOI Service). The best reference is Tony's presentation although it is mentioned in Jane's as well. "YADS implements a simple, safe and predictable recursive data model for describing resource collections. The aim is to assist in programming complex resource descriptions across multiple applications and to foster interoperability between them...So, the YADS model makes extensive use of bNodes to manage hierarchies of “fat” resources - i.e. resource islands, a resource decorated with properties. The bNodes are only used as a mechanism for managing containment."

This sounds a lot like RDF molecules and supports visualization (apparently). This seems like a good use of molecules that I hadn't previously thought of (Tony's talk gives an example of the London underground). The main homepage of YADS isn't around anymore - it'll be interesting to see if it's still being used/worked on.

Update: Tony has fixed up the YADS home page (there's also an older version).

Hadoop and Microsoft

2008-07-28T09:03:00.002+10:00

Pluggable Hadoop lists some extensions to Hadoop in the pipeline: job scheduling (including one based on Linux's completely fair scheduler), block placement, instrumentation, serialization, component lifecycle, and code cleanup (the analysis used Structure101).

I found the reason why HQL was removed from HBase (to be replaced by a Ruby DSL and to ensure that HBase wasn't confused with an SQL database) and moved to HRdfStore.

There's also rumours that Microsoft's recent investment in Apache may lead to them working on Hadoop too.

Save us China

2008-07-22T13:31:00.009+10:00

I was in Victoria when the ETS for Australia was announced (well the discussion papers). It's fairly funny, that replacing the world's worst plants even with other coal plants (Hazelwood is the world's worst), with Chineese brown coal plant technology, would reduce emissions by 30% to 40% (by just drying out the brown coal). It's still very poluting but it just shows how far behind Australia is. This has lead to greater compensation to Victorian polluters (which is just mad). At the same time Queensland is creating another coal port because we can't export the carbon fast enough.

The exclusions were annoying (aluminium, cement and some types of steel). Cement is annoying (5% of all CO2 apparently) as there exists green alternative technologies. The time is to invest not compensate.

Square brackets are scary

2008-07-22T10:48:00.002+10:00

For what may be an increasing trend of surfing the Web at 320x480 I noticed Cydia has a number of applications for Jailbroken iPhones (Java, Python and Ruby mainly). The mailing list on iPhone/Java doesn't have much on it except some interesting uses of JocStrap and UICaboodle (available from SVN by Jay Freeman). There's also the Sun blog that has some interesting sample applications using different Java implementations on the iPhone.

JRDF 0.5.5.1 Released

2008-07-04T22:21:00.001+10:00

Just a quick note about a new version of JRDF. It's been a short time between releases but it still contains one significant advance over the previous one and that's persistent graphs. It's still in the early stages but it's basic enough for simple use cases. It also contains text serialization (based on NTriples) that is useful for moving RDF molcules around nodes in a cluster (for example). A lot of this code is fairly much "spike" code and I expect that another release will be released after we exercise these new features more (and write some tests/rewrite the code).

Update: 0.5.5.2 is now available fixing many bugs and introducing FILTER support.

Of Mats and Cats

2008-07-04T09:38:00.000+10:00

No universal things Re: comparing XML and RDF data models was started by Bernard Vatant. This comes to the heart of whether people can know reality (well that's how I'd summarize the idea of universals see Beyond Concepts).

There were a few quotes that I found interesting:

It's been counter-productive in science for centuries. Physics had to go over the notion of universal thing to understand that light is neither a wave, nor a particle. Biology to go over the notion of taxa as rigid concepts based on phenotypes to understand genetics etc. Many examples can be found in all science domains. My day-to-day experience in ontology building, listening to domain experts, is indeed not that 'there are things that people are trying to describe', but that 'there are descriptions people take for granted they represent things before you ask, but really don't know exactly what those things are when you make them look closely'.

Bijan wrote:

I do think that the family of views in computational ontologies generally called "realist" is indeed naive and fundamentally wrong headed. Whether it's a "useful fiction" that helps people write better or more compatible ontologies is an open empirical question.

But I, for one, wouldn't bet on it.

I remember also a project where we were trying to get people to write simple triples. They got that they needed triples. But what they ended up putting into the tool was things like

S P O
"The cat is" "on the" "mat".
"Mary eats" "pudding" "on toast"

They just split up the sentences into somewhat equal parts!

I really feel like an interested amateur and my view is probably influenced by databases in computer science, where you are taking the non-realist approach. I say this because there are usually properties in databases that are not really based on reality but are a result of other requirements (like a column like "isDeleted" rather than actually deleting the statement).

Round of Links

2008-07-02T09:40:00.012+10:00

Apache Hadoop Wins Terabyte Sort Benchmark "One of Yahoo's Hadoop clusters sorted 1 terabyte of data in 209 seconds...This is the first time that either a Java or an open source program has won." There were just under 1000 nodes, the benchmark results are hosted by HP (a tad more detail here).

Microsoft buys Powerset one of the interesting things is that they use Hadoop (see their blog). It's hard to tell whether this is bad or good for Hadoop.

Google vs Microsoft - oh for structure.

Tom talking about GridGain from his presentation in February. C++ isn't as productive as Java?

Applets are back (according to Sun).

Why commenting is for n00bs. "And Haskell, OCaml and their ilk are part of a 45-year-old static-typing movement within academia to try to force people to model everything. Programmers hate that. These languages will never, ever enjoy any substantial commercial success, for the exact same reason the Semantic Web is a failure. You can't force people to provide metadata for everything they do. They'll hate you."

Some interesting discussion on Web 2.0 and the future of the web.

Rich text editor for browsers. Not free though.

Linked data and what it is.

ThoughtWorks Podcasts (the REST talk was what drew me to it).

Turtle specification. I've been looking at this for serialization of RDF molecules but it seems that you can't have blank nodes as objects using the nested syntax.

Semantic Web for bioinformatics.

Data structure stuff: Linear Bloom Filters, Bloom filters for Spell Checking, Optimal Bloom Filter replacements and scalable btree and B-tries for Disk-based String Management.

Ob. iPhone 2

2008-07-01T09:59:00.001+10:00

Good to see carriers actually putting up a bit of a fight for iPhone business. Telstra announces iPhone 3G details with $279, $30 a month on a 24 month, with free access to WiFi hotspots. This better be true.

Update: Optus releases pricing

ScalaCC

2008-06-30T10:43:00.004+10:00

Formal Language Processing in Scala which links to External DSLs made easy with Scala Parser Combinators that I'd read from here.

Although, just to keep it balanced I have noticed Steve Yegge's comments, under "Static Typing's Paper Tigers", on the complexity of Scala's typing (it does have a lot) and it has been pointed out that this does lead to problems with writing IDEs to support it.

JRDF 0.5.5

2008-06-26T11:52:00.002+10:00

The main difference in version 0.5.5 from the previous one is the inclusion of a RDF molecule store. Both in memory and disk based versions are supported and can be queried just like a normal triple store. This is also the first version that has been renamed URQL instead of SPARQL for the query evaluation. The SPARQL grammar is the same but it does not support the weird outliers that SPARQL has for empty graph patterns but follows relational (and other) algebras. There's also the usual bug fixes and other features.

Update: Due to a couple of bugs found in 0.5.5 there will be a 0.5.5.1 version released soon.

Beef of the Sea

2008-06-22T23:56:00.004+10:00

Everyone is probably sick of me talking about the Gruen Transfer. So what better what to continue to talk about it than to blog about it. Perhaps the best part of the show is The Pitch especially episode one's selling whale meat (this is the runner up) and making the Democrats electable (the second is best). Who would've thought deconstructing chocolate adverts would be interesting? One of the good things is that the show is available for download. There is also some good discussion in the forum and links to some other good adverts (although it possible should've been crows).

Bad Balmer

2008-06-17T20:37:00.001+10:00

Eight Years of Wrongness. Lists some of the things believed to have gone wrong with Microsoft in the last 10 years or so. They include: losing the DOJ and EU cases, Vista, XBox, IE, Zune, and Windows Mobile. Linked mainly because they use Fake Steve as a source of analysis.

Apple Sprouts

2008-06-17T09:20:00.009+10:00

AppleInsider has some details on SproutCore. The official web site says, "makes Javascript fun and easy" - and it's just a Ruby gem install away. They also link to some previous talk about Cocoa for Windows.

Apple's trojan horse in the runtime wars has been well known for a while.

The photo demo looks a lot like the MobileMe Gallery that was presented at WWDC 2008 (SproutCore doesn't seem to work too well under IE 7 and the rotation only works in Safari). Gallery has less functionality than things like Photoshop Express although the integration is obviously better.

There's also an interesting Javascript library for drawing 2D objects (UML, workflows, etc) that I've been shown recently called Draw 2D.

The Curse of the Floppy Penises

2008-06-13T14:24:00.015+10:00

A Western floppy penis is more valuable than preventing blindness in an African eye (see neglected diseases). This is part of the story in the video of the launch of "The Health Commons". The video talks about how hundreds of thousands of people go blind from "river blindness". It has very little value associated with it and drug companies focus on more valuable drugs to do with baldness and erectile disfunction. The video goes on to talk about how the network changes things and how there's a lack of process change in science to take advantage of these effects. If you can leverage network effects then this hopefully reduces the cost of drug discovery making drug development in less valuable diseases viable. The white paper covers some more of this in detail.

It also talks about an idea that I've often thought of as useful - the collection of failed experiments, "This deeply set inability to capture collective learning dooms everyone to revisit infinitely many blind alleys. The currency of scientific publication encourages individual scientists to hoard rather than share data that they will never have the time or resources to exhaustively mine. And, the wealth of “negative” information gleaned from clinical trial data is mostly lost to the need for companies to safeguard their commercial investments."

The general idea seems to share and standardize all aspects of research and science.

Ob. iPhone

2008-06-12T11:29:00.007+10:00

So I've been trying to find more information from a variety of sources on pricing.

The closest to reality that I've been able to find is these leaked details from Optus (via Gizmodo):
"The iPhone will only be available on a 24 month contract – no outright purchase, with the 8GB model to sell at AUD $220, and the 16GB model at $330, with only the 16GB model in white as Steve Jobs announced at the WWDC keynote.

Accessories will only be available through Apple stores – Optus will only carry the iPhone 3G itself, and the all important voice and data plans are as follows: $79 cap for $300 worth of calls and 1GB of data, or a $99 cap with $400 worth of calls and a 3G data download limit.

Visual voicemail is included, and the cap is whittled away in 35c per 30 second chunks, 25c per SMS message and the always annoying but always present flagfall which is set at 30c."

This makes it over twice as expensive as the ATT plans (and I think they had unlimited data). This is where I get cranky about Australian carriers and their stupid plans. It would probably count me out at those prices.

Update: No more Apple rumours. As Brad says in the comments, this is wrong.
Update 2: Looks like the UK is getting a good deal.
Update 3: Gizmodo link gone...nothing to see here.

Evolution and SUVs

2008-06-11T09:15:00.000+10:00

Two things that have I've been interested in before: SUVs and evolution.

The first mirrors what is happening in Australia too where small cars are winning over larger ones and Falcon sales have dropped by half. The F150 isn't really that popular here.

The observation that bacteria have evolved to process other nutrients is interesting. But the related articles tend to be a bit more forceful: such as the Bible's many inaccuracies, the many occurrences of homosexuality and how it actually works in nature and that accepting evolution does not mean rejecting morality.

Linked Data, FOAF, and OWL DL

2008-06-10T09:52:00.031+10:00

So I spent a little time a while ago looking through all the different ways ontologies support linked data. Some of my data I wish to link together is not RDF but documents that define a subject. For example, a protein will have peer reviewed documents that define it. It's not RDF but it is important.

The tutorial on linked data has a little bit of information: "In order to make it easier for Linked Data clients to understand the relation between http://dbpedia.org/resource/Alec_Empire, http://dbpedia.org/data/Alec_Empire, and http://dbpedia.org/page/Alec_Empire, the URIs can be interlinked using the rdfs:isDefinedBy and the foaf:page property as recommended in the Cool URI paper."

The Cool URIs paper, Section 4.5 says: "The rdfs:isDefinedBy statement links the person to the document containing its RDF description and allows RDF browsers to distinguish this main resource from other auxiliary resources that just happen to be mentioned in the document. We use rdfs:isDefinedBy instead of its weaker superproperty rdfs:seeAlso because the content at /data/alice is authoritative."

There is also some discussion about linking in URI-based Naming Systems for Science.

Now my use case is linking things to documents that define that thing. So rdfs:seeAlso is not appropriate as it "might provide additional information about the subject resource". And rdfs:isDefinedBy is also out as it is used to link RDF documents together. I need a property that defines a thing, is authoritative but isn't linking RDF (it's for humans). I also would like to keep my ontology within OWL DL.

FOAF has a page property. I've used the OWL DL version of FOAF before and FOAF cleaner (or should that be RDFS cleaner). So it seemed like a good match. However, its inverse is topic which isn't good. Because I'm linking the thing to the page - it's not a topic. So scrub that.

RSS has a link property which extends Dublin Core's identifier. This seems more like it. However, I'd like to extend my own version of link and I'm stuck because as soon as you use RDFS vocabularies in OWL DL you're in OWL Full territory. It'd be nice to stay in OWL DL. There is an OWL DL version of Dublin Core. All of the Dublin Core properties are nicely converted to annotation properties. However, you're still stuck because you can't make sub-properties without going into OWL Full. I like the idea of annotation and semantically Dublic Core seems to be a suitable vocabulary of annotation properties. Extending Dublin Core is out of OWL DL - which is shame because it's probably the closest match to what I wanted.

As an aside, annotation properties are outside the reasoning engine. The idea is that you don't want an OWL reasoner or RDF application necessarily inferring over this data or trying to look it up in order for the document to be understood. So the way they do it in OWL DL is to have annotation properties that are outside of/special to the usual statements. Sub-properties require reasoning, so limiting them makes some sense but it does hamper extensibility - it'd be nice to express them and turn on the reasoning only when asking about those properties (I think Pellet has this feature but I didn't look up the details).

The other vocabulary I looked at was SIOC's link. Again, this seems like a close match but again it's RDFS.

In the end, I just created another annotation property called link.

In summary:

For my requirements, the suggestions for linking data seems to only work for RDF and RDFS ontologies. Reusing RDFS from OWL DL or OWL DL from RDFS doesn't look feasible as one isn't a subset of the other (an old problem I guess).

Current, popular Semantic Web vocabularies are in RDFS. Why aren't there more popular OWL DL versions of these things? Is the lack of extensibility holding it back?

Is my expectation wrong - should I stick within OWL DL or is an RDFS and OWL DL combination okay?

Why not allow annotation properties to have sub-properties?

Maybe the OWL DL specification does have suitable properties for linking certain data but I don't understand which is the right one.

Update: The Neurocommon's URI documentation protocol is quite similar as well. Except that, it seems to be too specific as it ties the name with a single thing that defines it. All the parts of Step 5 could potentially be eliminated with what I'm thinking of.

Somewhere

2008-05-30T16:35:00.000+10:00

Alarm Bells Sound for the Amazon

Brazil's land mass and farming industry make it one of the most agriculturally productive countries in the world. It has already been dubbed "the world's feeding bowl" and is exporting more and more to emerging economies, such as India and China.

As China's middle-class continues to grow, so, too, does its demand for food. Brazil exports 10 million tons of soybeans to China a year for both animal feed and human consumption, trade that is crucial to Brazil's economic development.

And it's not just poverty that's an issue.

The state of Para has some of the worst human rights abuses in Brazil. People are trafficked from across the impoverished northeast of the country to work in slavelike conditions in the sawmills, illegal charcoal ovens and cattle farms.

They usually work in horrific conditions, with no basic rights and existing on roughly $5 a day. If they try to seek help from the authorities, they are threatened with death.

There's also the WHO page on "Deaths from Climate Change".

RDF Processing

2008-05-26T08:49:00.007+10:00

One of the interesting things about biological data, and probably other types, is that a lot of it is not quite the right structure. That's not to say that there's not people working to improve it, the Gene Ontology seems to be updated almost daily, but data in any structure maybe wrong for a particular purpose.

Biologists make a habit, out of necessity, of just hacking and transforming large amounts of data to suite their particular need. Sometimes, these hacks get more generalized and abstracted like GO Slims. We've been using GO Slims in BioMANTA for sub-cellular location (going from 2000 terms to 500). GO contains lots and lots of information and you don't need it all at once and more often you don't need it at the maximum level of granularity that it has. Some categories only have one or two known instances, for example. You may even need to whittle this down further (from say 500 to 200). For example, when we are determining the quality of an interaction we only care where the proteins exist generally in an organism. If the two proteins are recorded to interact but one is in the heart and the other in the liver then it's unlikely that they will react in the host organism. The part of the liver or the heart and other finer structural detail is not required for this kind of work (AFAIK anyway).

The point is, a lot of our work is processing not querying RDF. What's the difference between the two and what effect does it have?

For a start, querying assumes, at least to some degree, that the data is selective - that the results you're getting is vastly smaller than your original data. In processing, you're taking all of the data or large chunks of it (by sets of predicates, for example) and changing or producing more data based on the original set.

Also, writing is at least as important as reading the data. So data structures optimized for lots of writes, temporary, concurrent, is of greater importance than those built around more familiar requirements for a database.

Sorting and processing distinct items is a lot more important too. When processing millions of data entries it can be quite inefficient if the data has a large number of duplicates and needs to be sorted. Processing can also be decentralized - or perhaps maybe more decentralized.

To top it off, the data still has to be queried. So this doesn't remove the need for efficient, read only data structures to perform selective queries for the usual analysis, reporting, etc. So none of the existing problems goes away.

git + RDF = versioned RDF

2008-05-12T13:28:00.003+10:00

Reading, Git for Computer Scientists, and it seems like if you turn the blob into a set of triples you pretty much have versioned RDF (or molecules even).

I'm also wondering, if Digg is so pro-Semantic Web, where's the http://digg.com/semweb?