More Data

Neurocommons have released their integrated RDF datasets. It is composed of different modules or bundles including MeSH, Medline, OBO and others.

Who Holds the Power?

On Monday I went and saw John Wilbanks talk on "Publishing in Today`s Environment" (I can't seem to find a decent URL but a good overview of some of the topics discussed is "The Open Access Interviews: John Wilbanks"). I didn't take many notes but a few things have stuck with me:

• Libraries providing a role as repositories of data (both public and private).


• The idea of "free as in puppy" in relation to digital curation - a great metaphor. When someone gives you a puppy it is initially free but the upkeep of it is anything but.


• The Queensland Government has its own Creative Commons initiative and the Australian Government is following suite.


• There's a power struggle occurring between researchers and publishers to make data, papers and the like freely available. The power used to be on the side of publishers but as the producers band together (by country, university, faculty and so on) the power is going back to them.


• Creative Commons are continuing to fight, in cunning ways, to get back to decent copyright law.

The kind of behavior that publishers have exhibited appears to be on the way out and seems to be going the other way. This is where databases interoperate with each other, papers can link to the original data, and tools, data and papers are all part of an integrated experience. The linking and integration of papers and other artifacts seems to be a one way process, it's hard to imagine a developer, scientist, arts graduate or anyone else spending huge amounts of money and time finally wresting control from one bunch of people only to tie it to another.

This leads to the T-Mobile G1 announcement. The announcement was very underwhelming - guys speaking for the first 10-15 minutes saying what a wonderful job they had done and the guy talking about how good an experience the phone was at playing Pacman. The more intersting part is the open platform. Much like the publishing area the ability to have open access will be a massive differentiator. It reminded me a bit about the discussion on how AOL, Prodigy and the other closed networks quickly died when faced with the open web:

By contrast, the proprietary networks of CompuServe, AOL, Prodigy, and Minitel were out beating the bushes for content, arranging to provide it through the straightforward economic model of being paid by people who would spend connect time browsing it. If anything, we would expect the proprietary networks to offer more, and for a while they did. But they also had a natural desire to act as gatekeepers—to validate anything appearing on their network, to cut individual deals for revenue sharing with their content providers, and to keep their customers from affecting the network’s technology. These tendencies meant that their rates of growth and differentiation were slow.

The closed versus open network is not quite the whole picture with respect to the iPhone versus Android. While there is competition between iTunes versus Amazon, Streetview versus normal Google Maps and there will be other content battles, I don't think that's the source of the real innovation. I think it's probably the technical innovation that the online providers couldn't match that was decisive and will be for phones:

The software driving these communities was stagnant: subscribers who were both interested in the communities’ content and technically minded had few outlets through which to contribute technical improvements to the way the communities were built. Instead, any improvements were orchestrated centrally. As the initial offerings of the proprietary networks plateaued, the Internet saw developments in technology that in turn led to developments in content and ultimately in social and economic interaction: the Web and Web sites, online shopping, peer-to-peer networking, wikis, and blogs.

Is developing on an iPhone going to lead to more technical innovation over Android? Does the ability to have open source code on Android beat the Apple NDAs?

Apple will probably recognize that it's the developers that will ultimately have the power but like publishing it depends on the actions of both parties. At the moment both of these new phone platforms are a little limited - the ability for people to innovate on iPhones using iTunes, Mail is out but it doesn't seem terribly better for Android (being DRM free is a good start though). If history is any guide, it would seem to favor open development over closed.

Update: Similar more succinct explanation.

Update: No Pragamatic book due to NDA - the outrage!

For Silver

So now I can finally run Silverlight on Chrome on Windows, all the ducks are aligned.

Make Every Web Site a Semantic Web Site

Back in March this was news but I completely missed it. Dapper has a rather nice way of turning web sites into data - XML, JSON, etc but it also includes a semanify service. It includes using existing namespaces (FOAF, GSS, Creative Commons, Media RSS and Dublin Core) supported by Yahoo's search engine.

This is covered in more depth in Semantify Hacks - Creating a your own RDF schema using Dapper:

So now, building a Dapp means you also built your own RDF compatible schema, that you can use wherever by just pointing to the webservice:

http://www.dapper.net/websiteServices/dapp-scheme.php?dappName=MYDAPP

The given example is MSN's search engine which you can see in all its RDF/XML glory.

ReadWriteWeb has step-by-step instructions.

Into Thick Air

Chrome, JavaScript, and Flash: Two (Mostly) Opposing Views the second comment closely follows a reason update on CounterNotions, Google Chrome: Bad news for Adobe.

But a full-fledged browser. One that behaves, however, as a platform to host applications best tied to cloud computing with built-in local persistence for offline computing. Sure, in its current form Chrome can’t compete with Silverlight or Flex/AIR for what Adobe calls “expressiveness,” meme-speak for rich graphics, animations, integrated video and other visual UI goodies.

Chrome may shut it off for good. It’s possible that various open source Chrome technologies could melt into Safari and Firefox. But –– whether as a stand-alone product or a progenitor of fast, powerful and expressive browsers –– Chrome signals to anybody but the diehard Microsoft constituents that the browser itself, not a proprietary plug-in or a separate runtime, is the future of RIAs. With its huge ecosystem, Microsoft can live with that. At least until its enterprise monopoly seriously erodes. But Adobe cannot.

In a world where the online pie is divided among the .NET army of Microsoft, the browser-gang of Apple+Mozilla+Google, and the lone Adobe, it’s not difficult to predict whose share will shrink into insignificance. If the exclusion of Flash from the iPhone wasn’t a wake-up call for Adobe, Chrome should certainly be one.

Most of the commentary is focused on the browser within an operating system angle. Although one of the easter eggs is a familiar screensaver. I think it's more helpful to concentrate on the fact that these browsers are getting rich enough to remove the applications embedded within browsers. There is already a lot of functionality developed or being developed such as SVG and storage (part of HTML 5). Chrome ships with Gears though and Webkit, to see how HTML5 and Gears relates see Aaron Boodman's talk on implementing HTML5 in Gears. They create two namespaces one for implementing the standard APIs and one for non-standard APIs - it seems like it has quite a solid development process behind it. Is there really a lot of reason left to support these proprietary applications within applications?

More MapReduce Groovy

A very good post on Cascading (covered previously), GOODBYE MAPREDUCE, HELLO CASCADING

Cascading’s logical model abstracts away MapReduce into a convenient tuples, pipes, and taps model. Data is represented as “Tuples”, a named list of objects. For example, I can have a tuple (”url”, “stats”), where “url” is a Hadoop “Text” object and “stats” is my own “UrlStats” complex object, containing methods for getting “numberOfHits” and “averageTimeSpent”. Tuples are kept together in “streams”, and all tuples in a stream have the exact same fields.

An operation on a stream of tuples is called a “Pipe”. There are a few kinds of pipes, each encompassing a category of transformations on a tuple stream. For instance, the “Each” pipe will apply a custom function to each individual tuple. The “GroupBy” pipe will group tuples together by a set of fields, and the “Every” pipe will apply an “aggregator function” to all tuples in a group at once.

One of the most powerful features of Cascading is the ability to fork and merge pipes together.

Once you have constructed your operations into a “pipe assembly”, you then tell Cascading how to retrieve and persist the data using an abstraction called “Tap”. “Taps” know how to convert stored data into Tuples and vice versa, and have complete control over how and where the data is stored. Cascading has a lot of built-in taps - using SequenceFiles and Text formats via HDFS are two examples. If you want to store data in your own format, you can define your own Tap. We have done this here at Rapleaf and it has worked seamlessly.

Senior Re-Searcher

Someone who continues to search (re-search) Google until he finds what is required. Senior in that he remembers a time before Google.

Stats for Nerds

I read "Is google chrome under / partially reporting RAM usage ?" which seems to be the typical problem with Microsoft's task manager. A better way to check out Chrome vs others is to hit Shift-Escape, click on "Stats for Nerds" and it has a comparison there.

Antimetabole

Bill Clinton had an interesting one recently in his Democratic convention speech: "People the world over have always been more impressed by the power of our example than by the example of our power."

Thesis and Antithesis

David Anderson on Agile. He introduces his talk by discussing some very broad ideas, including how agility has been applied to other areas. He talks about how the agile manifesto seems to have become more about belief and superstition than a scientific model. It's also rarely clicked through - to see the principles behind the manifesto.

The agile community has found it is better to develop in a failure tolerant environment. This was a reaction to the focus on more and more accurate estimates which lead to analysis patterns (or antipatterns). They are antipatterns because there was never enough detail in the analysis to provide accurate enough estimates and to deal with the unexpected.

Through his talk he listed a bunch of thought bubbles on software development:
* Value is providing functionality fastest.
* Knowledge work is perishable.
* Perfect is the enemy of good enough.
* Develop a high trust, high social capital.
* Have a highly collaborative culture.
* Reflect and adjust.
* Sustainable pace.
* Craftmanship.
* Value is contextual, context is temporal.
* Waste over scale.

Traditional industries have relied on bigger batch sizes to provide better economies of scale. With software development the transaction costs causes large batch sizes to work against you. Smaller batches allow you release more business value more quickly and reduce waste (waste over scale).

He introduces kanban which has ideas such as: pull, flow, and regulate work in progress. It does not use time boxed iterations and little or no planning or estimations (at least from a traditional agile approach). It still has constant improvement and delivery. It drops the idea of a generalist approach to labor which is seen by some as waterfall in disguise. The reason this comes about is because at an enterprise scale it's not feasible to hire a lot of experts who are excellent generalists - the labor pool does not exist. This leads to a tension with typical lean principals of reducing waste by having generalists.

He sees two main changes coming in the future: software factories (software product lines - which includes DSLs) and cloud services (deploying web services in the cloud). Architecture and modeling will come back in fashion as a value chain allows incentives for delivering common behavior. He's talking about a 100 fold improvement in productivity through using these ideas.

Finally, he talks about how CMM/CMMI is actually quite a good indication of the ability for software projects to succeed and the industry as a whole. He mentions a report on how agile and CMMI can work together, the idea being that not only is organizational maturity a good idea but it actually means that agile methods can be implemented better (it's called "CMMI or Agile: Why Not Embrace Both?!" - I only found a few references, for example Agile+CMMI Panel @ SEPG).

Via.

Still have Richard P. Gabriel's talk to go through too.

Answering the Right Question

From a reddit comment on Ubiquity:

It's a natural-language interface to the entire web. And not a lame "find me the page at http://xxxxx", but "find the page, filter it of information, extract and cross-reference this data with multiple other data-sets and deliver the results to me in a format and medium of my choosing".

It's not really naturally language anymore than "Sam and Max" was natural language but the point of the filtering, finding, and integrating seems spot on to me - that's answering the right question.

It's a bit like Linus talking about how Subversion sucks (about half way through is a good place to start) - they made branching cheap - who cares.

It's this whole Unix pipes for the web thing I guess.

Jesus Camp

This seems like quite an amazing depiction of American fundamentalists. It seems as if bigotry and fundamentalism is being turned into a competition with the Middle East. If you don't like to be scared about American evangelicals look away.

• Harry Potter is the Devil - JESUS CAMP - the one that started it all - kinda weird but interesting.


• Bill Maher on Jesus Camp - talks about this (from last year).


• Jesus Camp Bush Insanity (or Child Abuse). Your call. - laying on of hands to a cardboard George Bush, the mixture of religion and politics - very scary.


• Oprah Denies Christ - she tries to reason that there must be more than one way to go to heaven - but of course her audience members don't allow her to get away with that one. Scarier.


• CBC Interview with 'Jesus Camp' Filmmakers - scariest.

Finally a Reason for HTML Email

Ubiquity for Firefox. I've been a little hestitant recently to put the stuff I've been finding interesting. But this has to go up - it's a really great example of user driven mashups and what this Semantic Web thing should be all about.

Update: More information at "If You Want To Create a Mashup, Just Ask Your Browser. Mozilla Labs Launches Ubiquity." and the Mozilla Labs blog.

MapReduce Groovy

Cascading "The processing API lets the developer quickly assemble complex distributed processes without having to "think" in MapReduce. And to efficiently schedule them based on their dependencies and other available meta-data."

Also, HBase now supports transactions (using Optimistic Concurrency Control).

Via “Beyond Relational Databases”.

The Trouble with Mini-Wheats

Australian Mini-Wheats are apparently the only Mini-Wheats with triticale, oats, barley, wheat, and rye. Why is that interesting? Triticale or quadrotriticale (cause we're in the future) was in "The Trouble with Tribbles". How many Star Trek references do you eat?

Information Poo

• Inspiron 910 hopefully will come out on Friday. Inspiron means cheap in Dell marketing? I guess it's better than Dell E. There's something wrong with looking forward to a Dell. I mean there is no Dell Insider or Inspiron Rumors.


• SearchMonkey Tutorials. SearchMonkey uses a format similar to RDF called DataRSS. The FAQ describes the parts of the Semantic Web that work with the parts of SearchMonkey - no reasoning just the triples.


• Introduction to High-Level Programming With Scala. A compiled language that is finally catching up with Ruby's duck typing.


• Dr Horrible Ringtones.


• Freebase and Parallax talks about the excellent demo making the rounds. It's available as an open source project. Who knew George Bush has had more than one assassination attempt made on him? That's the power of the Semantic Web.


• The goodness of Google apps talks about another happy corporate customer (except for Google Code - integrating GitHub does sound good) and mentions Guido's code review tool part of rietveld a Google App Engine application.


• ccREL: The Creative Commons Rights Expression Language. Using RDFa so you don't repeat yourself.


• SCOPE, Automatic Hadoop Optimization and Cassandra (Facebook's BigTable) all via Greg. Cassandra is a Google Code project and includes some good documentation.


• OWL ED papers including an ontology versioning system and OWL constraints.

One Machine

Kevin Kelly predicting the next 5,000 days of the web. The specifications for the current web, the one machine, is interesting. It uses 5% of the electricty on Earth and holds and processes the equivalent of one human brain (1 HB). If technology continues growing at the current rate, 6 billion HB will be achieved somewhere between 2020 and 2040, outstripping the current human population (and hopefully the population then as well). He also mentions that every bit will be a web bit, that all bits go through the web, and you see that happening with traditionally non-web data usage like word processing or mobile phone traffic (actually internet would probably be more accurate). This coincides with the convergence of digital and atomic "worlds", rather than moving from one to the other (like in "The Matrix") he suggests that it'll be integrated and that we are the extension of the web rather than the other way around.

The second half of the talk is about the Semantic Web (found from here, here and here). He describes his own definition of the Semantic Web by showing how it fits into what has gone before. The stages of networking includes: connecting by site from one computer to another, connecting by page and linking between them and the last two stages (I didn't see a clear distinction between the two) is by data, idea or item.

His web site also mentions that computers are beating some of the best players at Go (the article is quite bad - teraflops as a measure of storage - this one is better).

Real Developers don't use Ruby

Hadoop: When grownups do open source. It's quite an amusing read - especially the part about the word count example on 9,000 blogs, the digg at Twitter, Starfish being practically useless (using MySQL and no Reduce phase) and the bit about understanding something being harder than writing a Ruby version of it.

"Twitter decided they would be cute and trendy. They wrote their code in Ruby: the official state language of the hipster-developer nation. Doug Cutting, on the other hand, decided he would get xxxx done, and wrote Hadoop in Java. Starling was hidden away in some corner and forgotten (it's hosted at RubyForge...). Hadoop lives prominently at the Apache Software Foundation. Starling is a re-hash of an existing Java Enterprise API called JMS that has several open source implementations. Hadoop is an implementation of Google's MapReduce, a system that publicly only existed on paper. Hadoop has the added benefit of actually working."

Ahh the joys of installing Visual Studio - enough time to install IntelliJ, run it up, and catch up on news.

Pigeon Programming

Intellij 8 continues with "all you need is a space bar and meta combinations to program" functionality: "Pressing Ctrl-Shift-Space twice allows you to find values of the expected types which are "two steps away" (can be retrieved through a chained method call)." It seems to be supporting what I'd consider a code smell.

Burnator

Ack, using ISOs on Windows XP fits in that time period where I think I'll do it often enough to remember what software to use but ends up being long enough that I don't. So the two I've used the most is InfraRecord and IsoRecorder (a secret).