Outer Joins aren't Primitive

Optional data in SPARQL seems to be equivalent to left outer join in SQL. As it turns out, outer joins can be composed of disjunctions. This is similar to the original MAYBE function suggested to be added to Kowari (although that suggestions is quite a deal simpler). The below paper outlines algorithms to do outer queries more efficiently. They require computing the anti-join of certain relations - an antij-oin being the set difference between two tables (or MINUS operation). Here is a good explanation of semi-joins and anti-joins.

Outerjoins as Disjunctions "The outerjoin operator is currently available in the query language of several major DBMSs, and it is included in the proposed SQL2 standard draft. However, “associativity problems” of the operator have been pointed out since its introduction. In this paper we propose a shift in the intuition behind outerjoin: Instead of computing the join while also preserving its arguments, outerjoin delivers tuples that come either from the join or from the arguments. Queries with joins and outerjoins deliver tuples that come from one out of several joins, where a single relation is a trivial join. An advantage of this view is that, in contrast to preservation, disjunction is commutative and associative, which is a significant property for intuition, formalisms, and generation of execution plans.Based on a disjunctive normal form, we show that some data merging queries cannot be evaluated by means of binary outerjoins, and give alternative procedures to evaluate those queries. We also explore several evaluation strategies for outerjoin queries, including the use of semijoin programs to reduce base relations."

Also related, Outer Join in Edutella where each part of the outer query is done individually.

A Really Interactive Query Language

SQLBuilder "SQLBuilder uses clever overriding of operators to make Python expressions build SQL expressions -- so long as you start with a Magic Object that knows how to fake it."

An example:

>>> from SQLBuilder import *
>>> person = table.person
# person is now equivalent to the Person.q object from the SQLObject
# documentation
>>> person
person
>>> person.first_name
person.first_name
>>> person.first_name == 'John'
person.first_name = 'John'

Via, SQL API "I'd rather SQLObject be built on some ORM-neutral layer, where you can move down to that layer when SQLObject doesn't fit your problem; as opposed to now, where you kind of have to work around SQLObject."

This is almost exactly like something I was thinking about, to prevent semantically incorrect SQL queries. Add an AJAX interface on this and it would be cool and useful.

DRY and Embedded Program Code

What if you could define a user interface and surf it via a telephone or browser and that the data and state from one to the other was able to be shared across multiple users?

Beyond interactive voice response "English’s hope, he tells Inskeep in the interview, is that companies will admit how infuriating their systems often are...the practice of automatically collecting customers’ account numbers, and then making those customers repeat the numbers to an agent when they finally connect to one -- my top IVR gripe"

"Voice calls must be able to recruit data channels, and vice versa. That way, an agent could attach an IM session to your voice call and push you the URL in real-time chat. It might even be appropriate to extend the data session with screen sharing, so the agent can watch and assist. If things still don’t work out and the whole matter must be referred to someone else, you’d like to be able to initiate voice or data communication -- or both -- in a context-preserving way."

Via, Rethinking customer service. This points to XBL2 and an effort to provide "...a declarative format for applications and user interfaces...based on an existing application/UI format, such as Mozilla's XUL, Microsoft's XAML, Macromedia's MXML or Laszlo Systems' LZX..."

Model Driven, Semantic Web Enabled, Science Commons

Semantic Web eyed for life sciences data "The Semantic Web involves a concept in which data from multiple sources and ontologies can be integrated into a single information space. Experiment design automation (XDA) software vendor Teranode, which focuses on software for life sciences, plans to collaborate with Science Commons to build a neurology repository for the Semantic Web."

NeuroCommons is part of the ScienceCommons project, it is going to provide a database and annotations of scientific data in (presumably) RDF.

Teranode explains with their XDA product, why model driven and why the semantic web.

A related post, via Etymon.com Federated Databases in Science "The astronomy, chemistry, and geospatial communities were active well over a decade ago in collaborating with information scientists on federated databases through various open standards. Molecular biology is a field that currently has considerable needs in this area, stimulated by the Human Genome Project. Developing common standards through consensus is of course not a technological solution. The Web is successful because it exploits the relationships among a huge number of people making individual judgements that only people can make. Even the Semantic Web, if it ever has a chance of working, would have to depend on a very large base of common metadata standards, and that can only result from the slow process of people coming together and agreeing. There are many things that information technology cannot do on its own. The semantic integration of knowledge still remains a human activity."

Graphical Batch Files

AutoMate This is a pretty interesting application that basically provides similar functionality that OSX Automator provides. While you can create customized tasks in VBA it has lots of interesting inbuilt functionality manipulating Excel, FTP, terminal emulation, keyboard and windows manipulation.

Related, but much simpler is a piece of software called, AutoIt which was "...initially designed for PC "roll out" situations to reliably configure thousands of PCs, but with the arrival of v3 it has become a powerful language able to cope with most scripting needs." Basically allowing simple automation of window, mouse and keyboard events.

One Way Java is Better than Ruby

The unbridled humanity of APIs "But I think the Java guy has a point: 78 methods on your list objects isn't good. Less methods is good. Unless the result is stupid. Now, let's be honest here, Java is stupid. Dumb, idiotic, maybe written by people who aren't programmers; I just don't know how else to make sense of it. list.get(list.size() - 1) should be embarrassing. list.last or list[-1]? I think [-1] reads well enough, and fits into a very elegant set of functionality involving slices and whatnot. But I also think list.last is entirely justifiable. OTOH, list.get(0) isn't embarrassing, so list.first isn't as compelling."

"Maybe an interesting parallel is 0 vs. 1 indexing. 1 clearly seems more humane. I personally count starting from 1. I'm naturally inclined to index from 1. Languages go both ways on the choice...Of course Smalltalk indexes from 1, so no one gets everything right."

Humane Interfaces "Part of the reason this argument could go on forever is that Ruby’s Array is both an example of arguments for Humane design, and arguments against it...java.util.List isn’t really a shining example of good interface design either...Having two otherwise equivalent ways to perform the same operation is bad user-interface design, and it’s bad library interface design, because the existence of the synonyms actually adds to your cognitive load by making you choose between them."

Also, Why Ruby Shouldn’t Be Your Next Programming Language (Maybe).

Time you enjoy wasting, was not wasted

• "TestDriven.NET makes it easy to run unit tests with a single click, anywhere in your Visual Studio solutions. It supports all versions of Microsoft Visual Studio .NET meaning you don't have to worry about compatibility issues and fully integrates with all major unit testing frameworks including NUnit, MbUnit, & MS Team System." Discussion on using TestDriven.NET with Express. Via James.


• Obie Fernandez from ThoughtWorks on Ruby and the Semantic Web "Java Doesn’t Work for Ontologies...Polymorphism in RDF and OWL is very different than in Java which makes for admittedly clumsy API...Why Deep Integration Enhances Programming with Ruby...Gives Ruby programmer powerful inferencing capabilities against the underlying RDF database". Related to Scripting the Semantic Web and Northrop Buys Tucana and Continues Kowari.


• OO in One Sentence: Keep It DRY, Shy, and Tell The Other Guy Available here. Covers how OO methods are not function calls but message passing.


• Beyond Story Cards: Agile Requirements Collaboration "Using this idea of "latest responsible moment," we postpone creating our detailed requirements specification. In fact, we can do the specification for a story at the same time as we actually implement the story. (I'll be talking about that more in a little bit.) We don't need to know every detail about a feature ahead of time. Going into the level of detail is expensive, and if (or when) our plans change, that's wasted effort."
  

Making AJAX cool

Why Ajax Sucks (Most of the Time) "For new or inexperienced Web designers, I stand by my original recommendation. Ajax: Just Say No."

"Ajax breaks the unified model of the Web and introduce a new way of looking at data that has not been well integrated into the other aspects of the Web. With ajax, the user's view of information on the screen is now determined by a sequence of navigation actions rather than a single navigation action.

Navigation does not work with ajax since the unit of navigation is different from the unit of view. If users create a bookmark in their browser they may not get the same view back when they follow the bookmark at a later date since the bookmark doesn't include a representation of the state of the content on the page.

Even worse, URLs stop working: the addressing information shown at the top of the browser no longer constitutes a complete specification of the information shown in the window."

A little light I/O

Comparing Two High-Performance I/O Design Patterns "It is clear from the charts that C++ is still the preferable approach for high performance communication solutions, but Java on Linux comes quite close. However, the overall Java performance was weakened by poor results on Windows. One reason for that may be that the Java 1.4 nio package is based on select()-style API. Ð It is true, Java NIO package is kind of Reactor pattern based on select()-style API (see [7, 8]). Java NIO allows to write your own select()-style provider (equivalent of TProactor waiting strategies). Looking at Java NIO implementation for Windows (to do this enough to examine import symbols in jdk1.5.0\jre\bin\nio.dll), we can make a conclusion that Java NIO 1.4.2 and 1.5.0 for Windows is based on WSAEventSelect () API. That is better than select(), but slower than IOCompletionPortÕs for significant number of connections. . Should the 1.5 version of Java's nio be based on IOCompletionPorts, then that should improve performance. If Java NIO would use IOCompletionPorts, than conversion of Proactor pattern to Reactor pattern should be made inside nio.dll. Although such conversion is more complicated than Reactor- >Proactor conversion, but it can be implemented in frames of Java NIO interfaces. (this the topic of next arcticle, but we can provide algorithm). At this time, no TProactor performance tests were done on JDK 1.5."

Available in Java and C++ at Terabit.

Links to Share and Enjoy

Another list of links:

• XML 2005: Tipping Sacred Cows "Which brings us to one of our sacred cows: for decades we've had SQL for relational databases, and soon we'll have XQuery for general XML, and SPARQL for RDF...what if it was possible to construct a generalized query language, loosely coupled enough to work with any underlying data model? The mathematical basis for this was monoids. The presentation didn't actually define this fairly abstract term, only skipping from trivial examples like or to a fully worked representation of a generalized query. Erik's dynamic presentation style is such that I was not able to copy down the full example before he had moved on to the next slide. Whatever the details, it's a valuable topic in that it gets listeners to question their assumptions and see in new ways."


• Dabble combines the best of group spreadsheets, custom databases, and intranet web applications into a new way to manage and share your information online. A lot of the same conversations in Agile databases seem to occuring - has related to functionality, data integrity, migration (string to first name, last name, for example), data types and the like. Includes a blog and demo (movie). Merging is coming in version 2.0 and it's RAM based. It just cries out for an RDF data model. Via, Dabble is Bloody Brilliant.


• Problems with the $100 laptop "The time will certainly come when the appropriate tool to promote economic development will be a laptop produced very inexpensively in large volume. Before that point it will be necessary to implement systems that provide infrastructure which the laptop will need, in addition to producing tangible economic benefits for their users. OLPC is to be commended for raising issues and focusing attention, and for posing some technological challenges in a highly visible way...large sums of money are to be committed to the project in advance to fund manufacturing in deals where the customers are government ministries and not the end users." Also, $100 laptop.


• Two interesting articles: Breaking The Quality–Speed Compromise "The most important thing we can do to break the compromises we impose on customers is to move testing forward and put it in-line with (or prior to) coding. Build suites of automated unit and acceptance tests, integrate code frequently, run the tests as often as possible. In other words, find and fix the defects before they even count as defects." and Is Agile Software Development Sustainable? "So if agile practices are a “disruptive technology” compared to traditional software development processes, then it would be quite in character for them to start by addressing small systems."


• Exploratory Testing on Agile Projects Can Be a Good Fit "Why should agile teams do exploratory testing?: "Because an agile development project can accept new and unanticipated functionality so fast, it is impossible to reason out the consequences of every decision ahead of time. In other words, agile programs are more subject to unintended consequences of choices simply because choices happen so much faster. This is where exploratory testing saves the day. Because the program always runs, it is always ready to be explored.""


• Matthew De George on Cranky Middle Manager. Explaining how to apply market economies to management and more. Forgive Matthew for his hierachical view, it's all about graphs of course. I'm a bit slow in finding this.


• A different way to vommit in another yearly milestone: Tiger Moth Joy Flights.

Best things in development are free

One of the key differences between Java and .NET development is cost. To get the right Microsoft solution costs thousands of dollars. And what you get is something very different to an IntelliJ, NetBeans or Eclipse. You get vendor integration (or lock in if you prefer) and competition against the community. It may appear attractive to some but it seems odd to me to actively fight integration into existing, open solutions. It does seem though that the open side is winning (I wish Sun had've done the same thing when choosing a logging API).

Unit testing and source control are obvious ones. It's amazing to see that their entry level IDE does not come with this. There are great free solutions in NUnit and MbUnit. MbUnit is especially cool (released recently) allowing all sorts of built in test fixtures. And there's always Ankh for Subversion integration. There are lots of alternatives.

A summary from a Microsoft developer: Hey, Shareholders! VS 2005 is *Fantastic* and our Developers Love Microsoft! "I might wander in early on Monday to meander through the crowds celebrating the big Visual Studio launch. But my heart is heavy that we shoveled what we could together and Won't Fix-ed this release out the door. Microsoft has just opened a very big door to competition in the IDE space. Or at least towards people jealously holding onto VS 2003 and saying, "CLR 2.0? Screw that! The last time I tried to use generics my machine locked up!" Big freakin' mistake. Microsoft should be ashamed."

Little Ink

• Semantic Web as Webized Database "In the absense of the close coupling of designers, developers, users, and applications that is found in successful database implementations, what do the semantic web technologies offer in the way of establishing a shared view of the corespondence between the data and real world?" Links to Adam Bosworth's Learning from THE WEB.


• "IRIS is a semantic desktop application framework that enables users to create a “personal map” across their office-related information objects. IRIS includes a machine-learning platform to help automate this process. It provides “dashboard” views, contextual navigation, and relationship-based structure across an extensible suite of office applications, including a calendar, web and file browser, e-mail client, and instant messaging client." Screenshots.


• A relational algebra for SPARQL "Despite being in the Last Call stage of the W3C recommendation track, the SPARQL query language document currently lacks mathematical rigor and fails to accurately define the semantics for some cases...SPARQL doesn’t use a special value to indicate missing information, but simply leaves variables unbound. There’s no explicit heading. The SPARQL model does not, for example, distinguish between an OPTIONAL variable that is unbound in some solutions, and a variable that is not used in the query at all."

Faking It

A RATIONAL DESIGN PROCESS: HOW AND WHY TO FAKE IT "(1) In most cases the people who commission the building of a software system do not know exactly what
they want and are unable to tell us all that they know.

(2) Even if we knew the requirements, there are many other facts that we need to know to design the software. Many of the details only become known to us as we progress in the implementation. Some of the things that we learn invalidate our design and we must backtrack. Because we try to minimize lost work, the resulting design may be one that would not result from a rational design process.

(3) Even if we knew all of the relevant facts before we started, experience shows that human beings are unable to comprehend fully the plethora of details that must be taken into account in order to design and build a correct system. The process of designing the software is one in which0 we attempt to separate concerns so that we are working with a manageable amount of information. However, until we have separated the concerns, we are bound to make errors.

(4) Even if we could master all of the detail needed, all but the most trivial projects are subject to change for external reasons. Some of those changes may invalidate previous design decisions. The resulting design is not one that would have been produced by a rational design process.

(5) Human errors can only be avoided if one can avoid the use of humans. Even after the concerns are separated, errors will be made.

(6) We are often burdened by preconceived design ideas, ideas that we invented, acquired on related projects, or heard about in a class. Sometimes we undertake a project in order to try out or use a favourite idea. Such ideas may not be derived from our requirements by a rational process.

(7) Often we are encouraged, for economic reasons, to use software that was developed for some other project. In other situations, we may be encouraged to share our software with another ongoing project. The resulting software may not be the ideal software for either project, i.e., not the software that we would develop based on its requirements alone, but it is good enough and will save effort."

Does writing software show a lot about human nature? Via Why There Is No Rational Software Design Process.

JRDF 0.3.4.2 is out

Another bug fix release. This time it was problems in the parser with RDF/XML literals and a fix for EscapeUtil. The first was an interesting problem, not directly to do with fixing the problem per se, but trying to maneuver the code in such a way as to only use the standard Java SDK. The second was a simple regex change and finding out how Unicode support has changed from 1.4 to 1.5 (and how it breaks certain regexs from 1.4). It now means that JRDF 0.3.4.2 works under 1.4 and 1.5 as expected.

Both of these problems reinforce the idea that without tests to prove code works almost always means that it doesn't.

Microsoft has an interesting and incorrect words on TDD (from here).

UPDATE: Microsoft seems to have pulled the above mentioned article, although it's still available in Google's cache.

UPDATE 2: More comments artima developer forum.

UPDATE 3: Microsoft Gets TDD Completely Wrong "Microsoft has completely missed the point of TDD. They got it wrong. Do not follow their guidelines: they will decrease productivity. You'll find that the process they've described doesn't work. If you stick with it, you'll find yourself writing increasingly bad code to work around its problems."

Doomed to Repeat History

Detecting Semantic Errors in SQL Queries, for example: "SELECT * FROM EMP WHERE JOB = ’CLERK’ AND JOB = ’MANAGER’". Obviously, JOB cannot be two values - yet you are allowed to express it. This and other examples are queries that are simply wrong and should be detected as such. It seems that this entire area of research exists because it's possible to write semantically incorrect queries in the first place. While the first example above may require something like better feedback at the command line, other examples like HAVING and DISTINCT exist because of the language design. It would be neat to have an SQL interpreter that would prevent these incorrect queries from being submittted (similar to IntelliJ or Word). It also covers solving problems with subqueries (using Skolemization) and Null Values.

From the SQLLint page.

Military Intelligence and IQL

Smart Searching "Sources of intelligence in the field include feeds from UAVs, intelligence, surveillance and reconnaissance data from a vast array of sensors and overhead platforms, signal intelligence, satellites, film and video, not to mention all the data from the open source world."

"To conduct research and analysis effectively, DIA relies on a broad inventory of technology tools from such companies as Endeca Technologies, Basis Technology, Inxight Software, Insightful, Attensity, Convera, NetOwl and Clearforest."

"Many of the search engines familiar to consumers, such as Google, are based on Boolean logic to perform a search with complex, long queries. But the Boolean language is not as expressive as it could be in the kind of query one can pose. “The InFact Query Language (IQL) can express in three words what would take 20 lines using Boolean language,” said Marchisio.

Although Insightful offers a four-hour course on using IQL, it is currently working to increase usability in order to eliminate the need for the course and to reach a broader audience among those who are not all super users or might not have the time to learn how to use the technology."

Examples of the InFact query language can be performed on the live demo here. For example: "USA > invade > Iraq - returns links to all sentences mentioning USA invading Iraq", "[organization] > win> contract - returns links to sentences mentioning who won contracts" and "* > attack > 1st Infantry Division - returns links to sentences mentioning the division being attacked".

Also related to this area, using Semantic Web technologies is Ontology Works offering integration into legacy systems and terabyte scale knowledge servers.

Google Base data belongs to us

"I’ve read the little background material on Google’s Base and still can’t see whether the material you put there can be found by other search engines. I also cannot find evidence of an API that shares any standards for tags and structure. Is Base open or closed? So far, closed."

"I wish I were hearing more noise from the microformats guys to act as competitors — or at least as pressure on Google for openness and standards."

Google Base seems like a roach motel for your data. Its not Google's data - its your data. Danny has some thoughts on this.

Apparently, the answer is put your data on the web. The question remains, what will eBay's and database companies response be.

Old Skool

This arrived in my letter box (a graphical view of the included games is here). Another year not dying now has the added benefit of being able to relive part of my childhood in joystick form. For a mere $35, there are plenty of options for hacking: putting it in an old floppy disk drive, putting it in a case as well as hooking up a PS/2 keyboard and monitor. More information on Wikipedia, schematics and on a DTV hacking forum.

Kiwi Country

As a warning, this is not a usual post about RDF, the semantic web, Java, software development, etc.

This is just a little list I made on holiday based on my limited interactions while touring the south island of New Zealand:

• Money - Finally a country where the denominations are the correct size - the $2 coin is bigger than the $1 coin. It annoys me that denominations aren't based on size - both the US and Australia are guilty of this. Another example of some sensible thinking, the emergency number is 111 rather than 000 (New Zealand was first of course). You're having a heart attack, you've got a rotary phone, which number would you prefer to call.


• Lack of infrastructure - this is related to population I guess but the lack of guard rails, petrol stations, dual carriage ways, etc. especially around mountainous roads was very disconcerting.


• No newsagents - I really can't imagine living in a place that doesn't have newsagents or good book stores. In the US and Australia there's usually a healthy selection of magazines and newspapers - the only magazines that I could find were in a bookstore and there was a Mac one, a PC one and that's about it.


• Food - was pretty much the same with Coke tasting the same (although 20mls less in a can), more pie shops, good white wines (chardonnay that was drinkable), chocolate fish and alcohol in supermarkets.


• Language - while I started truncating my vowels by the end of the trip I also noticed that most people used "wee" rather than "little" a wee bit more than I was used to.


• Another country with long history of stupidly introduced species. Somewhat of a surprise was that most of the really bad ones are from Australia. Possums, magpies, wallabies, stoats, weasels and rabbits seem to be the main culprits for a country whose native fauna consists mainly of birds. The idea of a ferret eating a penguin alive rids you of some of the cuteness associated with mammals. I knew about the possums beforehand (which are pretty awful anyway) but the damage is pretty hideous and its good to see some productive use is made of them with possum fur combined with wool in clothing. The bird highlights were the kiwi, kakapo (introduced to me in Last Chance to See and recently highlighted in Kakapo Crisis), kea (also a concept extractor) and fiordland crested penguins (who have lost the fear of land based predation). The future does look good with many of them being introduced on islands where they are safe.


• Real estate or how beautiful places are being overrun by rich foreigners. This not unique to New Zealand or Australia but it was sad to hear that working class locals are quickly being out-priced by rich foreigners and may not be able to afford to live in the same place as where they have grown up.


• It would appear that the Maori culture is vastly better integrated and embraced than native culture in Australia (and many other parts of the world I would guess). There is a dedicated Maori television channel, radio programs, Maori is the official language and there seems to be much greater cultural interaction. Its not without it problems and injustices but it seems New Zealand has a much better history of treating people decently. This includes it being the first country to give everyone over 21 the vote (in 1893) and generally being a progressive socialist state. Another example, universal voting wasn't granted until the 1960s in Australia whereas in New Zealand it was there from the start (it did require voters to have individual title of the land however).


• Water, water everywhere. I'm fairly convinced that some places, like South Australia, are fairly silly places to live, especially compared with somewhere like New Zealand - which is basically paradise without the snakes. New Zealand is almost arrogantly wet and fertile. The sheer number of animals (mainly sheep) in a paddock was impressive.


• I don't want to reinforce stereotypes but the most annoying tourists would have to be American. It wasn't the loud complaining per se but more the insistence to understand everything and it had to be within their own cultural context. There seemed to be a lack of adapting and accepting things and it's just annoying to always suppose that something is worse just because it's different. I know this isn't an unfamiliar sentiment - the converse is often said to be true - that others don't stand up for themselves enough. I'm trying to be even handed here, maybe everybody does it and sure not all Americans are the same, maybe its just the ones that go on holidays, maybe I'm a cultural fascist, but it was really, really annoying.


• And one last thing, the iPod compatible bed.

Lazy Links

* MKSearch Beta 1 Released - includes web crawler, HTML metadata extractor, and RDF storage using Sesame. Also, MG4J (Managing Gigabytes for Java).
* Open Source Java Application Management: BlueGlue and MyJavaPack. A little different to Ivy (also interesting IvyCruise).
* Concurrency JSR-166 Interest Site includes interesting posts like Java Memory Model versus dotnet Memory Model mentions the forthcoming problems with Java code when multi-core systems are rolled out. Also, Concurrent Skip List Map (coming to Java 6).