Thursday, January 03, 2008
Intellij Under OSX
Seems that the bug with IntelliJ under OS X 10.5 (Leopard) (works for me now anyway) is fixed in currently nightly builds and will be in 7.0.3. More information here.
Saturday, December 29, 2007
Wii!
I couldn't be more impressed. For the first time in about 25 years (I think the last time was H.E.R.O. for the Atari 2600) my mum has sat down (well stood up too) and played a computer game - Wii golf, bowling and tennis. While I remained victorious, it wasn't by much (especially golf). It reminded me of a description of the arrival of Pong by Nolan Bushnell where women would hustle men (induce to gamble over the outcome of a game) in bars.
Monday, December 24, 2007
Fat Controller
When does MapReduce make sense and what architecture is appropriate? I don't really know; wheras Tom has some ideas. I like the idea of MapReduce in the Google architecture (as cloned by Hadoop). I like the use of a distributed file system to evenly smear the data across the nodes in the cluster. However, these choices don't always seem appropriate.
The easiest and most obvious example where this causes a problem is that sometimes you want to ensure a single entry or exit in your system (when to start or stop). Another obvious example is where the overhead of network latency overwhelms your ability to parallelize the processing. More than that, it seems that if you can order things globally then the processing should be more efficient but it's unclear to me where the line is between that and the more distributed MapReduce processing (and whether adding that complexity is always worth it).
Google's third lecture (slides) answers some of the basic questions such as why use a distributed file system. It also lists decisions made such as read optimization, mutation handling (serializing writes and atomic appends), no caching (no need due to large file size), fewer but larger files (64MB chunks) and how the file handling is essentially garbage collection. They have implemented appends as it was a frequent operation. This is something that Hadoop has yet to do and can be an issue, especially for databases (which includes the requirements for appends and truncates attached to that issue).
There is obviously some adaption needed to alogirthms to run in a MapReduce cluster. Lecture 5 gives some examples of MapReduce algorithms. It includes a breadth first search of a graph and PageRank. Breadth first is chosen so there doesn't need to be any backtracking. For graph searching, they suggested creating an adjacency matrix - 1 indicating a link in the matrix and 0 indicating no link. To transfer it efficiently they use a sparse matrix (where you only record the links - very much like column databases of course). MapReduce is similar, with the processing split as a single row per page. For both of these process there is a non-MapReduce component. For example, in the PageRank algorithm a process exists to determine convergence of page values.
The easiest and most obvious example where this causes a problem is that sometimes you want to ensure a single entry or exit in your system (when to start or stop). Another obvious example is where the overhead of network latency overwhelms your ability to parallelize the processing. More than that, it seems that if you can order things globally then the processing should be more efficient but it's unclear to me where the line is between that and the more distributed MapReduce processing (and whether adding that complexity is always worth it).
Google's third lecture (slides) answers some of the basic questions such as why use a distributed file system. It also lists decisions made such as read optimization, mutation handling (serializing writes and atomic appends), no caching (no need due to large file size), fewer but larger files (64MB chunks) and how the file handling is essentially garbage collection. They have implemented appends as it was a frequent operation. This is something that Hadoop has yet to do and can be an issue, especially for databases (which includes the requirements for appends and truncates attached to that issue).
There is obviously some adaption needed to alogirthms to run in a MapReduce cluster. Lecture 5 gives some examples of MapReduce algorithms. It includes a breadth first search of a graph and PageRank. Breadth first is chosen so there doesn't need to be any backtracking. For graph searching, they suggested creating an adjacency matrix - 1 indicating a link in the matrix and 0 indicating no link. To transfer it efficiently they use a sparse matrix (where you only record the links - very much like column databases of course). MapReduce is similar, with the processing split as a single row per page. For both of these process there is a non-MapReduce component. For example, in the PageRank algorithm a process exists to determine convergence of page values.
Sunday, December 23, 2007
Stacked
Apple to tweak 'Stacks' in Mac OS X Leopard 10.5.2 Update Getting better - now all they have to do is fix Spaces and Java.
Friday, December 14, 2007
Timeless Software
Barry Boehm (having read it many times it's pronounced "beam"), "A View of 20th and 21st Century Software Engineering", lists the timeless qualities of a good programmer that have been discovered through the decades. Some of them include: don't neglect the sciences, avoid sequential processing, avoid cowboy programming, what's good for products is good for process, and adaptability trumps repeatability.
The powerpoint for the presentation is also online.
Update: I have notes from this talk but little time to review them. There's three things that stick out in my mind though.
The first is slide 5. This lists the most common causes of project failure. Most of these are not technical but centre around project management and people. I think it can be successfully argued that the other issues, mainly technical problems, are also people problems too, especially things like "lack of executive support".
Slide 9 has a Hegelian view of the progress made in writing software. That is, there is a process of advancement that involves theses, antitheses and syntheses. For example, software as a craft was an overreaction to software being considered analogous to hardware development and agile methods are a reaction (some may say overreaction) to the waterfall methodology.
Thirdly, Slide 30 has what initially looks like a good diagram to help decide how agile your project should be. I had issues with this. Much of my experience with agile projects is that they are actually better planned than most traditionally run projects. I also don't see how the criticality of the defects is appropriate either, as the number of defects is far fewer and less critical in well run agile projects too. Maybe I've never been on a well run planned project.
This made me wonder, why does software development still seem so far removed from science? Or to put in another way, why is it so dependent on personal experience? OO is quite frequently vilified but the reason people supported and continue to do so was because reuse increased and costs decreased - people were just following the numbers. Before OO, it was structured programming. And in contrast, what seemed like good ideas, such as proving the correctness of programs, have not been widely adopted. Barry points out that the reasons, since its inception in the 1970s, was because it did not prevent defects occurring in the specification and it was not scalable. This lead to the ideas of prototyping and RAD which were developed to overcome these perceived failures.
The powerpoint for the presentation is also online.
Update: I have notes from this talk but little time to review them. There's three things that stick out in my mind though.
The first is slide 5. This lists the most common causes of project failure. Most of these are not technical but centre around project management and people. I think it can be successfully argued that the other issues, mainly technical problems, are also people problems too, especially things like "lack of executive support".
Slide 9 has a Hegelian view of the progress made in writing software. That is, there is a process of advancement that involves theses, antitheses and syntheses. For example, software as a craft was an overreaction to software being considered analogous to hardware development and agile methods are a reaction (some may say overreaction) to the waterfall methodology.
Thirdly, Slide 30 has what initially looks like a good diagram to help decide how agile your project should be. I had issues with this. Much of my experience with agile projects is that they are actually better planned than most traditionally run projects. I also don't see how the criticality of the defects is appropriate either, as the number of defects is far fewer and less critical in well run agile projects too. Maybe I've never been on a well run planned project.
This made me wonder, why does software development still seem so far removed from science? Or to put in another way, why is it so dependent on personal experience? OO is quite frequently vilified but the reason people supported and continue to do so was because reuse increased and costs decreased - people were just following the numbers. Before OO, it was structured programming. And in contrast, what seemed like good ideas, such as proving the correctness of programs, have not been widely adopted. Barry points out that the reasons, since its inception in the 1970s, was because it did not prevent defects occurring in the specification and it was not scalable. This lead to the ideas of prototyping and RAD which were developed to overcome these perceived failures.
Scala Tipping the Scales
With the announcement of the Scala book and a couple of posts on Reddit: "The awesomeness of Scala is implicit" and "Why Scala?". This coincided with me reading up on what the Workingmouse people (like Tony) were up to with Scalaz.
I've only read Chapter 1 of the book but it's fairly compelling reading mentioning that it's OO, functional, high level, verifiable, etc. It also gives examples of how succinct Scala is compared to Java.
Compared to:
This works for me much like the Ruby language and the books. You can understand where you came from and how it's better. It seems much more compelling than giving a sermon about how all Java users are fornicators of the devil and that they corrupt young children. It seems that most of the industry when selecting a computer language, to continue the religious theme, act like born-agains - jumping from one state to another, "I used to be a sinner managing my own memory, using pointer arithmetic and now I have seen the one true, pure language".
Update: There's also an interview with Bill Venners on Scala. He compares it with LINQ, why keeping static typing is important, how using the Option type removes nulls, that it is both more OO than Java (it removes static methods and primitives) while being fully functional, describes singleton objects and how it achieves the goal as a scalable language. One of the interesting points he makes is that strong types makes you solve problems in a particular way.
I've only read Chapter 1 of the book but it's fairly compelling reading mentioning that it's OO, functional, high level, verifiable, etc. It also gives examples of how succinct Scala is compared to Java.
boolean nameHasUpperCase = false;
for (int i = 0; i < name.length(); ++i) {
if (Character.isUpperCase(name.charAt(i))) {
nameHasUpperCase = true;
break;
}
}
Compared to:
val nameHasUpperCase = name.exists(_.isUpperCase)
This works for me much like the Ruby language and the books. You can understand where you came from and how it's better. It seems much more compelling than giving a sermon about how all Java users are fornicators of the devil and that they corrupt young children. It seems that most of the industry when selecting a computer language, to continue the religious theme, act like born-agains - jumping from one state to another, "I used to be a sinner managing my own memory, using pointer arithmetic and now I have seen the one true, pure language".
Update: There's also an interview with Bill Venners on Scala. He compares it with LINQ, why keeping static typing is important, how using the Option type removes nulls, that it is both more OO than Java (it removes static methods and primitives) while being fully functional, describes singleton objects and how it achieves the goal as a scalable language. One of the interesting points he makes is that strong types makes you solve problems in a particular way.
Wednesday, December 12, 2007
LINQed Data
I did a quick search for LINQ and RDF and there are two LINQ providers for .NET:
- Andrew Matthews' LINQtoRDF and LINQtoRDF Designer, which is a Google code project.
- Hartmut Maennel's A LINQ provider for RDF files. Available as a zip file
Wednesday, November 28, 2007
Academic Software and BioMANTA
On Monday I gave a presentation at the ACB all hands meeting on the BioMANTA project. It covered the basics: the integration process, ontology design and the architecture.
There were some very incomprehensible presentations. But of the ones I did understand the lipid raft modeling (which looked a bit like Conway's Game of Life) was perhaps the coolest. There was quite a few presentations of software that involved: "we did it this way, the specifications changed, next time we'll do it right because what we have at the moment is a mess". It's very frustrating to see that change is still not accepted as the norm.
Two posts I read recently reminded me of this too: "Architecture and innovation" and "Why is it so difficult to develop systems?". Both describe the poor state of academic software. Many projects seem to suffer from this problem, not just academic ones, although academic ones seem to be prone to adding technology because it's cool/trendy/whatever which may help get it published but usually obscures the real novel aspects (or worse they don't have anything except the cool or trendy technologies).
There were some very incomprehensible presentations. But of the ones I did understand the lipid raft modeling (which looked a bit like Conway's Game of Life) was perhaps the coolest. There was quite a few presentations of software that involved: "we did it this way, the specifications changed, next time we'll do it right because what we have at the moment is a mess". It's very frustrating to see that change is still not accepted as the norm.
Two posts I read recently reminded me of this too: "Architecture and innovation" and "Why is it so difficult to develop systems?". Both describe the poor state of academic software. Many projects seem to suffer from this problem, not just academic ones, although academic ones seem to be prone to adding technology because it's cool/trendy/whatever which may help get it published but usually obscures the real novel aspects (or worse they don't have anything except the cool or trendy technologies).
But my impression of the last 10-15 years (especially W3C and Grid/eScience projects) is that they rapid become overcomplicated, overextended and fail to get people using them.
Ultimately much of the database and repository technology is too complicated for what we need at the start of the process. I am involved in one project where the database requires an expert to spend six months tooling it up. I thought DSpace was the right way to go to reposit my data but it wasn’t. I (or rather Jim) put 150,000+ molecules into it but they aren’t indexed by Google and we can’t get them out en masse. Next time we’ll simply use web pages.
By contrast we find that individual scientists, if given the choice, revert to two or three simple, well-proven systems:
* the hierarchical filesystem
* the spreadsheet
A major reason these hide complexity is that they have no learning curve, and have literally millions of users or years’ experience. We take the filesystem for granted, but it’s actually a brilliant invention. The credit goes to Denis Ritchie in ca. 1969. (I well remember my backing store being composed of punched tape and cards).
If you want differential access to resources, and record locking and audit trails and rollback and integrity of commital and you are building it from scratch, it will be a lot of work. And you lose sight of your users.
So we’re looking seriously at systems based on simpler technology than databases - such as RDF triple stores coupled to the filesystem and XML.
Present and Future Scalability
Measurable Targets for Scalable Reasoning
An interesting aspect of this paper is its description of some of the difficulties in comparing data loading and querying across triple stores. For example, at load time this can include forward chaining (inferencing) and what the complexity of the data model is (whether named graphs and other metadata is used). Query evaluation can vary due to backward chaining, result set size and the types of queries.
While some triple stores can now load up to 40,000 triples a second (BigOWLIM) the average seems to be around 10,000 triples a second for a billion triples. The target in the next few years is 20-100 billion triples at 100,000 triples second. The rate of 100,000 triples per second is the upper range but I would imagine that to load the data in a reasonable time this is what people have to aim towards. Otherwise, 100 billion triples is going to take 100 days.
This analysis is based on published results of several of the most scalable engines: ORACLE, AllegroGraph, DAML DB, Openlink Virtuoso, and BigOWLIM. The targets are defined with respect to two of the currently most popular performance measuring sticks: the LUBM repository benchmark (and its UOBM modification) and the OWL version of UNIPROT - the richest database of protein-related information.
An interesting aspect of this paper is its description of some of the difficulties in comparing data loading and querying across triple stores. For example, at load time this can include forward chaining (inferencing) and what the complexity of the data model is (whether named graphs and other metadata is used). Query evaluation can vary due to backward chaining, result set size and the types of queries.
While some triple stores can now load up to 40,000 triples a second (BigOWLIM) the average seems to be around 10,000 triples a second for a billion triples. The target in the next few years is 20-100 billion triples at 100,000 triples second. The rate of 100,000 triples per second is the upper range but I would imagine that to load the data in a reasonable time this is what people have to aim towards. Otherwise, 100 billion triples is going to take 100 days.
Thursday, November 22, 2007
Why Riding Your Bike Doesn't Help Global Warming
An interesting article, "Climate Change – an alternative approach", which highlights that the current supply of oil fails to meet demand. So reducing your usage of CO2 emissions, like riding your bike to work, does not help - the drop in demand is quickly met by consumption somewhere else. The author is suggesting a more viable approach is to focus on production rather than consumption.
According to the author, the answer lies in the oil and coal producing nations reducing production which is much easier to coordinate than billions of consumers. This doesn't detract from investing in reducing consumption and the usual suspects (Australia, India, USA and China) are involved on both sides of production and consumption.
He was talking about flying to Sydney and stated that if you chose not to fly you were making an immediate carbon saving (as apposed to offsetting the flight where the saving was at least delayed if it ever happened at all). Does tearing up your ticket to Sydney reduce carbon emissions? Ask the question, have some fossil fuels been left in the ground that would otherwise be extracted? The answer, absolutely not, and I’m not talking about how the plane’s still going to fly without you.
I’m talking about the fact that oil extraction is not determined by demand, it’s determined by supply. It has been since earlier this decade when the market price diverged markedly from the production costs.
According to the author, the answer lies in the oil and coal producing nations reducing production which is much easier to coordinate than billions of consumers. This doesn't detract from investing in reducing consumption and the usual suspects (Australia, India, USA and China) are involved on both sides of production and consumption.
Friday, November 16, 2007
SPARQL isn't Unix Pipes
It wasn't supposed to be this way. I was just trying to get what I wrote in 2004 acknowledged. All I wanted then, as now, was aggregate functions and a matching data model. I did a bit more research in 2006 (mostly in my spare time while I had a day job to go to) and thought that people could read it and understand it. I even spent some time over the last Christmas holidays making a more gentler introduction to it all.
SPARQL is a proposed recommendation - which is one step away from being a published standard. So, I put my objections in to the SPARQL working group. From what I can tell people either didn't understand or thought that I was some kind of weirdo. The unhappiest part of this is the summary of my objection, "The Working Group is unfamiliar with any existing query languages that meet the commenter's design goals."
All I wanted was closure of operations, where the query language matches the data model it queries. Maybe this is a very odd thing to want. No one seems to know what the relational model really is either. Maybe it's a bad example.
Maybe a better example is Unix pipes. Unix pipes have operations (sort, cut, etc.) that take in text and output text. That is, it takes the same input as output or something known as closure. So you can take input from one tool and string them together in any order you want. Sometimes it's more efficient to do one operation first over another. In SPARQL you can't do that as the first operation of every query turns it into variable bindings.
I was hoping that SPARQL would be the Unix pipes of RDF. This would mean that the operations like join, filter, restrict (matching triples) and so on take in an RDF graph (or graphs) and output an RDF graph (or graphs). This gives tremendous flexibility in that you can create new operations that all work on the same model. It also means that a lot of the extra complexity that is part of SPARQL (for example, CONSTRUCT and ASK) go away.
This is not to say that SPARQL doesn't have value and shouldn't be supported. It is just a missed opportunity. It could have avoided repeating mistakes made with SQL (like not standarizing aggregate functions, having a consistent data model and so on).
Update: I re-read this recently. It struck me that maybe I was being a little unclear about what I expected as input and output in the RDF pipes view of SPARQL. Really, it's not a single RDF graph per se that are being processed but sets of triples. It's not really a big difference - as RDF graphs are just sets of triples but it's more that the triples being processed don't have to come from one graph. There's no restriction on what I'm talking about above to process, in one go, triples from many different graphs. The criticism is the same though - SPARQL breaks triples into variable bindings. Having multiple graph processing (or sets of triple processing) just requires the graph that the triple came from recorded (the quad in most systems). It certainly something that could be added to JRDF's SPARQL implementation.
SPARQL is a proposed recommendation - which is one step away from being a published standard. So, I put my objections in to the SPARQL working group. From what I can tell people either didn't understand or thought that I was some kind of weirdo. The unhappiest part of this is the summary of my objection, "The Working Group is unfamiliar with any existing query languages that meet the commenter's design goals."
All I wanted was closure of operations, where the query language matches the data model it queries. Maybe this is a very odd thing to want. No one seems to know what the relational model really is either. Maybe it's a bad example.
Maybe a better example is Unix pipes. Unix pipes have operations (sort, cut, etc.) that take in text and output text. That is, it takes the same input as output or something known as closure. So you can take input from one tool and string them together in any order you want. Sometimes it's more efficient to do one operation first over another. In SPARQL you can't do that as the first operation of every query turns it into variable bindings.
I was hoping that SPARQL would be the Unix pipes of RDF. This would mean that the operations like join, filter, restrict (matching triples) and so on take in an RDF graph (or graphs) and output an RDF graph (or graphs). This gives tremendous flexibility in that you can create new operations that all work on the same model. It also means that a lot of the extra complexity that is part of SPARQL (for example, CONSTRUCT and ASK) go away.
This is not to say that SPARQL doesn't have value and shouldn't be supported. It is just a missed opportunity. It could have avoided repeating mistakes made with SQL (like not standarizing aggregate functions, having a consistent data model and so on).
Update: I re-read this recently. It struck me that maybe I was being a little unclear about what I expected as input and output in the RDF pipes view of SPARQL. Really, it's not a single RDF graph per se that are being processed but sets of triples. It's not really a big difference - as RDF graphs are just sets of triples but it's more that the triples being processed don't have to come from one graph. There's no restriction on what I'm talking about above to process, in one go, triples from many different graphs. The criticism is the same though - SPARQL breaks triples into variable bindings. Having multiple graph processing (or sets of triple processing) just requires the graph that the triple came from recorded (the quad in most systems). It certainly something that could be added to JRDF's SPARQL implementation.
Thursday, November 15, 2007
Sesame Native Store
I'm very impressed at the moment with OpenRDF's native store as others have been in the past. One of the best things is how easy it was to work into the existing JRDF code.
As I've said before I've been searching for an on disk solution for loading and simple processing of RDF/XML. In the experiments I've been doing OpenRDF's btree index is much faster than any other solution (again not unexpected based on previous tests). The nodepool/string pool or ValueStore though is a bit slower than both Bdb and Db4o.
Loading 100,000 triples on my MacBook Pro 2GHz takes 37 secs with pure Sesame, 27 with the Sesame index and Db4o value store, 35 with Bdb value store and ehCache is still going (> 5 minutes). A million takes around 5 minutes with Sesame index and Db4o nodepool (about 3,400 triples/second) and 3 minutes with a Sesame index and memory nodepool (about 5500 triples/second).
There's lots of cleanup to go and there's no caching or anything clever going on at the moment, as I'm trying to hit deadlines. 0.5.2 is going to be a lot faster than 0.5.1 for this stuff.
Update: I've done some testing on some fairly low-end servers (PowerEdge SC440, Xeon 1.86GHz, 2GB RAM) and the results are quite impressive. With 100,000 triples averaging around 11,000 triples/second and 10 million averaging 9,451 triples/second.
Update 2: JRDF 0.5.2 is out. This is a fairly minor release for end user functionality but meets the desired goal of creating, reading and writing lots of RDF/XML quickly. Just to give some more figures: Bdb/Sesame/db4o (SortedDiskJRDFFactory) is 30% faster for adds and 10% slower for writing out RDF/XML than Bdb/Sesame (SortedBdbJRDFFactory). Both have roughly the same performance for finds. I removed ehcache as it was too slow compared to the other approaches.
As I've said before I've been searching for an on disk solution for loading and simple processing of RDF/XML. In the experiments I've been doing OpenRDF's btree index is much faster than any other solution (again not unexpected based on previous tests). The nodepool/string pool or ValueStore though is a bit slower than both Bdb and Db4o.
Loading 100,000 triples on my MacBook Pro 2GHz takes 37 secs with pure Sesame, 27 with the Sesame index and Db4o value store, 35 with Bdb value store and ehCache is still going (> 5 minutes). A million takes around 5 minutes with Sesame index and Db4o nodepool (about 3,400 triples/second) and 3 minutes with a Sesame index and memory nodepool (about 5500 triples/second).
There's lots of cleanup to go and there's no caching or anything clever going on at the moment, as I'm trying to hit deadlines. 0.5.2 is going to be a lot faster than 0.5.1 for this stuff.
Update: I've done some testing on some fairly low-end servers (PowerEdge SC440, Xeon 1.86GHz, 2GB RAM) and the results are quite impressive. With 100,000 triples averaging around 11,000 triples/second and 10 million averaging 9,451 triples/second.
Update 2: JRDF 0.5.2 is out. This is a fairly minor release for end user functionality but meets the desired goal of creating, reading and writing lots of RDF/XML quickly. Just to give some more figures: Bdb/Sesame/db4o (SortedDiskJRDFFactory) is 30% faster for adds and 10% slower for writing out RDF/XML than Bdb/Sesame (SortedBdbJRDFFactory). Both have roughly the same performance for finds. I removed ehcache as it was too slow compared to the other approaches.
Friday, November 09, 2007
JRDF 0.5.1 Released
This release is mainly a bug release. There are improvements and fixes to the Resource API, datatype support and persistence. Another persistence library has been added, db4o, which has some different characteristics compared to the BDB implementation. However, it's generally a little slower that BDB. The persistence offered is currently only useful for processing large RDF files in environments with low memory requirements.
Also, the bug fixes made to One JAR have been integrated, so JRDF no longer has its own version.
Available here.
Also, the bug fixes made to One JAR have been integrated, so JRDF no longer has its own version.
Available here.
Saturday, October 27, 2007
No Java 6 for You!
Just in case you're like me and you upgraded to Leopard only to find Java 6 no longer works and Java 5 unstable, here's a fix:
At least it comes with better Ruby support and a RubyCocoa bridge.
Update: I don't think I like Leopard. I don't like: the 3D or the 2D look of the dock, the semi-transparent menu bar, Spaces behaviour - you can't have multiple windows spanning desktops from the same application (like two browser windows in separate desktops), the removal of text lists (why not have Fan, Grid and List?), in Quick Look you can go to a page in a PDF or Word file but when you click on it it goes to the first page, and Java support (IntelliJ and others seems to have weird refresh issues and you can't seem to allocate it to a virtual desktop).
There's of course a lot to like though too (tab terminals, RSS reader, better Spotlight, Cover flow, Safari, etc).
Update 2: So the Spaces thing. If you minimise a window, it goes to the dock, change to another virtual desktop, then expand, it goes to the desktop that the window was originally in. As noted in the comments though (and it is in the guided tour), if you activate Spaces (the default is F8), then move the window to a new virtual desktop then it is tied to it. There are two other ways: click and hold on the window and switch to another desktop or click and hold on the window, move it to the edge of the screen, wait, and it will go to the next desktop.
To me, some of the behavior breaks the illusion/metaphor of the virtual desktop and seems unnecessarily difficult. I'd prefer non-click and hold options supported as well (pretty much like other features like copying files, etc).
The networking improvements (non-blocking, built in VNC) overcomes what was probably one of the worst things about OS X.
The IntelliJ issue that I mentioned is logged as issue 16084.
-- First, delete ~/Library/Java/Caches/deployment.properties
-- Move aside your Java 1.6 directory. The 1.6 preview on Tiger does not work on Leopard.
% cd /System/Library/Frameworks/JavaVM.framework/Versions
% sudo mv 1.6.0 Tiger_1.6
% sudo rm 1.6
At least it comes with better Ruby support and a RubyCocoa bridge.
Update: I don't think I like Leopard. I don't like: the 3D or the 2D look of the dock, the semi-transparent menu bar, Spaces behaviour - you can't have multiple windows spanning desktops from the same application (like two browser windows in separate desktops), the removal of text lists (why not have Fan, Grid and List?), in Quick Look you can go to a page in a PDF or Word file but when you click on it it goes to the first page, and Java support (IntelliJ and others seems to have weird refresh issues and you can't seem to allocate it to a virtual desktop).
There's of course a lot to like though too (tab terminals, RSS reader, better Spotlight, Cover flow, Safari, etc).
Update 2: So the Spaces thing. If you minimise a window, it goes to the dock, change to another virtual desktop, then expand, it goes to the desktop that the window was originally in. As noted in the comments though (and it is in the guided tour), if you activate Spaces (the default is F8), then move the window to a new virtual desktop then it is tied to it. There are two other ways: click and hold on the window and switch to another desktop or click and hold on the window, move it to the edge of the screen, wait, and it will go to the next desktop.
To me, some of the behavior breaks the illusion/metaphor of the virtual desktop and seems unnecessarily difficult. I'd prefer non-click and hold options supported as well (pretty much like other features like copying files, etc).
The networking improvements (non-blocking, built in VNC) overcomes what was probably one of the worst things about OS X.
The IntelliJ issue that I mentioned is logged as issue 16084.
Saturday, September 22, 2007
Migration
With the announcement of JRuby in Glassfish or the end of mongrel it seems to offer Sun the hope of capturing more developers not just from Ruby but from .NET. Many people from both Java and C# worlds jumped to Ruby and it occurred to me that .NET developers using Netbeans for Ruby development may notice that there's a suprisingly good C# clone under the covers.
Wednesday, September 19, 2007
Beautiful or Otherwise
Two of the chapters from Beautiful Code Alberto Savoia chapter Beautiful Tests and Simon Peyton Jones on concurrency are available in PDF form. Beautiful Tests covers most of the different kinds of tests and how to make changes to code to make it more testable and starts to cover creating and validating theories.
Speaking of tests, Scala, DSLs, Behavior Driven Development?, talks about how Java is poor at creating DSLs specifically compared to Scala. And what's the application? Behavior driven development in a Java project called beanSpec (which superficially looks similar to Instinct probably as they're both based on RSpec and using the same stack example). So you have this neat sort of convergence where people are looking at testing Java better in ways that are more declarative (functional even).
Making Java more functional is on the cards for Java 7 in, Will Java 7 be Beautiful?, it links to point free programming in Java and Haskell and how the new language proposals (closures) make Java 7 look a lot like Haskell with the suggestion that all Java 7 functions should be curried.
While the syntax is potentially getting more beautiful, the user interfaces are traditionally pretty poor in Java (even with OS X support) but even that is changing. I've been following Chet Haase's blog and recently Filthy Rich Clients was made available to purchase. It's all about improving the look and feel of Java and to finally approach the richness of native OS X and Windows applications. He even has some links about language proposals too, including bringing back line numbers. Some examples of their work are on the web site and on Roman Guy's blog which includes an entry called Beautiful Swing.
Roman also linked to a movie of *7 the prototype handheld device running Green (Java). Who knew Duke had a house?
Speaking of tests, Scala, DSLs, Behavior Driven Development?, talks about how Java is poor at creating DSLs specifically compared to Scala. And what's the application? Behavior driven development in a Java project called beanSpec (which superficially looks similar to Instinct probably as they're both based on RSpec and using the same stack example). So you have this neat sort of convergence where people are looking at testing Java better in ways that are more declarative (functional even).
Making Java more functional is on the cards for Java 7 in, Will Java 7 be Beautiful?, it links to point free programming in Java and Haskell and how the new language proposals (closures) make Java 7 look a lot like Haskell with the suggestion that all Java 7 functions should be curried.
While the syntax is potentially getting more beautiful, the user interfaces are traditionally pretty poor in Java (even with OS X support) but even that is changing. I've been following Chet Haase's blog and recently Filthy Rich Clients was made available to purchase. It's all about improving the look and feel of Java and to finally approach the richness of native OS X and Windows applications. He even has some links about language proposals too, including bringing back line numbers. Some examples of their work are on the web site and on Roman Guy's blog which includes an entry called Beautiful Swing.
Roman also linked to a movie of *7 the prototype handheld device running Green (Java). Who knew Duke had a house?
Thursday, September 13, 2007
A Real LINQ Clone for Java
Introducing Quaere - Language integrated queryies for Java
Via, Quaere: LINQ for Java.
The Quaere DSL is very flexible and it lets you perform a wide range of queries against any data structure that is an array, or implements the java.lang.Iterable or the org.quaere.Queryable interface. Below is an overview of the querying and other features available through the DSL interface, the underlying query expression model and query engine. See the examples section to gain an understanding of how these features are used.
* Ability to perform queries against arrays or data structure implementing the Iterable interface.
* An internal DSL (based on static imports and fluent interfaces) that lets you integrate the query language with regular Java code. No preprocessing or code generation steps are required to use the DSL, simply add a reference to the quaere.jar file (and its dependencies).
* A large number of querying operators including restriction, selection, projection, set, partitioning, grouping, ordering, quantification, aggregation and conversion operators.
* Support for lambda expressions.
* The ability to dynamically define and instantiate anonymous classes.
* Many new “keywords” for Java 1.5 and later.
Via, Quaere: LINQ for Java.
Friday, September 07, 2007
Column Databases Reach Slashdot
In a typically incorrect charactization of the issue Slashdot is covering column databases: Relational database pioneer says technology is obsolete via Are Relational Databases Obsolete?. A better way to explain it, is that column databases are designed to get around the current problems with modern computer architectures such as latency in memory and hard disks as well as achieving better CPU utilization (typically databases have very little parallelism).
While it's early days yet, column databases storing something like RDF may get faster than row-oriented SQL databases. I posted about column databases storing RDF previously.
And from the cited blog:
Seeing as though I've just been looking into this here are a few references:
Update 2: While I'm normally pretty skeptical about most things, column databases have had their own fair share of problems in the past.
While it's early days yet, column databases storing something like RDF may get faster than row-oriented SQL databases. I posted about column databases storing RDF previously.
"Vertica beats all row stores on the planet -- typically by a factor of 50," he wrote. "The only engines that come closer are other column stores, which Vertica typically beats by around a factor of 10."
And from the cited blog:
In addition, it provides built-in features appropriate to the needs of 2007 customers. These include:
o Linear scalability over a shared-nothing hardware grid
o Automatic high availability
o Automatic use of materialized views
o "No knobs" -- minimum DBA requirements
Seeing as though I've just been looking into this here are a few references:
- The End of an Architectural Era (It’s Time for a Complete Rewrite) One of the links from my previous post written by Michael Stonebraker and others. It provides a deeper explanation of column databases than in the above articles.
- Decomposition Storage Model the paper that started it all (from 1985). Section 2.5 is especially relevant as it mentions how to store directed graphs and parent-child relationships.
- Database Architecture Optimized for the New Bottleneck: Memory Access
- Monet: A Next-generation DBMS Kernel for Query-intensive Applications
Update 2: While I'm normally pretty skeptical about most things, column databases have had their own fair share of problems in the past.
Thursday, September 06, 2007
Perfect Storm (of Erlang)
It would be remiss of me not to keep track one of the other contenders for scalability. The first hint was a few days ago with, "CouchDB: Thinking beyond the RDBMS":
CouchDB and for even cooler kids, there's a Ruby API too.
The other bit that adds to this is the RDF JSON specification or RDFON. The first specification doesn't support nesting which means only named blank nodes are supported - which I think is a bug - naming things, especially things that don't naturally have names, can be a pain. The second was criticized as being a diversion away from N3.
I'm for a way to make it easier for programmers and web browsers to have better access to RDF data. Whether there needs to be another format to describe RDF is a bit unclear though - although being able to store it in a scalable way, perform queries and update it through REST seems like a positive. JSON is in that fuzzy area between code and data with its own drawbacks (like security).
Just to keep the pro scale-out/Hadoop/MapReduce etc. architecture going here's a recent posting on the more general issue:
Update: Dare corrects some mistakes made in the above posting about CouchDB. The way I read the original was it wasn't saying it was replacing relational databases just offering a better solution for certain kinds of problems. Where, "thinking beyond" is not being constrained by the expected features of a database.
CouchDB on first look seems like the future of database without the weight that is SQL and write consistency.
It stores document in a flat space.
There are no schemas. But you do store (and retrieve) JSON objects. Cool kids rejoice.
And all this happens using Real REST (you know, the one with PUT, DELETE and no envelopes to hide stuff), so it doesn’t matter that CouchDB is implemented in Erlang. (In fact, Erlang is a feature)
CouchDB and for even cooler kids, there's a Ruby API too.
The other bit that adds to this is the RDF JSON specification or RDFON. The first specification doesn't support nesting which means only named blank nodes are supported - which I think is a bug - naming things, especially things that don't naturally have names, can be a pain. The second was criticized as being a diversion away from N3.
I'm for a way to make it easier for programmers and web browsers to have better access to RDF data. Whether there needs to be another format to describe RDF is a bit unclear though - although being able to store it in a scalable way, perform queries and update it through REST seems like a positive. JSON is in that fuzzy area between code and data with its own drawbacks (like security).
Just to keep the pro scale-out/Hadoop/MapReduce etc. architecture going here's a recent posting on the more general issue:
Functional programming paradigms, the map/reduce pattern, and to a lesser extent distributed and parallel processing in general are subjects not widely understood by most quasi-technical management. Further, the notion of commodity machines with guarenteed lack of reliability as a means of achieving high performance and high scalability is essentially counterintuitive. Even referring newcomers to what I regard as the seminal papers on these topics (the papers written by Ghemewat, Dean et al at Google, and yes I know it all started with LISP but my management wasn't even alive in the 1970s although I was :-)), people steeped in a long tradition of "shared everything" database architectures still don't quite get it. I spend considerable amounts of time in what amounts to management de-programming: No MySQL can't do this, and Oracle can't either, except with Oracle it will cost you a lot more to find that out.
Update: Dare corrects some mistakes made in the above posting about CouchDB. The way I read the original was it wasn't saying it was replacing relational databases just offering a better solution for certain kinds of problems. Where, "thinking beyond" is not being constrained by the expected features of a database.
Wednesday, September 05, 2007
Subscribe to:
Posts (Atom)