Removing Large Files from Git

When I've used git I've used it pretty much CVS, SVN and any other version control system I've used before - I've checked in binary files.  Whether that's dlls or jars or gems I've checked them in.  Pretty much everywhere I've worked people have said this is a problem and I've tended to argue it solves a lot of problems - one of the main ones is when where the repositories and management software along with that fails - I still have a working system from a checkout/clone/etc.

The price of this is that sometimes you need to cleanup old binary files.  Git makes this complicated but once you've found a couple of tools then it's relatively straightforward.

Stackoverflow has a Perl and Ruby script that wraps around a few git commands to list all files in a repository that's above a certain file size "Find files in git repo over x megabytes, that don't exist in HEAD".  The main gist of it is (in Ruby):

IO.popen("git rev-list #{head}", 'r') do |rev_list|

  rev_list.each_line do |commit|

    commit.chomp!
    for object in `git ls-tree -zrl #{commit}`.split("\0")

      bits, type, sha, size, path = object.split(/\s+/, 5)
      size = size.to_i

      big_files[sha] = [path, size, commit] if size >= treshold
    end
  end
end

big_files.each do |sha, (path, size, commit)|
  where = `git show -s #{commit} --format='%h: %cr'`.chomp
  puts "%4.1fM\t%s\t(%s)" % [size.to_f / Megabyte, path, where]
end

Then to remove the old files from the repository:

git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch [full file path]' -- --all
git push --force

Then to cleanup any used space in the git repository:

rm -rf .git/refs/original/
rm -rf .git/logs/
git reflog expire --expire=now --all
git gc --aggressive --prune=now

Global Code Retreat 2012

I went to the local Global Code Retreat held on the 3rd of December.  Overall, it was an amazing event - very well hosted and attended.  The basic structure of the day was 6 or so 45 minute sessions trying to implement something with a different person each time.  At the end of the 45 minutes, no matter how far you had got, you deleted your solution.

The problem was "The Game of Life".  I'm pretty familiar with this problem having come across "Conway's Game of Life" early on in a magazine like Compute! or Byte.

However, if you walked away with a really awesome solution to "The Game of Life" you probably missed the point - most of the things that were being taught were hidden.

The solution was really beside the point.  One of the main reasons is to repeat solving the problem from scratch based on an idea called kata (movements practiced by yourself or in pairs).  This is something that I had come across in "The Pragmatic Programmer" which at the time reminded me of the time I had spent with projects at home - reimplementing the same thing over and over again.

Steve Yegge mentions the same thing in his article "Practicing Programming".  He mentions that even as you program in your day job you may not actually be practicing programming.  Repetition in solving the same problems seems to be about keeping the problem fixed and then changing how you approach it and freeing you from any time constraints.  Most programming jobs involve solving the solution once (or if you're lucky doing a proof of concept and then implementing it again).

The first time around it was awful.  I didn't know what I was doing, my environment was a little bit shaky, we couldn't agree on a language and I spent a lot of the time just setting it up.

It made me become aware that for the first time, practically ever, my personal computer had diverged from my work computer.  Not in the "normal" Windows at work, Linux and OS X at home - but what I do at home and at work have diverged to the point where I'm learning stuff in many directions and there's almost no overlap between the two.

The second time was much better.  There was less discussion on languages to use, how to approach the problem, how do you test drive it, whose computer to use and so on.  There was still discussion but we both shared a bit more context this time which made the discussion flow.  A big difference to the first time.

The third time around changed the format a little to where you couldn't talk to the person but you could only express requirements through tests.  So this sorted out the people who were testing from those who weren't.  But it also seemed to reduce the clutter around what needed to be done.  Tests are much less ambiguous compared to talking through requirements and so once you setup a rhythm of tests it became much easier.  Also, the whole room was very quiet.  You could imagine that a team doing silent TDD and pair programming wouldn't be the noisiest group in the room (for once).

Each round thereafter changed the programming requirements: no loops, methods no more than 3 lines, and no if statements.

What did I learn?  Heaps.

I ended up doing Ruby quite a bit and mostly the solution came out at about 30 lines of production code and 30 lines of tests and you could pretty much do it in the time allocated.  I also did solutions C# and Haskell.  The Haskell solution came out at about 30 lines total - both tests and production code - and met every constraint (no loops, small functions, no if statements).

Doing the same problem over and over again is surprisingly effective and nothing replaces sitting with a person to learn a new language or to be exposed to a variety of solutions.  One of the tricks - and you find this frequently with pair programming - you have to be very good at communication - both by saying what you're doing but also getting the other person to explain themselves.

I also learnt:

• Even with something as well defined and familiar as "The Game of Life", the solutions were varied and some of the requirements (based on the rules on Wikipedia) were redundant.
• My brain is very weak compared to how well Google search works.
• Between each new attempt you tend to reflect on each previous solution and see the negatives and positives.
• By continually starting a new project setup time was greatly reduced - dependencies that get in the way were slowly reduced - editors, libraries, searching the web, etc.
• That it's good to throw code away.  It frees you up by allowing you to try different approaches or learn something new (like a different language).
• Think before you hack.
• A functional approach seemed to be where the answers were converging - meeting all the programming constraints that were given.

Real Developers don't use Ruby

Hadoop: When grownups do open source. It's quite an amusing read - especially the part about the word count example on 9,000 blogs, the digg at Twitter, Starfish being practically useless (using MySQL and no Reduce phase) and the bit about understanding something being harder than writing a Ruby version of it.

"Twitter decided they would be cute and trendy. They wrote their code in Ruby: the official state language of the hipster-developer nation. Doug Cutting, on the other hand, decided he would get xxxx done, and wrote Hadoop in Java. Starling was hidden away in some corner and forgotten (it's hosted at RubyForge...). Hadoop lives prominently at the Apache Software Foundation. Starling is a re-hash of an existing Java Enterprise API called JMS that has several open source implementations. Hadoop is an implementation of Google's MapReduce, a system that publicly only existed on paper. Hadoop has the added benefit of actually working."

Ahh the joys of installing Visual Studio - enough time to install IntelliJ, run it up, and catch up on news.

No Java 6 for You!

Just in case you're like me and you upgraded to Leopard only to find Java 6 no longer works and Java 5 unstable, here's a fix:

-- First, delete ~/Library/Java/Caches/deployment.properties
-- Move aside your Java 1.6 directory. The 1.6 preview on Tiger does not work on Leopard.

% cd /System/Library/Frameworks/JavaVM.framework/Versions
% sudo mv 1.6.0 Tiger_1.6
% sudo rm 1.6

At least it comes with better Ruby support and a RubyCocoa bridge.

Update: I don't think I like Leopard. I don't like: the 3D or the 2D look of the dock, the semi-transparent menu bar, Spaces behaviour - you can't have multiple windows spanning desktops from the same application (like two browser windows in separate desktops), the removal of text lists (why not have Fan, Grid and List?), in Quick Look you can go to a page in a PDF or Word file but when you click on it it goes to the first page, and Java support (IntelliJ and others seems to have weird refresh issues and you can't seem to allocate it to a virtual desktop).

There's of course a lot to like though too (tab terminals, RSS reader, better Spotlight, Cover flow, Safari, etc).

Update 2: So the Spaces thing. If you minimise a window, it goes to the dock, change to another virtual desktop, then expand, it goes to the desktop that the window was originally in. As noted in the comments though (and it is in the guided tour), if you activate Spaces (the default is F8), then move the window to a new virtual desktop then it is tied to it. There are two other ways: click and hold on the window and switch to another desktop or click and hold on the window, move it to the edge of the screen, wait, and it will go to the next desktop.

To me, some of the behavior breaks the illusion/metaphor of the virtual desktop and seems unnecessarily difficult. I'd prefer non-click and hold options supported as well (pretty much like other features like copying files, etc).

The networking improvements (non-blocking, built in VNC) overcomes what was probably one of the worst things about OS X.

The IntelliJ issue that I mentioned is logged as issue 16084.

Migration

With the announcement of JRuby in Glassfish or the end of mongrel it seems to offer Sun the hope of capturing more developers not just from Ruby but from .NET. Many people from both Java and C# worlds jumped to Ruby and it occurred to me that .NET developers using Netbeans for Ruby development may notice that there's a suprisingly good C# clone under the covers.

DERI at Google

The Semantic Web at Google (I found the movie here).

Starting from the end, Stefan was asked about inferencing and relationships. He basically responded that linked data is more practical and immediately useful and that the affect of Description Logics has been over estimated.

The highlight for me was the demo on how to construct user interfaces automatically (from about 20 minutes in). The algorithms are described in more detail in, Extending faceted navigation for RDF data.

They also talked a little about how Ruby was a good language for Semantic Web applications and referenced, ActiveRDF: Object-Oriented Semantic Web Programming.

The applications and tools demoed: SIOC Project, Active RDF, JeromeDL (digital library), BrowseRDF (automatically generated, facted UI) and S3B (social semantic search and browsing).

On the Web No One Can Agree on the Schema

• "Why the Semantic Web will Fail" responses include "Business Doesn’t Cooperate? Are You Kidding?" and "Why the Semantic Web Marketing Message Has Failed". Which businesses or organisations cooperate internally, let alone externally? Couldn't this have been the same argument as to why TCP/IP, HTML, RSS, XML, relational databases, or any technology would've failed?
• Ajax Anti-Patterns and more includes: reduce polling, not inspecting callbacks, using XML, and reducing work on the server.
• "What's wrong with Ruby?" Suggests that Ruby won't make programming easier and doesn't include a substantially better way of handling abstractions, has non-intuitive operator overloading, and that dynamic modification of code is evil. The real answer is freedom from coding for CPUs.
• "Microcultures and the WII" "When I visit a friend, we listen to a radio station only airing electro remixes of C64 songs, which you can receive on the net. See, its the dude from Kaiserslautern who enjoys electro remixes of C64 songs who is the market of tomorrow." Maybe microcultures are bigger than you think. For all those that dig things like a capella versions of IK+ (the video is interesting).

Ruby Interfaces and Design By Contract

Interfaces in Ruby (attempted summary)

Technically speaking Ruby does not need interfaces since it is dynamically typed. Any argument list that at least satisfies the arity condition can be handed into a method.

Since people appreciate the usage of interfaces and even more Design by Contract (DbC) some mechanics were developed that mimic interfaces and certain aspects of DbC. I list them in order of increasing complexity.

(1) Document which methods a method argument must implement or its expected type.

(2) Define a module containing the methods that all raise an exception like this:

module FooInterface
def bar(a,b) raise "bar(a,b) must be overridden"; end
end

class FooClass
include FooInterface

def bar(a,b) a+b; end
end

(3) Create a framework (like the one Paul Brannan suggested here [1]) for defining interface methods and for checking that a particular instance implements these methods.

(4) Try to support even more of DbC by dealing with preconditions and postconditions like I tried in [2].

Recently announced 0.1 version of Ruby Design by Contract and Java style interfaces for ruby.

Tupelo, Tupelo

Tupelo is a project to create a content repository to support data-driven scientific applications. More specifically it support JSR 170 (Content Repository API), WebDAV and URIQA (which are the extended HTTP verbs for getting RDF graphs). It implements a per object ACL and version control system.

It uses Kowari, 3Store, and Jena as backing stores. It also has a rather interesting API for dealing with triples called Context (more information in the cookbook). There are some smaller things too like using 1.5's varargs to add triples, transitive closures on queries, object to resource mapping and something that I'd meant to do for JRDF a while ago, have a ResourceVisitor that actually visits resources.

The presentation lists similar projects: SRB, Slide/SAM, Fedora, DSpace and Jackrabbit. All of which (except for SRB) I'm pretty familar with.

And ActiveRDF, a Ruby API for accessing RDF, went 1.0!

Win Some, Lose Some

In, Rocking with Ruby, I made the outlandish claims that: "Continuations are going to be a key piece of infrastructure.". So maybe not, with the recent news that Seaside is going away from it: "One thing I’d like to do is reduce the dependence of Seaside on continuations - they drove a lot of the initial interest in the framework but they’re becoming (or seeming) much less important over time, and the use cases to which they’re best suited are these days often addressed with AJAX instead."

The other claim that seems to have worked out though was, "That future languages and platforms will probably be deployed on .NET and Java VMs. The competition between the two seems to have a positive impact on both - locking out any competitors. That means, there's something to look forward to in Java 7 and .NET 3." This follows the news about JRuby: "JRuby has been getting more and more attention from folks within Sun, Rubyists around the world, and especially from Java developers anxious to escape from their Java-only prisons."

Iron VMs

Will It Be JRuby vs. IronPython? "On the heels of Microsoft's release of IronPython 1.0 comes the news that Sun has hired the two primary JRuby developers.

I don't know if Sun intended it, but the juxtaposition of these two news item gives the effect that the two managed execution environments—The JVM and .NET—have each chosen an anointed champion scripting language for their platform.

The orphan that's left out of the spotlight is Jython. I feel kind of sad for Jython and its lone publicly known active maintainer Frank Wierzbicki."

JRuby Steps Into the Sun "The potential for Ruby on the JVM has not escaped notice at Sun, and so we'll be focusing on making JRuby as complete, performant, and solid as possible. We'll then proceed on to help build out broader tool support for Ruby, answering calls by many in the industry for a "better" or "smarter" Ruby development experience."

Java 7 with Closures

Closures for Java "I'm co-author of a draft proposal for adding closures and local functions to the Java programming language for the Dolphin (JDK 7) release. An abbreviated version of its current state is reproduced below. It was carefully designed to interoperate with the current idiom of one-method interfaces."

Based on the example, "int(int) plus2b = (int x) : x+2;" you know it must be Java when you have the type in there three times. Something like Groovy's closures or Ruby's seems better syntactically. There's also some other ideas to add such as currying and iterators.

JRuby now does Rails**

"** We are able to generate and run the cookbook demo from rolling with rails tutorial (http://www.onlamp.com/pub/a/onlamp/2005/01/20/rails.html) with what we have and all appears to work. With this said, it is likely that there are several aspects of rails that are not working correctly. See docs/README.rails for known issues/instructions in release."

JRuby "WEBrick runs...Ruby on Rails runs on top of WEBrick (and generation scripts work)**"

Time for a free lunch:

"I believe I've found a 'free lunch' in Ruby on rails. Oh, it's not always free. If I need to do two-phased commit or hardcore object relational mapping, this lunch may cost me more than I'm willing to pay. But often enough, it's for all practical purposes free.

* I can train a team of Rails developers faster than I can teach a new Java developer Spring plus Hibernate plus whatever web mvc you want plus all of the other frameworks and tools Java developers have to know.
* I can build my applications much faster than I could before.
* For many applications, the latency in the database is the overriding concern, so I don't even notice differences in performance.
* I can trivially expose web services, letting other applications, potentially written in other languages, quickly access my Rails services.

Now, I know that some will tell me that the lunch really isn't free. But you can tell that to my customers that pay a fraction of the price they'd pay for a Java application, and get something that's easier to maintain, just as fast, and on an earlier schedule. From that exec's perspective, the lunch is free."

Logo in Ruby

If It's Not Nailed Down, Steal It "Pattern Matching, S-Expressions, and Domain Specific Languages in Ruby...Pattern matching is a relatively rare language feature found in Standard ML, OCaml0, Haskell, Common Lisp (CLOS), and a handful of others. It’s a form of multiple dispatch, so pattern matching functions run different code when called with different arguments. Pattern Matching lets you do this dispatch based on type, value, and even internal structure of the function’s arguments...Ruby gives us everything we need to represent S-expressions. Ruby list literals are written in enclosing brackets ([...]) and use commas as separators. Symbols are written as barewords proceeded by a colon (:)...Okay, enough talk, let’s implement a simple version of the Logo programming language in Ruby. Remember Logo3? That language with the silly turtle that draws? "

Simpsons Ruby Mixin

Homerpalooza "I used to be with it, but then they changed what "it" was. Now, what I'm with isn't it, and what's "it" seems weird and scary to me."

Why Aren't My Assertions Counted? "The problem typically comes when you put assertions in your library modules. This isn't hard. All you have to do is put include Test::Unit::Assertions in your module or helper class and then you can include calls to assert, assert_equal and all the other assert methods. It works, except that your assertions aren't counted.

The reason why is that the method add_assertion does the counting. This is automatically called by all the assert methods. When they call the add_assertion method of TestCase, the count shows up in your test results no problem. But the add_assertion method of the Assertions module is just a no op. This is why they don't affect the assertion count. The solution is to add a functional add_assertion method back to your helper module/class that ties back to your test case."

When the language has the []= operator and it can be overloaded is somewhat scary. But when you fix what seems like a bug by changing the behaviour of the underlying class it seems crazy. I'd like to be able to say with certainty that the lack of interfaces is a mistake (or not) just like people seem to consider checked exceptions. Or that the ability to abitrarily add behaviour to existing objects is going to destroy open source and make code unmaintainable.

Why do you need new languages? Everyone knows programming languages attained perfection in 1995. It's a scientific fact.

The New OO

So now that Ruby is officially the next new thing it's time for the next level of books. Bruce Tate's From Java to Ruby or From Java To Ruby "Java to Ruby is packed with interviews of Ruby customers and developers, so you can see what types of projects are likely to succeed, and which ones are likely to fail. Ruby and Rails may be the answer, but first, you need to be sure you're asking the right question. By addressing risk and fitness of purpose, Java to Ruby makes sure you're asking the right questions first."

RDF on Rails

While searching for Ruby APIs for RDF I came across: ActiveRDF: object-oriented RDF in Ruby "Although most developers are object-oriented, programming RDF is triple-oriented. Bridging this gap, by developing a truly object-oriented API that uses domain terminology, is not straightforward, because of the dynamic and semi-structured nature of RDF and the open-world semantics of RDF Schema.

We present ActiveRDF, our object-oriented library for accessing RDF data. ActiveRDF is completely dynamic, offers full manipulation and querying of RDF data, does not rely on a schema and can be used against different data-stores. In addition, the integration with the popular Rails framework enables very easy development of Semantic Web applications."

"The development of such APIs has been attempted using a statically-typed language (Java) in RdfReactor, Elmo and Jastor. These approaches ignore the flexible and semi-structured nature of RDF data and instead:
1. assume the existence of a schema, because they rely on the RDF Schema to generate corresponding classes,
2. assume the stability of the schema, because they require manual regeneration and recompilation if the schema changes and
3. assume the conformance of RDF data to such a schema, because they do not allow objects with different structure than their class definition.

Unfortunately, these three assumptions are generally wrong, and severely restrict the usage of RDF. A dynamic scripting language on the other hand is very well suited for exposing RDF data and allows us to address the above issues."

The homepage: ActiveRDF. Uses YARS and Redland.

Recent related W3C note: "A Semantic Web Primer for Object-Oriented Software Developers".

Related previous posting: "Scripting the Semantic Web".

Update: It looks like Henry Story noticed ActiveRDF as a good idea too.

Update 2: Brian Gilman recent posting to the Life Science list links to the BioRuby project.

Things I've Read

* Evangelical Scientists Refute Gravity With New 'Intelligent Falling' Theory related to Boing Boing's $250,000 Intelligent Design challenge (UPDATED: $1 million) and Flying Spaghetti Monster.
* JUnit-Addons and GSBase so at least only two software projects have to reinvent the wheel.
* Non-blocking Data Sharing in Multiprocessor Real-Time Systems "In this paper, we present an efficient non-blocking solution to the general readers/writers inter-task communication problem our solution allows any arbitrary number of readers and writers to perform their respective operations."
* Why I Prefer SOA to REST "The problem is that although it is easy to model resources as services as shown in the example in many cases it is quite difficult to model a service as a resource. For example, a service that validates a credit card number can be modeled as a validateCreditCardNumber(string cardNumber) service. On the other hand it is unintuitive how one would model the service as a resource. For this reason I prefer to think about distributed applications in terms of services as opposed to resources." Now people can just argue about what is intuitive (credit card gateways on the Web existed before SOAP - must have been REST I guess).
* I Need a New Language: Rel? "I want a language for table programming. I think you can write programs in this language that do everything we expect of an application programming language -- building GUIs, reacting to mouse events, listening to sockets -- everything. Don't model your domain as objects. Model it as relations...I can imagine programs that have a relvar (Date's term for a relational variable: essentially a table or a view) for MouseState."
* Package Scoping And Unit Testing "Package scoping particularly shines during unit testing. Some programmers argue that you should only test through the public API. Don't be silly. Limiting your tests to the public API contradicts the spirit of unit testing and subjects you to unnecessary dependency pain. I prefer to isolate and limit the amount of code I test at one time, and test as close to the code as possible." Somewhat related, JSR 277 - Java Module System
* Web as Platform Mash-Ups "There have been a lot of excellent posts and articles this week about APIs, the Web as Platform, web sites as software companies, and so forth..."
* Ruby, Python, "Power" "There are different opinions on the relative power of Ruby and Python. I'm not much more authoritative than other resources (though I'm not less authoritative either; most comparisons between the two languages are flawed). Ultimately I don't believe there are many (any?) places where one language is more "powerful" than the other (and not just in the "they are both Turing complete" sense)"