Quality without a Name

Wikipedia on fail fast including a PDF highlighting it as a way to write software.

Searching for this kind of work comes across some nice hybrid articles: Fail fast (for XP), James Shore on the Quality without a Name, also came across Patterns for Scalability, Availability and Stability (including Java 7's ParallelArray).

Thesis and Antithesis

David Anderson on Agile. He introduces his talk by discussing some very broad ideas, including how agility has been applied to other areas. He talks about how the agile manifesto seems to have become more about belief and superstition than a scientific model. It's also rarely clicked through - to see the principles behind the manifesto.

The agile community has found it is better to develop in a failure tolerant environment. This was a reaction to the focus on more and more accurate estimates which lead to analysis patterns (or antipatterns). They are antipatterns because there was never enough detail in the analysis to provide accurate enough estimates and to deal with the unexpected.

Through his talk he listed a bunch of thought bubbles on software development:
* Value is providing functionality fastest.
* Knowledge work is perishable.
* Perfect is the enemy of good enough.
* Develop a high trust, high social capital.
* Have a highly collaborative culture.
* Reflect and adjust.
* Sustainable pace.
* Craftmanship.
* Value is contextual, context is temporal.
* Waste over scale.

Traditional industries have relied on bigger batch sizes to provide better economies of scale. With software development the transaction costs causes large batch sizes to work against you. Smaller batches allow you release more business value more quickly and reduce waste (waste over scale).

He introduces kanban which has ideas such as: pull, flow, and regulate work in progress. It does not use time boxed iterations and little or no planning or estimations (at least from a traditional agile approach). It still has constant improvement and delivery. It drops the idea of a generalist approach to labor which is seen by some as waterfall in disguise. The reason this comes about is because at an enterprise scale it's not feasible to hire a lot of experts who are excellent generalists - the labor pool does not exist. This leads to a tension with typical lean principals of reducing waste by having generalists.

He sees two main changes coming in the future: software factories (software product lines - which includes DSLs) and cloud services (deploying web services in the cloud). Architecture and modeling will come back in fashion as a value chain allows incentives for delivering common behavior. He's talking about a 100 fold improvement in productivity through using these ideas.

Finally, he talks about how CMM/CMMI is actually quite a good indication of the ability for software projects to succeed and the industry as a whole. He mentions a report on how agile and CMMI can work together, the idea being that not only is organizational maturity a good idea but it actually means that agile methods can be implemented better (it's called "CMMI or Agile: Why Not Embrace Both?!" - I only found a few references, for example Agile+CMMI Panel @ SEPG).

Via.

Still have Richard P. Gabriel's talk to go through too.

Beautiful Engineering

One Bridge Doesn’t Fit All

American bridge engineering largely overlooks that efficiency, economy and elegance can be mutually reinforcing ideals. This is largely because engineers are not taught outstanding examples that express these ideals.

A 2000 report by the Federal Highway Administration indicated that an average of about 2,500 new bridges are completed each year; each could be an opportunity for better design. The best will be elegant and safe while being economical to build.

The key is to require that every bridge have one engineer who makes the conceptual design, understands construction and has a strong aesthetic motivation and personal attachment to the work. This will require not only a new ethos in professional practice, but also a new focus in the way engineers are educated, one modeled on the approach of those Swiss professors, Wilhelm Ritter and Pierre Lardy.

Via Bridges and code.

Everything You Know is Wrong

• OWL can be, as the French would say, databasesque. Using OWL to implement database like integrity constraints.


• It's time to throw away your database (a paper describing the advantages of column databases) and replace it with something more flexible and scalable (using C-Store to store RDF). And column databases mean no NULLs. The problem isn't that your data is normalized it's that it's not normalized enough (also mentioned here).


• Following a question that I had earlier this year about where is the good code to learn from, I've read about 3/4 of Beautiful Code. And while I have a hard time finding C/C++ code beautiful it certainly covers some good examples of code and the process behind writing it (including Beautiful Tests, Beautiful Concurrency and MapReduce).


• Latency Moore's curse.


• Nova links to why what you thought about genes is wrong (I've actually heard this before but hadn't come across any articles).


• Scaling laws in Biology. Humans run on 100 watts or used to. Now we have cars etc. our power requirements are now 11,000 watts (making us the biggest creatures that have ever lived). And the magical number: 4.


• The Enemies of Reason is now on Google Video.

Patterns In Software - Part 4

Patterns of Software: Tales from the Software Community (freely available in PDF). The chapter I'll be covering this time is: "The Failure of Pattern Languages".

So it begins with the punch that the last few chapters have been building up to:

Alexander’s story does not end with the publication of A Pattern Language in 1977. It went on and still goes on. And Alexander did not sit still after he wrote the patterns. Like any scientist he tried them out.

And they did not work.

The reason, quoted by Gabriel is that other processes other than architecture play a more fundamental role. That a building is not just the process of slapping together a bunch of very well thought out patterns based on emperical evidence. It's the result of many other processes like finance, zoning, construction, etc.

...one problem with the building process is lump-sum development. In such development few resources are brought to bear on the problems of repair and piecemeal growth. Instead, a large sum of money is dedicated to building a large artifact, and that artifact is allowed to deteriorate somewhat, and anything that is found lacking in the design or construction is ignored or minimally addressed until it is feasible to abandon the building and construct a replacement.

Gabriel gives an example from a previous chapter about how these processes can destroy the otherwise good qualities of development. For example, the process of getting a mortgage and paying it off directly influences the types of buildings built and used. Generally, you invest a large amount of money in a property, so large, that you usually can't afford to make piecemeal modifications because you can barely afford paying it off. This is okay as long as people are jumping from house to house. This avoids fixing up the problems there maybe with these houses until things get really bad and they are knocked down and rebuilt. Perhaps losing what was wrong with them in the first place.

So if you retain these old processes you still get the old results. Now it's clear in software that this is also the case - a nicely architected and controlled software project does not necessarily lead to high quality software. How often have you worked on a software project that you knew was going to be re-written in a few years anyway?

Gabriel suggest that the answer lies in good code and coders not in the typical separation of analysis, design and implementation:

And isn’t the old-style software methodology to put design in the hands of analysts and designers and to put coding in the hands of lowly coders, sometimes offshore coders who can be paid the lowest wages to do the least important work?

Methodologists who insist on separating analysis and design from coding are missing the essential feature of design: The design is in the code, not in a document or in a diagram. Half a programmer’s time is spent exploring the code, not in typing it in or changing it. When you look at the code you see its design, and that’s most of what you’re looking at, and it’s mostly while coding that you’re designing.

And finally, that the typical software patterns are not following Christopher Alexander's original concept of a pattern language:

When I look at software patterns and pattern languages, I don’t see the quality without a name in them, either. Recall that Alexander said that both the patterns themselves and the pattern language have the quality. In many cases today the pattern languages are written quickly, sort of like students doing homework problems. I heard one neophyte pattern writer say that when writing patterns he just writes what he knows and can write four or five patterns at a sitting.

The answer is to build software piecemeal, incrementally, partially designed and reflecting - much more like a Turkish rug and that's the next chapter, "The Bead Game, Rugs, and Beauty".

Patterns In Software - Part 3

Patterns of Software: Tales from the Software Community (freely available in PDF). I was going to talk about, "The Failure of Pattern Languages" but I thought that a little discussion on the preceding chapter, "Pattern Languages", is required. This is somewhat of a generalisation of the preceding chapters and a discussion on Alexandar's "A Pattern Language", which offered the template for the software pattern books.

A comparison is made between art and science - art is creation, science is description - and that there are few activities that combine both. The exceptions given are computer science and architecture (I'd suggest maybe biological science is another). Trying to avoid the separation of construction and design is a mistake in computer science. Much or almost all of computer science is design and that it is expensive because it does combine both aspects for example, "What is Software Design?".

So this is specifically what all the discussion was a few years ago about TDD - being Test Driven Design (not Development). And the focus on behaviour rather than testing, for example Tom's recent Instinct project.

I want to set the stage for the accepted view of how architecture is done, at least in the pre-Alexander world. Architects are hired to solve the problem of how to construct a building or buildings that meet certain constraints as specified by the future inhabitants or people in a position to specify what those inhabitants will need. The architect generally interviews those future inhabitants, studies the site, refers to the local building code, observes neighboring buildings, considers the building materials and construction protocols in the area, and then is inspired, in the manner of all artists, to create a set of drawings which a general contractor, in occasional conference with the architect, reduces to a physical object.

Following from the suggestion given in the book, that the inhabitants are the programmers, it's obvious that "software architects" should ask the inhabitants of the software the best way to solve their problems. But that's not enough, more often than not they dictate solutions without knowledge of any of the complexity that their "solutions" might bring. This aligns with Alexander's suggestion for architects.

The first of Alexander’s contributions to architecture was to reject the separate architect and builder model and to posit user-centered design—in which users (inhabitants) design their own buildings—and the architect-builder who would blend the activities of design and construction. This was viewed by architects as high treason enough, let alone the curious quest for an elusive quality that Alexander cannot name.

The mechanism he proposed to accomplish his new model was the pattern language. A pattern language is a set of patterns used by a process to generate artifacts. These artifacts can be considered complexes of patterns. Each pattern is a kind of rule that states a problem to be solved and a solution to that problem.

I recently started to read a copy of "A Pattern Language" and I was struck by the same thing that Gabriel was, especially in comparison with the GoF book:

One thing that strikes many readers of the pattern language is the degree to which Alexander talks about people and their activities in the patterns — the patterns are a response to his arguments about how life is best lived. Because of this, many regard "A Pattern Language" as a manual of how to live and how physical surroundings can support and enhance living.

The patterns discussed in Alexander's book are all about what makes an area good for people - side walks (foot paths), water features, stairs, the height of crossings, making a city accessible to bicycle and foot traffic, etc - all these things make the environment livable and most ideas are supported by empirical evidence. The world of software patterns, by comparison, seems much more barren and arbitrary.

So there are some obviously good things about patterns:

Patterns seem to be a partial solution to the over abstraction problem I talked about in “Abstraction Descant.” They are a way to take advantage of common patterns without building costly, confusing, and unnecessary abstractions when the goal is merely to write something understandable. That is, when there are more idioms to use, using them is far better than inventing a new vocabulary.

Patterns in software are a good idea, but I have rarely seen the attention to the human side—to living within software—that I see in Alexander’s patterns. One architect commented on Alexander that he regarded the patterns as providing information, sort of heuristics, about architecture, and I have to admit the patterns I’ve seen about software do that—they provide information that the software practitioner just plain ought to know.

But this also highlights one of the problems with patterns. They discuss the context of the solution to a problem without first defining and more closely examining the problem itself, especially in a more general way. While it's often discussed in this chapter that patterns interact with other patterns and that they must be free of "systematic variations" the problems themselves don't seem to require this property. When people are critical of patterns, this is what I think they are objecting to, rather than the patterns themselves. He quotes Alexander saying: "You see that the patterns are very much alive and evolving."

And in a foreshadowing of what's to come:

The worst thing is this: Alexander had the chance in the 1970s to try out his pattern language. He observed the results of others trying it out and he even tried it out himself several times. And guess what: It didn’t work.

The next chapter, "The Failure of Pattern Languages", discusses the failure to apply these pattern but also a proposed solution to the problem: that the process should also have "the quality without a name".

Patterns in Software - Part 2

Patterns of Software: Tales from the Software Community (freely available in PDF). "Quality Without a Name" spends a lot of time taking Alexander's objective "meaning for beauty, for the aliveness that certain buildings, places, and human activities have" and applying it to software. This chapter also has a fairly good definition of patterns:

Patterns certainly have an appeal to people who wish to design and construct systems because they are a means to capture common sense and are a way to capture abstractions that are not easily captured otherwise.

The second thing (and second paragraph) is a re-definition, for most people in the software industry, on who the users of the software are:

...when you read Alexander, it is clear that a “user” is an inhabitant—someone who lives in the thing constructed. The thing constructed is under constant repair by its inhabitants, and end users of software do not constantly repair the software, though some might want to.

So these are not the regular definitions of quality or user. It's certainly not the TQM definition of quality: the fitness to a standard or set of requirements. It's much closer to the principle of quality in XP and other agile methodologies. Meeting requirements is rejected as the definition of quality because they are often contradictory, he quotes Alexander on his experience with Bay Area Rapid Transit (BART) system:

So it became clear that the free functioning of the system did not purely depend on meeting a set of requirements. It had to do, rather, with the system coming to terms with itself and being in balance with the forces that were generated internal to the system, not in accordance with some arbitrary set of requirements we stated...What bothered me was that the correct analysis of the ticket booth could not be based purely on one’s goals, that there were realities emerging from the center of the system itself and that whether you succeeded or not had to do with whether you created a configuration that was stable with respect to these realities.

He also takes some time using other (somewhat confusing) words such as alive, whole, comfortable, free, exact, egoless, and eternal to further refine what "the quality without a name" is. The purpose for this is clear though, it's to rediscover objective quality or the combination of fact and value.

We in software are not so lucky—all of our artifacts were conceived and constructed firmly in the system of fact separated from value. But there are programs we can look at and about which we say, “no way I’m maintaining that kluge.” And there are other programs about which we can say, “Wow, who wrote this!” So the quality without a name for software must exist.

Some of them I'd consider fairly vague but others have some parallel with XP principles and similar ideas. In order to describe how a system is alive Gabriel uses the metaphor of a fire and a fireplace. The structure of the logs, chimneys and so on is a well thought out system which supports a self-sustaining process that once set in motion reaches a predefined end - a small pile of ashes. And "whole" is a property where something is self consistent. This is what occurs when code is reflected upon and refactored. And while Gabriel doesn't spend much time talking about a system being egoless, this concept should be familiar to most agile practitioners. XP and other methodologies encourage the idea of egoless programming and collective ownership.

All of these concepts and "the quality without a name" all seem to me to be descriptions of development methodologies, practices, principles and values and the particular kind of software that is developed as a result of applying these processes. That is, software with this quality is a result of a software process with the same quality.

He ends this chapter with "some things" (I'd say requirements) for software which possesses the "quality without a name". I'd summarize it as:

• Not written under unrealistic deadlines.
• Modules and abstractions not too large and small enough to understand and remember what they do.
• Code constantly repaired (he'd probably say refactored today).
• A fractal nature of code - looking at the large or the small the code is coherent.

He ends it with this lament:

I wish we had a common body of programs with the quality, because then we could talk about them and understand. As it is, programs are secret and protected, so we rarely see any but those we write ourselves. Imagine a world in which houses were hidden from view. How would Alexander have found the quality with no name?

Since this was written there's been a lot of code made public but I still can't think of any that I've really looked at and appreciated (and in fact I tend to avoid looking at Java code written under the Apache licence for example). Any suggestions?

Patterns in Software - Part 1

Patterns of Software: Tales from the Software Community (freely available in PDF format) is something I came across last time I was talking about design patterns.

First off, I think this book is very well written - style, language, structure - it's all good. I wish I could write as well as this guy. Christopher Alexander is in the preface (the guy who is attributed to inventing patterns).

Secondly, I'm going to cover the chapters that I've liked as I get to them. The first is "Abstraction Descant".

In this chapter he introduces the concept of compression which is: "...the characteristic of a piece of text that the meaning of any part of it is “larger” than that particular piece has by itself."

Examples are macros, function or class names. There is a danger though that compression and abstraction can be overused.

"The problem is that people are taught to value abstraction above all else, and object-oriented languages and their philosophy of use emphasizes reuse (compression), which is generally good. However, sometimes the passion for abstraction is so strong that it is used inappropriately — it is forced in the same way as it is with larger, more complex, and typically ad hoc abstractions."

"Another problem with complex abstraction arises from the observation that abstractions are about ignorance...Some complex abstractions, however, contain information about the implementation that is legitimately required, such as its performance, the algorithm, coding tricks, and resource usage—keep in mind that almost all interaction issues are about resource conflicts."

The next point is that software evolves through time, in little pieces by programmers not designers. The danger is the designers use abstractions (usually in ignorance) whereas coders should write code and design it in order to make it more habitable.

"...creating habitable software that can be effectively maintained, recognizing that the reality of software development is piecemeal growth and to plan accordingly, and to understand that the power of object-oriented programming is compression, which carries a terrific requirement for careful use of inheritance—relate to how we use abstraction and how much we use it."

"In programming, if a set of large abstractions does nearly the right thing, it is tempting to use them and to bend the structure of the surrounding program to fit them. This can lead to uninhabitable programs."

"Worse: You can fight this temptation and choose not to use them. This choice also can lead to uninhabitable programs because you will be using parts similar but subtly different from possibly familiar ones. The only way to avoid this is to use small blocks rather than large ones, or to use blocks well-designed and tested by experts."

He also has a comment on teaching programming where we need to "learn from the classics".

"How much time do we spend reading in our ordinary education? And from our reading we gain a foundation for writing...But in programming we just learn the language and solve a bunch of short puzzles. Sort of like writing 50 limericks and then off to write books."

The next point took me a while to digest but I think it's probably the most useful part of the chapter. It is that we rarely spend time creating new control abstractions (like loops) and most of the time creating abstractions of data. He argues that the two should go hand in hand.

"Let’s look at another problem with abstractions: Data and control abstractions are generally best when they are codesigned and this is rarely done anymore. Consider, for example, the Fortran abstractions of arrays and iteration. Arrays are abstractions designed to represent vectors and matrices. Iteration is a control abstraction useful for traversing vectors and arrays. Think, for example, of how easy it is to implement summation over the elements of a vector. This is because arrays and DO loops were codesigned."

"But an interesting thing happened to Lisp in the early 1980s: the use of macros to define control structures became forbidden style. Not only did some organizations outlaw such use of macros, but the cognoscenti began sneering at programmers who used them that way. Procedural abstractions are acceptable, but not control abstractions. The only acceptable control abstractions in Lisp today are function invocation, do loops, while loops, go statements (sort of), non-local exits, and a few mapping operations (such as mapcar in Lisp)."

"Regardless of what you make of this view of data versus control abstraction, it is certainly true that because almost every programming language does not allow any sort of meaningful user-defined control abstractions, there is always a mismatch in abstraction levels between control and data. If there is a good reason for allowing data abstractions, why isn’t that a good reason for allowing control abstractions; and if there is a good reason to disallow control abstractions, why isn’t that a good reason to disallow data abstractions?"

Finally a word on the general use of patterns.

"Common patterns are similar in nature though not detail to the patterns that Christopher Alexander uses in his so-called pattern languages. A pattern language is a language for generating buildings and towns with organic order. Patterns generally specify the components of a portion of a building or a place and how those components are related to other patterns."

More of this is covered in "The Failure of Pattern Languages" but the next part will be "The Quality Without a Name".