Showing posts with label software development. Show all posts
Showing posts with label software development. Show all posts

Monday, February 28, 2011

Linkage

Tuesday, March 31, 2009

One Way to Describe What I Did Today

I've thought a bit about how best to describe what I did today. I spent a whole day, knowingly, adding more technical debt rather than two days removing existing debt. Now writing bad code is not linear and it's hard to judge but from what I can tell there is now twice as much debt as yesterday. And this is seen as progress.

Thursday, October 02, 2008

Coffee Inspired


  • The Global graphs in JRDF was inspired by the work done in MSG (minimum self-contained graphs, published in the RDFSync paper) and RDF Molecules. The former links to an implementation of DBin (P2P Semantic Web client) and there's also GVS (Graph Versioning System).
  • It's a trap (cloud computing) It's a fairly typical Stallman statement - not wrong but not aware of the compromises people make. It is obvious that one of the reasons vendors are excited about cloud computing is because it is a chance for them to try and own your data (or at least make switching too hard). But you do have to do more than put data in the cloud - you do have to have executable services there. There are open source cloud computing infrastructure that you can run-up on your own servers like Hadoop or CouchDB. And it's not just back to mainframes and renting CPU time out by the hour it really is different to what has gone before.

  • Live the Cloud Life lists the cloud computing applications in categories such as email, documents, data, music, photo editing and storing and browser synchronisation. Some categories are missing like RSS reading and calendaring.

  • Apple drops NDA - the outrage worked.

  • The main eResearch 2008 conference is over and some of the papers are available.

  • Speaking of which, all development, and that definitely includes software, in whatever organisation (universities, governments, banks, etc) should have failure as an option. One sign of a truly stuffed culture is to never have a project fail.

  • Muradora a refactoring of Fedora to allow pluggable authentication and enable metadata editing.

  • Acer Aspire One links: Dual Monitor Support,
    Installing Firefox 3 on Acer Aspire One Linux and
    Updated repositories. It's a shame that OpenOffice doesn't support presenters view yet.

Monday, September 01, 2008

Thesis and Antithesis

David Anderson on Agile. He introduces his talk by discussing some very broad ideas, including how agility has been applied to other areas. He talks about how the agile manifesto seems to have become more about belief and superstition than a scientific model. It's also rarely clicked through - to see the principles behind the manifesto.

The agile community has found it is better to develop in a failure tolerant environment. This was a reaction to the focus on more and more accurate estimates which lead to analysis patterns (or antipatterns). They are antipatterns because there was never enough detail in the analysis to provide accurate enough estimates and to deal with the unexpected.

Through his talk he listed a bunch of thought bubbles on software development:
* Value is providing functionality fastest.
* Knowledge work is perishable.
* Perfect is the enemy of good enough.
* Develop a high trust, high social capital.
* Have a highly collaborative culture.
* Reflect and adjust.
* Sustainable pace.
* Craftmanship.
* Value is contextual, context is temporal.
* Waste over scale.

Traditional industries have relied on bigger batch sizes to provide better economies of scale. With software development the transaction costs causes large batch sizes to work against you. Smaller batches allow you release more business value more quickly and reduce waste (waste over scale).

He introduces kanban which has ideas such as: pull, flow, and regulate work in progress. It does not use time boxed iterations and little or no planning or estimations (at least from a traditional agile approach). It still has constant improvement and delivery. It drops the idea of a generalist approach to labor which is seen by some as waterfall in disguise. The reason this comes about is because at an enterprise scale it's not feasible to hire a lot of experts who are excellent generalists - the labor pool does not exist. This leads to a tension with typical lean principals of reducing waste by having generalists.

He sees two main changes coming in the future: software factories (software product lines - which includes DSLs) and cloud services (deploying web services in the cloud). Architecture and modeling will come back in fashion as a value chain allows incentives for delivering common behavior. He's talking about a 100 fold improvement in productivity through using these ideas.

Finally, he talks about how CMM/CMMI is actually quite a good indication of the ability for software projects to succeed and the industry as a whole. He mentions a report on how agile and CMMI can work together, the idea being that not only is organizational maturity a good idea but it actually means that agile methods can be implemented better (it's called "CMMI or Agile: Why Not Embrace Both?!" - I only found a few references, for example Agile+CMMI Panel @ SEPG).

Via.

Still have Richard P. Gabriel's talk to go through too.

Friday, December 14, 2007

Timeless Software

Barry Boehm (having read it many times it's pronounced "beam"), "A View of 20th and 21st Century Software Engineering", lists the timeless qualities of a good programmer that have been discovered through the decades. Some of them include: don't neglect the sciences, avoid sequential processing, avoid cowboy programming, what's good for products is good for process, and adaptability trumps repeatability.

The powerpoint for the presentation is also online.

Update: I have notes from this talk but little time to review them. There's three things that stick out in my mind though.

The first is slide 5. This lists the most common causes of project failure. Most of these are not technical but centre around project management and people. I think it can be successfully argued that the other issues, mainly technical problems, are also people problems too, especially things like "lack of executive support".

Slide 9 has a Hegelian view of the progress made in writing software. That is, there is a process of advancement that involves theses, antitheses and syntheses. For example, software as a craft was an overreaction to software being considered analogous to hardware development and agile methods are a reaction (some may say overreaction) to the waterfall methodology.

Thirdly, Slide 30 has what initially looks like a good diagram to help decide how agile your project should be. I had issues with this. Much of my experience with agile projects is that they are actually better planned than most traditionally run projects. I also don't see how the criticality of the defects is appropriate either, as the number of defects is far fewer and less critical in well run agile projects too. Maybe I've never been on a well run planned project.

This made me wonder, why does software development still seem so far removed from science? Or to put in another way, why is it so dependent on personal experience? OO is quite frequently vilified but the reason people supported and continue to do so was because reuse increased and costs decreased - people were just following the numbers. Before OO, it was structured programming. And in contrast, what seemed like good ideas, such as proving the correctness of programs, have not been widely adopted. Barry points out that the reasons, since its inception in the 1970s, was because it did not prevent defects occurring in the specification and it was not scalable. This lead to the ideas of prototyping and RAD which were developed to overcome these perceived failures.

Saturday, August 25, 2007

Beautiful Engineering

One Bridge Doesn’t Fit All
American bridge engineering largely overlooks that efficiency, economy and elegance can be mutually reinforcing ideals. This is largely because engineers are not taught outstanding examples that express these ideals.

A 2000 report by the Federal Highway Administration indicated that an average of about 2,500 new bridges are completed each year; each could be an opportunity for better design. The best will be elegant and safe while being economical to build.

The key is to require that every bridge have one engineer who makes the conceptual design, understands construction and has a strong aesthetic motivation and personal attachment to the work. This will require not only a new ethos in professional practice, but also a new focus in the way engineers are educated, one modeled on the approach of those Swiss professors, Wilhelm Ritter and Pierre Lardy.

Via Bridges and code.

Wednesday, August 22, 2007

Result

The previous issue of IEEE is avaiable to Xplore subscribers. It is an issue dedicated to TDD and agile development.

Mock Objects: Find Out Who Your Friends Are highlights the oft discussed criticisms of test driven development including: breaking encapsulation, hard to read, increased code fragility and whether it is a valuable use of developer time. Of course, these differences can be resolved in a typical XP way:
My point is that plenty of well-written code is produced without relying on mocks or Tell, Don’t Ask. On the other hand, I also encounter plenty of TDD novices who produce poor code and poor tests. Can Steve and Nat’s approach consistently help novices write better code and tests? It could be. At this point, I remain open minded and am ready to pair-program with Steve or Nat to learn more.

The second article, is about the studied effect of TDD on the quality of code, TDD: The Art of Fearless Programming. Pretty much every study showed an increased effort (15-60%) and an increase in quality (5-267%). The ratio of effort to quality is 1:2 - so it seems to pay. Example definitions of quality were "reduction in defects" or "increase in functional tests passing". In general:
All researchers seem to agree that TDD encourages better task focus and test coverage. The mere fact of more tests doesn’t necessarily mean that software quality will be better, but the increased programmer attention to test design is nevertheless encouraging. If we view testing as sampling a very large population of potential behaviors, more tests mean a more thorough sample. To the extent that each test can find an important problem that none of the others can find, the tests are useful, especially if you can run them cheaply.

Via Mock Objects

Tuesday, April 03, 2007

Patterns In Software - Part 4

Patterns of Software: Tales from the Software Community (freely available in PDF). The chapter I'll be covering this time is: "The Failure of Pattern Languages".

So it begins with the punch that the last few chapters have been building up to:

Alexander’s story does not end with the publication of A Pattern Language in 1977. It went on and still goes on. And Alexander did not sit still after he wrote the patterns. Like any scientist he tried them out.

And they did not work.


The reason, quoted by Gabriel is that other processes other than architecture play a more fundamental role. That a building is not just the process of slapping together a bunch of very well thought out patterns based on emperical evidence. It's the result of many other processes like finance, zoning, construction, etc.

...one problem with the building process is lump-sum development. In such development few resources are brought to bear on the problems of repair and piecemeal growth. Instead, a large sum of money is dedicated to building a large artifact, and that artifact is allowed to deteriorate somewhat, and anything that is found lacking in the design or construction is ignored or minimally addressed until it is feasible to abandon the building and construct a replacement.


Gabriel gives an example from a previous chapter about how these processes can destroy the otherwise good qualities of development. For example, the process of getting a mortgage and paying it off directly influences the types of buildings built and used. Generally, you invest a large amount of money in a property, so large, that you usually can't afford to make piecemeal modifications because you can barely afford paying it off. This is okay as long as people are jumping from house to house. This avoids fixing up the problems there maybe with these houses until things get really bad and they are knocked down and rebuilt. Perhaps losing what was wrong with them in the first place.

So if you retain these old processes you still get the old results. Now it's clear in software that this is also the case - a nicely architected and controlled software project does not necessarily lead to high quality software. How often have you worked on a software project that you knew was going to be re-written in a few years anyway?

Gabriel suggest that the answer lies in good code and coders not in the typical separation of analysis, design and implementation:

And isn’t the old-style software methodology to put design in the hands of analysts and designers and to put coding in the hands of lowly coders, sometimes offshore coders who can be paid the lowest wages to do the least important work?

Methodologists who insist on separating analysis and design from coding are missing the essential feature of design: The design is in the code, not in a document or in a diagram. Half a programmer’s time is spent exploring the code, not in typing it in or changing it. When you look at the code you see its design, and that’s most of what you’re looking at, and it’s mostly while coding that you’re designing.


And finally, that the typical software patterns are not following Christopher Alexander's original concept of a pattern language:

When I look at software patterns and pattern languages, I don’t see the quality without a name in them, either. Recall that Alexander said that both the patterns themselves and the pattern language have the quality. In many cases today the pattern languages are written quickly, sort of like students doing homework problems. I heard one neophyte pattern writer say that when writing patterns he just writes what he knows and can write four or five patterns at a sitting.


The answer is to build software piecemeal, incrementally, partially designed and reflecting - much more like a Turkish rug and that's the next chapter, "The Bead Game, Rugs, and Beauty".

Friday, March 09, 2007

Incremental Adobe

Adobe edits the development cycle

The change we made was going from a traditional waterfall method to an incremental development model. Before, we would specify features up front, work on features until a "feature complete" date, and then (supposedly) revise the features based on alpha and beta testing and fix bugs. But we were scrambling so hard to get all the committed features in by the feature complete date - working nights and weekends up to the deadline - that the program was always very buggy at that point. We'd be desperately finding and fixing bugs, with little time to revise features based on tester feedback.

At the end of every cycle, we faced a huge "bugalanch" that required us to work many nights and weekends again. Of the three variables: features, schedule, and quality, the company sets the schedule and it's only slightly negotiable. Until feature complete, we could adjust the feature knob. But when we hit that milestone, quality sucked and we had only a fixed amount of time until the end. From there to the end, cutting features was not an option and all we could do was trade off our quality of life to get the quality of the product to the level we wanted by the ship date. We've never sacrificed product quality to get the product out the door, but we've sacrificed our home lives.


The quality of the program was higher throughout the development cycle, and there have been fewer total bugs. Instead of the bug count climbing towards a (frighteningly high) peak at "feature complete", it stayed at a much lower plateau throughout development. And we were able to incorporate more feedback from outside testers because we didn't switch into "frantic bug fix mode" so early.

Monday, February 26, 2007

Patterns In Software - Part 3

Patterns of Software: Tales from the Software Community (freely available in PDF). I was going to talk about, "The Failure of Pattern Languages" but I thought that a little discussion on the preceding chapter, "Pattern Languages", is required. This is somewhat of a generalisation of the preceding chapters and a discussion on Alexandar's "A Pattern Language", which offered the template for the software pattern books.

A comparison is made between art and science - art is creation, science is description - and that there are few activities that combine both. The exceptions given are computer science and architecture (I'd suggest maybe biological science is another). Trying to avoid the separation of construction and design is a mistake in computer science. Much or almost all of computer science is design and that it is expensive because it does combine both aspects for example, "What is Software Design?".

So this is specifically what all the discussion was a few years ago about TDD - being Test Driven Design (not Development). And the focus on behaviour rather than testing, for example Tom's recent Instinct project.

I want to set the stage for the accepted view of how architecture is done, at least in the pre-Alexander world. Architects are hired to solve the problem of how to construct a building or buildings that meet certain constraints as specified by the future inhabitants or people in a position to specify what those inhabitants will need. The architect generally interviews those future inhabitants, studies the site, refers to the local building code, observes neighboring buildings, considers the building materials and construction protocols in the area, and then is inspired, in the manner of all artists, to create a set of drawings which a general contractor, in occasional conference with the architect, reduces to a physical object.


Following from the suggestion given in the book, that the inhabitants are the programmers, it's obvious that "software architects" should ask the inhabitants of the software the best way to solve their problems. But that's not enough, more often than not they dictate solutions without knowledge of any of the complexity that their "solutions" might bring. This aligns with Alexander's suggestion for architects.

The first of Alexander’s contributions to architecture was to reject the separate architect and builder model and to posit user-centered design—in which users (inhabitants) design their own buildings—and the architect-builder who would blend the activities of design and construction. This was viewed by architects as high treason enough, let alone the curious quest for an elusive quality that Alexander cannot name.

The mechanism he proposed to accomplish his new model was the pattern language. A pattern language is a set of patterns used by a process to generate artifacts. These artifacts can be considered complexes of patterns. Each pattern is a kind of rule that states a problem to be solved and a solution to that problem.


I recently started to read a copy of "A Pattern Language" and I was struck by the same thing that Gabriel was, especially in comparison with the GoF book:

One thing that strikes many readers of the pattern language is the degree to which Alexander talks about people and their activities in the patterns — the patterns are a response to his arguments about how life is best lived. Because of this, many regard "A Pattern Language" as a manual of how to live and how physical surroundings can support and enhance living.


The patterns discussed in Alexander's book are all about what makes an area good for people - side walks (foot paths), water features, stairs, the height of crossings, making a city accessible to bicycle and foot traffic, etc - all these things make the environment livable and most ideas are supported by empirical evidence. The world of software patterns, by comparison, seems much more barren and arbitrary.

So there are some obviously good things about patterns:

Patterns seem to be a partial solution to the over abstraction problem I talked about in “Abstraction Descant.” They are a way to take advantage of common patterns without building costly, confusing, and unnecessary abstractions when the goal is merely to write something understandable. That is, when there are more idioms to use, using them is far better than inventing a new vocabulary.


Patterns in software are a good idea, but I have rarely seen the attention to the human side—to living within software—that I see in Alexander’s patterns. One architect commented on Alexander that he regarded the patterns as providing information, sort of heuristics, about architecture, and I have to admit the patterns I’ve seen about software do that—they provide information that the software practitioner just plain ought to know.


But this also highlights one of the problems with patterns. They discuss the context of the solution to a problem without first defining and more closely examining the problem itself, especially in a more general way. While it's often discussed in this chapter that patterns interact with other patterns and that they must be free of "systematic variations" the problems themselves don't seem to require this property. When people are critical of patterns, this is what I think they are objecting to, rather than the patterns themselves. He quotes Alexander saying: "You see that the patterns are very much alive and evolving."

And in a foreshadowing of what's to come:

The worst thing is this: Alexander had the chance in the 1970s to try out his pattern language. He observed the results of others trying it out and he even tried it out himself several times. And guess what: It didn’t work.


The next chapter, "The Failure of Pattern Languages", discusses the failure to apply these pattern but also a proposed solution to the problem: that the process should also have "the quality without a name".

Thursday, February 01, 2007

Need for Speed

Bosworth On Why AJAX Failed, Then Succeeded links to which isn't the first time he has examined AJAX's success and failures or why other technologies fail such as RDF. Some commentary was interesting in the comments and a few blogs, including this summary:


All in all the speaker (intentionally) kept his talk simple and stupid but here are some points I think are worth noting:
  • Products fail because of [lack of] human psychology not poor engineering
  • Top 3 keypoints of UI:
    1. Speed
    2. Speed
    3. Speed
  • Your app should be a tool, it shouldn't leave the user feeling like one
  • Search isn't perfect but it is far better than any human can do so it is tolerable.
  • Analyze the frequency of use of feature and implement those first
  • If your app takes more than 2 seconds to load - forget about it
  • The Motorola RAZR was cool enough so the poor UI (narrow keypad) didnt matter

Wednesday, January 24, 2007

Patterns in Software - Part 2

Patterns of Software: Tales from the Software Community (freely available in PDF). "Quality Without a Name" spends a lot of time taking Alexander's objective "meaning for beauty, for the aliveness that certain buildings, places, and human activities have" and applying it to software. This chapter also has a fairly good definition of patterns:

Patterns certainly have an appeal to people who wish to design and construct systems because they are a means to capture common sense and are a way to capture abstractions that are not easily captured otherwise.


The second thing (and second paragraph) is a re-definition, for most people in the software industry, on who the users of the software are:

...when you read Alexander, it is clear that a “user” is an inhabitant—someone who lives in the thing constructed. The thing constructed is under constant repair by its inhabitants, and end users of software do not constantly repair the software, though some might want to.


So these are not the regular definitions of quality or user. It's certainly not the TQM definition of quality: the fitness to a standard or set of requirements. It's much closer to the principle of quality in XP and other agile methodologies. Meeting requirements is rejected as the definition of quality because they are often contradictory, he quotes Alexander on his experience with Bay Area Rapid Transit (BART) system:

So it became clear that the free functioning of the system did not purely depend on meeting a set of requirements. It had to do, rather, with the system coming to terms with itself and being in balance with the forces that were generated internal to the system, not in accordance with some arbitrary set of requirements we stated...What bothered me was that the correct analysis of the ticket booth could not be based purely on one’s goals, that there were realities emerging from the center of the system itself and that whether you succeeded or not had to do with whether you created a configuration that was stable with respect to these realities.


He also takes some time using other (somewhat confusing) words such as alive, whole, comfortable, free, exact, egoless, and eternal to further refine what "the quality without a name" is. The purpose for this is clear though, it's to rediscover objective quality or the combination of fact and value.

We in software are not so lucky—all of our artifacts were conceived and constructed firmly in the system of fact separated from value. But there are programs we can look at and about which we say, “no way I’m maintaining that kluge.” And there are other programs about which we can say, “Wow, who wrote this!” So the quality without a name for software must exist.


Some of them I'd consider fairly vague but others have some parallel with XP principles and similar ideas. In order to describe how a system is alive Gabriel uses the metaphor of a fire and a fireplace. The structure of the logs, chimneys and so on is a well thought out system which supports a self-sustaining process that once set in motion reaches a predefined end - a small pile of ashes. And "whole" is a property where something is self consistent. This is what occurs when code is reflected upon and refactored. And while Gabriel doesn't spend much time talking about a system being egoless, this concept should be familiar to most agile practitioners. XP and other methodologies encourage the idea of egoless programming and collective ownership.

All of these concepts and "the quality without a name" all seem to me to be descriptions of development methodologies, practices, principles and values and the particular kind of software that is developed as a result of applying these processes. That is, software with this quality is a result of a software process with the same quality.

He ends this chapter with "some things" (I'd say requirements) for software which possesses the "quality without a name". I'd summarize it as:
  • Not written under unrealistic deadlines.
  • Modules and abstractions not too large and small enough to understand and remember what they do.
  • Code constantly repaired (he'd probably say refactored today).
  • A fractal nature of code - looking at the large or the small the code is coherent.

He ends it with this lament:

I wish we had a common body of programs with the quality, because then we could talk about them and understand. As it is, programs are secret and protected, so we rarely see any but those we write ourselves. Imagine a world in which houses were hidden from view. How would Alexander have found the quality with no name?


Since this was written there's been a lot of code made public but I still can't think of any that I've really looked at and appreciated (and in fact I tend to avoid looking at Java code written under the Apache licence for example). Any suggestions?

Friday, January 19, 2007

Patterns in Software - Part 1

Patterns of Software: Tales from the Software Community (freely available in PDF format) is something I came across last time I was talking about design patterns.

First off, I think this book is very well written - style, language, structure - it's all good. I wish I could write as well as this guy. Christopher Alexander is in the preface (the guy who is attributed to inventing patterns).

Secondly, I'm going to cover the chapters that I've liked as I get to them. The first is "Abstraction Descant".

In this chapter he introduces the concept of compression which is: "...the characteristic of a piece of text that the meaning of any part of it is “larger” than that particular piece has by itself."

Examples are macros, function or class names. There is a danger though that compression and abstraction can be overused.

"The problem is that people are taught to value abstraction above all else, and object-oriented languages and their philosophy of use emphasizes reuse (compression), which is generally good. However, sometimes the passion for abstraction is so strong that it is used inappropriately — it is forced in the same way as it is with larger, more complex, and typically ad hoc abstractions."

"Another problem with complex abstraction arises from the observation that abstractions are about ignorance...Some complex abstractions, however, contain information about the implementation that is legitimately required, such as its performance, the algorithm, coding tricks, and resource usage—keep in mind that almost all interaction issues are about resource conflicts."


The next point is that software evolves through time, in little pieces by programmers not designers. The danger is the designers use abstractions (usually in ignorance) whereas coders should write code and design it in order to make it more habitable.

"...creating habitable software that can be effectively maintained, recognizing that the reality of software development is piecemeal growth and to plan accordingly, and to understand that the power of object-oriented programming is compression, which carries a terrific requirement for careful use of inheritance—relate to how we use abstraction and how much we use it."

"In programming, if a set of large abstractions does nearly the right thing, it is tempting to use them and to bend the structure of the surrounding program to fit them. This can lead to uninhabitable programs."

"Worse: You can fight this temptation and choose not to use them. This choice also can lead to uninhabitable programs because you will be using parts similar but subtly different from possibly familiar ones. The only way to avoid this is to use small blocks rather than large ones, or to use blocks well-designed and tested by experts."


He also has a comment on teaching programming where we need to "learn from the classics".

"How much time do we spend reading in our ordinary education? And from our reading we gain a foundation for writing...But in programming we just learn the language and solve a bunch of short puzzles. Sort of like writing 50 limericks and then off to write books."


The next point took me a while to digest but I think it's probably the most useful part of the chapter. It is that we rarely spend time creating new control abstractions (like loops) and most of the time creating abstractions of data. He argues that the two should go hand in hand.

"Let’s look at another problem with abstractions: Data and control abstractions are generally best when they are codesigned and this is rarely done anymore. Consider, for example, the Fortran abstractions of arrays and iteration. Arrays are abstractions designed to represent vectors and matrices. Iteration is a control abstraction useful for traversing vectors and arrays. Think, for example, of how easy it is to implement summation over the elements of a vector. This is because arrays and DO loops were codesigned."

"But an interesting thing happened to Lisp in the early 1980s: the use of macros to define control structures became forbidden style. Not only did some organizations outlaw such use of macros, but the cognoscenti began sneering at programmers who used them that way. Procedural abstractions are acceptable, but not control abstractions. The only acceptable control abstractions in Lisp today are function invocation, do loops, while loops, go statements (sort of), non-local exits, and a few mapping operations (such as mapcar in Lisp)."

"Regardless of what you make of this view of data versus control abstraction, it is certainly true that because almost every programming language does not allow any sort of meaningful user-defined control abstractions, there is always a mismatch in abstraction levels between control and data. If there is a good reason for allowing data abstractions, why isn’t that a good reason for allowing control abstractions; and if there is a good reason to disallow control abstractions, why isn’t that a good reason to disallow data abstractions?"


Finally a word on the general use of patterns.

"Common patterns are similar in nature though not detail to the patterns that Christopher Alexander uses in his so-called pattern languages. A pattern language is a language for generating buildings and towns with organic order. Patterns generally specify the components of a portion of a building or a place and how those components are related to other patterns."


More of this is covered in "The Failure of Pattern Languages" but the next part will be "The Quality Without a Name".

Monday, January 15, 2007

Quality is Free

The business value of software quality "Organizations that develop low-quality software, whether for internal use or for sale, are always looking backward, spending time and money on fixing defects in "finished" products."

"One common misconception about quality is that it can be traded for improved development speed, reduced development budgets, or added functionality. In practice, however, most organizations find the opposite is true. In the long run, improved quality enables teams to deliver more projects on time, at lower cost, with more features. We should realize that Meskimen's law -- "There's never time to do it right, but always time to do it over" -- is a tongue-in-cheek adage for a reason. A development team that continuously ensures quality does it right the first time. By not introducing defects into the system throughout the entire development process, a team eliminates the time and cost required to find and fix those defects later on."

And why don't customers care about quality: "The simple answer is: “Because defective software works.” The reason it works, however, is because software doesn’t wear out, rot, or otherwise deteriorate. Once it is fixed, it will continue to work as long as it is used in precisely the same way."

Sunday, January 14, 2007

The Software of Star Wars

Code Reads #7: Parnas's "Star Wars" paper links to the well known paper, "Software Aspects of Strategic Defense Systems", where he quotes a description of what a software developer actually does, "How, then, do we end up with any big programs that work at all? "The answer is simple: Programming is a trial and error craft. People write programs without any expectation that they will be right the first time. They spend at least as much time testing and correcting errors as they spent writing the initial program.""

"With these observations in mind, Parnas casts a cold eye on the SDI project, which aimed to produce working systems that could identify and target incoming enemy missiles in a matter of minutes. The system couldn't be tested under real-world conditions; it would be expected to function effectively even when some of its pieces had been disabled by enemy attack; and it was intended to be foolproof (since, with incoming H-bomb-armed ICBMs, 90 percent wasn't good enough). No such system had ever been built; Parnas maintained that no such system could be built within the next 20 years, using either existing methods or those (like AI or "automatic programming") on the horizon."

Logic is highlighted as the mathematics of programming much like continuous mathematics in electrical and mechanical engineering. A lot of the reasons why SDI was unfeasible was that the requirements gathering - it may seem that the customer is a ruthless, dictatorship, with the finger on total annihalation - but it's rarely true.

There are a couple of important questions that probably have better answers now. Can we provide proofs for code larger than 500 lines (in whatever language)? Is hardware failure considered in these proofs? Is rules based programming more efficient?

I'm glad I found this article, it's good because I vaguely remember hearing that the SDI project was not technically feasible (games like this demonstrate the error of linear scaling) but I had not idea that one of the problems was the development of the software (obvious in retrospect). This is slightly different to the management lessons learnt about the space shuttle but similarly illuminating.

The author of the initial blog entry also interviewed Joel Spolsky in which he mentions the idea of programmer fatigue. His book is available of the 16th of this month, called "Dreaming in Code", it talks about software development and the Chandler project. Maybe setting out to be the OS/360 project of the next generation?

Monday, July 17, 2006

Achieving Quality

Technology: Can You Automate Software Quality? "A lot of the debate has been focused on testing. Total Quality, however, would suggest that although testing is necessary, it's not sufficient. Testing focuses on inspection, not on prevention. To over-simplify, you test in hopes of demonstrating that the software has no defects (because you have a good high-quality development process), not to detect the defects that are present, but should not be there (because you don't have a good high quality development process). After several years of significant effort, the code my client was developing (and testing) still isn't of the high quality they are looking for. So we decided to go back, start from some basics, and look again at the issue of software quality."

Goes through the processes to ensure quality software: code quality (using TDD), functional quality (giving the customer what they want), non-functional quality (security, privacy, compliance, etc), deployment and production quality, and maintenance.

"This probably seems like a lot of additional work for the development organization. And it is. But the costs of defective code in the production environment (both direct, in terms of sustaining engineering, and indirect, in terms of lost revenue and reputation) was becoming significant. Management felt it had no choice but to focus on improved quality, and turn to the productivity issues later. So far, despite the added burden of all these quality–related activities (many of which already took place, but simply weren't very effective) we have not seen a slowdown in software availability. Some teams are actually moving faster than before, despite the new work they have to perform, because they spend much less time and effort on remediation and last-minute adjustments to code to fix issues that only show up as the code is transitioned to production."

Wednesday, June 28, 2006

Agile Anchors

XP2006 Patterns lists a summary of experiences in adopting agile development practices including anti-patterns: Unpaid Debt, Lots of Bugs Found by Customer. Some metrics to help drive agile software development with this not so contraversial claim, "Code coverage metrics measure the percentage of methods and classes under test. Developers should strive for 100% test coverage on classes that are behaviorally intense. The CCN can be used as a baseline for the number of tests required per method. If a method has CCN of three, a good starting point to exercise all paths through code would be three tests. Additional tests may be required that exercise error conditions and alternate behaviors. A robust suite of unit tests provides the courage necessary to refactor." Also mentioned, in "Examining Test-Driven Development, "The reality is that your unit tests form a large part of your design specification, and similarly acceptance tests form a portion of your requirements specification..."

Wednesday, June 21, 2006

Test First Saves Time

To obtain good code, writing tests and code is faster then code alone "Ron Jeffies, one of the XP gurus stated that "in order to obtain good code, writing tests and code is faster then just code". To find out if this is true or not let's make a small experiment."

Tuesday, May 30, 2006

The Greyness of Gold

Steve McMcConnell's Software Project Survival Guide mentions gold plating a project, specifically: "Implement only what is required. Developers, managers, and customers often think of small, easy changes that seem to make the software better. These changes often have much more far-reaching impacts than anticipated by the specific developer who will implement the change. Do not let additional complexity creep into the project through gold-plating."

This may fly in the face of ideas like refactoring (although refactoring is generally an attempt to remove complexity not add it). To put this into context, though, the team that he's talking about is well above the quality curve: "...the average MIS shop would need about 14 calendar months and 110 staff-months to deliver a 100,000 line-of-code MIS system, and it would typically contain about 850 defects when delivered. The NASA SEL [the team in question] would deliver a system of that size with about the same amount of time and effort, but it would contain only about 50 defects."

So when you have a manager talking about not wanting to have a gold plated solution check to see if your project is delivering on the 50 defects/100,000 lines of code first.

Of course, the original point is true too - you can continually seek the perfect solution to a business problem, probably forever, and in doing so add complexity to a system. Business processes aren't always rational so there may not be a good solution to the problem. So continually swapping ways of solving a problem can be an untenable situation.

I know that some people want to refactor code forever and others are happy just to write code and see if it works somehow. There's obviously a balance.

To me saying, "I don't want a gold plated solution" is asking not try to achieve a certain level of quality and certainly most IT shops should be trying to do more not less.

Thursday, March 30, 2006

Vista Chicken - System/360 all Over Again

Discontent at Microsoft. "What I saw in MS was PM's pushing hard for features:
* even if it meant that the test combinations would be very large, so the product couldn't not be tested properly.
* even if it couldn't be done properly in the time allocated. After all an estimate of time was made, now all of those features mus go in the product evne if things are taking longer than expected.
* even if the product was falling apart at the seems b/c every other pm was doing the same thing.

In fact, people often played schedule chicken. It didn't matter if you were running late by the metric of the day as long as another group was running later."

"WinFS is a great example of a file system designed by lunatic engineers and inbred GPM teams (led by a totally lunatic DirPM) without a clue as to what a real customer even looks like. Complexity in the design for complexity sake is the kiss of death. Complexity without a clear, or even muddy, picture of the problem you are actually trying to solve for the actual customer is the kiss of death. Not having customers involved at every step of the design and development process is just arrogance. Believing you know better than the customer is just stupid."

Also, Exceptions to Brooks’ Law "Brooks’s Law: adding manpower to a late software project makes it later...It depends who the manpower is...Some teams can absorb more change than others...There are worse things than being later...There are different ways to add manpower...It depends on why the project was late to begin with...Adding people can be combined with other management action."

Fortune interview with Fred Brooks: "One is to officially slip the schedule. And officially doing it has many benefits over unofficially letting it slip...That is, if you're going to take a slip, get everybody onboard, get organized, and take a six-month slip, even though you may at the moment feel as if you're only four months late."