The Rapid Release Tautology


Several responses to How to Ruin Your Ability to Release Software asked "How does the company mentioned in the final paragraph do it?" The failed release management strategy of the first company is common. If that's so obviously wrong, how can anyone else do it right?

The secret is simple. Projects with frequent software releases can release their software frequently because they release their software frequently.

I know that sounds flippant. Before you write angry comments, let me digress for a moment.

Making the Impossible Possible

One of my favorite stories of last year is Booting Linux in Five Seconds. I'd interviewed Arjan van de Ven the month before on a related topic, and he let slip that he and Auke Kok had eliminated most of the cold boot time for GNU/Linux operating systems.

Other people had tried several approaches to reduce boot time. Most seemed to believe that starting services in parallel was the right approach, but few systems cracked the minute mark.

Arjan and Auke took a different approach. They set a goal: a five-second boot time. They relentlessly set priorities and threw out everything that stood in the way. The Linux kernel is endlessly configurable, but the configuration of any particular machine is reasonably static. Why probe for new hardware on every boot? Throw it out. Why recompile keyboard maps for every launch of Throw that out.

I don't mean to trivialize the process. They overcame some interesting technical challenges ("How do you use available CPU and IO resources to their limit?"). Yet their approach was deceptively simple. They removed everything that stood in the way of achieving their goals.

How to Release Software Frequently

The only realistic way I know of to release software frequently is to start to release software frequently. You will probably fail the first few times you try. If your release process is painful, it will continue to be painful until you identify and eliminate the causes of the pain.

First, your team needs to agree to make changes. They may be radical and painful. Without a collective agreement to try the new approach, you will likely fail. Don't bother trying without this.

(By "collective agreement" I mean that everyone involved in actively producing a release must agree. There's no reason you have to ship a new CD to customers every week or every month. That may be a good thing, but project-only weekly releases are a good thing in and of themselves. I'd use them even if I worked on a shrinkwrapped application with an 18 month sales cycle.)

Second, you need to establish the duration of your release cycle (See Iteration Planning). I prefer weekly releases for full-time teams, but two weeks is fine and a month is workable. The most important attribute of this process is a single, very subtle change in your mindset: tying the release date to the calendar, not to any particular feature.

Before you hit that comment box again, please take a moment to consider the implications, both good and bad. Ready? Good.

By tying a release date to a calendar ("We release a stable, usable version of our software every Wednesday morning at 10 am"), you have two options. Either your software is usable and releasable every Wednesday morning at 10 am, or it isn't. If you want to meet the commitment you've made, you need to do something radically different.

Obviously one of those changes is to reduce the amount of time and work it takes to produce a release. That must be a quick process. It must be an automated and repeatable process. It must not block on the knowledge or activity or presence of any one single person. This is not easy to achieve, but you get a chance to improve it once a week, every week. This will be painful. It will improve over time. (One of the best places to start is to improve the speed of your test suite.)

Done Means Done

The biggest change is that trunk must always be stable. You can work on whatever features you like, as long as you keep the trunk of the project in a releasable state. You must be relentless.

Part of this stability process is never merging features until they are completely and utterly finished. The agile world calls this Done Done. In contrast to my colleague's project, Wednesday morning at 9 am is not the time to check in a feature that you just made compile and are pretty sure that QA won't find any bugs, at least you hope so.

If a feature isn't done -- verified usable as intended -- you can't merge it to the trunk. It won't be part of this week's release. That's not so bad, however. There's always next week's release.

What You Can't Do

Of course, there are more implications. The Iteration Planning link suggests that you need to identify week-sized units of progress and prioritize them relentlessly. I agree, but that won't help much until you can release software frequently.

Note that there are several things you can't do anymore:

  • Hold feature freezes. The release process is too frequent for that. Good. Keep your code Done Done instead.
  • Play games with your bug tracker.
  • Hold back releases more than a couple of hours. If a release slips even by a couple of hours, something has gone wrong. Identify it and figure out how to fix it, right away.
  • Segregate QA and development. The days in which you can throw code over the wall and hope someone else will tell you if it's Done Done are gone. You can't afford to wait to find out. Get feedback immediately!
  • Maintain long-lived branches in your SCM, hoping to land them eventually. You don't have time for the divergence from trunk. Keep your branches small and short and your features small.

Is this realistic? I believe so. We've done it on Parrot for almost two and a half years now with volunteer labor. We're not perfect by any means, but we're predictable and we're improving.

None of this is out of reach of any decent team which wants to improve its processes. It's not easy, but it's possible -- and I believe it's better than every other alternative in software development.


Happy to read this explanation of your previous post, which I found really interesting. But indeed this solution is not possible without a fully automated testing process, isn't it ?
With this approach, do you still have major/minor releases ? Or is each release at the same level ?
Last but not least, how do clients react to this rather unusual approach ? And your management ?

If you release your product to customers once every 18 months -- or, to be somewhat less severe, if you released the boxed product that is all that your customers purchase and install every 18 months -- then aren't your interim "releases" just release candidates?

They might be getting installed within the organization, or used by a few customers (who might in other cases be called beta testers), but they're not really being installed and getting all the use you'd expect from customers who think of themselves as being current.

I would definitely not argue that there is no benefit to being able to cut a release at any time. We do it internally, and it's great. I just want to be sure that I understand what you think about the semitriannual cd shippers.

Modern Perl: The Book

cover image for Modern Perl: the book

The best Perl Programmers read Modern Perl: The Book.

sponsored by the How to Make a Smoothie guide



About this Entry

This page contains a single entry by chromatic published on April 6, 2009 3:13 PM.

The Release Candidate Trap was the previous entry in this blog.

A Test-Infected Culture is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Powered by the Perl programming language

what is programming?