What does "Stable" Mean?

| 3 Comments

By now it's clear that the people behind the Perl 5 release process consider "stability" one of their primary constraints. Support isn't easy to define for community-produced software; neither is "stable". That's no reason not to try.

Expectations of Stability

A reasonable first definition of "stable" is that an average user should be able to follow the documented process of obtaining, configuring, and building the software on a supported platform. After that, the user should expect that the automated tests pass.

There may be glitches in that process, but any software release for which the developers cannot make that guarantee is not stable.

Any disagreements? Good.

Does "stable" guarantee that all tests will pass on your system? No; your system may have unique characteristics that the developers did not anticipate. No plan survives first contact with the enemy (and feel free to quote me on this truism: no test plan survives first contact with entropy). Careful design and revision can ameliorate this, as can extended testing in heterogenous environments, but the only true test of whether the software works as intended is when real users use it for real purposes.

This is why we have bug trackers, patches, and subsequent releases.

Does "stable" guarantee that all previous uses of the software will continue to work as expected? No; your specific uses may rely on the presence of bugs, unspecified features, happy accidents, or other behaviors not captured in the test suite. Again, careful attention to detail can ameliorate this, but you can never guarantee it unless you vet all uses of your software.

This is why we have bug trackers, patches, subsequent releases, deprecation policies, and comprehensive documentation.

Does "stable" guarantee that the software will meet all of your needs? No; the software only does what it says it will do. You may be able to coax it to perform other functions beyond those documented and supported, but if you depart from what the developers expect, intend, and promise, you are on your own.

This is why we have roadmaps, overviews, and comprehensive documentation.

Does "stable" guarantee that there are no bugs? No; though whole-program analysis and computer-aided proofs can assist in verifying algorithms, this is infeasable for most software. There will be bugs of implementation, bugs of unclear specification, bugs of poor documentation, bugs of misunderstood requirements, bugs of portability, bugs of testing, and bugs of many other types.

This is why we have bug trackers, bug reports, subsequent releases, and you get the picture by now.

Does "stable" guarantee that you can replace the previous version of the software with the new version in your production environments on the day of release and sleep soundly that night? No; I cannot imagine anyone who would not call you a fool for doing so.

That's why you have comprehensive test suites, test plans, deployment environments, and access to the bug tracker, release notes, deprecation plans, support policies, and lines of communication to the developers.

That's why you have access to testing releases (not that anyone uses them).

That's why maintaining your software is your responsibility -- not because developers are jerks who hate you and would burn down your office if they could, but because the only effective way to prove that a piece of software is sufficient for your needs is for you to verify it against your needs.

Implications

Should developers get sloppy about their release processes? Of course not. Automated testing, exploratory testing, smoke testing, and other techniques to keep code quality high are amazingly important. If anything, most projects don't take them seriously enough.

Do developers get a free pass if a release has embarrassing bugs? Of course not. Mistakes are mistakes, and software that's not suitable for its intended uses needs immediate attention.

Are users part of the testing and verification process? They always have been, especially for general purpose tools such as a programming language. If software developers could get it all right the first time, we wouldn't need to argue over release processes or write automated test suites or include FAQs and disclaimers in our documentation. You wouldn't need to test software against your own purposes.

We don't live in that world, though.

Can regressions and embarrassing bugs be rare? Of course. The quality of the code and development process is exceedingly important. A well-maintained test suite verifying a well-factored and maintainable codebase updated regularly with thoughtful analysis of feedback, bug reports, and feature requests from active users can achieve very high quality. Mistakes will happen, but there are ways to reduce their risk and frequency.

Is it worth pretending that we can achieve perfect stability by delaying the release of software? In my opinion, no. It's more important to me to reduce risk by the flexibility and agility of being able to release software frequently.

I suppose that a development process that has to coordinate dozens of separate projects to converge upon a stable point on multiple platforms simultaneously while operating on all sorts of other constraints can eventually converge on that miraculous point of stability... but it'll still have bugs. There'll still be patches. Users will still need to test their own software against it.

Are users are better off waiting months and months for fixes to problems they have right now than they are dealing with mythical bugs and regressions and problems they might have in a future so-called "stable" release? Given that you can't know those problems until users encounter them, and given that users don't test release candidates, they'll discover these problems sooner or later, and only after a release.

The question is whether it's better to address those problems sooner or later. My bias is toward fixing problems as people report them. Certainly released software with fixes for known bugs must be more stable than unreleased and untested software.

3 Comments

A historical approach would be interesting. What was Jarkko Hietaniemi's thinking about this, in view of his Jack Cohen URL:http://www.drjackcohen.net/ quote: There is a special biological word for 'stable': dead.

Anyway, I don't think this is just a problem with perl, or software in general. The last (minor) changes in English spelling were made by Noah Webster 200 years ago. Japan conducted significant spelling reform after WWII. Perhaps a humiliating defeat is the only thing that will shake up the status quo.

By the way, why is change in perl5 an issue, with perl6 around the corner?

Lots of luck with keeping things moving in perl6.

Perl 5 will be relevant for a long time, regardless of Perl 6. Giving up on it now would be terrible for both projects. I, for one, would love to see some of chromatic's suggestions have an effect on the P5 development process.

A historical approach would be interesting. What was Jarkko Hietaniemi's thinking about this, in view of his Jack Cohen URL:http://www.drjackcohen.net/ quote: There is a special biological word for 'stable': dead.

Anyway, I don't think this is just a problem with perl, or software in general. The last (minor) changes in English spelling were made by Noah Webster 200 years ago. Japan conducted significant spelling reform after WWII. Perhaps a humiliating defeat is the only thing that will shake up the status quo.

By the way, why is change in perl5 an issue, with perl6 around the corner?

Lots of luck with keeping things moving in perl6.

Modern Perl: The Book

cover image for Modern Perl: the book

The best Perl Programmers read Modern Perl: The Book.

affiliated with ModernPerl.net

Categories

Pages

About this Entry

This page contains a single entry by chromatic published on June 9, 2009 2:04 PM.

Who Gets to Criticize Your Free Software Project? was the previous entry in this blog.

How to Change a Running System is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.


Sponsored by Blender Recipe Reviews and the Trendshare how to invest guide

Powered by the Perl programming language

what is programming?