Over the Waterfall
Method acting for software developers
Comment Let's throw some rocks at Waterfall development. You know, the development method which says that you have to produce about 10kg of specification document (there’s a standard, usually, for the weight of paper appropriate to different project types).
You put this on the shelf to gather dust, as some sort of protective talisman, while you use what you can remember of it to direct the enjoyable part of your job, which is cutting code.
This specification is very important, so you mustn’t change it. When you deliver something two years later that's no longer needed, because the business process went and changed on you, this is proof-positive that you were Doing It Properly and coding to spec - and therefore deserve your annual bonus. And, of course, the Waterfall says that you mustn’t test anything until just before delivery, so that any defects in the bits of your system that are still needed are disruptive and expensive by the time that you find them.
This is all very wonderful and gives you a great weapon to beat the standards police and other non-programming busybodies over the head with. Except, this terrible Waterfall Method doesn't really exist.
Even so, Tom Gilb took me to task recently because he thought that I was suggesting that Winston Royce had proposed a method like this, around 1970. Well, I'm sorry, Tom: Winston Royce really did describe a Waterfall Method in a paper presented to IEEE WESCON in 1970 (Managing the Development of Large Software Systems, Proceedings of IEEE WESCON, August 1970, 1-9). But if one reads past the first page one discovers that what he was talking about was very different to the Waterfall Method I caricatured above, the one that everybody loves to hate.
Royce’s theoretical discussion recognizes, for example, the need for a degree of iterative development and for end-user involvement (two things that successful project managers using a Waterfall model usually adopt anyway) - and the assumed absence of which form the basis of much criticism of the Waterfall.
For a good description of Royce's thesis, with annotations from an up-to-date point of view, read chapter one of Software Project Management by his son, Walker Royce (published while he was VP and General Manager at Rational Software Corporation, now part of IBM). Walker Royce says (op. cit.):
"I have always been overwhelmed by the insight presented in this paper. While most of the industry has spent considerable energy bashing the waterfall model approach, I find only minor flaws in the theory even when it is applied in the context of today's technology. The criticism should have been targeted at the practice of the approach, which incorporated various unsound and unworkable elements. I suspect that most critics never really understood this theory; they just understood the default practice."
According to Gilb, what Winston Royce actually described 36 years ago has more or less evolved into the EVO (evolutionary method) that he now espouses. But at the Unicom seminar where I met Gilb, Graham Dick of process improvement consultants Lamri, pointed out that the Waterfall Method -as commonly accepted - is still widely used - and one attendee lamented that she couldn't get rid of it, try as she might.
Maybe some people are using a dysfunctional version of the Waterfall Method, as caricatured at the start of this article. But most are probably using something much closer to the Royce recommendations, as this promises to make satisfying external stakeholders in a “fixed price” project (who want to know what they’ll get and how much it will cost) straightforward.
I once had to look after a Waterfall Method for a major bank. We applied commonsense - eventually. We removed defects early, by analysing requirements, by questioning them and looking for inconsistencies and omissions, by validating data models, by unit-testing code as it was written.
The Requirements Spec and the Waterfall Lifecycle were used as guidelines, not as prescriptive mandates; and Requirements were allowed to change, under control. However, we made some attempt to feed back the financial and temporal consequences of requirements changes to those requesting them. And we didn't so much document the Method in a Method Bible, as promote it through instructor-led “good practice” training and mentoring.
There were still some issues, such as a lamentable tendency to interpret a statement like "consider doing X at this point" in what method documentation we did have, as: "you must do X, even if it makes no sense”; but with a good project manager it was a workable development method that delivered real business benefit.
These days, there are a plethora of development methods or processes, ranging from Rational/IBM's Unified Process through the OMG’s MDA to Kent Beck’s eXtreme Programming and Agile methods such as DSDM. Any one of these can be made to look stupid by “malicious compliance” to the letter rather than the spirit of the method.
And, any company that has made a mess of a Waterfall Method, developing projects Late, Over Budget and Wrong, must ask itself why it thinks it can make new methods work – whether the reason for its past development failures was the development method it adopted - or the way it adopted it.
Unless, of course, it now recognises the reasons for its past failures and addresses them – the Process Improvement which Borland, amongst others, now espouses.
I suspect that a mature, process-oriented, improvement-focused company with decent project managers will make a success of any development method, even something old-fashioned. But a company that wants to follow a mindless recipe for development success will fail - no matter how new and shiny the methods it adopts. ®
David Norfolk is the author of IT Governance, published by Thorogood. More details here.