Agile vs Waterfall
It’s time to settle the long-running debate of the best way to develop software; Agile vs Waterfall. I’m going to describe and compare the two approaches, discuss their pros and cons, and determine a winner.
What are Agile and Waterfall?
These words generally refer to two very different approaches to organising, planning and executing software development. In the 1960s, there was no formal approach or methodology to software development.
In 1970, Winston Royce published a famous paper “Managing the development of large software systems”, in which he describes a sequential or phased-gate approach.
The emphasis here is on planning and certainty.
Structuring the work
Another important point to note is that it is a sequential / phase-gate and single-pass. I’ll explain what these all mean, and what problems they pose.
Sequential: this means that the software development work is broken down into certain types or blocks, and that they happen in a certain order.
Phase-gate: this means that the work of one type cannot begin until all the work of the previous type has been completed and “signed off” (i.e. there is a gate at the end of a phase, and you cannot pass the gate until you’ve met all the criteria for finishing that phase).
Single-pass: this means that the work flows through these phases once and once only. Once a phase has been completed it can never be returned to or begun again. It is a one-way trip.
These seem like reasonable rules that can help bring order and control to a software development process. But in reality, there are actually enormous problems here. Agile software development came about largely as an attempt to solve these problems.
The problem with sequential work is that it often makes sense to have different people working on different things at the same time. For example, what are the developers doing in the Analysis phase? What are the testers doing in the Development phase? It makes much more sense to have people working on whatever needs to be done at that particular time.
The problem with Phase-gate work is that the entire project can become stuck (or not even really started) if even a tiny part of one type of work cannot be resolved.
For example, if there is one requirement that needs further elaboration or needs approval from a stakeholder who cannot be reached, then that phase-gate becomes as a giant bottleneck and no work can proceed until it is resolved.
There are stories of Waterfall projects that spent months or sometimes years waiting for all the requirements to be agreed on and approved. By the time this happened, many of those requirements may have become outdated or irrelevant.
The agile alternative to structuring the work
Agile software development arose in the 1990s to solve many of the problems plaguing large software projects that were following these models. Their approach to work sequencing was very different. Work was broken down into small pieces, and different tasks were often performed in parallel.
There were no big phase-gates that were needed to progress the project. Instead of big separate blocks of one type followed by a block of work of another type, agile suggested working in smaller chunks or batches that were repeated again and again. This had several advantages:
- People of different skill sets working together at all stages of the project (development, testing and analysis could all happen at once)
- No big phase-gate checks that could hold up work indefinitely
- People can iterate, go back and change or improve their work after learning more about their product and customers
- Development can begin almost immediately, reducing time to market and increasing learning.
Winner: Agile by a clear mile. Big chunks of work with phase gates are a clumsy and inefficient way of working.
Organisational structure
Although it is not directly discussed in the original Waterfall model described by Royce, Waterfall projects are almost always accompanied by a “component” or “specialist” based organisational structure. That is, the teams in the organisation are each built around specialists in one of those phases in the Waterfall model.
So you have a team of analysts who do the analysis work, a team of developers who do the development work, a team of testers who do all the testing work, and so on. At a higher level you usually have a PMO (Project Management Office), who are responsible for organising the whole thing.
The thinking here is that specialists are the best people to do the job, and the best way to build specialisation is to gather people together around a particular skill and get them to work together and focus on improving their specialisation. There are two problems here, however:
- “silo” teams like this mean there are handovers between teams, that can cause hold-ups and blockages
- Specialist silos mean that people can’t easily learn from and understand other people from other domains.
The agile alternative to structuring the organisation
Agile practitioners suggest instead forming “cross-functional” teams: forming small “squads” of people with a mix of skills. This means a single team can be capable of designing, building, testing and releasing a contained piece of software by themselves. This has two advantages over the Waterfall approach:
- A cross-functional team doesn’t need anything from any other teams, so won’t have dependencies on other teams (fewer handovers and approvals)
- A cross-functional team are more likely to share skills and knowledge and build up the overall capabilities of all the people in the team.
Winner: Agile. While specialisation seems like a good idea, handovers between silo teams can create waste and friction. Also, do not underestimate the value of having a diverse mix of people in a team.
Development approach
The development work on Waterfall projects is generally done on a large batch basis. Developers will each create a fork of the main branch, work on their allotted work for a while until it is complete from their perspective, and then all merge and integrate back into the main branch.
This is a terrible model, since there are major integration problems (often called “integration hell”) when these separate branches all get merged back together. It simply increases the complexity and risk of the project with no benefit whatsoever. Even the most die-hard of Waterfall purists are starting to move away from this model to the more Continuous Integration style of Agile practitioners.
The agile alternative to the Waterfall development approach
Most people doing Agile development will use a Continuous Integration model: each developer works in small batches, continuously merging back and “re-basing” (updated their local repository) as they go. Since there are many small changes being made at a very rapid pace, this is usually combined with Test-Driven Development.
TDD involves developers writing a small test, quickly coding until the test is passing, then refactoring (or improving) the code they have written. Developers merge back into the master branch by raising “pull requests”, which means other developers view and comment on their code before it can be merged back in.
Winner: Agile. This is the most clear-cut distinction of them all. There is simply no logical reason to adopt the Waterfall method of “integration hell”.
Requirements and change control
This is probably the most widely known and debated distinction between agile and Waterfall.
Traditional Waterfall projects follow the PMI (Project Management Institute) philosophy, which assumes that all requirements are correct and should be locked down at the beginning of the project.
They also stipulate that any deviation from the initial requirements is detrimental to the success of the project, and should be resisted via change control process.
That process means that changes must be presented to, reviewed and approved by a Change Control Board (CCB). This is usually a lengthy and expensive process. And the CCB often rejects changes, since they are a deviation from the original plan of the project.
The agile alternative to requirements and change
The Agile Manifesto declares that it “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage”.
Rather than operate from the assumption of fixed scope, Agile projects instead assume that scope can be flexible. And that the best scope items are often discovered after the work begins. Rather than try to resist or restrict these changes, they are welcomed as moving closer towards realising value.
Winner: If your priority is certainty (see more on that below), then Waterfall is better here. But if your priority is value, as it should be for any business, then agile is much more suitable. Projects and products live and die by how much value they created for customers and companies, not by how closely they followed a project plan or list of requirements.
Quality
Waterfall projects tend to place testing or quality assurance as a major activity to be performed towards the end, once all development work has been completed. Some of the work may have defects, which means further development work is required to fix those defects.
The agile alternative to quality
Although not specifically mentioned in the Agile Manifesto, people practising agile software development usually view quality in a very different. They adopt the Lean principle of building quality in, rather than just checking for it at the end. That is, testing is something that is done very early and very often, in many tiny batches.
If the developers are practising Test Driven Development, and have many automated unit tests, then testing is something that could be happening every few seconds. Sometimes thousands of tests will be run every day, or even more frequently!
This allows for early detection of defects, rapid changes in the codebase, and low risks for deployment and launches. There is a much lower cost of fixing defects if they are found early on rather than at the end.
Winner: Agile, easily. Finding defects earlier is cheaper and less risky than doing big batch testing at the end. Test automation is becoming crucial for application development. And even some Waterfall projects have started adopting these principles. Much like Continuous Integration, they are simply standard practices that all software developers are moving towards. No matter how they feel about agile.
Introspection
Waterfall projects tend to have a PIR or Post-Implementation Review at the end of the project. This is a big meeting where everyone discusses how the project went: what went well, what went badly, and what has everybody learned.
While it is good to reflect on what you have learned, doing it once at the end is too late. All the mistakes have already been made and there is little to be done about them.
During the project, there are usually a lot of meetings and committees, such as PWG (Project Working Group) and PSG (Project Steering Group). These are however meetings of managers and stakeholders, not the people doing the actual work to deliver the project. So they are often just a reporting exercise rather than a real attempt at introspection and improvement.
The agile alternative to introspection
Agile projects tend to follow the Lean principle of Kaizen or Continuous Improvement. This means that the team (the people actually doing the work on the project, not managers or stakeholders) regularly discuss how they can change and improve the way they work.
This is sometimes done for example at a Sprint Retrospective, in Scrum (although I think you should actually be practicing Continuous Improvement all the time, not once a sprint).
Winner: Agile. Introspection and improvement are vital to a team performing and delivering great work in the long run. Doing it once at the end of a project is too little too late, and PSGs and PWGs don’t count.
Certainty
Waterfall projects are concerned with certainty: locking down scope, defining schedules and timelines, and ensuring everything goes according to plan. While this sounds good in theory, and is often required if the work is contracted by a government agency or must follow some international standards or compliance legislation.
Of course, things can still often go awry.
The big phase gates, silo teams and late integration and testing all pose huge risks for any sizeable software project. Late discovery of defects or change requests can wreak havoc in plans, causing teams to go on “death marches” to keep to the original schedule.
And often, teams discover new scope or variations in estimation once they are part way through the project, not at the beginning. So much of the confidence in “certainty” on Waterfall projects is misleading or misplaced.
The agile alternative to certainty
Agile projects take a very different view. Rather than trying to fix scope, they try to fix cost and/or time, and let scope vary. Since the best scope items are often discovered after you have started and especially once you have made a first release to customers, they start with a very small or very vague list of scope items, and intend to change them as the project progresses.
This works extremely well for firms working on new products or entering new markets, such as startups. It is obviously less well suited to big projects where the scope is all well known and understood up-front.
Winner: Waterfall provides a much stronger degree of certainty as to what scope will be produced by the end of the work. However, keep in mind that most companies these days are operating under conditions of great uncertainty, where changing of scope and requirements can be greatly beneficial, rather than a hazard.
Summary
Overall, agile software development follows much better ideas for testing, scope, integration and sequencing, and forms smarter cross-functional teams.
If you are working under a strict government contract where all the value is known up-front and all scope must be produced exactly as per specification, then Waterfall may be a better approach. Though you should still follow the agile practices of small batches for integration and testing.
For any companies building products in competitive markets with change customer and market conditions (which is 99% of industries these days), agile is a better approach for maximising customer and company value.
