Uncertainty of benefits versus costs
I think a lot about estimation. I used to do a lot of estimation with my teams. When the organisation I’m currently working for started doing agile, we did loads of estimation. We were constantly being asked for estimates. High-level estimates, low-level estimates, feature level estimates, story level estimates. We would spend a lot of time doing various agile games, planning poker, silent sizing. All in the name of coming up with estimates. I think a lot of that effort wasn’t very valuable.
I don’t think everybody should stop doing estimates. I’m not a member of the #noestimates movement (though I find the idea interesting). I think people doing Agile should spend a small amount of time doing estimates, but not much more than that. Because the estimates aren’t very important. That is because of the relative uncertainty of benefits versus costs.
Estimates are done for two reasons: the first is to find out when something will be finished, i.e. the duration. The second is to find out the cost.
Most of the time we did estimates, it wasn’t because someone wanted to know when it would be done. It was so they could find out how much it would cost. And they wanted to know the cost because that would tell them whether it was worth building it or not.
That line of thinking is completely stupid and wrong. Because the uncertainty of benefits is higher than the uncertainty of costs.
Return on an investment is the benefits minus the costs
You don’t need an MBA to know that the return of an investment is the benefit minus the cost. If a project earns a benefit of $100K, and costs $80K, your return is $20K. Basic stuff. So if you want to know the return of a project, you need to know the benefits, and you need to know the costs. Well, we don’t know these things, because they happen in the future and we can’t see the future. So we try and estimate them.
Generally, a product owner or project manager will ask a scrum team or a solution architect or someone to come up with a cost estimate. Meanwhile, that person will run around and put together a benefit estimate, that will go in a business case. If the business case shows that the (estimated) benefit is higher than the (estimated) cost, by a reasonable amount (taking into account some factors like risk and time value of money and opportunity cost), then the business case will be funded and the project will go ahead. So what’s wrong with this picture?
Most people focus way too much on costs and not enough on benefits
In my experience, the average amount of time that people on a project spend estimating the costs of a project is orders of magnitude greater than the amount of time people spend estimating the benefits of the project. There Which would make sense if the uncertainty around costs is much higher than the uncertainty around benefits. But it isn’t, it’s the other way around.
If you are working on anything innovative, your benefit uncertainty is crazy high
If you’re doing some boring project that is similar to a hundred other projects, or if you are doing BAU (Business As Usual) work, your benefits aren’t risky at all. But you probably shouldn’t be spending time reading this blog. This blog is about agile software development, which is a methodology used for building new software under conditions of uncertainty.
If you’re building something that has market uncertainty, and/or customer uncertainty, and/or value proposition uncertainty, which a lot of people are these days, then the uncertainty of your benefits is high. Extremely high.
That is where your real risk and uncertainty is. Not in your costs. I’ve worked on a project that estimated it would have 700,000 users; we shut it down after a year when we realised it had seven, and some of those were probably fake or hackers. I’ve worked on a project that was expected to have sales of 20 a week. Not long after launch, it had sales of 2000 a week.
You need to consider the uncertainty of benefits versus costs
What does this tell us? It tells us that for any sufficiently innovative product, the uncertainty of the benefits of that product is much higher, orders of magnitude higher, than the uncertainty of the costs of that product. This is really important. Let’s go through that again, in a simplified form:
the uncertainty of your benefits is higher (much higher) than the uncertainty of your costs.
Print that out and stick it on your cubicle wall. Turn it into a GIF and set it as your desktop background. Get your phone to send you a reminder with that text every day at 7am. Burn it into your brain. Say it out loud and say it again: “The uncertainty of your benefits is higher than the uncertainty of your costs”.
What does that actually mean though?
OK say that I’m right and that is true. So what? What does this mean?
It means you should spend a lot less time trying to reduce the uncertainty of your costs, and a lot more time reducing the uncertainty of your benefits.
If your costs could be anywhere between $400K and $800K, but your benefits could be anywhere from $0 to $5million, then the benefits are by far the more important factor in determining your return.
So why do we put so much effort into estimation, which reduces the uncertainty of our costs (but not the uncertainty of our benefits)? Why do we not spend a lot of time trying to reduce the uncertainty of our benefits?
I’m not sure why. I think it’s a form of cognitive bias, where we feel we are more in control of costs so we try and understand them better, but we subconsciously feel the benefits are just left up to fate, and nobody has any idea of what they will be or of any way to control them, so why bother trying to estimate them?
Well, we can’t estimate them very effectively (just like we can’t estimate costs very effectively), but there is a methodology that can give us some useful information about benefits without a big upfront investment: the MVP (Minimum Viable Product), a concept from Lean Startup. Go read my article about MVP if you don’t know what it is.
But we do market research!
Market research is fine but it doesn’t give you real information about your benefits, just like estimation doesn’t give you real information about your costs. Surveys and focus groups can give you some clues but they don’t tell you how people will use a real product in a real environment. True research is done by collecting data with a real product (even if it’s a basic minimum one).
You find out if people like your product by giving them a product. You find out how much something will cost by building it. This is the whole point of the Minimum Viable Product: it’s the smallest thing you can build that will give you validated learning. And that learning is generally learning about how much customers value (or will value, once you’ve built the proper thing) your product.
This gives you information about your benefits, i.e. reduces the uncertainty of your benefits, which informs your decision as to whether to continue building or not.
To go back to our previous example, your MVP might tell you that your benefits will be more in the $1million to $2million range, in which case it is definitely worth building it no matter whether it would cost $400 or $800K, or it might tell you that your benefits will be more in the $200k to $400K range, in which case it is not worth building it no matter whether it would cost $400K or $800K.
But we can’t estimate based on data, it has to be guesses!
You’re probably thinking “but this is a trap! If we can only estimate once we’ve built the thing, how can we decide to build it or not, since we only know how much it will cost by building it?”.
Well, remember, your benefit risk is higher than your cost risk. Your cost estimation might be off by +/- 50%, but your benefit estimation could be off by +/-95% (or more). Also, estimation doesn’t have to be done in one big chunk at the beginning. If you’re doing agile, you should be breaking things down into pieces. You can then estimate them in pieces, as you go, rather than all in one lump at the beginning.
For example, you could break your work into “MVP, feature set A, feature set B, non-functionals”. Then you estimate the MVP (but not anything else), then build it (that might tell you whether it is worth even bothering to estimate feature set A or B, or which one is more important).
You also have information about how hard this thing is to build. You can then estimate feature set B for example (if your MVP tells you that is worth building). And maybe then non-functionals, once you’ve build feature set B. As you go, you learn more about the domain and the product, and your estimate will be more accurate and valuable. Trying to estimate everything up front is a waste of time because:
- you might not build all of it anyway (you might end up building hardly any of it, and that’s fine, remember, it’s not a crime to kill a bad product)
- you will learn a lot as you go and estimates done further down the track will be more valuable than ones done at the beginning.
So we should all stop doing estimates, right?
No I am not saying that we should all stop doing estimates for software development. We need to rethink why we are doing it and make sure we are doing it for the right reasons. And make sure we are doing other activities to reduce our benefit uncertainty. Remember, estimation can be done for two reasons:
- to tell us when something might be delivered
- to tell us how much something will cost i.e. is it worth building.
If you are doing it for the first reason, fine, though remember, an estimate is a forecast not a commitment. And the earlier the estimate is made, the less information is available to inform it. So the less valuable it will be and the less you should use it to enforce dates.
If you are doing it for the second reason, make sure you are spending about ten times as much time and effort on reducing the uncertainty of your benefits as you are in estimating the cost. Because that is where most of your uncertainty actually lies.
