
How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition
Chapter 6 of How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition, is titled: Quantifying Risk Through Modeling. Chapter 6 builds on the basics described in Chapter 4 (define the decision and data that will be needed) and Chapter 5 (determine what is known). Hubbard addresses the process of quantifying risk in two overarching themes. The first theme is the quantification of risk and the second is using the Monte Carlo analysis to model outcomes.
Risk is the possibility that a loss or some other bad thing will occur. The possibility that something equates to uncertainty. Risk is often expressed in qualitative terms such low, medium, high and the ever popular really high rather than in quantified terms. However, that qualitative approach generates ambiguity. Qualitative designations provide little measurement value outside of a sort of measurement placebo effect. Even though an absolute value for risk might not be knowable, risk can be quantified as a range of values. The qualification of risk is important, both in terms of understanding risk and in terms of usefulness for defining overall outcomes. Defining a range of risks makes understanding the amount of risk in any decision less ambiguous. Quantifying risk is also the foundation of further measurement needed for decision-making. Hubbard hammers home the point that measurement is done to inform some decision that is uncertain and has negative consequences if it turns out wrong.
The goal of Monte Carlo analysis is to provide input for better decision making under uncertainty. When you allow yourself to use ranges and probabilities, you really don’t really have to assume anything you don’t know for a fact (Chapter 5 showed us how to estimate based on what we know). All risks can be expressed by the range of uncertainty on the costs and benefits and the probabilities of events that may affect them. Turning a range of estimates into a set of predicted outcomes requires math. Monte Carlo analysis is a mathematical technique that uses the estimated uncertainty in decisions to furnish a range of possible outcomes and the probabilities they will occur.
Monte Carlo analysis can incorporate a wide range of scenarios and variables. Hubbard points out that it is easy to get carried away with the detail of the model. Models should only be as sophisticated as needed to add value to a decision. Remember, as Gene Hughson of Form Follows Function says, “all models are wrong.” Models are abstractions of real life, there is always detail you leave out, no matter how sophisticated the model becomes. Hubbard suggests that model users always ask whether a new more complex model is an improvement on any alternative model. Quantification provides a platform for making consistent choices in order to clearly state how risk-averse or risk tolerant any organization really is.
Hubbard closes this chapter by stating a risk paradox.
“If an organization uses quantitative risk analysis at all, it is usually for routine operation decisions. The largest, most risky decisions get the least amount of risk analysis.”
The combination of estimation (Chapter 5), quantifying risk and Monte Carlo analysis may seem complex keeps many decision makers from using the technique, this is especially true in software development, hence the paradox. For example, every software development estimation problem, whether Agile, lean or plan based, has a large degree of uncertainty embedded in the process and therefore, is a perfect candidate to use Monte Carlo analysis. However, very few estimator understand or use the technique. Learning Monte Carlo analysis (and using the one of the many tools to do the mathematical heavy lifting) or alternately hiring some to perform risk analysis are both paths to addressing adding quantitative data to decision making. When making decisions under conditions of uncertainty, Monte Carlo analysis is a necessity to do the math needed.
Previous Installments in Re-read Saturday, How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition
How To Measure Anything, Third Edition, Introduction
Chapter 1: The Challenge of Intangibles
Chapter 2: An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily
Chapter 3: The Illusions of Intangibles: Why Immeasurables Aren’t
Chapter 4: Clarifying the Measurement Problem
Chapter 5: Calibrated Estimates: How Much Do You Know Now?
January 24, 2016 at 1:20 am
[…] Chapter 6: Quantifying Risk through Modeling […]
January 24, 2016 at 5:30 pm
I have drank the kool-aid on this stuff. Forecasting using Monte Carlo simulation is a much better way.
Author Daniel S. Vacanti also has some words to say about this in his book “Action Agile Metrics for Predictability”, and Vacanti’s book also references Hubbard’s HTMA book.
Hubbard talks about selecting the correct probability distribution for you Monte Carlo simulation. Vacanti states you need worry about this if you have the data. And Vacanti’s book is all about collecting the data for processes (incoming and outgoing).
I use this exact approach to collect data about the scrum sprint processes. The clock starts when a user story is accepted into a sprint and the sprint begins. The clock ends when either that user story is deployed into production, postponed, or returned into the Backlog. Start – Stop cycle-time is what I report on, but I also collect two other intermediate events – dev-complete and business accepted to help figure-out how-to reduce the end-to-end cycle-time.
I then use this data to forecast using Monte Carlo simulations with the help of Daniel S. Vacanti’s Actionable Agile online tool.
March 31, 2016 at 10:57 pm
Wonderful feedback . . . perhaps we should talk about your approach on the podcast?
January 31, 2016 at 12:05 am
[…] Chapter 6: Quantifying Risk Through Modeling […]
February 6, 2016 at 11:56 pm
[…] Chapter 6: Quantifying Risk Through Modeling […]
February 13, 2016 at 11:57 pm
[…] Chapter 6: Quantifying Risk Through Modeling […]
February 16, 2016 at 11:55 pm
[…] an organization can craft a balanced set of Agile metrics. The goal of any measure is to reduce the uncertainty the organization is facing. Value and portfolio health metrics shift the discussion from the […]
February 27, 2016 at 11:56 pm
[…] Chapter 6: Quantifying Risk Through Modeling […]
March 5, 2016 at 11:56 pm
[…] Chapter 6: Quantifying Risk Through Modeling […]
March 12, 2016 at 11:58 pm
[…] Chapter 6: Quantifying Risk Through Modeling […]
March 19, 2016 at 11:56 pm
[…] Chapter 6: Quantifying Risk Through Modeling […]
March 26, 2016 at 11:56 pm
[…] Chapter 6: Quantifying Risk Through Modeling […]
April 2, 2016 at 11:57 pm
[…] Chapter 6: Quantifying Risk Through Modeling […]
March 9, 2017 at 11:55 pm
[…] Chapter 6: Quantifying Risk Through Modeling […]