
How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition
Chapter 5 of How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition, is titled: Calibrated Estimates: How Much Do You Know Now? Chapter 4 described how to define the decision that needs to be made and the data that will be needed to make that decision. Chapter 5 builds on the first step in Hubbard’s measurement processes by providing techniques in order to determine what you know as you begin the measurement process. Hubbard addresses two major topics in this chapter. The first is using estimation to quantify what you know, and second is his process for calibrating estimators.
Measurement is a tool to help make decisions. Once you have determined what decision you need to make and what data you need to do so, the next step is to determine how much you know. In many cases, it is easy to come to the conclusion that since you have not started measuring, you don’t know anything or you don’t know anything precisely enough to be useful. However, even if you don’t know exactly, you still know something. Estimation is a tool to establish a baseline of what is known. For example, using the recent gyrations of the Shanghai stock market as an illustration if you know what is absurd value for a daily drop and increase you can start to define a range. Until recently, the when the market falls by 7%, trading would be suspended. Knowing that gives us a lower boundary for the measurement of change in that market. We could establish an estimate of the upper limit by asking whether we would be confident that a 1,000% increase would be absurd. It would be easy to go through a process of ratcheting up or down until the answer is ‘yes that is probably absurd’ thereby establishing an upper boundary. Perfect? No, but it is a starting estimate and we can then collect data and use the feedback to improve what we know.
Estimates are generally stated as a range. For example, I am 90% confident that I will wake up unaided between 3 – 5 AM local time (unless I ingest sleeping aids, like pizza, right before bed). Expressing what we know about a number or variable as a range of probable values provides a confidence interval that we can assign a degree of confidence. We think the real value of a number is somewhere in that range. We can determine how good we are at subjective probability assessment by comparing our expected outcomes to actual outcomes. As Hubbard points out, the rub is that very few people are accurate estimators unless they have been trained, or as Hubbard calls it, calibrated (the book has several footnotes supporting this assertion and the Freakonomics podcast of Jan 14, 2016, also addresses this issue)
Almost everyone tends to be biased either toward overconfidence or underconfidence. Hubbard observes (and I agree) that the vast majority of estimators are overconfident. Overconfidence is defined as routinely overstating knowledge and being correct less often than expected. Alternately, under-confidence is when an individual understates knowledge and is correct much more often than expected. Estimating, or assessing uncertainty, is a skill that can be learned and improved. As an example, Hubbard a set of questions and asks the reader to estimate (as a range) answers at a 90% confidence. The set of questions shown in the book is similar to the types of questions Hubbard uses in his calibration workshop. Even in the exercise in the book, some questions seem too difficult to answer. However, using the boundary estimation technique used earlier I was able to harness what I did know. For example, I know it would be absurd to estimate the lower boundary of possible year Shakespeare was born before 1 AD.
Once a baseline is established there are a number of calibration techniques and tricks that you can learn (See Exhibit 5.4 in the book for the list). However, the first discussed in the book was to establish consequences for misestimating (this is very similar to the discussion on the Freakonomics podcast noted above). Hubbard uses the mechanism of betting money to establish a consequence for how well an estimator performed. Betting (real or pretend money) provided a feedback loop that provided significant calibration improvements. Another method involved asking people to identify potential problems for each of their estimates. For example, ask the estimator to assume his or her answer/estimate is wrong then and then explain why it was wrong. This technique is useful for helping to expose the unvoiced assumptions and biases, so they can be taken into account. I have recently started using this technique with colleagues on a board of directors I chair. In most cases it generates a discussion that helps tune estimates and decisions. These two are just a sample of the techniques noted in the book; Hubbard recommends that estimators learn and understand all of the techniques.
The process of determining what we know is a process of establishing a range (uncertainty) for the variables we intend to measure. Defining a range is an estimation problem. There is a wide range of techniques to establish what we know and our level of uncertainty. Initially, almost all estimators are not great at the process; however, we can improve our ability to subjectively assess what we know and then improve through calibration.
Previous Installments in Re-read Saturday, How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition
How To Measure Anything, Third Edition, Introduction
Chapter 1: The Challenge of Intangibles
Chapter 2: An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily
Chapter 3: The Illusions of Intangibles: Why Immeasurables Aren’t
Chapter 4: Clarifying the Measurement Problem
January 17, 2016 at 5:55 pm
[…] Chapter 5: Calibrated Estimates How Much Do You Know Now […]
January 17, 2016 at 7:29 pm
I learned a new skill reading this chapter and practicing the sample calibration tests.
My approach for using the calibration tests in this chapter and in the appendix of the book, is to take 5 questions at-a-time. That is, if you are time challenged.
I was happy that my 90% range did cover the actual value of the average percentage of Design in Software projects.
And I also recommend the Feakonomics podcast titled “How to Be Less Terrible at Predicting the Future” (January 14, 2016)
January 17, 2016 at 10:06 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. In Chapter five, we discussed estimation, calibration and what we know […]
January 23, 2016 at 11:57 pm
[…] builds on the basics described in Chapter 4 (define the decision and data that will be needed) and Chapter 5 (determine what is known). Hubbard addresses the process of quantifying risk in two overarching […]
January 24, 2016 at 10:11 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. In Chapter Six, we discussed using risk in quantitative analysis and the Monte Carlo […]
January 31, 2016 at 12:05 am
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
January 31, 2016 at 9:00 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. In Chapter Seven, we discuss the concept of the economic value of […]
February 6, 2016 at 11:56 pm
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
February 7, 2016 at 9:52 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. In Chapter Eight, we begin the transition from what to measure to how to […]
February 13, 2016 at 11:57 pm
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
February 14, 2016 at 9:50 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. In Chapter Nine, we tackle […]
February 21, 2016 at 10:14 pm
[…] We take a break for Podcamp Toronto and to begin the process of picking the next book. What are your suggestions? In the meantime catch up on the re-read of How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. […]
February 27, 2016 at 11:56 pm
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
February 28, 2016 at 9:56 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. In Chapter 10 we visited how to use Bayesian Statistics to account for having prior knowledge […]
March 5, 2016 at 11:56 pm
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
March 6, 2016 at 10:06 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. Chapter 11, begins section four of the book and is titled Preferences and Attitudes: The Softer […]
March 12, 2016 at 11:58 pm
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
March 13, 2016 at 9:58 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. In Chapter 12 we discussed The Ultimate Measurement Instrument: Human Judges. Humans can be a […]
March 19, 2016 at 11:56 pm
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
March 20, 2016 at 9:16 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. In Chapter 13 we discuss New Measurement Instruments for Management. Hubbard shifts gears in […]
March 26, 2016 at 11:56 pm
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
March 27, 2016 at 10:16 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. Chapter 14 is titled A Universal Measurement Method. In this chapter Hubbard provides the […]
April 2, 2016 at 11:57 pm
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
April 3, 2016 at 9:13 pm
[…] Finding the Value of “Intangibles in Business” Third Edition by Douglas W. Hubbard on the Software Process and Measurement Blog. In this week installment, we summarize our major takeaway and identify what we can do to improve […]
March 9, 2017 at 11:55 pm
[…] Chapter 5: Calibrated Estimates: How Much Do You Know Now? […]
September 19, 2018 at 4:03 pm
[…] not done them before or never received formal training on how to do estimations correctly, eg. calibration training. Who knew estimation training was a thing right? It is also naive to think that estimates was just […]
March 7, 2020 at 6:15 pm
[…] consistent and defensible way. Today, however, Open FAIR incorporates Monte Carlo simulation well as calibrated estimation and other techniques that will bring information risk management into the […]