How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition

Chapter 8 of How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition, begins the third section of the book.  Part III is focused on Measurement Methods.  Chapter 8 is titled, The Transition: From What to Measure to How to Measure. This is where you roll up your sleeves, crack your knuckles and get to work. Whenever you are beginning something new, the question of where to start emerges.  If I were to summarize the chapter in three sentences I would say:

  1. Understand the concepts of measurement and error.
  2. Break what you are measuring down into smaller pieces.
  3. Never reinvent the wheel; research how others have measured what you are interested in measuring.

In measurement, we collect data by observing something happening. Our instruments could be a survey, an experiment, a gauge, or many other mechanisms. Instruments have varying levels of precision.  For example, measuring the components of water using chromatography is far more precise than an employee attitude survey. Your choice of instrument will be driven by how much uncertainty the measurement needs to remove.   Remember the goal of measurement is to reduce uncertainty to an acceptable level.  We use standardized instruments because they are consistent and can be calibrated to account for certain types of error.  For example, a colleague developed a sizing instrument for software that was always 12% low when compared to a full IFPUG function point count. Knowing the error allowed him to calibrate the instrument.

Some variables are hard to measure because they represent a nebulous concepts or processes such as the value of information technology.   Decomposition is a step that starts with the variable you want to measure and then breaks it down into its component parts. That way you can identify where there is uncertainty, which parts are observable, which are “easy” to measure and which parts have value.  As noted in Chapter 7, not everything that is measurable has economic value.  Hubbard points out that there is a decomposition effect.  The effect is that as we decompose a metric it is possible to learn enough not to need new observation. Quoting Hubbard:

“The entire process of decomposition itself is a gradual conceptual revelation for those who think that something is immeasurable.”

It is easy to fall into the trap of believing that your measurement problem is unique. Hubbard suggests that we start the measurement process with the assumption that someone else done this before us.  If someone has developed a measurement solution (or close) for what we need, then we can tap into those ideas with some research.  The internet and library provide a rich source of secondary sources. Start by searching on your topic while including terms that will help filter fluff articles.  For example, if you are looking for information on measuring software productivity include terms like data, correlation or tables to the search criteria.  Consider trolling the reference links in Wikipedia articles.  Also, consider reviewing the bibliographies of somewhat related articles.

Once the research has been done observations need to be made to collect data. One technique for determining how to measure, as suggested by Hubbard, is to describe in detail how you see or detect the object being measured. This step is not always easy, especially for anyone that believes any specific object or concept can’t be measured. One the important pieces of advice from Hubbard in when generating a description is difficult, is that if you have any basis for the belief that an object exists, you are observing it.  Describing how you or detect something will provide a strong sign of how to observe the variable.  Determining where to start measuring is like hunting for the first few pieces of a jigsaw puzzle (the fun kind – not the one my wife has going that is just varieties of green and gray).

Here are some of the questions that Hubbard uses to begin:

  • Does it leave a trail of any kind? Almost all phenomena or processes generate some evidence that they occur.  For example, seeing a contrail is evidence that jet has passed overhead.
  • If the trail doesn’t already exist, can you observe it directly or at least, a sample of it? I am often struck by the shock on people’s faces when I suggest that they actually go and observe what is happening. Early in my career in process improvement, we had data at showed a large productivity fall off in key punch operators beginning 30 minutes before lunch.  We had no audit trail that “told” us what was happening and did not understand until we actually watched and measured what was happening.  Let’s leave the answer at a poor application design and move on.
  • If it doesn’t appear to leave behind a detectable trail of any kind, and direct one time observations do not seem workable, can you devise a way to begin to track it now? For example, the EU built the Large Hadron Collider to discover and the Higgs boson particle.
  • If tracking the existing conditions doesn’t suffice, can you force the phenomena to occur under conditions that allow easier observation? Experimentation in the corporate environment is not always easy and often times are not perceived as fair.  Remember, when experimenting on projects fairness issue might be as broad as who is allowed to participate in the experiment or as nitpicky as why team members are asked to expend effort on collecting extra data. Dan Airily, of the Wall Street Journal, in a recent blog addressed the fairness issue noting across an organization (or between teams), “ If you can figure out how to frame them as fair, they might become more palatable.”

All measurements include error.  There are two basic types of error. Systematic are those that are consistent and not just random variations from one observation to the next.  The error in the software measurement example earlier represents a systematic error.  You can account for systematic error through careful calibration.  The second type of error is random error.  Individual observations influenced by random error can’t, by definition, be precisely predicted. Understanding the amount and types of error present in any measurement affects its value.  While you can’t remove all error, that does not mean that you should not understand (mathematically) the error present and to get rid of the amount of error that is economically feasible.

Measurement done on a formal, systematic basis provides information that has value for making decisions.  A colleague has a business that, as a third party, measures the amount of software produced by a development organization and delivered to the client.  Based on that size the client pays the producer. In this example, measurement is used to make a payment decision (and often the estimated size is used to make a purchasing decision). All parties monitor the amount of systemic and random error in the transaction so that the cost and precision meet the need of all parties.  In earlier installments we discussed the economic value of perfect information (EVPI), in this example, I have been told that my colleague and his clients have discussed the EVPI of the information and that the actual cost is far below the EVPI.  The goal is to measure just enough to reduce their uncertainty in the transaction to acceptable levels by focusing on the observable portion of the transaction that is relevant to both parties.  I suspect that they all read this chapter before agreeing on what to measure.

Previous Installments in Re-read Saturday, How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition

  1. How To Measure Anything, Third Edition, Introduction
  2. Chapter 1: The Challenge of Intangibles
  3. Chapter 2: An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily
  4. Chapter 3: The Illusions of Intangibles: Why Immeasurables Aren’t
  5. Chapter 4: Clarifying the Measurement Problem
  6. Chapter 5: Calibrated Estimates: How Much Do You Know Now?
  7. Chapter 6: Quantifying Risk Through Modeling
  8. Chapter 7: Quantifying The Value of Information