Chapter 11 of How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition begins the fourth and final section of the book. This section addresses topics that are beyond the basics. Hubbard titled Chapter 11 Preferences and Attitudes: The Softer Side of Measurement. The softer side is a euphemism for attitudes and opinions. Here is a summary of the chapter in a few bullet points:
· Measure opinions and feelings.
· Design out bias in surveys and questions.
· Observe opinions and feelings through trade-offs.
· Use trade-offs to describe risk tolerance.
Anyone living in the United States knows that every election year there are a plethora opinion polls. One of my favorite blogs is Nate Silver’s FiveThrityEight, which shows a wealth of statistical information about sports, economics, culture, and politics (a form of sport). Much of the data presented is a reflection of opinions and attitudes. Often they are real predictors of behavior and product success.
Hubbard points out that there are two ways to observe preferences. The first is by examining what people say and the second watching what people do. When these two measurement paths conflict we can observe the difference between stated and revealed preferences (what people say and what people do). One of the most tried and true methods of collecting stated presences is the survey. Developing surveys instruments that are not biased requires both skill and an understanding of cognitive biases. Biased questions will affect the value of the data collected. Pew Research has a wonderful article on questionnaire design that discusses some of the issues and nuances of creating questions that accurately measure what Hubbard describes as the softer side of measurement. Hubbard provides five simple strategies for avoiding response bias (any person that feels they may have to create a survey review this list). The list includes:
- Keep the questions precise and short. Short, precise questions are easy to understand thus increasing the possibility that the respondent will answer the question you want, not another question based on their internal interpretation. I would include the advice that you avoid jargon. Jargon increases the likelihood differences in interpretation.
- Avoid loaded terms. Certain terms evoke either a positive or negative response. Adding terms that evoke an emotive reaction will bias the survey.
- Avoid leading questions. The most famous loaded question of all time is, “Have you stopped beating your wife?” The respondents that aren’t married or never beaten your wife in the first place answering no or yes provides biased data. (Note: the reference to wife beating is also loaded term (strategy 2).
- Avoid compound questions. Compound questions obscure what the response means. For example, if you wanted to know which of two Agile practices a team performed and asked if they did stand-up meetings or demonstrations it would be difficult to interpret an answer of “yes”. It could mean they did one but not both.
Reverse questions to avoid response set bias. Respondents typically recognize a pattern in how a survey is asking questions and modify their behavior based on their bias. Vary the how the questions are asked and vary how the scale represented.
In most metrics programs, surveys are the go-to tool for measuring attributes, opinions and collecting behavior data. Surveys are easy to execute (although as we have seen above not easy to get right). Directly asking people what they prefer, choose, desire and feel via survey vehicles is not the only way to learn about these things.
Techniques that use how a respondent assigns a value to a feature or opinion is a measure of how someone feels. Using value as a measure helps shift the measuring of an individual’s perspective to a more objective form. There are several techniques that to observe or capture value that is useful in measuring something that is often identified as intangible. Examples of items typically thought of as intangible can be measured using value include safety, risk, human life, and time-to-market to name a few. Two different approaches using valuation to generate measurable observations include:
- Willingness to pay measures by forcing a trade-off. How much money someone is willing to pay for an object is a reflection of value. Gather willingness to pay measures by asking how much someone would pay or by observing they spend their money (or budget). One of the grocery store chains I worked for in the past used to run A/B tests varying the prices of new items or classes of products to gather data from pricing decisions. Rather than asking they observed behavior (cash register data linked to price and customer via loyalty programs).
- The value of statistical life. An interesting twist to ‘willingness to pay’ is the value of statistical life (VSL). VSL is a measure of how much a person is willing to pay for a product that was incrementally safer (trade-off between money and life). For example, would a smoker be willing to pay more for a “safer” cigarette? In a similar vein, would an organization buying a software product spend more for a product with better security?
Both of these techniques can be used to measure risk tolerance using a trade-off model. Instinctively software development and maintenance organizations use the trade-off model to manage risk every day. The practice of curtailing testing so that a product can get to market faster is a reflection of risk tolerance. The decision is a reflection of a curve that balances quality and testing effort. Would our level of risk tolerance be the same for a medical device software compared to the software needed to print shipping labels? Plotting the points where a trade-off between cost and benefits is made will generate a curve between more reward and lower risk. Using techniques like willingness to pay provides a mechanism to quantify and expose the trade-off. Hubbard uses Modern Portfolio Theory (Markowitz Analysis) and investment boundary curves that show the boundaries between acceptable and unacceptable investments as an illustration of measuring risk tolerance based on estimated ROI from different investment mixes. The concept of using to a willingness to pay to generate a risk tolerance curve can be leveraged for IT portfolio management. Use the risk tolerance boundary to determine which risk/ROI trade-off is acceptable based on the organization’s culture.
The concepts in this chapter provide a set of techniques and guidance on how to deal with addressing subjective trade-offs. The investment boundary is a type graph that in the introduction to economics courses called a utility curve (I still remember the classic bread versus bombs utility curve from my Econ 101 class). The concept of giving an opinion a value and then making trade-offs is nearly universal. For example, in almost every development organization there is often a tension between productivity, quality, and time-to-market. That tension is a reflection of a trade-off that is always going on but rarely quantified so we can tangibly understand to choices we are making.
Past installments of the Re-read Saturday of How To Measure Anything, Third Edition, Introduction
- Chapter 1: The Challenge of Intangibles
- Chapter 2: An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily
- Chapter 3: The Illusions of Intangibles: Why Immeasurables Aren’t
- Chapter 4: Clarifying the Measurement Problem
- Chapter 5: Calibrated Estimates: How Much Do You Know Now?
- Chapter 6: Quantifying Risk Through Modeling
- Chapter 7: Quantifying The Value of Information
- Chapter 8 The Transition: From What to Measure to How to Measure
- Chapter 9: Sampling Reality: How Observing Some Things Tells Us about All Things
- Chapter 10: Bayes: Adding To What You Know Now