Measurement and metrics are lightning rods for discussion and argument in software development. One of the epithets used to disparage measures and metrics is the term ‘vanity metric’. Eric Ries, author of The Lean Startup, is often credited with coining the term ‘vanity metric’ to describe metrics that make people feel good, but are less useful for making decisions about the business. For example, I could measure Twitter followers or I could measure the number of blog reads or podcast listens that come from Twitter. The count of raw Twitter followers is a classic vanity metric.
In order to shortcut the discussion (and reduce the potential vitriol) of whether a measure or metric can be classified as actionable or vanity I ask four questions:
- Are there mechanisms in place to ensure the measure isn’t game-able. Does the metric reflect how work is being done or have guidelines in place so it can’t easily be manipulated without changing the outcome of the process? For example, I can buy 10,000 Twitter followers, but adding these users will not translate into blog readers or podcast listeners which is the important output.
- Do changes in the measure or metric correlate to changes in the business outcome? For example, measure of automated code coverage is typically positively correlated to product quality and to amount of value delivered. If a metric is not correlated, there is a strong possibility that the metric is a vanity metric.
- Does the metric provide an understanding of what is happening within the process being measured without confusion or ambiguity? For example, if a team measures the number test cases run and the number test cases number of increased (or decreased), what would the change mean?
- When the measure shows a change, can and will you be able to take action? If criteria 3 is true and you understand the signal being sent, can and will your organization do something about it? If true, can a decision be made? For example, an organization I recently met with measured overtime amongst developers. Coder and testers chronically put in 8 hours of overtime each week and had for over a year they either could not use the data to make a change or choose not use the data; this was a vanity metric.
If you can answer the four questions with a yes, the metric will be actionable. A no to any of the four questions generally indicates a vanity metric. Cutting out vanity metrics provides a better focus on the measures that provide value.