Over the last two weeks, we published three articles on vanity metrics:
One of those articles elicited an interesting discussion on Twitter. The discussion was predominately between @EtharUK (Ethar Alali) and @GregerWikstrand. The discussion started by using the blog entry from 4 Big Myths of Profile Pictures from OKCupid to illustrate vanity metrics, and then shifted to a discussion of whether vanity metrics can be recognized based on statistical validity. The nice thing about Twitter is that responses are forced to be succinct. See the whole conversation here.
The observations that vanity metrics are not necessarily unsound is important to remember. Just because a metric is statically sound does not make it useful. The four criteria to help recognize vanity metrics (gamability, linked to business outcomes, provides process knowledge and actionable) specifically did not include a statistical test because statistical validity is not discriminative for determining whether a metric is valuable or not. I have a shelf in my library that includes several statistical texts that I use in my day-to-day business (I have a large box of stats and quantitative methods books in the attic). When it comes to stats, I have game. I can comfortably say that a measure or metric can only have value IF the data is correct AND the measures and metrics derived from that data is are statistically valid.
Quantitative management requires good data. This statement seems so overtly obvious that it almost can go unstated. Almost being the critical word. While most data collection errors are not due to conscious cheating in an IT organization. The problem is typically loose or multiple definitions. This means it is difficult to get a good handle on what any piece of information means unless you do the due diligence to construct a definition that everyone understands and will use. For example, the “simple” measure of when a project ends is often different team to team. Does a project end at implementation, after some support period or at the end of the year? These are three common definitions I have observed in the same department within a single company. Garbage in, garbage out (or worse)!
Statistical validity is a discussion of whether the conclusions drawn from any specific statistical test are accurate and reliable. In order to draw correct conclusions, there are numerous statistical concepts that need to be understood, including things like a standard error, variability, and sample size. The type of test applied to a measure or metric will specify how to determine statistical validity for the conclusions drawn from the data. While statistics are one of the most popular water cooler discussions (albeit people think they are talking about sports or politics), almost no one acknowledges the depth of statistical knowledge needed to drive important decisions based on quantitative data. Note: I say this with the full knowledge that almost everyone that has been through college or university has had some formal exposure to statistics. All “good” measures or metrics must have statistical validity but just because a measure or metric is statically valid does not mean the conclusions drawn from them are useful.
Good data and statically validity are just the table stakes that get us to the point of determining whether a measure or metric can be classified as a vanity metric. Without both good data and statistical validity, any measure or metric is not valuable.