Jason Yip (@jchyip) commented on my post, Why Measure, pointing out the problems with many of measures that are used, and therefore the unintended behavior they cause because the wrong question is being asked. Asking the wrong question (or seeking the wrong answer) is exacerbated when measures and metrics are not linked to organizational goals. Measures and metrics that are not closely linked to the organization’s commitments cause the wrong behavior. The only way to truly know if your measures and metrics are linked to the organization goals is to map them back to the goals. There are several techniques for determining whether linkages exist. We begin simply with a mapping/matrix approach using a spreadsheet.
Here is the Spreadsheet-Based Metrics Mapping Process:
- Identify the organizational goals and objectives. While this should be as simple as asking the first person you run into, generally collecting the goals and objectives is more problematic and can generate quite a bit of conversation (a benefit in its own right). If you have to mine for the objectives, review the annual report or interview senior managers. As a general rule you are looking for the overall organization’s goals rather than those for a department. However, if you are serving an autonomous division within a conglomerate, your sights might be lower.
- List the organizational goals across the top of the spreadsheet.
- Inventory all the metrics and any measures that are in a standalone mode (A measure is a concrete or objective attribute that can be directly measured. Metrics are the combination of two or more measures or metrics, such as miles per hour).
- List the metrics and measures down the slide of the spreadsheet. The resulting matrix will look something like this:
- Convene a cross-functional team of approximately 5 -7 people (include management, team members, product owners and other IT stakeholders) and fill in the matrix. This is generally a two-pass process.
- Pass 1: Walk through each measure and develop a consensus of whether it directly incentivizes behavior that supports the goal and strength of the support that is generated. Use blanks and Harvey Balls to indicate degree of support visually. A blank represents no support and a fully closed in Harvey Ball as strong support. Time box this step to no more than 2 minutes per metric (average), because you want initial reactions.
- Pass 2: Challenge the team to walk through each metric and measure and to name the behavior they believe the metrics will incent. Then ask them to revisit or refine their initial observations.
Note: I consider the order important. The first step helps to identify participant’s natural, non-intellectualized view of the metric or measure, and also creates a minor anchor bias. The second step allows the participants to express their intellectual positions and to challenge the potentially more emotional position they may have initially espoused.
The outcome would look like the following:
- Where the linkages between metrics and goals are not strong and specific there are three options:
- Stop collecting the metric (this is my favorite IF there is no or just a tenuous linkage, or if there are other stronger links. Less is generally more in measurement).
- Change or refine the metric if you absolutely need the refined metric to cover a specific goal or to incent a specific behavior.
- Continue despite the lack of linkage and waste everyone’s time (this is usually driven by poor internal politics).
The mapping/matrix approach is easy to deploy and collaborative. Involving all the stakeholders in the measurement process in the mapping process creates a substantial level of buy-in. The process of refining the links between measurement and goals can generate a lot of information about the organization. For example, the process will expose whether all levels in the organization know what the organizational goals are and then whether they are progressing against them.