Measure Twice or Good Numbers Can Go Bad
Mark Twain once said “There are lies, damn lies, and statistics.” The same numbers can be used to support many causes. Even though numbers are just numbers, they can be used to tell a story.
What you do with the messages developed from the metrics you collect is important in its own right. Messages become tools (or weapons) to motivate. Motivation can range from the positive (look how good you are doing) to the negative (look how bad you are doing) or the ultimatum (do better or else). Here we will discuss what happens when there is no message or where the message and data aren’t synchronized (part two!)
Using Team Measures for Individuals:
Measurement is an intimate subject, because it exposes the person being measured to praise or ridicule. Management many times will begin with a group or team level focus only to shift inextricably to focus on the individual. The individual-view is fraught with difficulties, such as gaming and conflict, which typically becomes the norm for causing anti-team behavior, which, in the long run, will reduce quality, productivity and time-to-market (short-term gains, but long-term pain). The focus of measurement must stay at team level for measures that focus on the results of team behavior and only evolve to individual measures when measure relates to the output of an individual.
Don’t We Want To Be Average?:
Another classic mistake made with numbers is regression to the mean. Performance will tend to approximate the average performance demonstrated by the measures chosen. A method taken to address the mistaken is to:
1. Select the metric based on the behavior you want to induce,
2. Set goals to incent movement in the proper direction and away from average.
The proper direction is never to be average in the long run.
It’s All About People:
It is difficult to ascribe a motive to a number. It’s merely the tool of the person wielding it. Put a number in a corner and it will stay there minding its own business not attracting attention or detracting from anyone else. Add a person and the scenario begins to change. The wielder, whether Lord Voldemort, Dumbledore or some other high lord of metrics, becomes the determining factor on how the number will be represented. Measures and metrics can be used for good or evil. Even when the measures are properly selected and match the organization’s culture, “badness” can still occur through poor usage (people). There is one theory of management that requires public punishment of perceived laggards (keel haul the lubber) as a motivation technique. The theory is that punishment or the fear of punishment will lead to higher output. In real life, fire drills (team running around like crazy to explain the numbers) are a more natural output which absorb time that could be used to create value for IT’s consumers. The fire drills and attempts to game the numbers reduces value of the measurement specifically and management more generally.
Are Report Cards A Silver Bullet?:
Report cards are a common tool used to present a point-in-time view of an organization, project or person. At their best, report cards are benign tools, which are able to consolidate large quantities of data into a coherent story. However, creating coherence is not an easy feat. It requires a deft hand that allows a balanced, comprehensive view that is integrated with what is really important to the firm and the entity being measured. Unfortunately, since this is this difficult, often a stilted view is given based on that data which is easily gathered or important only to a few parties. This stilted view is surefire prescription for when Good Numbers Go Bad. The solution is to build report cards based on the input from all stakeholders. The report card needs to include flexibility in the reporting components so they can be tailored to include the relevant facts that are unique to a project or unit without disrupting its framework. Creating a common framework helps rein in out-of-control behavior by making it easy to compare performance (peer-pressure and comparison being the major strength of report cards).
Most of us have been introduced to report cards during school. In most cases, they were thrust upon us on a periodic basis, the report card presenting a summary of the basic school accounting. While we did not get to choose the metrics, at least we understood the report card, and the performance it represented seemed to be linked to our efforts. Good Numbers Go Bad when corporate report cards are implemented using team-level metrics as a proxy for individual performance. As I noted above, balance is critical to elicit expected behavior as well as application of metrics at the proper level of aggregation (teams to teams, people to people). Team metrics present information on how the whole team performed and was controlled, unless the metrics are applied to the unit that controlled performance it misses the mark.
The Beatings Will Continue Until . . .:
“One characteristic of a bad metrics program is to beat people up for reporting true performance.” — Miranda Mason, Accenture
Terms like “world-class” and “stretch” get used when setting goals. These types of goals are set to elicit teams or individuals to over-perform for a period of time. This thought process can cause inappropriate behaviors, in which the goal seekers act more like little children playing soccer (“kick the ball and everyone chases the ball”) rather than a coordinated unit. Everyone chases the ball rather than playing their position like a team,. Goals that make you forget teamwork are a perfect example of when Good Numbers Go Bad. Good measurement programs challenge this method of goal setting. Do not be impressed when you hear quotes like” we like to put it out there and see who will make it.”.
Goals are an important tool for organizations and can be used to shape behavior. Used correctly, both individual and team behaviors can be synchronized with the organization’s needs. However, when used incorrectly, high pressure goals can create opportunities for unethical behavior. An example of unethical behavior I heard recently was in an organization that promoted people for staying at work late. The thinking was that working more hours would increase productivity. In this case, a manager would check in at approximately 8 PM every evening ostensibly to do something in the office, but in reality to check to see who was in office. In this case, many people did nothing more than to go out to dinner then come back to work or just read a newspaper until the appointed hour. When the manager checked, there were many people working away at their desks. I suspect that little additional time was applied to delivering value to the organization or their customer. The manager should spend time determining the behavior (good and bad) that can be incentivized as he set metrics and the goals for those metrics. Spending the time on the psychology of measures will increase the likelihood that you will get what you want.
As noted earlier, the way numbers are used has a dramatic impact of whether long-term good will is extracted from the use and collection of metrics. The way numbers are used is set by the intersection of organizational policy and politics. The mere mention of the word politics connotes lascivious activities typically performed inside a demonic pentagram. However, all human interactions are political in nature. Interactions are political and organizations are collections of individuals. Many times the use of the word “political” is a code to indicate a wide range of negative motives (usually attributed to the guy in the next cube) or to hide the inability to act. When you are confronted with the line “we can’t challenge that, it is too political,” step back and ask what you are really being shown. Is it:
- • lack of power or will;
- • lack of understanding, or
- • lack of support?
Once you have identified the basis for the comment, you can build a strategic response.
Metrics, A Tool For Adding More Pressure?:
There are many types of pressure that can be exerted using metrics. Pressure is not necessarily a bad thing. Rather, it is the intent of the pressure that is the defining attribute of whether the metric/pressure combination is good or bad. Good Numbers Go Bad when measurement pressure is used to incent behavior that is outside of ethical norms. Pressure to achieve specific metrics, rather than a more constructive goal, can create an environment where the focus is misplaced. School systems that have shifted from a goal of creating an educated community to the goal of passing specific tests are a good example. An IT example that was once described to me was in an organization that measured productivity (with a single metric problem described before) in which a project gamed is productivity performance by under-reporting an effort (actually they hid it in another project). As in the discussion of using single metrics, creating a balanced view that targets organizational goals and needs is the prescription. When a balanced approach is applied, pressure can be applied to move the team or individual towards the organizational goals in a predictable (and ethical) manner.
- Measure Twice or Good Numbers Can Go Bad