Excerpt from When Good Numbers Go Bad
Thomas M. Cagley Jr.

(Recorded for the Software Process and Measurement Cast 32 – www.spamcast.net)

When Communication Makes Good Numbers Go Bad

 

One of the most tragic errors young metrics programs make can be classified as the Field of Dreams Syndrome; “measure it and they will find it useful.”  The issues that this type of thinking causes are typically recognized first as communication or training problems.  Questions surface such as: “Why isn’t anyone using our measures?” or “Why isn’t anyone interested?”  Dashboards and reports are created, but no one cares.  At least two problems are generally present, insular vision and lack of validation. 

 

Monologues:

 

Late night television is the home of the monologue.  Jay Leno and David Letterman use monologues to make us laugh.  Their only feedback is the laugh track.  The unidirectional flow of the information is an important feature of a monologue.  Late night comedy and metrics presentations really should have little similarity (albeit a bit of levity is probably a good thing).  Most metrics reports and presentations are approached as if they were monologues rather than dialogs. 

 

The monologue approach occurs for a number of reasons.  The first is the confusion of the volume/value attribute.  Metrics programs need to show value, and the two attributes of volume and value are sometimes confused (recall old bromides like ”the more the merrier”).  When these concepts are confused, it seems that the goal of a metrics presentation seems to be to show every bit of data ever collected crammed into charts (or slides) and then to tell anyone who will listen the perception of what they mean ( also known as death by slides).  Focusing on volume chokes the ability to hold a dialog.  It should be noted that volume and quality are unrelated attributes.  An old adage states, ”a designer has achieved perfection not when there is nothing left to add, but when there is nothing left to be taken away.” (Read any or all of Edward Tufte’s books.)  Design your presentation with the aim of evoking action by the recipient.  Simplicity and minimalism are concepts that need to be used when designing your presentation tool (show pictures but have the data).  Once you have a tool to aid your communication, the next step is to use the tool to facilitate a dialog as the basis for creating understanding (bi-directional).  A dialog (defined as an exchange of information and understanding) provides a platform for the metrics team to affect behavior the behavior of the organization and to absorb information about how work is being done.  Wikis and blogs are means of creating this type of dialog.

 

Another idea to combat monologue is to recognize that presentations and handouts are not the same thing. Presentations are structures to create dialogs; handouts are one way vehicles, monologues.

 

Beliefs:

 

Beliefs act as a powerful filter that can promote communication problems.  Deep-seated beliefs force the believer into a difficult position when it comes to challenging the status quo.  When change occurs without the ability to challenge, it leads to confusion and possibly to conflict.  This is a scenario where Good Numbers Go Bad.  Beliefs do not have to be based on mathematical or scientific fact but can be driven by common understandings of how things work which may or may not be correct.  Once upon a time most people believed the world was flat.  This belief constrained behavior.    

 

As an example, I was recently exposed to a senior executive who firmly believed that education and training were not related to improved capability of his organization.  If acted upon, the outcome of this belief will potentially be lower productivity, lower innovation, lower capability and potentially the need to outsource.  The workforce would not stay current or gain new skills, creating a spiral downward.  I certainly wish I could have asked whether the executive thought it was important for his children to be educated and whether that education would impact their future capability.  Facts and the relationships between facts can and are abridged through beliefs.  Metrics professionals must continually create awareness so that everyone in the metrics equation keeps an open, questioning mind to extract the full value from numbers.

 

Just Plain Wrong:

 

One of the final classes of communication errors occurs when the information gleaned from a chart, graph or single number, is published by the metrics team and is wrong.  In my mind the most frightening words are “my interpretation of this graph is that the earth is flat.”   Misinterpretation can be caused by a number of problems ranging from education and knowledge of the interpreter, active misinterpretation or the act of spreading misinformation (or that belief thing in the last section).  Regardless of why the interpretation is wrong, the damage is done.  As soon as the misinterpretation is perceived the metrics program will be viewed as non-neutral and potentially biased.  When measurement drives activity based on misinterpretation, the results can be erroneous business decisions with lasting implications.  Motivation is discussed later; however, when the reason for the error is perceived to be caused by misinterpretation, a bad taste will be left in people’s mouths for a long period of time.

 

Zombie Hypothesis:

 

One of the worst errors made by humans is not publicly recognizing a mistake and trying to tough it out.  The affliction can be encapsulated by the phrase, “throwing good money after bad.”  When applied to a metrics program, this affliction can lead to a scarcity of funds for metrics and SPI investment opportunities.  Not facing up to your mistakes causes a scenario where Good Numbers Go Bad.  The cost and effort needed to gather, analyze, report and react to the measures being collected will eclipse the value derived if you are living a lie.  The Zombie Hypothesis is a variant of the Law of Crappy Process which implies that the worst, most incorrect data will become the de facto (italics) standard (real or perceived) for your measurement program.  When a problems is found, recognize it, fix the process(es) then the definitions and re-implement the measurement.  The effectiveness and efficiency of the measurement program will be improved.  More importantly, you will inhabit the moral high ground of knowing you are measuring the right thing in the right way.