Audio Version:  SPaMCAST 161

When talking about metrics to measure agile processes one should begin by defining “why” you want to measure followed immediately by “what” we will do with the data when we know it. As I have noted before, there are only two reasons to measure: The first is to generate specific behaviors and second to predict the future.  Organizational goals provide the rational for what to measure and the type of measure determines whether it drives behavior or provides direction.

Change only occurs when three states are satisfied. There needs to be a trigger (1), those that are being asked to change must have the ability (2) and the motivation (3) to change.  Measurement as a tool to predict the future can provide the trigger and motivation for change.  Goals are a mechanism to order how we interpret the data generated as we measure but they are passed through many filters until they arrive at the individual where action happens.  The hierarchy begins with strategy and then is interpreted organizationally and operationally until individuals place their stamp on it and act.  Transparency and balance are required to help ensure that each interpretation stays true to the organizational goal.

Effective measurement is a balance.  On one side collection of the measures which are then synthesized into metrics.  Collection signals what is important to the organization which is balanced by the insights, actions, change and transformations that are generated.  Balance in measurement is much akin to the second law of thermodynamics.  To paraphrase, for every measure there must be an equal and opposite reaction.   A simple metrics model to enforce balance includes metrics in four categories

      1. Productivity
      1. Quality
      1. Predictability
      1. Value

The model help focus measurement in order to deliver value based on actions taken to make the business better. 

Agile measurement requires embracing seven philosophies as they relate to measurement.  Agile measurement must:

  • Reinforce desired AGILE behavior
  • Focus on results
  • Measure trends
  • Be easy to collect
  • Include context
  • Create real conversation

If measurement incents behavior, incent the right behavior!  As an example, developing a tasks level schedule for the whole project then measuring tasks completed against that schedule flies in the face of the daily planning and re-planning required to support effectiveness of self-managed and self-organized teams.   

Interim steps are far less important than results.  Does the code work?  What did we finish?  Can it be delivered? By focusing on results we are far less likely to miss the big picture.  I recently read a status report that proudly announced that the team had met 95% of the dates during the current release.  The one they missed was the final delivery date . . .

In most cases any individual observation is a reflection of the process.  Deming used the term “common cause”.   The only way to address a “common cause” problem is to change the process.  Trends are useful in describing the capacity of the process (predicting the future).  I use a rule that individual observations are studied only when they are three standard deviations away from the mean. In other words, study individual observations when they are different enough to matter; otherwise, reflect on the trend of the observations.

Collecting information, any information, takes time and effort.  Firstly, any effort that you take away from delivering functionality reduces ITs value to the company.  Secondly, there is always a resistance to work that is perceived as overhead.  Automate as much as possible to reduce the impact and make sure the information collected can be consumed at the point of collection, as well as in the hallowed halls of the PMO.

What does a number really mean?  It has been said that a set of metrics have never solved a problem and while I don’t think in absolutes I would suggest that numbers without context are far less valuable and more open to highly variable interpretations that will require even more effort to contain and manage.

Generating a conversation is a corollary to the need for context.  Conversation causes data to be synthesized and contextualized so that common organizational knowledge is generated.  Knowledge is the real value. 

Only measure what is absolutely needed.  For every metric I would ask you to think about what you will do with the information you are requesting and then how perceived overhead will affect acceptance before you act.

Principles keep us honest and focused on delivering value to the organization.  Principles that an organization develops, adapts and adopts are a reflection of what the organization values.

A subset of all of the metrics in the world makes sense in the agile environment.  The pallet the Metrics Minute suggests is as follows.  The term pallet is used as a considered opinion, a pallet suggests selecting those that are really needed to paint the picture that has value. The measures in the pallet are as follows: 

      • ROI (Value)
      • Customer Satisfaction (value)
      • Team Satisfaction (value)
      • Business Value – Burn-up (Value)
      • Velocity – Burn-down (predictability)
      • Sprint Escape Rate (Predictability)
      • Test Case Pass Rate  – Automated (Quality)
      • Technical Debt (Quality)
      • Work-In-Process (Productivity)
      • Cycle Time (Productivity)

Each of these metrics will be (or have been) covered in detail in other Metrics Minute entries so that you can determine if these metrics are useful in answering the questions you have about your agile development program. Rarely will any individual project, team or organization need to use all of the metrics in the agile metrics pallet.   The pallet is just . . .  a pallet. Pick only those that you need for balanced view of your development process.  Create your own color by blending metrics.