Bringing in the big guns, or the Bigfoot...

Bringing in the big guns, or the Bigfoot…

In many cases the simple matrix approach to identifying the links between organizational goals and metrics will suffice, however when a measurement program doesn’t exist or if it has gone totally off the tracks, more disciplined approaches can be VERY useful. The most famous of the disciplined approaches/frameworks is Goal, Question and Metric (GQM) popularized by Dr. Victor Basili. Others frameworks such as the Practical Software Measurement (PSM) draw heavily from the work done on GQM. The goals all of these frameworks are to provide a means to link data collection to the questions generated to support business goals.

A typical GQM approach follows the following steps:

  1. Develop a cross functional team of 5 – 7 and coach/facilitator.  The team should a mixture of disciplines.  Generally the team represents a cross section of IT management with support from someone that will lead the measurement team after the process is complete. The cross functional advice is critical if the metrics you are going to develop need to encompass all the functions within a typical IT organization.
  2. Identify and document the organizational goals. This is thought of as the conceptual level in the GQM. The process coach should ensure that the team has a solid understanding of the goals. An example of a goal might be:

Improve our responsiveness to customer issues.

  1. Develop a set of question that, when answered, will assess how each of the goals are being met. Generally each goal generates several questions. Examples:

What is the current average age of customer-reported defects?

What is the defect reporting rate after projects are released to customers?

  1. Review the overall set of goals and questions. I am a big fan of minimalism. Keeping the set of questions to a minimum helps to minimize the overhead needed to support data gathering and analysis. Eliminate any redundant questions.
  2. Identify metrics that will provide data that is needed to answer each of the questions. There is generally a many to many relationship between questions and metrics (a question might require a number of metrics and a single metric might be able to provide insight for several questions).  I generally suggest that the team over identify metrics, as not all of the identified metrics may be implementable (at least in the short run).
  3. Identify the measures needed to generate the identified metrics and either begin to collect them or begin the project(s) needed to implement them.

The process is deceptively simple. However the GQM process usually requires several sessions to generate a comprehensive pallet of measures.  Occasionally I will break the team into subgroups after identifying the organizational goals.  Examples of subgroups include example development and operations (if possible include one person from the other group’s primary discipline on the other subgroup). When this is needed I always have the team come back together to review and refine the questions and again when they identify metrics. Many times the split teams generate metrics that are significantly overlapped. Despite the overlaps the use of subgroups exposes differing points of view more easily.

Measurement frameworks such as GQM build a pallet of metrics and measures based on the stated goals of the organization. The process for building that pallet follows a path of decomposition that ensures both the linkage between metrics and goals but traceability through the step question step. The question step makes sure that the metrics team and the organizations has spent the timed to examine why any individual measure or metric makes sense and delivers value.