Can we really measure anything?

Organizations and teams come to agile—for that matter, any concept, framework or technique—for a wide variety of reasons.  Even if we are just the keeping up with the neighbors, we need feedback to know if we have met our goal.  We need feedback because—to quote Paul Gibbons, author of The Science of Successful Organizational Change (Re-read Saturday)—“we confuse what we think ought to work” with what does work (quote from SPaMCAST 480).  Feedback is required when we are trying to determine if the time and effort invested to adopt agile delivered the expected results.  The typical results promised from an agile transformation fall into nine overall categories.  Each of these can be used to generate questions which can be used to measure and assess impact.

  • Lower Cost – Are teams delivering functionality at less cost than they did when using different methods?
  • Faster Completion – Are teams delivering projects and programs faster?
  • Frequent Deliveries – Are teams able to release functionality more often?
  • Transparency – Are the agendas, policies, conditions, and decisions available to everyone involved in delivering value?
  • Business and Customer Focus – Is the work that teams do important to the business and/or the organization’s customers?
  • Engagement – Are teams, stakeholders, and customers working collaboratively in a concerted fashion to deliver value?
  • Predictability – Are teams able to state what they will do, when they will do it and then deliver on that promise?
  • Flexibility – Can the team change direction to meet business needs?
  • Increased Quality – Are teams delivering more usable functionality? Does the functionality have fewer defects and meet needs of the business?

Arguably not all of these promised benefits are direct benefits of adopting agile.  However, all of these benefits are used to sell the cost of adopting agile. Therefore it is imperative that you be able to answer whether we have met the promises and goals set when committing to an agile transformation. Peter Drucker, the management pundit,  has been quoted as saying “The purpose of business is to create and keep a customer.” The impact of adopting agile in the business or technical components of the organization should have an impact on the purpose of the organization.  Most organizations adopt agile to improve their ability to deliver value and gain customers.  At the business level, these tactical goals that support Drucker’s more lofty goal are typically measured by reduced cost and/or increased profit.  Both are proxies for creating and keeping customers.  Evan Leybourn recently wrote that companies are not in the business of making money.  In a comment, someone pointed out that making profits was a requirement for maintaining the ability to create customers; therefore, it is a crucial part of a business’s survival equation.

The Agile Manifesto does not make any promises about performance or about how people will interact.  The Manifesto does describe values and principles that guide adoption and behavior in broad terms.  There are multiple paths for achieving the values and principles in the manifesto.  People almost always sell agile as more than a set of values and principles.  People sell agile to leaders with promises of faster, better, cheaper or otherwise improved functionality.  Given these promises it behooves all practitioners to be able to answer whether those promises were delivered with more than a shoulder shrug.

 

Next – Suggested Measures and Metrics

Listen Now
Subscribe: Apple Podcast
Check out the podcast on Google Play Music

SPaMCAST 473 features our essay on 6 Important Flow Metrics!  Getting the most value out of a process is important to any leader.  Balancing getting the most value with getting value sooner complicates the discussion.  In some cases, getting some value sooner is worth more than the same value delivered later.  Guiding the delivery of value is more complicated than a rank ordering a list of user stories and then magically hoping that everything will happen in the most effective and efficient manner possible.  Measurement is an important tool to help teams and organizations ask the right questions.  The 6 flow metrics provide process transparency into organizations that leverage continuous flow, scrumban, and/or Scrum as the basis for their Agile implementations.

We will also complete our discussion of part 3 (3 of 3) of chapter 20 of Tame The Flow: Hyper-Productive Knowledge-Work Performance, The TameFlow Approach and Its Application to Scrum and Kanban  (buy a copy here).

 

Re-Read Saturday News

This week,  we tackle Chapter 8 of Actionable Agile Metrics for Predictability: An Introduction by Daniel S. Vacanti. Chapter 8 is titled, Conversion of Flow Part II.  Remember that requirement in Little’s Law.that work that enters the process, completes and leaves.  We do a deeper dive on why that is important.  Buy your copy today and read along! (more…)

 How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition

How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition

Chapter 3 of How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition, is titled The Illusion of Intangibles: Why Immeasurables Aren’t.  In this chapter Hubbard explores three misconceptions of measurement that lead people to believe they can measure something, three reasons why people think something should not be measured and four useful measurement assumptions.

Hubbard begins the chapter with a discussion of the  reasons people most commonly suggest that something can’t be measured. The misconceptions are summarized into three categories:

  1. The concept of measurement. – The first misconception stems from not understanding the definition of measurement. Hubbard defines measurement as an observation that quantifiably reduce uncertainty. If we considered that most of the business decisions are faced with are based on imperfect information, therefore, made under uncertainty. The   measurement is an activity that adds information that improves on prior knowledge. Like the bias that causes people to avoid tackling risks that can’t be reduced to zero, some people will avoid measurement if they can’t reduce uncertainty to zero. Rarely if ever does measurement eliminate all uncertainty but rather  reduces uncertainty, however; all measures that reduce enough uncertainty is often valuable. The concept of the need for measurement data that reduce uncertainty can be seen in portfolio management questions which decide which projects will be funded even before all of the requirements are known.

    All types of attributes can be measured regardless of whether they are qualitative (for example team capabilities) or quantitative (for example project cost). Another consideration when an understanding measurement is that there are numerous measurement scales. Different measurement scales include nominal, ordinal, interval and  ratio scales.  Each scale allows different statistical operations and can present different conceptual challenges. It is imperative to understand the how can be used and the mathematical operations that can be leveraged for each (we will explore these on the blog in the near future).

    Hubbard concludes this section with a discussion of two of his basic assumptions. The first is the assumption is that there is a prior state of uncertainty that be quantified or estimated. And that uncertainty is a feature of the observer not necessarily that of the thing being observed.  This is a basic argument of Bayesian statistic. In Bayesian statistics both the initial and change in uncertainty will be quantified.

    Bottom line, it is imperative to understand the definition of measurement, measurement scales and Bayesian statistics so that you can apply the concepts of measurement to reducing uncertainty.

  1. The object of measurement. – The second misconception stems from the use of sloppy language or a failure to  define what is being measured. In order to measure something, we must unambiguously state what the object of measurement means. For example, many organizations wish to understand the productivity of development and maintenance teams but don’t construct a precise  definition  of the concept AND why we care / why the measure is valuable.

    Bottom Line: Decomposing what is going to be measured from vague to explicit should always be the first step in the measurement process.

  1. Methods of measurement. – The third misconception is a reflection of not understanding what constitutes measurement. The process and procedures are often constrained to direct measurement such as counting the daily receipts at a retail store. Most of the difficult measurement in the business (or a variety of other) environments must be done using indirect measurement. For example, in Chapter 2 Hubbard used the example of  Eratosthenes measurement of the circumference of the earth. Eratosthenes used observations of shadows and the angle of the sun to indirectly determine the circumference. A direct measure would have been if he had used a really long tape measure (pretty close to impossible).

    A second topic related to this misconception is thought that valuable measurement requires either measuring the whole population or a large sample. Studying populations is often impractical.  Hubbard shares the rule of five (proper random samples of five observations) or the single sample majority rule yield can dramatically significantly narrow the range of uncertainty.

    Bottom line: Don’t rely on your intuition about sample size.  The natural tendency is to believe a large sample is needed to reduce uncertainty, therefore, many managers will either decide that measurement is not possible managers because they are uncomfortable with sampling.

Even when it possible to measure the argument often turns to why you shouldn’t. Hubbard summarizes the “shouldn’t” into three categories.

  1. Cost too much / economic objections. – Hubbard suggests that most variables that are measured yield little or no informational value, they do not reduce uncertainty in decisions. The value delivered by measurement must outweigh the cost (this is one of the reasons you should “why” you want any measure) of collecting and analysis.

    Bottom Line:  Calculate the value of information based on the reducing uncertainty. Variables that have enough value justify deliberate measurement is justified. Hubbard suggests (and I concur) when someone says something is too expensive or too hard to measure the question in return should be compared to what.

  1. Measures lack usefulness or meaningfulness. – It is often said that you can prove anything with statistics as a reason to point out that measurement is not very meaningful. Hubbard suggests you “you can prove anything” the statement is patently unprovable. What is really meant is that numbers can be used to confuse people especially those without skills in statistics.

    Bottom Line: Investing in statistical knowledge is important for anyone that needs to make decisions and wants to outperform expert judgment.

  1. Ethical objection – Measurement can be construed as dehumanizing. Reducing everything can be thought of as taking all human factors out of a decision, however, measurement does not suggest there are only black and white answers. Measurement increase information while reducing uncertainty.  Hubbard provides a great quote, ”   the preference for ignorance is over even marginal reduction ignorance is never moral.”

    Bottom Line: Information and reduction of uncertainty are  neither moral or immoral.

The chapter is capped by four useful measurement assumptions that provide a summary for the chapter.

  1. Everything has been measured before. Do your research before you start!
  2. You have data than you think. – Consider both direct and indirect data.
  3. You need far fewer data points than you think. – Remember the rule of five.
  4. New observations are more accessible than you think. – Sample and use simple measures.

More than occasionally I have been told the measuring is meaningless since the project or piece work being measured is unique and that the past does not predict the future.  Interesting these same people yell the loudest when I suggest that if the past does not count that team members can be considered fungible assets and trade in and out of a project.  Measurement and knowledge of the past reduce almost always reduces uncertainty.

Previous Installments in Re-read Saturday, How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition

How To Measure Anything, Third Edition, Introduction

HTMA, Chapter 1

HTMA, Chapter 2

 

 

Untitled

“I never knew anybody . . . who found life simple. I think a life or a time looks simple when you leave out the details.” – Ursula K. Le GuinThe Birthday of the World and Other Stories

The act of measurement either reflects how work was done, how it is being done and what is possible in the future. A measurement framework that supports all of these goals is going to have to reflect some of the details and complexity that are found in the development (broad sense) environment. The simple Agile measurement framework uses the relationships between the areas of productivity, quality, predictability and value to account and reflect real world complexity and to help generate some balance. Each quadrant of the model interacts with the other to a greater or lesser extent. The following matrix maps the nuances between the quadrants.

Impact Matrix

Impact Matrix

The labor productivity quadrant most directly influences the value quadrant. Lower productivity (output per unit of effort) equates to higher costs and less value that can be delivered. Pressure to increase productivity and lower cost can cause higher levels of technical debt, therefore lower levels of quality. Erratic levels of productivity can be translated into time-to-market variability.

Predictability, typically expressed as velocity or time-to-market, most directly interacts with quality at two levels. The first is terms of customer satisfaction. Delivering functionality at a rate or date that is at odds with what is anticipated will typically have a negative impact on customer satisfaction (quality). Crashing the schedule to meet a date (and be perceived as predictable) will generally cause the team to cut corners, which yields technical debt and higher levels of defects. Lower quality is generally thought to reduce the perceived value of the functionality delivered.

Quality, measured as technical debit or delivered defects, has direct links to predictability (noted earlier) and value. The linkage from quality to value is direct. Software (or any other deliverable) that has lower quality than anticipated will be held in lower regard and be perceived as being less useful. We have noted a moderate relationship between labor productivity and quality through technical debt. This relationship can also be seen through the mechanism of fixing defects. Every hour spent fixing defects is an hour that would normally be spent developing or enhancing functionality.

Value, measure as business value or return on investment, is very strongly related to productivity and value (as noted earlier).

Based on the relationships we can see that a focus on a single area of the model could cause a negative impact on performance in a different quadrant. For example, a single minded focus on efficiency can lead to reduced value quality and more strongly less value to stakeholders. The model would suggest the need to measure and set performance level agreements for value if labor productivity is going to be stressed.

The simple Agile measurement framework provides a means to understand the relationships between the four macro categories of measurement that have been organized into quadrant. Knowledge of those relationships can help an organization or team to structure how they measure to ensure approach taken is balanced.

Agile Metrics Framework

Agile Metrics Framework

What gets measured depends on the team’s and the organization’s reporting needs and the measurement goals. For instance, an organization that needs to quantitatively prove their transformation will need to consider measures (and metrics) that can be generated consistently across project teams. Organizations whose teams are standalone and do not anticipate the need for baselining or benchmarking can easily leverage team-based relative measures, such as function points. The simple metrics framework suggested here with potential metrics is shown below: [

  1. Labor Productivity Quadrant
    1. Labor productivity is generally expressed as a functional measure of size per person month, for example: function points per person month or use case points per person month. Labor productivity is typically the measure of choice when an overall transformation program needs to prove efficiency results. These measures are easily comparable between teams and have industry benchmarks available for comparison. The drawbacks are the perceived level of effort to generate the measure and the invasiveness of the process used to generate the size component of the metric.
    2. Story completion (variants include measure of percentage story completion) is a relatively easy metric for teams to collect and leverage. The simplest form of this measure is a simple count of the number of stories competed in a sprint (or period of time for Kanban). Adding a time component creates a rate of completion (a metric) which can be used as a variant of velocity.
  2. Quality Quadrant
    1. Customer satisfaction is a measure of how satisfied the customers (or stakeholders) of a project are with the team performance or functionality delivered. Customer satisfaction can measured using techniques as simple as asking specific stakeholder how they feel about the project or very formal techniques, such as surveys and calculations such as Net Promoters. The higher the formality the more effort that will be required to collect and analyze the metric and the more comparable the metric will be between teams across the life of long running projects.
    2. Delivered defects are a count of the number of defects found after the code (or other deliverable) is marked as done. This measure is generally considered one of the more important reflection of quality, because all code or other deliverables are potentially implementable after they have be marked as done, which means any defects found, regardless of by whom, could have been found in production. Defects found in production can negatively impact customer satisfaction and the overall business.
    3. Usability is typically a measure of compliance against a set of industry and/or organizational standards. Most teams build usability compliance into the definition of done, therefore what is delivered as done is compliant. The metric is used as a mechanism to reflect progress while functionality is being built.
  3. Predictability Quadrant
    1. Velocity is the average number of units of work delivered in a sprint (or any other repeating unit of time). Typically velocity is expressed as the average number of story points a team delivers per sprint. While similar to productivity, velocity is typically used to represent team predictability. Predictability metrics can be used to generate release plans (when will some group of functionality be ready for production) and in sprint planning.
    2. Time-to-market is very similar to velocity reflecting how fast functionality is developed and delivered. Time to market is generally used to reflect plan-based (non-Agile) projects or in organizations using functional metrics (e.g. function points). An example of a time-to-market metric would be the number of function points per calendar. Note: like velocity, time-to-market metrics are generally averages and used in planning exercises or in benchmark.
    3. Effort burn-down is a measure of a team’s estimate of the number of hours of effort remaining in a sprint to deliver the functionality that was committed to during planning. This almost always used as mechanism for teams to anticipate whether will complete work by the end of the sprint. There are numerous variations on effort burn-down chart including story points, task and feature burn-down charts. In every case some measure of work is count (e.g. hours, tasks, story points) and then as they are completed, used up or new instances discovered, the number remaining is either incremented or decremented.
  4. Value Quadrant
    1. Business value is an estimate of the net value being delivered in a unit of work (e.g. story, epic or feature). While business value is the Holy Grail in this category, it is generally very difficult to estimate at a story or feature level, therefore tracking value tends at a higher level such as a release.
    2. Features delivered is a proxy for business value. This measure is typically a count of the number of features delivered in sprint. Variants of this measure include counting stories or epic (larger user stories). Features and stories reflect requirements therefore as the number of features delivered increase the value users and the product owner perceive.

The measures and metrics noted above barely scratch the surface of what can be measured. What should be measured is dependent on the needs and goals of the team and the organization. In ALL cases the measures and metrics must be vetted to ensure they meet the Agile measurement philosophies.

Metrics Are About Prediction

Metrics Are About Prediction

There are three reasons to measure. The first is to guide specific behaviors. The second is to provide information on the status of process. And the third is as a tool to help predict the future. At a team level it is easy to take a very narrow view of metrics and measurement, however the organization is another significant stakeholder in the collection and consumption of metrics information. Teams and other organizational stakeholders have different metrics needs for each of three basic reasons for measuring.  Part of maturing as an Agile organization is the development of a common understanding of metrics needs that includes the differences between groups.  Reaching a common understanding is a step toward developing the mechanisms to accommodate all of the relevant metrics needs within the organization.

 

Reason to Measure

Agile Team Perspective Organizational Perspective
Guide Behaviors The goal of metrics and tools at a team level are to support tactical behaviors focused on the delivery the functionality the team has committed to delivering.  Metrics can be delivered with tools such as card walls (the simple metric of a card moving across the board), burn-down charts or story completion charts.  These tools (also known as information radiators) provide information that teams generally find useful for guiding behaviors such as swarming, collaboration and continuous re-planning. The goal of measurement that guides behavior at the organizational level is to reinforce desired overall Agile behavior. The metrics needed to support and reinforce Agile behavior will evolve as an organization completes its Agile transformation.  Examples of organization metrics that guide behavior include Ka8znztcskills/capabilities tracking (gamification – Gamification is a mechanism that leverages the competitive attributes of the target audience). [As the transformation matures, measurement against Agile Maturity Models can be leveraged to guide behavior.
Provide Status Tactical Perspective: The team shares status on a daily basis during the stand-up/Scrum meeting while leveraging tools line the card wall and burn-down charts as metrics and informational radiators. Burn-down chart provides team level status information that can by share across multiple layers of the organization hierarchy, however team level data tends to be seen as too granular as projects morph into programs and status is passed up the organizational hierarchy. Program level burn-up charts and story maps provide quantifiable measurement feedback that is accessible to senior leaders.
Predict Future Scrum and Scrumban teams need to be able see the work in front of them to understand how to plan both at a short, medium term and long term basis.  Tools like burn-down (short term), burn-up (program level view), story maps and product roadmaps (both long-term) provide a quantified view of progress. Organizations need to develop tactical and strategic plans that are supported by software functionality.  Portfolio metrics and information radiators (story maps and product roadmaps) leverage naturally occurring data from project performance.

Different stakeholders have different measurement needs and perspectives.  Occasionally there is a suggestion that the only measurement data that Agile should generate is what the team needs.  While teams and other organizational stakeholders, such as product, IT and executive management, can (and should) use similar tools, organizational data needs extend to being able to monitor and guide the Agile transformation and other process improvement efforts. Those needs will require everyone involved collect a wider range of data and generate different metrics.

398137720_d2b6404e4a_b

When driving to work or watching a baseball game, we can observe Newton’s Third Law of Motion, “for every action there is an equal and opposite reaction.” The engine explodes gasoline the engine and transmission makes the wheels turn generating motion.  A baseball player swings his bat and the ball flies from the stadium. Measurement has a similar set of effects. In Agile Metrics: Philosophy we discussed the impact of the effort required for measurement on teams (every hour spent measuring is one less hour building functionality – equal and opposite). Equally as important is the impact of what is being measured on behavior. What is measured signals what is important to the project teams and the organization, therefore influencing both what work is done and how the work is done. All measurement programs must have balance so as not to let one measure create runaway behavior that generates too much of a good think and overwhelms other desired behaviors. To paraphrase the Third Law of Motion: for every measure there must be an opposite measure. A simple four quadrant metrics framework can be used to shape a balanced approach to measurement.  The framework includes four measurement quadrants that push and pull against each other to minimize runaway behavior.

  1. Labor Productivity – Labor productivity measures the efficiency of the transformation of labor into something of higher value. For example, the productivity of a software project would be measured as the amount of software that is produced per hour (or more typically a person-month). The metric is generally represented as an average value for team over a period of time. Productivity is very similar to common Agile metric, velocity which measures how many stories or story points a sprint a team averages. A singular focus on productivity and velocity can push teams toward working more efficiently or can cause teams to cut corners and undervalue quality and value.
  2. Quality – Quality measures the compliance of the function delivered against a standard. This typically assessed by some form of comparison. Testing, for example, is a comparison of functionality against the standard of predicted behavior. Standards can include technical coding standards, security standards, and measures of technical debt (corners cut), customer satisfaction or defects delivered.
  3. Predictability – Predictability measures a team’s (or organization’s) ability to deliver on its commitments. Teams commit to the number of stories (or story points) they will delivery in a sprint and similarly IT departments and even organizations commit to delivering levels of value and functionality their stakeholders. Measures such time-to-market (of which there are many variants), the number incomplete stories or program-level burn-up charts are often used to measure and highlight predictability.
  4. Value – Value measures the worth or usefulness of the functionality (or other deliverable) delivered. Value can be a difficult concept to quantify, especially when the functionality (or any deliverable) does not interface with the ultimate customer. However most projects have gone through a qualification process with some form of cost/benefit analysis. While this process can be very informal in some organizations, someone has gone through a process to rationalize the benefits that are expected from the project or feature they are requesting. A typical measure for value is return on investment (ROI).

A balanced metrics model will help ensure that one measure does not generate unanticipated negative results. For example, an over-focus on quality can cause organizations to spend more time and effort on testing than needed; causing costs to rise and value to diminish. There is no single formula for a balanced set of metrics, each organization will need to generate a balance based on their business context.