How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition

Chapter 12 of How to Measure Anything, Finding the Value of “Intangibles in Business” Third Edition is the second chapter in the final section of the book.  Hubbard titled Chapter 12 The Ultimate Measurement Instrument: Human Judges.  The majority of HTMA has focused on different statistical tools and techniques.  This chapter examines the human as a measurement tool.  Here is a summary of the chapter in a few bullet points:

  • Expert judgement is often impacted by cognitive biases.
  • Improve unaided expert judgment by using simple (sic) statistical techniques.
  • Above all else, don’t use a method that adds more error to the initial estimate.


Turkey Trot

Turkey Trot

Empathy is defined as understanding what another person is experiencing from their frame of reference. Empathy is more than mere understanding; requiring more of a cognitive connection between two people or a group. Empathy is a very valuable concept because it forms a basis for trust, enables communication and possibly facilitates the development of altruism. However, poorly practiced empathy can lead to problematic behaviors. The first is reinforcing boundaries and defining outsiders, which makes it hard to teams of teams to interact. Secondly is misapplied empathy when there is no basis of trust; the illusion of empathy can be perceived as manipulation. (more…)

Listen Now

Subscribe on iTunes

The Software Process and Measurement Cast 345 features our essay on Cognitive Biases and two new columns. The essay on cognitive bias provides important tools for anyone that works on a team or interfaces with other people! A sample for the podcast:

“The discussion of cognitive biases is not a theoretical exercise. Even a relatively simple process such as sprint planning in Scrum is influenced by the cognitive biases of the participants. Even the planning process itself is built to use cognitive biases like the anchor bias to help the team come to consensus efficiently. How all the members of the team perceive their environment and the work they commit to delivering will influence the probability of success therefore cognitive biases need to be understood and managed.”

The first of the new columns is Jeremy Berriault’s QA Corner.  Jeremy’s first QA Corner discusses root cause analysis and some errors that people make when doing root cause analysis. Jeremy, is a leader in the world of quality assurance and testing and was originally interviewed on the Software Process and Measurement Cast 274.

The second new column features Steve Tendon discussing his great new book, Hyper-Productive Knowledge Work Performance.  Our intent is to discuss the book chapter by chapter.  This is very much like the re-read we do on blog weekly but with the author.  Steve has offered the SPaMCAST listeners are great discount if you use the link shown above.

As part of the chapter by chapter discussion of Steve’s book we are embedding homework questions.  The first question we pose is “Is the concept of hyper-productivity transferable from one group or company to another?” Send your comments to

Call to action!

Reviews of the Podcast help to attract new listeners.  Can you write a review of the Software Process and Measurement Cast and post it on the podcatcher of your choice?  Whether you listen on ITunes or any other podcatcher, a review will help to grow the podcast!  Thank you in advance!

Re-Read Saturday News

The Re-Read Saturday focus on Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement began on February 21nd. The Goal has been hugely influential because it introduced the Theory of Constraints, which is central to lean thinking. The book is written as a business novel. Visit the Software Process and Measurement Blog and catch up on the re-read.

Note: If you don’t have a copy of the book, buy one.  If you use the link below it will support the Software Process and Measurement blog and podcast.

Dead Tree Version or Kindle Version 

Next . . . The Mythical Man-Month Get a copy now and start reading! We will start in four weeks!

Upcoming Events

June 9 – 12
San Diego, California
I will be speaking on June 10.  My presentation is titled “Agile Estimation Using Functional Metrics.”

Let me know if you are attending!

Also upcoming conferences I will be involved in include and SQTM in September. More on these great conferences next week.

Next SPaMCast

The next Software Process and Measurement Cast will feature our will interview with Jon Quigley.  We discussed configuration management and his new book Configuration Management: Theory, Practice, and Application. Jon co-authored the book with Kim Robertson. Configuration management is one of the most critical practices anyone building a product, writing a piece of code or working on a project with other must learn or face the consequences!

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.

What you see is dependent on what you expect.

What you see depends on what you expect.

Cognitive biases are a pattern of behavior that reflects a deviation in judgment that occurs under particular situations. Biases develop as filters or shortcuts that help us perceive information quickly in a manner that turns out to be beneficial to a decision-making process. Biases that help us recognize patterns helped early humans stay alive by recognizing predators. The shortcut kept our ancestors alive even if there were false positives (you can survive lots of false positives, but you aren’t likely to survive a false negative). Project teams (Agile or not) use or fall prey to a wide range of biases that affect perceptions and decisions. A sample of common biases that affect project teams in this category include:

Anchor bias refers to the tendency to rely heavily on one piece of information when making decisions. This bias is often seen when early estimates for a project or a tasks are made. The instant they are placed on the table they become a reference point to which all changes will be compared.

Clustering illusion (or clustering bias) is the tendency to see patterns in clusters or streaks of in a smaller sample of data inside larger data sets. For example, a team I recently worked with had an average velocity of 30 story points per sprint (ranged between 20 – 36) had three sprints in a row that delivered 40+ story points. While nothing significant had changed with how the team was working, outsiders saw a pattern and believed something out of the ordinary was occurring. (FYI – if there is no statistical significance to the data what we are seeing is “common cause” variance.)

The curse of knowledge bias generates a filter that blocks the ability think about a topic from a different and generally less informed perspective. In many cases being an expert on a topic makes it very difficult to see an out-of-the-box solution. This is one of the reasons significant changes in IT platforms or concepts come as new players enter the organization that have less experience with current tools and techniques. In a similar manner to the curse of knowledge, the status quo bias or the tendency to want things to stay relatively the same, creates a perception filter that helps the individual or team seek out and fixate data and concepts which makes them comfortable so they do not need to change.

An availability cascade causes a concept to become more and more plausible the more it is repeated publicly.  An availability cascade generates a self-reinforcing feedback loop. Perhaps that is why Pokemon became more popular the more it was shown on the Cartoon Network. Daniel Pink, author of To Sell Is Human, in a Salesforce.Com webinar on July 9th pointed out that repetition increases process fluency, which makes the repeated item seem to be more true through repetition. Sales, marketing and 24 hour news channels understand and use the availability cascade bias to great effect.

A final example of biases that affect behavior and perception is optimism bias. Optimism bias is the tendency to be overoptimistic about favorable outcomes. This bias is often exhibited in status reports or in promises made early in a project. These are generally not lies, but rather due to optimism bias we believe that we can deliver. Dr. Ricardo Validri in Software Process and Measurement Cast 84 suggests that optimism bias is one of major reasons IT personnel are poor estimators.

This is a sample of cognitive biases that affect how we perceive information and then how we make decisions. Each of the biases reflects some basic component of human psychology and has been found to be generally beneficial. However all biases can create blind spots. A good coach or leader will first be aware of his or her biases and then help the team understand their blind spots while not abandoning the shortcuts that can help us perceive what is important and make timely decisions.

Many of us have a cognitive bias toward eating scorpions!

Many of us have a cognitive bias toward eating scorpions!

Cognitive biases are patterns of behavior that reflect a deviation in judgment that occurs under particular situations. The phrase cognitive bias was introduced by Amos Tversky and Daniel Kahneman in 1972. Biases can affect how information is perceived, how teams and individuals behave and even our perception of ourselves. Biases are a part of nearly every human interaction, so we need to understand the potential biases that are in play if we are going to help teams grow and evolve.

Project teams make decisions on continuous basis. Most decisions are made based on how the decision maker perceives the information he or she has at hand. One bias that can affect how information is perceived is the illusory correlation. The illusory correlation is the perception of a relationship between two or more variables when no relationship exists. An example would be that a team that works more hours a week has higher productivity because working longer gives the perception of creating more output. The perception of a relationship causes you to pay less attention to other factors, such as the higher level of effort they are expending. There are numerous biases that affect how information is perceived, and these biases can impact the outcome of decisions or even whether we make needed decisions at all.

Biases can affect behavior. Neglect of probability is a type of cognitive bias common in IT organizations that are planning and estimating projects or in risk management. For example, most estimates should be represented as a range based on probability. Techniques like Monte Carlo analysis can be used to generate a range of probability based estimates to address type of bias. However, almost all estimates are represented as a single number and regardless of all the caveats attached, and the continuum of probability is ignored. Lottery ticket sales are another reflection of the neglect of probability bias; buying one or 10 doesn’t materially affect the probability of winning, but that does not stop those who think buying ten tickets increases their chances of winning. In both cases neglecting probability affects how we behave and make decisions.

Biases can affect our motivation. For example, a self-serving attributional bias, occurs when success is attributed to internal factors and failures are attributed to external factors. This type of bias can occur at an individual level or at the team level. While self-serving bias can improve self-esteem (or a team self-esteem) it can also cloud judgment by causing an overestimation of capability. For example, if a team is able to deliver more than their long-term productivity or velocity would predict, the team might then perceive that they have increased their capability to deliver. If no fundamental changes have occurred such as an infusion of knowledge, training or new tools, the higher velocity may not be attributable to the team. A good coach will help teams examine these types of biases during retrospectives.

Bias are powerful psychological filters that affect how both individuals and teams perceive the world around them and then how they behave. Biases reflect shortcuts in how we interpret and react to stimuli. In many cases these reactions are valuable; however they can also cause problems (as many shortcuts can). Understanding how biases impact how individuals and teams perceive the world around them can help team make better decisions and therefore deliver value more effectively.


Consider an elastic band that has been stretched between two points. If the elastic hasn’t lost its stretch, as soon as it is released at one end it will snap back. Organizational culture is like that elastic band. We pull and stretch to make changes and then we want them to settle in. However, we need to anchor the change so that when we change focus the changes don’t disappear. The eighth step in Kotter’s eight-stage model of change discusses this need to anchor the change to avoid reversion.

Culture describes the typical behaviors of a group and the meaning ascribed to those behaviors. Kotter describes culture as the reflection of shared values and group norms. All groups have a specific culture that allows them to operate in a predictable manner. Within a group or organization, culture allows members to interpret behavior and communication, and therefore build bonds of trust. When culture is disrupted bond are scrambled and behavior becomes difficult to predict until culture is reset. If a change program declares victory before the culture is reset, the group or organization tends to revert to back to the original cultural norm.

Culture is powerful because:

  1. The individuals within any group are selected to be part of the group and then indoctrinated into the culture. Cognitive biases are a powerful force that pushes people to hire and interact with people that are like them, homogenizing and reinforcing culture. Culture is further reinforced by training, standards and processes that are used to reduce the level of behavioral variance in the organization. Standardization and indoctrination help lock in culture.
  2. Culture exerts itself through the actions of each individual. While in a small firm, the combination of the number of people in the firm and proximity to the leaders of the change make culture change easier (not easy just easier).  However when we consider mid-sized or large firms in which hundreds or thousands of people need to make a consistent and permanent change to how they act, change gets really complicated. Since culture reflects and is reinforced by how people work, real change requires change each how each affected person behaves which is significantly more difficult to change than words in the personnel manual.
  3. Much actions taken in an organization is not driven by conscious decision which makes it hard to challenge or discuss. A significant amount of our work behavior is governed by shared values and muscle memory. I often hear the statement “that’s just the way it is done here” when I ask why a team has taken a specific action. Many of these actions are unconscious and therefore tend to go unrecognized until challenged from the outside. Pushing people away from comfortable patterns of behavior generates cognitive dissonance.

Less power is needed overcome entrenched culture if the change can build on the organization’s base culture rather than having to confront it. Building on to the current culture will often generate some early momentum towards change because those being asked to change see less risk. Alternately change that is at odds with the current culture will require significantly more effort and a greater sense of urgency to generate and sustain.

Kotter argues that culture changes trail behavior. Put another way, culture change happens last. Each of the stages in the model for change are designed to build urgency, momentum and support for organizational changes. Vision provides the direction for the change. Results provide proof that the change works and is better than what it replaced. Continuous communication of vision, direction and results break through the barriers of resistance. Breaking down the layers of resistance challenges old values and pushes people to admit that the change is better. When barriers can’t or won’t change sometimes change means changing key people.  Nihilistic behavior in the face of results can’t be allowed to exist. Kotter finally points out that in order to anchor long-term change the organization will need to ensure that both succession planning and promotions reinforce the change rather than allow reversion.

Peter Drucker said, “Culture eats strategy for breakfast.” Innumerable people have suggested a corollary that says “Culture eats change for breakfast.” The Eight Stage Model for Significant Change provides a strategy for overcoming the power of an entrenched culture to generate lasting change.

Re-read Summary to-date

Change is a fact of life. John P. Kotter’s book, Leading Change, defines his famous eight-stage model for change. The first stage of the model is establishing a sense of urgency. A sense of urgency provides the energy and rational for any large, long-term change program. Once a sense of urgency has been established, the second stage in the eight-stage model for change is the establishment of a guiding coalition. If a sense of urgency provides energy to drive change, a guiding coalition provides the power for making change happen. A vision, built on the foundation of urgency and a guiding coalition, represents a picture of a state of being at some point in the future. Developing a vision and strategy is only a start, the vision and strategy must be clearly and consistently communicated to build the critical mass needed to make change actually happen. Once an organization wound up and primed, the people within the organization must be empowered and let loose to create change. Short-term wins provide the feedback and credibility needed to deliver on the change vision. The benefits and feedback from the short-term wins and other environmental feedback are critical for consolidating gains and producing more change. Once a change has been made it needs to anchored so that that the organization does not revert to older, comfortable behaviors throwing away the gains they have invest blood, sweat and tears to create.

Trying to read the context...

Trying to read the context…

Many people poke fun at consultants because they invariably use the phrase “it depends,” even when discussing lunch. “It depends” is code for the context of the situation matters to the answer. When I asked a sample of my readers and listeners to weigh in on whether being on-budget, on-scope or on-schedule was the most important attribute of project success, most answered without using the dreaded “it depends”. An excerpt for a recent Twitter conversation in response to the essay about being on-budget, emphasized that in real life the definition of success will be influenced context.

UntitledIn the sample I surveyed, being on-schedule was the most important attribute with on-scope a close second and on-budget a distant third. If I had added constraints to the mix the answer would probably have been different. One respondent, while explaining why they felt that scope was the key attribute of success, made a strong argument for context (without calling it out directly), “Budgets and schedules can be modified when justified, and functionality can be phased in over time.” The statement could easily be interpreted as as a statement that success criteria, at least in the short run, depends on the goals of the business and the context of the project.

We can all conceive of scenarios where a specific context lead one attribute to be viewed as the most important in project success ASSUMING the software does what it is supposed to do and in an adequate manner. As an example of context, one respondent noted that in his environment and for some projects, “regulatory requirements that say it MUST deploy on a specific date.” Failure to meet the date can result in fines or penalties. In a similar vein, another respondent stated, “Projects are often tied to an organization’s business milestones or regulatory requirements so missing the date is not acceptable.” In both cases the context of a project could lead a project team to focus on one attribute over another.

Why did I ask the question without context? As I have noted in earlier essays, we are all subject to cognitive biases. These bias act as filters and shortcuts that help us interpret data. The context of a project will provide information about the goals and constraints of the project. Our cognitive biases are usefully in helping us interpret project context. Which is why project teams with diverse perspectives tend to make better choices when sorting out which attribute is most important. Is one attribute truly more important than another? It depends!

IMG_0696I recently asked  a sample of  the Software Process Measurement blog readers  and listeners of the Software Process and Measurement Cast how they defined project success. I constrained the answers to the classic project management three-legged stool of on-schedule, on-budget and on-scope, to avoid the predominant answer: “it depends.” Even though many of the respondents found the question difficult, everyone had a succinct opinion. The ranked responses were:

  1. On-schedule
  2. On-scope
  3. On-budget

When I asked which of the three “ons” defined project success, the top response hands down was on-schedule with nearly twice as many responses as on-scope. On-budget was a distant third, but interestingly, it was as mentioned as the second most important response on more than a few responses.

I am not surprised by how the responses stacked up. Schedule is the most easily measured and monitored of the success criteria. Everyone knows how to read a calendar, and given that most projects are created with a due date, whether or not a project is on-time is obvious.

On-scope, while representing the voice of the customer, is more difficult for most projects to interpret and measure. Projects (whether Agile or plan-based) generally evolve over the life of the project. That evolution means that the picture any stakeholder, developer or product owner had in their head at the beginning of the project may not represent what has been delivered when the project is completed. Since scope is less tangible, it is perceived to be less important (or at least more easily debated). Being on-scope may be more of a dis-satisfier (if what is delivered not close to what was wanted people will be dissatisfied, however just being on-target or close does not move the needle) than a satisfier.

The final leg on our stool was on-budget. In most cases the true budget of a project is an outcome impacted by schedule and scope, therefore is a metric that all project managers monitor but have few levers to control. Given that budget performance at a project level is deterministic the respondents perceived it as less important. Had I asked a room of IT finance personnel, I suspect that the answer might have been different.

One of the most interesting observations is that even without context everyone had an opinion about the definition of success. While context, if given, may have shifted the respondent’s perspective, their initial response represents the respondent’s natural cognitive bias. When asked to identify whether being on-schedule, on-budget or on-scope was the most important attribute of project success everyone I asked had an opinion. And, that opinion focused on the very tangible and measurable calendar metric: on-schedule. The basic definition of success that a project manager or leader carries with themselves is important because that opinion will guide how they will try to influence project behavior.

Note over the next few days we will explore the rationale behind why each leg of the stool was important to those who responded.

Overly Self-interested Behavior Is Bad For Teams!

Overly Self-interested Behavior Is Bad For Teams!

Hand Drawn Chart Saturday

Teams are an important concept in most IT organizations, regardless of their development philosophy. Philosophies like Agile may put more emphasis on teams, but even in organizations that do not embrace Agile philosophies, teams are important. Dan Ariely in his Ted Talk, “What Makes Us Feel Good About Our Work” suggested that overly self-interested, cynical behavior can negatively impact organizations by reducing their ability to communicate and innovate. The same problem can occur, albeit on a small scale, at the team level. In a recent presentation, a fellow Agile coach described a team engaged in overly self-interested behavior. He described a scenario in an organization that cuts the bottom 10% of all groups annually and stated vision that IT should maximize the value it deliverers to its customers. After losing a popular team member the previous year, the team had decided to make sure that his replacement was given the worst assignments in order to ensure he stayed on the low end of the performance scale in the coming year. Their goal was to ensure that the core team stayed intact during the next review cycle. In their mind, keeping the core team together ensured that they would deliver more value to their internal customers. The behavior of the team attempted to circumvent the idea that adding new and more highly qualified personnel would lead to improved performance. Viewed from the point of view of organizational policy, the whole team was acting in an overly self-interested behavior manner, but from the point of view of core team they are acting rationally and within their interpretation of the rules as seen through IT’s vision of value delivery. The team did not believe that their behavior was at odds with the behavior the organization wanted to incent.

What we perceive as overly self-interested at a team level is based on collective cognitive biases. Biases are powerful psychological filters that affect how both individuals and teams perceive the world around themselves and then guide how they behave. Biases reflect shortcuts in how we interpret and react to stimulus. In many cases, these reactions are valuable, however they can also cause problems (as many shortcuts occasionally can). Understanding how biases impact how individuals and teams perceive the world around them can help teams make better decisions and therefore deliver value more effectively. In the example, the core team had decided to protect itself by defining the new person as an outsider, which allowed them to hold them at arm’s length. In-group favoritism is a typical team level bias that causes teams to favor those inside the team’s boundaries over those perceived to be on the outside. This type of bias negatively affect how outsiders are perceived to perform. Negative perceptions of outsiders will effect whether we listen to their ideas and whether we trust them to deliver value.

Teams need to break overly self-interested patterns of behavior by striving ensure that they are pursuing a goal that is greater than team’s own self-interest. At a project level, product owners need to clearly identify and communicate the overall value proposition, so the team has something greater than themselves to pursue. When behavior goes off track coaching can help to identify issues that the team can’t see and diagnosis themselves. However, in many cases overly self-interested behavior at the team level can be a reflection of poor philosophies at the organization level. Organizational philosophies, like decimation or objectives that foster individual competition rarely support intergroup communication and innovation.

Fantasies are as ethereal as a cloud.

Fantasies are as ethereal as a cloud.

There are a number of fantasies about estimation that non-IT people and even some experienced software development professionals have. They are that 1) estimates are like retail prices, a predictable fixed price, 2) estimates can always be negotiated to a smaller number with impunity, and 3) in order to be accurate estimates must be precise. The belief in any of these fantasy fallacies will have negative.

The first fantasy is that is that custom projects can be priced like a cup of coffee. We fall prey to these fantasies because we are human and we want software projects to be as predictable as buying that cup of coffee.  When you go to most coffee shops, whether in North America, South America, Europe or India to buy a cup of coffee, the price is posted above the register.  In my local Starbucks I can get a cup of coffee for a few dollars, I just read the menu and pay the amount. The same is true for buying an app on my iPhone or a software package. Software project estimates are built on imperfect information that ranges from partial requirements to evolving technologies and, worse yet, include the interaction of people and the chaos that portends.  From a purely mathematical perspective these imperfections mean that the actual effort, cost and duration of the project will be variable.  How variable is influenced by the process used, the amount of learning that is required to deliver the project and the number of people involved. These are just a few critical factors that drive project performance. This variability in knowledge is why mature estimation organizations almost always express an estimate either as a range or as a probability, and why some organizations suggest that estimation is impossible. Agile projects re-estimate every sprint based on the feedback from the previous sprint using the concept of velocity.  Many waterfall projects re-estimate at the beginning of every new phase so that the current estimate utilizes the information the team has learned through experience.  Even when a fixed price is offered, the organization agreeing to a fixed price will have done an analysis to determine whether they can deliver for that price and what the probability is that the project will really cost (with a profit). This would be the process followed for any project to say they were x% confident of an estimate. When projects run short on time, resources or money and they can’t beg for more, they will begin to make compromises ranging from cutting corners (we don’t need to test that) to jettisoning scope (lets push that feature to phase two).  Many of these decisions will be made quickly and without enough thought, which will hurt IT’s reputation and increase project risk.

A second classic fantasy is that you can always brow beat the team into making the estimate smaller.  This fantasy can be true.  A good negotiator will leverage at least two physiological traits to whittle away at an estimate.  The first trait is the natural optimism of IT personnel, which we discussed in Software Project Estimation: Types of Estimates.  The problem is that negotiating the estimate downward (rather than negotiating over scope or resources) can lead to padding of estimates or to technical debt driven by pressure on profit margin or on career prospects. Estimators that know they are going to be pushed to reduce any estimate regardless of how well it is built will sometimes cheat and pad the estimate. So, when they are pushed to cut they can do so without hurting the project. This behavior is only a short term fix.  Sooner or later (and usually sooner) sponsors and managers figure out the tactic (perhaps because they used it themselves) and begin demanding even deeper cuts.  The classic estimation joke is that every first estimate should be cut in half and then sent back to be re-estimated.  A second side effect of this fantasy is that when the estimate is compressed and the requirements are not reduced, the probability of the team needing to cut corners increases.  Cutting corners can result in technical debt or just plan mistakes.  In extreme circumstances, teams will take big gambles on solutions in an attempt to be on budget.

A third fantasy is that precision equals accuracy. Precision is defined as exactness.  A precise estimate for a project might be that a project will cost $28,944 USD and require 432 hours, will take 43 days beginning January 1st and completing February 12th. Whether the estimate is accurate, defined as close to actual performance, is unknown.  This is precision bias, a form of cogitative bias, in which precision and accuracy are conflated.  In most cases in precision bias occurs the high precision infers higher accuracy.  The level of precision gives the impression that it is highly accurate.  The probability of a highly precise estimate being accurate is nearly zero, however add few decimal places and see how much more easily it is to be believed. As we have noted before, wrong budgets and/or estimates will increase the risk of project failure.

When I teach estimation I usually begin with the statement that all estimates are wrong.  This is done for theatrical effect, however it is perfectly true.  Any estimate that is a single, precise number that has gone through several negotiations (read that as revised down) is nearly always wrong. However, if when we jettison the false veneer of precision, integrate uncertainty and stop randomly padding estimates, we can construct a much more accurate prediction of how a project will perform.  Always remember that an estimate is a prediction, not a price.