Aging is good for scotch, not for software development.

Aging is good for scotch, not for software development.

Lean is the systematic removal of waste within a process, known as muda in Japanese.  Delay is often one of the predominant forms of waste in any process. Cost of delay (CoD) is a method of placing a value on the waste inherent in delay. In manufacturing or other industries, the cost of delay is often obvious. Simple examples of the CoD might include the cost of warehousing incomplete items or holding stock in that room of a store that is waiting where it can’t be sold. In software development the concept CoD is not as easy to see, but no less real. In software development (including development, enhancements and maintenance) CoD is generally conceived as a measure of the reduction in value caused as work sits when it is not completed. Delay is the time a unit of work is not being actively being worked on. For example, if it takes a day to write and unit test a piece of code and then that piece of code has to wait a month to be tested then delay is measured as one month. The total delay can be determined by summing total time the unit of work spends sitting around until it gets to a point where the final user of the software can actually use the it! Conceptually CoD is similar to financial techniques like Return on Investment and Net Present Value (NPV). NPV normalizes benefits and costs in the future based on a discount rate (such as the presumed interest rate over the period or the organization’s internal rate of return). Simply put, a dollar invested at ten percent interest today will be worth more in a year than the dollar that sat waiting to be approved to be invested for 364 days. The difference in value is the cost of delay. Delay can generate cost many ways including:

Delay to gaining value: Until a unit of work is delivered, it can’t be used. For example, when we discussed Test Driven Development we define the following user story:

As a beer logo glass collector, I want to be able to log a logo glass into my collection so I do not recollect the same glass!

If this story was completed, but implementation of the story was delayed a week and I bought a duplicate glass for$5.35 (including sales tax) during the delay, the tangible cost of delay would be $5.35.

Opportunity costs: In the example above, if I had the $5.35 in my pocket I could have done something else with it. While this might be considered an intangible cost, if the $5.35 had been invested in another value generating asset or feature, the cost of delay can be quite real.

Value of a feature decays over time: In most organizations being first to market with an app or feature is viewed as valuable. The value is often couched in terms of market share or the ability to charge a premium. Being late to market can often be difference between success and failure.

The concepts of delay and the cost of delay generates a focus on queues (where work sits while waiting to be worked on) and speed to delivery. In general, the faster a story or feature can be delivered, the faster the organization can begin to accrue the inherent value. Understanding the CoD provides data to understand the trade-offs between starting one thing and then waiting to deliver and starting and completing something else instead. If we assume that the work we are doing has value if it is completed then then letting it sit around and age generally doesn’t make sense unless we are making cheese or wine.

How do you weight costs and benefits?

How do you weight costs and benefits?

Introducing any process improvement requires weighing the costs and benefits.  Most major improvements come with carefully crafted cost/benefit analyses. Where the benefits outweigh the costs, changes are implemented.  The goal is to the decide where to expend precious process improvement capital so that it has the most benefit.  Many of the benefits and criticisms of TDD and other test-first techniques can be quantified, but others are less tangible.  In many cases the final decision rests on personal biases, beliefs and past experiences. In the end, our organization and environment may yield different results – there is no perfect ROI model.

In a perfect world, we would pilot TDD with a typical team to determine the impact. For example, will the cost of change be higher than anticipated (or lower), or will the improvement in quality be worth the effort to learn and implement?  In many cases pilot teams for important process improvements are staffed with the best and the brightest.  Unfortunately these are the people that will deliver results regardless of barriers.  The results are rarely perfectly extensible because the capabilities of the best and brightest aren’t generally the same as a normal team. Another typical failure many organizations make is to reflect on the results of a team that embraces TDD (or any other change) on their own.  Expecting the results of teams that self-selects to extend to the larger organization is problematic.  Those that self-select change are more apt to work harder to make TDD work.

In a perfect world we would be able to test our assumptions about the impact of TDD in our organization based on the impact typical groups. However, sometimes we have to rely on the results of the best team or the results of true believers.  Either case is better than having to rely solely on studies of from outside the organization.  Just remember to temper your interpretation of the data when you are weighing your interpretation of costs and benefits.

Metrics Minute: Return on Investment (ROI)
Thomas M. Cagley Jr.

Audio Version:  SPaMCAST 163

Definition

ROI is a standard measure of project profitability that is the discounted profits over the life of the project expressed as a percentage of initial investment.  ROI is a classic financial metric that when applied to projects, is typically used as a technique for project acceptance or, in retrospect, as a tool to evaluate overall performance. When applied in agile projects, ROI can be used to determine priority of epics, themes or even stories.

Formula

ROI = (Average Net Benefits) ÷ (Initial Costs)

Average Net Benefits is the residual value that results from a project after adding total benefits for a specific period and subtracting all expenses for the project for that period.

Initial Costs includes any expense related to the initial development of the project.

Usage

The primary use of ROI is to compare a potential project to other possible projects in order to determine which makes the most sense to pursue (like Return on Assets – ROA).  Projects with a higher ROI are more likely to be included in the portfolio of projects.  When applying ROI to agile projects I suggest at the very least using ROI as a tool to prioritize epics and features into a release plan.

Alternately ROI can be used as a tool to evaluate continued investment in a product or a portfolio of products.  In this case the metric would be periodically recalculated to determine whether an overall project portfolio represents an efficient use of assets or if segments of the portfolio offer high a higher return.

Issues

There are several issues with the calculation and use of ROI as a decision tool including poor math, failing to make an apples-to-apples comparison or in extreme cases the combination of poor math and bad comparisons.  The most significant issues are:

Return on Investment is a financial tool designed to help build a business case to support decision making by evaluating the forecasted impact to your organization’s bottom line.  ROI or any other financial analyses should not be your only evaluation criteria.  High ROI’s may be an artifact of wishful thinking or creative accounting for benefits.  Rigorous follow-up helps stop this type of behavior.

A second issue is failing to account for the time value of money.  The time value of money suggests that a benefit or cost today has a higher absolute value than the same value or cost in the future.  Techniques like Net Present Value normalize benefits and costs in the future based on a discount rate (such as the presumed interest rate over the period or the organizations internal rate of return).

Comparisons of factors like ROI for two projects must be across similar timeframes so that an apples-to-apples comparison can be made.  In environments that are rapidly evolving shorten the comparison period to favor projects that deliver benefits earlier rather than later.  The comparison problem can be exacerbated when used for the smaller work packages usually seen in agile projects. Gross value may be more easily determined for work packages like stories.

A final issue is that intangible benefits may affect the rational for project selection which forces organizations to address valuation of intangible benefits. This issue is a relative of the “numbers don’t tell the whole story” argument. As in the discussion of comparing work items above, as the work packages become more granular, it may become more difficult to accurately evaluate intangible or non-business value in order to calculate ROI. Create rules for how you will allocate or value intangible benefits and costs. Valuation of intangible benefits requires a judgment call and unless that judgment is based on rules or guidelines those judgment calls will be open to question or manipulation.

Related Metrics

  • Return on Equity (ROE)
  • Return on Assets (ROA)
  • Payback Period
  • Internal Rate of Return (IRR)
  • Net Present Value
  • Total cost of ownership

Criticisms

Criticisms of ROI, like the issues noted earlier, revolve around how the metric is used.

This first criticism is that comparisons across different types of projects are difficult due differing cost and benefit structures that effect ROI.  An example of problems that occur when comparing differing types of projects would be making an investment decision about an infrastructure project versus a project that generates revenue. It is usually more difficult to identify and monetize the benefits of an infrastructure project which makes these types of projects more difficult to fund even though they may be required to enable projects with a more classic benefit stream. This criticism is true; however, I would suggest this is a failure to enforce a linkage between infrastructure projects to the impact on revenue streams.

A second criticism: Valuing intangible costs and benefits is at best an inexact science and is open to internal political games. This criticism is the most troublesome. There isn’t general agreement on how to value the intangible.   I would suggest not using a one-size-fits-all solution; rather I would suggest ensuring that valuations of intangible cost or benefits are consistently evaluated based on a risk adjusted discount rate. The higher the risk, the higher the discount rate would be set. A formal approach to measuring or estimating risk will be required to avoid manipulation of the analysis.

A third criticism of ROI centers on using a single point in time to qualify projects. The calculation of ROI generally reflects a snapshot of at particular moment in time. Unless you live under a rock it is difficult to forget that the business environment is – – – dynamic. The idea that an ROI calculation can be done and evaluated once to ensure that the organizations portfolio of projects continually maximizes value is ludicrous. To deal with the issue I suggest periodic revaluations of the portfolio to ensure that the business environment does not change enough to require changing the composition of the project portfolio.  Note: I would suggest that there is a need re-evaluate projects more often if they in the gray area between acceptance and rejection.