Technical Debt


Play Now!

Alex Omeyer and I discussed defining, tracking, and measuring technical debt. Technical debt generates a liability that every organization must deal with. As Alex states, “tech debt might not be the sexiest topic” but if teams and organizations don’t address their liabilities it will be difficult to deliver the value customers need and expect. 

We also talk about the practicalities of pivoting life ambitions and entrepreneurship.

(more…)

Listen Now

Subscribe on iTunes

Software Process and Measurement Cast 350 features our interview with Arlene Minkiewicz. Arlene and I talked technical debt.  Our discussion included the definition of technical debt, a set of philosophies for technical debt and perhaps a few solutions. Regardless of philosophy or approach, a little technical debt goes a long way!

Arlene F. Minkiewicz is the Chief Scientist at PRICE Systems, LLC with over 30 years of experience at PRICE building cost models.  She leads the cost research activity for TruePlanning, the suite of cost estimating products that PRICE provides.  She is a software measurement expert dedicated to finding creative solutions focused on helping make software development professionals successful.  She is widely published and speaks frequently on software related topics.  She has a BS in Electrical Engineering from Lehigh University and an MS in Computer Science from Drexel University.

Email: Arlene.Minkiewicz@PRICESystems.com
Twiitter: @arlenemink
Website:  www.pricesystems.com

Call to Action!

I have a challenge for the Software Process and Measurement Cast listeners for the next few weeks. I would like you to find one person that you think would like the podcast and introduce them to the cast. This might mean sending them the URL or teaching them how to download podcasts. If you like the podcast and think it is valuable they will be thankful to you for introducing them to the Software Process and Measurement Cast. Thank you in advance!

Re-Read Saturday News

We have just begun the Re-Read Saturday of The Mythical Man-Month (buy it here to support the blog and podcast). We are off to rousing start beginning with the Tar Pit. Get a copy now and start reading!

The Re-Read Saturday and other great articles can be found on the Software Process and Measurement Blog.

Remember: We just completed the Re-Read Saturday of Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement which began on February 21nd. What did you think?  Did the re-read cause you to read The Goal for a refresher? Visit the Software Process and Measurement Blog and review the whole re-read.

Note: If you don’t have a copy of the book, buy one. If you use the link below it will support the Software Process and Measurement blog and podcast.

Dead Tree Version or Kindle Version 

Upcoming Events

Software Quality and Test Management 

September 13 – 18, 2015

San Diego, California

http://qualitymanagementconference.com/

I will be speaking on the impact of cognitive biases on teams!  Let me know if you are attending!

 

More on other great conferences soon!

Next SPaMCast

The next Software Process and Measurement Cast will feature our essay on distributed Agile. Distributed Agile is not just Scrum and other Agile techniques over a distance.  As distribution and cultural diversity increase what worked for a co-located team will often fall short.  We will identify solutions next week!

 

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.

Listen to the Software Process and Measurement Cast.

Software Process and Measurement Cast number 301 features our essay on technical debt. Technical debt is the work not done or the shortcuts taken when delivering a product. We all take shortcuts, but at what cost? The essay begins:

Technical debt is a term coined by Ward Cunningham to represent the work not done or the shortcuts taken when delivering a product. In almost every circumstance there are multiple paths than can be taken to deliver a functional product.  For example, when documenting the code you are writing there is a difference between explaining exactly what the code does in detail and being terse and a bit oblique (I can hear the rationalization, “they can just read the code”). The code runs, but if there is ever a problem it will take longer to diagnose the problem. Whether fixing a defect or rewriting the code, if there is a delay caused by figuring out the code, that represents the ‘debt’ of technical debt.  Technical debt is applied to software, but the phrase can be extended to any deliverable or product.  The work that is not done may or may not be fixed in the future.  Until the technical debt is paid back, the debt accrues interest.  Whether or not that interest is important depends on your situation.

Listen to the rest on the Software Process and Measurement Cast 301

 

Next

Software Process and Measurement Cast number 302 will our interview with Larry Maccherone of Rally Software. We talked about Agile and metrics.  Can you combine Agile and metrics without creating an oxymoron?

Upcoming Events

I will be presenting at the International Conference on Software Quality and Test Management in San Diego, CA on October 1.  I have a great discount code!!!! Contact me if you are interested!

I will be presenting at the North East Quality Council 60th Conference October 21st and 22nd in Springfield, MA.

More on all of these great events in the near future! I look forward to seeing all SPaMCAST readers and listeners that attend these great events!

The Software Process and Measurement Cast has a sponsor.

As many you know I do at least one webinar for the IT Metrics and Productivity Institute (ITMPI) every year. The ITMPI provides a great service to the IT profession. ITMPI’s mission is to pull together the expertise and educational efforts of the world’s leading IT thought leaders and to create a single online destination where IT practitioners and executives can meet all of their educational and professional development needs. The ITMPI offers a premium membership that gives members unlimited free access to 400 PDU accredited webinar recordings, and waives the PDU processing fees on all live and recorded webinars. The Software Process and Measurement Cast some support if you sign up here. All the revenue our sponsorship generates goes for bandwidth, hosting and new cool equipment to create more and better content for you. Support the SPaMCAST and learn from the ITMPI.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.

 

 

Development Debt

Development Debt

In Technical Debt, we noted that technical debt is a metaphor coined by Ward Cunningham to represent the work not done or the shortcuts taken when delivering a product. Cunningham uses the term to describe issues in software code. As with any good metaphor, it can be used to stand in for short cuts in other deliverables that effect the delivery of customer value.  In fact, a quick Google search shows that there have been numerous other uses. These “extensions” of technical debt include:

  • Quality Debt,
  • Design Debt,
  • Configuration Management Debt,
  • User Experience Debt, and
  • Architectural Debt.

Before I suggest that this extension of the technical debt metaphor represents a jumped the shark moment,  I’d like to propose one more extension: process debt.  Process debt reflects the shortcuts taken the process of doing work that doesn’t end up in the code.  Short cuts in the process can affect the outcome of sprint, release, whole project or an organization’s ability to deliver value. For example, teams that abandon retrospectives are incurring process debt that could have long-term impact.

Short-term process debt can be incurred when a process is abridged due to a specific one-time incident. For example, failing to update an application’s support documentation while implementing an emergency change. In the long run, support personnel might deliver poor advice based on outdated documentation, thereby reducing customer satisfaction.  The one-time nature of the scenario suggests that the team would not continually incur debt purposefully. However if process debt becomes chronic, an anti-process bubble could form around the team.  Consider how a leader with a poor attitude can infect a team.  High quantities of process debt can reflect a type of bad lead syndrome that will be very difficult to remediate.

An example of process debt I recently discussed with a fellow coach occurred when a team that was supposed to do a code check-in every evening for their nightly build called a week hiatus.  The overall process was that code was to be checked in, followed by a consolidated build of the software, and then it was to be subjected to a set of automated smoke and regression tests. A significant amount of effort had been put into making sure the process was executed, however the person that oversaw the process got the flu and no one else had been trained. The lack of executing a nightly build process allowed the code to drift enough so that it could not be integrated. It took twelve days and a significant amount of rework to get to a point where a nightly build could be done again.  The process debt was incurred when a backup was not trained and the process stopped being used.

Peer pressure, coaching and process and product quality assurance (PPQA) audits are tools for finding, avoiding and, when needed, remediating process debt.  In all three cases someone, whether a fellow team member or an outsider, will need to look at how the work has been done in order to understand how the process was followed.

Process debt equates to shortcuts in the use of the process that can impact a team, a project, project deliverables or the long-term ability of the organization to deliver value to their customers.  What process debt is not is experimentation done by teams consciously in order to improve. Process debt is not change made to a standard process that does not fit their context adopted during retrospectives. Process debt is incurred when teams abandon a process without a better process. Process debt stands as a worthy extension of the technical metaphor that can help us understand that shortcuts in how we do our work can have ramifications, even if the code is not affected.

A dark alley...

A dark alley…

If someone were to approach you on a dark and stormy night as you stood below an underpowered streetlight and ask if you wanted some “technical debt,” I am sure answer would be an emphatic NO. While there are reasons we would intentionally accept technical debt, any development or support team would be better off if they could avoid being shouldered with it.  One strategy to get rid of it is to transfer the ownership of technical debt onto someone else.  Buying packaged software can do this – COTS (commercial off the shelf software) or SAAS (software as a service) can be used as a strategy to avoid owning technical debt.

When an organization buys a supported package or purchases SAAS, they avoid developing specific software development experience (who wants to code an accounting package if you can buy it), shortening the time needed to move from needs-recognition to using the software and avoiding maintaining a staff to maintain the software. In short, the day-to-day management of technical debt becomes someone else’s problem . . . sort of.  One of the laws of software (or at least should be) is that all software includes accrued technical debt. Some portion of the cost initial and annual support or subscription cost of the software pays for the management and maintenance of the software technical debt.

Most software companies do not explicitly publish measures of technical debt in their software, therefore as a user of the software you are more or less blind to level of the technical debt in the package. Why does it matter?  The level of technical debt can still affect your organization even when you don’t “own” the debt. Technical debt in COTS or SAAS impacts the ease of installation, ease of interfacing with your legacy systems, support costs, the existence of latent defects, and maybe most importantly, function agility.  Function agility represents how quickly new features and functions can be rolled into the software.  The longer it takes to develop and integrate new features, the lower the function agility of a software package. The level of technical debt in a package or in SAAS can only be inferred by observing these areas of performance. Review performance in these areas when you are doing your due diligence, and remember that since technical debt tends to build up, performance generally gets worse before it gets better.  And of course, changing software packages is very similar to trying to switch horses while riding (only done by trained stuntmen in the movies, not in real life). So, try to get it right the first time.

A simple cost analysis for determining whether to shoulder the technical debt or to transfer it elsewhere would use the following criteria (assuming the original build/buy decision has been made).

Costs to internally own the technical debt:

  1. Support Staff Costs                            _____
  2. Support Staff Overhead                    _____
  3. Support Staff Tools                            _____
  4. Support Staff Training                      _____

A. Total Costs                                              _____

Costs of someone else owning the technical debt:

  1. Support Cost/Subscription Cost   _____
  2. Support Staff Costs                            _____
  3. Support Staff Overhead                    _____
  4. Support Staff Tools                            _____
  5. Support Staff Training                      _____

B. Total Costs                                                      _____

Net                   Cost        (A – B)                         _____

This is a relatively simple view of the support costs. You should note that there will more than likely be some residual support staff costs needed to interface the new package with legacy systems or just to act as a help desk to initially triage problems. Intangibles, some of which are affected by technical debt such as function agility, will need to be factored into the decision process.

Transferring the servicing and management of technical debt of software by buying packages or using SAAS can make sense.  In many cases COTS and SAAS can reduce amount organizations spend on support and maintenance, including technical debt.  A cost benefit analysis can easily be done to verify whether this is true.  What is less obvious is whether the software’s owner will manage the level of technical debt well. If technical debt is managed poorly, the ability of the package to keep pace with the necessary functionally may suffer. Caveat emptor: do your due diligence and look before you leap.

You can more aggressively remediate debt if you have more monitors, right?

You can more aggressively remediate debt if you have more monitors, right?

One of the more enjoyable parts of writing these blog entries is the interesting conversations I get into.  The conversation I had at 3:30 AM this morning was even more interesting than normal (I was discussing this topic at 3:30 AM because I needed to find a convenient time for a multi-time zone phone call that would not conflict with running a 10k before 8 AM!) The discussion was about whether an aggressive approach to remediating technical debt can have measurable productivity benefits. My friend had some data that argues that technical debt should be remediated continually (at least at some level).

In my experience, most organizations only react to technical debt when it reaches some trigger level.  That is not to say that that they don’t attempt to avoid technical debt through reviews, pair programming, tools and other techniques as a normal practice.  There are two basic categories of triggers.  The first is a limit that fits into the organization’s scheme for measuring technical debt. The second is when someone decides that too much time or money is being spent just on keeping an application or portfolio running.  When the latter case is extreme, words like replacement or outsourcing tend to get used, rather than remediation or refactoring. In either case, the logic is that effort and budget is better spent on delivering functionality than on refactoring, until the technical debt begins to impede progress.

My friend suggested that rather than waiting, some portion of all project time should be consciously and publicly committed to remediating known technical debt above and beyond the normal technical debt avoidance techniques.  The argument is that the technical debt is very similar to the scenario that teams face when they decide to improve how they will work in a retrospective.  We could easily say that addressing issues found in a retrospective represents the remediation of process debt.  Why wouldn’t we pursue technical debt in a similar manner? I think the logic is hard to argue with, if it can be shown to benefit the customer and the organization.

As noted earlier, my friend came with data.  My phone friend described two moderately mature Agile teams. Both teams included 7 people, were mostly dedicated resources (did not work on projects outside the team), most people on the both teams had been together doing a mix of Scrum, xP and lean for 3 years, and that they have approximately the same level of technical capability. The organization uses IFPUG function points to measure team productivity to identify macro level process improvements and best practices.  One team actively includes one story each sprint to either refactor some portion of code or to address a specific item of technical debt from their backlog.  The other does not follow this practice.   The data from the last three quarterly releases is shown below:

Untitled

Team Apple was the team that actively remediated.  The data, while not statistically valid, suggests that an active remediation strategy increases productivity.  What we were not able to conclude was whether the cost/benefit of remediation (effort, a direct cost and delayed functionality, an opportunity cost) was positive.  The organization’s current thinking is a big “probably”.

Does aggressive or active technical debt remediation make sense?  The answer is probably not cut and dry and we certainly need more data to draw a conclusion for these two teams. In order to decide on an approach, each organization is going to have to evaluate their own context.  For example, as discussed in Technical Debt, remediating consciously accrued technical debt applications that have a very short planned life probably does not make sense. If you are dealing with a core ERP application the answer may be very different. However, if productivity is enhanced by aggressively remediating technical debt, waiting until you hit some arbitrary trigger might not make sense.

DSC_0448In Technical Debt we defined technical debt as the work not done or the shortcuts taken when delivering a product. Then we discussed the sources of technical debt in Technical Debt: Where Does It Come From? The next question is can we measure technical debt? If we have a consistent measure, we can determine whether the technical debt in our code is growing or shrinking, or even whether the level of debt passes some threshold of acceptability which requires remediation.  There are three common approaches to identifying and measuring technical debt. Each of these approaches have pluses and minuses. They are:

  1. Tool Based Approaches: There are several software tools that scan code to determine whether the code meets coding or structural standards (extensibility is a structural standard). These tools include Cast AIP, SonarQube, SonarJ, Structure101 and others.  Some tools are proprietary and some are open source.  Tools are very useful for identifying technical debt that has crept into an application unintentionally and that impacts code quality, but less useful for identifying partial requirements, dropped testing or intentional architecture deviations.
  2. Custom Measures:  Many organizations create measures of technical debt.  Almost all measures of technical debt are proxies, including automated test code coverage, audit results and size to line-of-code ratios ([Lines of code/ Function Points Delivered]/Industry Function Point Language Specific Backfire Calibration Number). Since there any number of custom techniques, it is difficult to determine where these can be best applied. However, audit techniques are very valuable to identify technical debt generated by process problems that are not reported.  For example, when pressed a developer may not complete all of his or her unit testing (the same behavior will occur in independent testing).  Unless it is self-reported or observed by a peer, this type of behavior may leave unrecognized technical debt in the code that can’t be seen through a scan. Audits are a mechanism to formally inspect code for standards and processes that can’t be automated.
  3. Self-Reporting: Team-level self-reporting is a fantastic mechanism for tracking intentionally accrued technical debt.  The list of identified technical debt can be counted, sized or valued.  The act of sizing or valuing converts the list into a true measure that can be prioritized.  The sizing mechanisms I have used include function points, story points and effort to correct (usually hours or days). When using value instead of size or effort I always done using a currency (e.g. Dollars, Rupees, Euros). Everyone understands money. I usually ask teams I am working with to generate a value for each piece of technical debt they intentionally accrue as they make the decision.  More than once I have seen team change their decision to incur the technical debt when they considered the value of the item they are considering.  Self-reporting is very useful for capturing intentional accrued technical debt, such as partially implemented requirements, intentional architectural deviations and knowingly constrained testing.

Technical debt is a powerful tool to help teams and organizations think about the quality of their code base.  Measures are also important tools to understand how much technical debt exists and whether or not it makes sense to remediate the technical debt.  Each category of measuring technical debt is useful, however each has its own strengths and weaknesses.  A combination is usually needed to get the best view of the technical debt to meet the needs of a specific team or organization.

These are people acting with a lot of intention!

These are people acting with a lot of intention!

In Technical Debt, we defined technical debt as the work not done or the shortcuts taken when delivering a product. Technical debt can be accrued intentionally or unintentionally.  Intentionally accruing technical debt means that the developer or team has made a conscious choice and hopefully mitigated the risk. If the debt is accrued unintentionally, the level of risk is higher because the impacted is unknown. Looking into where the technical debt comes from can help us infer whether it was intentional or not.

The taxonomy of the sources of technical debt is as follows:

  1. Code that isn’t extensible will require significantly more effort to enhance or maintain.  A team I managed early in my career came across a module late one night that had to be rewritten from scratch because we could not extend the function in the face of a priority one problem (it is a long story and you will need to buy me a beer to hear it).  Concepts like design patterns, object orientation or modular programming will help the team learn how to make their code extensible. Good standards, code reviews and pair programming are tools to ensure this technical debt is avoided. Generally this type of technical debt is accrued unintentionally, and unless processes or work patterns (like pair programming) are in place it can ripple through a project increasing effort and increasing the risk to delivering maximum value to the customer.
  2. Code that is hard to understand is by definition hard to maintain.  I once had to test an old DOS COBOL program that was so large that it had embedded assembler calls that unpacked and repacked specific bytes so that each bit could be used as a variable.  This program was a horror story to understand and test therefore it was rarely enhanced. This ‘bit’ of technical debt was explicitly accrued due to a compiler limitation and the lack of will to upgrade.  This is easy for team members to causally to agree accept, and is usually coupled with the agreement to refactor in the near future (which never comes). The ease which this type of technical debt can be accrued means that it can accumulated unintentionally. The impact is usually only obvious when the team needs to maintain or enhance the code making future work more risky.   As with extensibility – code reviews, standards and pair programming are tools to reduce this type of debt.  I also suggest periodically adding refactoring stories to all project and application backlogs.
  3. Code that knowingly represents incomplete requirements is a type of technical debt that is a bit of gray area, and could arguably be considered a defect. In order for this category to be technical debt, the team (including the product owner) needs to knowingly decide not to develop a whole breadth of a requirement.  I have seen teams take this option to make specific delivery dates. Phasing projects, re-arranging release plans or breaking up user stories are approaches to using this tactic in a transparent fashion.  Doing is this by accident is a defect, and therefore the impact is unpredictable until it is noticed.
  4. Code, modules or applications that are difficult to implement costs everyone involved in the implementation and installation effort and aggravation. Implementation approaches like code usually have multiple possibilities (the hard way and the easy way). Development teams that include members of operations generally will stray toward the easy way, however choices are sometimes made in order for a team to make a date (e.g. compliance requirements or end of year processing are dates that generally aren’t negotiable) that pushes the team to toward the hard way and that accrue technical debt. Including operations personnel in the development teams (or minimally including operations personnel in reviews and demos) are tools to help avoid unconsciously accruing this type of technical debt. Assuming basic reviews, implementation standards and a trained or experienced team, this type of technical debt is generally accrued intentionally and the impact is predictable.
  5. Code that does not meet organizational standards (code or architecture) is generally intentionally accrued when teams are faced with an unmovable deadline. The ability to compare the code to standards makes the impact of not meeting those standards more predictable therefore less risky but only if the choice is made on purpose and transparently.  This type of debt can be accrued unconsciously only when standards are either not very well understood or are not applied consistently.  Make sure all team members are trained on the organization’s standards, hold reviews and make sure they are applied consistently.  One final note: good portfolio management and estimation of releases and projects are useful to ensure that teams do not start work with a due date that is too aggressive and will cause them to cut corners.
  6. Code that is not fully tested.  I was a quality assurance manager once upon a time.  In organizations that test last in their systems development life cycle, there will always be pressure to test less when a sprint is nearly over or a release is due. This form of technical debt, even thought it  comes perilously close to being a defect (or defects), is generally accrued unintentionally. The lack of predictable impact makes this type of technical debt more risky.  Techniques like test driven development, acceptance test driven development or behavior driven development are tools to avoid having to accrue this type of technical debt.

Technical debt can be a combination of intentional choices made by rational teams.  When we accrue technical debt unintentionally, we increase the risk.  That risk might not only affect the current project, but also our ability to enhance and maintain the code going forward. Knowing the sources of technical debt and how we can combat unintentionally accruing that debt is a positive step toward reducing risk and managing the technical debt backlogs all projects accrue.

Student loans are a different kind of debt.

Student loans are a different kind of debt.

Technical debt is a term coined by Ward Cunningham to represent the work not done or the shortcuts taken when delivering a product. In almost every circumstance there are multiple paths than can be taken to deliver a functional product.  For example, when documenting the code you are writing there is a difference between explaining exactly what the code does in detail and being terse and a bit oblique (I can hear the rationalization, “they can just read the code”). The code runs, but if there is ever a problem it will  take longer to diagnose the problem. Whether fixing a defect or rewriting the code, if there is a delay caused by figuring out the code, that represents the ‘debt’ of technical debt.  Technical debt is applied to software, but the phrase can be extended to any deliverable or product.  The work that is not done may or may not be fixed in the future.  Until the technical debt is paid back, the debt accrues interest.  Whether or not that interest is important depends on your situation.

In a perfect world there would be no technical debt.  Work would be done according to all organizational standards and no corners would be cut.  However we live in reality.  Calendar time is the enemy of perfection.  Today’s business environment is dynamic with advantages appearing and disappearing rapidly. The ability to bring products to market quickly to capture a market advantage that might be fleeting can lead to technical debt. A limited amount of technical debt can be used to accelerate delivery.  Clearly some technical debt can be acceptable.  The question is, when does the debt have to be repaid?

Possibly never. The technical debt accrued when building functions or applications that have a short lifespan may never need to be addressed. This is where the metaphor of accruing interest breaks down a bit.  If these short term functions are retired quickly enough the debt need not be paid back.  However functions or applications that have a long term future will need to be refactored to remove some or all of the technical debt.

Technical debt is a metaphor for all of the corners we cut as software is developed.  Technical debt is not a metaphor for defects in the code. Technical debt creates drag by making code harder to understand, more likely to break when changed or slightly off the architectural standard. The effects of technical debt can usually be measured in the additional amount of effort required to make any specific change or in the amount of planned refactoring a team feels they need to allocate to keep the development process moving efficiently.  The choices to manage technical debt are fairly stark, in the long run technical debt has to either be repaid, serviced to keep it manageable or the code needs to be retired.

Leadership and technical practices must be balanced

Leadership and technical practices must be balanced

What is Agile?  It is easy to be lulled into believing that Scrum and the practices encompassed by that framework is all that is need to be a mature Agile organization. A broader view of Agile requires a balance between both leadership practices and technical practices.

In Eastern philosophy, Yin and Yang reflect the two complementary forces that make up the aspects of life; male and female, heaven and earth, chocolate and peanut butter or planning and construction. Leadership practices include planning, control and collaboration activities.  Scrum, the most popular Agile framework, fits into the category of leadership practices.  Technical practices include automated testing, build scripts, user stories, emergent design, pair programmer and other activities needed to create functionality. Extreme Programming (xP) and Test Driven Development (TDD) are methodologies that have significantly more technical focus. The combination of Yin and the Yang of leadership and technical practices yields a more complete development environment.  Just doing one or the other set of practices does not deliver a project.

To be a mature Agile organization, both the technical and leadership aspects of the development process have to be addressed.  The leadership practices, such as planning and collaboration, help synchronize and focus the technical processes in order deliver the most value possible.

Next Page »