Many common measures of software quality include defects. A collection of defects and information about defects can be a rich source of information to assess or improve the functional, structural and process aspects of software delivery. Because of the apparent ease of defect collection and management (apparent because it really is never that easy) and the amount information that can glean from defect data, the number of defect-related measures and metrics found in organizations is wide and varied. Unfortunately, many defect measures are often used incorrectly or are expected to be predictive.
Defect measures that are useful while work is in process (or pretty close) include:
- Defect Density is a measure of how many defects are in a piece of software during a defined period of development divided by the size of the module. This ratio can be used before delivery. Defect density is typically used to reflect structural quality.
- Defect Discovery Rate is a count of the number of defects being discovered over time. Discovery rates tend to follow common patterns, for example in most cases more defects are found the first time a piece of code is tested than later in the development process. Discovery rates can be used to assess all aspects of software quality depending on the type of testing. Discovery rates for user acceptance testing or formal delivery tests are a reflection of functional quality, while discovery rates for unit and integration tests can be used to provide information on structural quality.
- Cumulative Discovered Defects is the number of defects discovered for a specific piece of software, release or project summed up and presented over a period of time. The time increments used are typically a reflection of the type of work being done. I have seen large real-time systems using automated testing show cumulative discovered defect charts minute by minute. A classic pattern seen in cumulative discovered defect charts is the S-curve. In an S-Curve, the number of defects discovered accumulates quickly in the middle of the testing or reviews and then tapers off when testing is complete or nearly complete. If the rate of discovery does not taper off, a problem exists. Cumulative Discovered Defect metrics are related to discovery rate metrics. This category of defect measures are often used real time during the development process and are generally used to assess structural quality.
- Defect Counts by Type This type of defect measure can be used to monitor process, structure or functional aspects of quality depending on when data on defects is collected AND the taxonomy used to categorize defects. This simple measure is almost always the starting point for other defect measures. The simplicity of this measure makes it easy to start and stop. For example, I have seen a team diagnose a process problem in a retrospective, and then collect data for a few sprints to validate the problem and then the fix.
In all cases, some work needs to be completed before defect data can be collected; however, data collection does not have to wait until the software is implemented in production. These measures are most valuable during the development process.