One of the strongest indications of the quality of a piece of software is the number of defects found when it is used. In software, defects are generated by a flaw that causes the code to fail to perform as required. Even in organizations that don’t spend the time and effort to collect information on defects before the software is delivered collect information on defects that crop up after delivery. Four classic defect measures are used “post” delivery. Each of the four measures is used to improve the functional, structural and process aspects of software delivery. They are:
- Simple Backlog or List is the simplest defect measure. Relatively early in my career, I took over a testing group that supported a few hundred developers. During the interview process, I determined that a lot of defects were being found in production; however, no one was quite sure how many were “a lot.” Enter the need to capture those defects and generate a simple backlog.
Variants on the simple backlog include defect aging reports, counts by defect type, and quality of fix (how many times it takes to fix a defect) to name a few.
Backlogs of delivered defects can be a reflection of all aspects of the development process. Variants that include defect type or insertion point (where the defect come from) can be used to target a specific aspect of quality.
- Defect Arrival Rate (post implementation) is similar to the Defect Discovery and Cumulative Discovery Rates used to monitor defects during software development. This metric is most often used to monitor the implementation of programs or other types of large-scale effort. I used this tool to monitor bank mergers and conversions. Arrival rate metrics collected during the development process are most often a measure of structural quality; however, after functionality is in production it is much more difficult to tie this metric to a specific aspect of quality. More information is needed to determine whether the defect data is tied to functional, structural or process quality.
- Defect Removal Efficiency (DRE) is the ratio of the defects found before implementation and removed to the total defects found through some period after delivery (typically thirty to ninety days). DRE is often used to evaluate the quality of the process used to develop the software (or other product). DRE includes information from all defect insertion tasks (such as analysis, design, and coding) and all defect removal tasks (reviews, testing and usage).
- Mean Time to Failure is the expected length of time a device or piece of software is expected to last in operation before it breaks. Failure is typically held to mean the device or software ceases to operate (defining the word failure is always a bit fraught). This is most sophisticated of the classic quality metrics (really a measure of reliability) and is most often used to evaluate devices. Mean time to failure is almost always used to evaluate the functional aspect of quality.
The number and types of delivered defects are an obvious reflection of software quality. The more defects a user of any sort finds, the lower the quality of the software. However, simple counts of defects or backlogs tend to provide less information value anytime we get past a few defects.