Simple and Cheap aren't necessarily the same!

Simple and Cheap aren’t necessarily the same!

Attribute and usage models are fairly typical frameworks for translating a strategic definition of quality into tactical measurements of quality. The quality of the software or product is measured by observing the impact of what is delivered. The software or product quality can be influenced by the development process (process quality); however, process-like development, project management, and testing are not being directly measured. The attributes and usage components you select are a reflection of the organization’s goals and missions. For example, one would expect an airline’s definition and model of quality to prominently feature safety while a financial institution would tend to highlight data and physical security. Leveraging the ISO “quality in use” model as an example to develop a measurement pallet (a bunch of metrics that can be used based on specific needs) provides the following example:

  1. Effectiveness is a measure of how successful a project is in producing the desired result. In cases where a project does not deliver revenue, I find that translating improvements into cash allows the use of tools that most investors (anyone with a retirement account) can understand. The more effective a project the higher the revenue or cost savings. Effectiveness measures include:
    1. Return on Investment (ROI): ROI is the granddaddy of all projects measures. ROI is the ratio of the benefit generated by the project and the cost of the project. One of the variants of this metric that can be used is return-on-assets.
    2. Internal Rate of Return (IRR): IRR measures the net present value of the profitability or return based on an expected rate of return.
  2. Efficiency is a measure of the impact of the software on the amount of work performed with the software per unit of effort.
    1. Productivity is the ratio of how much product is produced per unit of work. Productivity is increased by increasing the output while holding work the same or reducing the amount of work required to produce a unit of work.
    2. Cycle Time is a measure of the amount of time it takes to product or to do a job.
  3. Satisfaction is a measure of whether the software meets a need or desire.
    1. Customer Satisfaction is typically measured through a survey of attributes that are important to the end customer. The answers are then summarized into an overall customer satisfaction rating.
    2. Net Promoter is calculated by subtracting the number of customers that are neutral or unlikely to promote your product from those that are very likely to recommend your product.
  4. Safety is a measure of whether the operation of the software causes or reduces the danger. Measuring safety is often critical in life or safety-critical systems (systems in which a malfunction can impact human life or safety).
    1. Mean Time to Failure measures the average time between failures of the product or software. Increasing the average is an effect of higher quality.
    2. Failure/Production defect trends provide visual evidence changes in quality.
  5. Usability is a measure of the ease of use, functionality and learnability.
    1. Completion rates measure whether or not users can accomplish their goals. Increased completion rates reflect higher quality.
    2. UI defects affect the users of the product or software’s ability to accomplish their goals. UI defects and completion rate are often very closely related.


Which attributes, and then which metrics, are important for an organization is impacted by many factors. Organizations creating software to control medical devices will put more emphasis on safety measures than the IT department coding a website for your local pub. There is no one size fits all model for defining and measurement quality. Approaches like defect management, quality attribute models or quality use models provide frameworks to facilitate a robust discussion of how to measure the quality of delivered software.