Many practitioners see Agile acceptance testing as focused solely on the business facing functionality. This is a misunderstanding; acceptance testing is more varied. The body of knowledge that supports the International Software Testing Qualifications Board’s testing certifications deconstructs acceptance testing into four categories:
- User Acceptance Testing focuses mainly on the functionality requested by the business user(s).
- The Operational Acceptance test or Production Acceptance test validates whether the system meets the organization’s requirements for operation. These requirements are often considered technical or non-functional.
- Contract Acceptance testing compares results against the criteria documented in the contract. Contract criteria can and often do overlap with criteria from other forms of acceptance testing.
- Compliance Acceptance testing validates that the product adheres to relevant regulations and standards. Compliance is often driven by government or industry regulation.
Fitting each of these versions of acceptance testing into Agile framework requires forethought and planning. For example, most Agile and lean frameworks spread Agile User Acceptance Testing (AUAT) steps throughout the development framework so that AUAT is not a gate at the end of development. Spreading the user acceptance activities into user story creation and grooming, demonstrations and integration steps for scaled projects reduce the possibility of work being completed that does not meet the needs of the business. The other three types of acceptance testing can leverage many of the same techniques; however, sometimes each of the different types might need a bit of augmentation to acceptance testing goals.
Operations acceptance can be addressed by considering operations personnel as a user or customer of the product being built. Users have acceptance criteria, participate in demonstrations and in some cases participate directly on or with the team(s) doing the work. DevOps addresses the idea of interweaving development and operational personnel. In addition, non-functional and technical operational requirements should be built directly into the definition of done, so that teams ensure they are covering operational needs as they solve the business problem.
Using Agile acceptance testing methods on project or product work that requires external parties and contracts is often more complicated than just leveraging common Agile practices or adding components into the definition of done. In this scenario, the contract vehicles have to be crafted to recognize and use both the Agile planning cadence (iteration and release timing) and the feedback loops the cadence generates as a contract control mechanism. The level of trust and formality will dictate who has to be involved in contract acceptance. For example, in some formal contract scenarios, purchasing agents, executives, and legal representatives might be required in addition to typical team members. Using the iteration cadence as the contract review point, rather than classic stage gates, may add additional overhead to the end of iteration ceremonies such as demonstrations, however, expending the effort will reduce the potential for rework by avoiding surprises later in the development cycle.
Agile compliance acceptance testing is typically approached using the same mechanisms as Agile operational acceptance testing, with the exception of the injection of an auditor or compliance officer as a participant.
The four types of acceptance testing have similar characteristics. They can be addressed using many of the basic Agile techniques with a few tweaks, such as involving auditors, compliance officers, purchasing agents and (gasp) lawyers, when needed. In order to make the full range of acceptance testing work, the broader organization needs to understand the nuances of the Agile techniques that are being used. And that my dear readers is a culture change, which is sometimes just a wee bit hard.