Agile Acceptance Testing Anti-Patterns

Don’t just throw it over the wall.

A user acceptance test (AUAT) is a process that confirms that what is produced meets the business needs and requirements. User acceptance testing is a form of validation done by executing the product. For a software product, UAT is performed by executing the code in the user’s environment and comparing the outcome to the users needs. In the classic V-model of testing, acceptance testing is the final proof that the requirements have been transformed into functional code. Agile frameworks have incorporated the steps for acceptance testing numerous ways. Not all of the patterns that have been adopted for Agile User Acceptance Testing (AUAT) are effective, and in some cases they are anti-effective.

Post Development AUAT: Waiting to do any user acceptance testing until all of the code has been developed and integrated has been a widely used, bad practice for as long as I can remember. It is one of those practices that almost every practitioner in the software development world would agree ought to be different, but when push comes to shove, gets adopted. The Agile version of this practice is incorporating all AUAT activities into a hardening sprint. In this scenario the code is placed in a production or production-like environment and AUAT testing is performed. Everyone holds their breath as they wait for surprises to be found. Finding problems that affect the business value of the product this late in the process is always costly. The post development AUAT is often combined with other anti-patterns which only exacerbate the pain.

Fix: Leverage the natural UAT activities built into most Agile frameworks, such as story level acceptance criteria, embedded product owners, demonstrations and continuous integration with testing.

Users Responsibility Only: In many organizations, user acceptance testing is done by users of the software. Often business users have never had any professional training on testing, and therefore, perform only cursory testing. Note: This is not a condemnation of the level of effort or interest that UAT participant have in the process. Some Agile-ish organizations have elected to continue this practices (almost always coupled with post-development UAT).

Fix: The whole Agile team, including the product owner, should actively participate and facilitate user acceptance testing. The word facilitate is chosen carefully. The Agile team should not perform the UAT for the users, but rather help them plan and execute the process.

Tossing The Software Product Over the Wall: In this scenario, one group creates the product and then declares that it is done and tosses it over the proverbial wall, where another group takes possession. In most circumstances, communication between the two groups is limited. This scenario is most common were the development and the enhancements and support are done by different companies (common in the government and large, commercial organizations). The lack of team continuity and communication often leads to delivering poor quality. This scenario is almost always combined with the UAT after development anti-pattern and sometimes with the user’s only anti-pattern. .

Fix: Where this behavior is not legally mandated, just don’t do it. The use of a DevOps type models in which the development team, where user subject matter experts (users), and maintenance personnel interact across the product lifecycle, are a useful tool for breaking through the wall.

AUAT in Production: This scenario, like the others, happens regardless of whether an organization is using plan-based or Agile methods. The impact of AUAT done in production is often worse because it reflects a poor understanding of Agile and it is probable that other practices might be askew.

Fix: Leverage the natural UAT activities built into most Agile frameworks, such as story level acceptance criteria, embedded product owners, demonstrations and continuous integration with testing. Also, review all Agile practices to ensure they are effective and are delivering.

Agile user acceptance testing is most effective when users and team members work together to validate that what is being built over the entire development process. Anything else is an anti-pattern.