ACUT isn't all sunshine and flowers.

AUAT isn’t all sunshine and flowers.

Agile user acceptance testing (AUAT) confirms that the output of a project meets the business’ needs and requirements. The concept of acceptance testing early and often is almost inarguable, whether you are using Agile or any other method. AUAT generates early customer feedback, which increases customer satisfaction and reduces the potential for delivering defects. The problem is that implementing an effective and efficient AUAT isn’t always easy. Several complicating factors include:

  1. Having the right technical environment. Pithythoughts strongly argued in his comments (read them here) that acceptance testing without exposure to the true production environment is insufficient to really determine whether the work is “done and that business value has been delivered.” I believe that no seasoned developer or IT pro would argue, however it rarely happens except for new products or for relatively self-contained applications.  For example, consider the expense of having an environment that mirrors production for SAP on top of multiple test and development environments. Environmental compromises are made, and each level of abstraction away from the production mirror increases the risk that what seems ‘acceptable’ might not be in the real world.
  2. Acceptance test cases can be fragile. One of the central tenants in Agile projects is to embrace change. In many cases that change will need to be reflected in the acceptance test cases and criteria. Given that acceptance tests are known for suffering from false positive test failures (they are perceived to be true when they are not) they need to be carefully evaluated as the project evolves or the feedback the team gets from Agile acceptance testing can be counterproductive. Acceptance test cases need to be refactored just like code.
  3. Acceptance Test Driven Development (ATDD) can encourage big upfront design rather than emergent design favored in Agile projects. The development of acceptance criteria can lead the product owner (or other stakeholders) involved in developing the test cases to focus on “how” business value will be delivered technically rather than on the functionality or the “what” will be delivered. Fixing the design too early any project increases the likelihood of rework, and many times is the reason for rejecting change later in the project.  An emergent design allows teams to design only what is needed and then to learn from implementing and demonstrating that design.  Acceptance testing is part of the feedback loop needed to create an evolving design.
  4. Some product owners are unable or unwilling to write acceptance criteria and tests. Many times, product owners have day jobs in addition to their role as the voice to the business, which makes it harder to participate in an Agile team.  In rare cases, some product owners look at the activity of writing Agile user acceptance test cases as being beneath them. I once heard a “product owner” state that writing test cases was an IT job or maybe a clerk’s job – certainly not something he should do. Regardless of whether you are using Agile or waterfall, user acceptance testing is a critical step towards implementation. I find that in these scenarios, generating the proper level of product owner involvement takes a significant amount of time and effort. In the short run, partner a business analyst with the product owner to shepherd the creation of acceptance criteria and then pursue a long term coaching opportunity to change the product owner’s attitude.

Agile user acceptance testing is a method of pursuing feedback from the business early and often by focusing the product owner (and by extension the rest of the business) on considering and then documenting the proof they will need to accept what the project will deliver. AUAT is not easy nor is it free.  There are numerous issues that need to be dealt with to make sure Agile acceptance testing is done correctly.  Without an environment that is as close to production it will be difficult to interpret the results.  Just because you capture acceptance criteria once does not mean you are done, acceptance criteria need to be continually review and refactored just like code.  If the process you are using to generate the acceptance criteria mean that you need to develop the solution too early, you need to step back and take the process more incrementally. Finally remember the word user in the Agile user acceptance testing otherwise you are just doing developer-based functionality testing.  Agile user acceptance testing is not all sunshine and flowers, but the outcome is generally worth the hassle.