In goes the money and out comes the soda? It is a test!

In goes the money and out comes the soda? It is a test!

Acceptance testing is necessity when developing any product. My brother, the homebuilder, includes acceptance testing through out the building process. His process includes planned and unplanned walkthroughs and backlog reviews with his clients as the house is built. He has even developed checklists for clients that have never had a custom home built. The process culminates with a final walk through to ensure the homeowner is happy. The process of user acceptance testing in Agile development has many similarities, including: participation by users, building UAT into how the teams and teams-of-teams work and testing user acceptance throughout the product development life cycle.

Acceptance testing is a type of black box testing. The tester knows the inputs and has an expected result in mind, but the window into how the input is transformed is opaque. An example of a black box test for a soda machine would be putting money into a soda machine, pressing the selection button and getting the correct frosty beverage. The tester does not need to be aware of all the steps between hitting the selector and receiving the drink. The story-level AUAT can be incorporated into the the day-to-day activity of an Agile team. Incorporating AUAT activities includes:

  1. Adding the requirement for the development of acceptance tests into the definition of ready to develop. (This will be bookended by the definition of done)
  2. Ensuring that the product owner or a well-regarded subject matter expert for the business participate in defining the acceptance criteria for stories and features.
  3. Reviewing acceptance criteria as part of the story grooming process.
  4. Using Acceptance Test Driven Development (ATDD) or other Test First Development methods. ATDD builds collaboration between the developers, testers and the business into the process by writing acceptance tests before developers begin coding.
  5. Incorporating the satisfaction of the acceptance criteria into the definition of done.
  6. Leveraging the classic Agile demo lead by the product owner to stakeholders performed at the end of each sprint. Completed (done) stories are demonstrated and stakeholders interact with them to make sure their needs are being addressed and to solicit feedback.
  7. Performing a final AUAT step using a soft roll-out or first use technique to generate feedback to collect final user feedback in a truly production environment. One of the most common problems all tests have is that they are executed in an environment that closely mirrors production. The word close is generally the issue, and until the code is run in a true production environment what exactly will happen is unknown. The concept of first use feedback borders on one the more problematic approaches that of throwing code over the wall and testing in production. This should never be the first time acceptance, integration or performance is tested, but rather treated as a mechanism to broaden the pool of feedback available to the team.

In a scaled Agile project acceptance testing at the story level is a step in a larger process of planning and actions. This process typically starts by developing acceptance criteria for features and epics which are then groomed and decomposed into stories.  Once the stories are developed and combined  a final acceptance test at the system or application level is needed to ensure what has been developed works as a whole package and meets the users needs.

Are there other techniques that you use to implement AUAT at the team level?

In the next blog entry we will address ideas for scaling AUAT.

Balloon glows require more expertise than a single team!

Balloon glows require more expertise than a single team!


Over the years I have heard many reasons for performing some form of user acceptance testing. Some of those reasons are somewhat humorous, such as “UAT is on the checklist, therefore we have to do it” while some are profound, such as reducing risk of production failures and lack of acceptance. Regardless of the reason acceptance testing does not happen by magic, someone has to plan and execute acceptance testing.  Even in the most automated environment acceptance testing requires a personal touch and in Agile, acceptance testing is a group affair.

The agile literature and pundits talk a great deal about the need for Agile teams to be cross functional. A cross-functional Agile team should include all of the relevant functional and technical expertise needed to deliver the stories they have committed to delivering. Occasionally this idea is taken too far and teams believe they can’t or don’t need to reach beyond their boundaries for knowledge or expertise. This perception is rarely true. Agile teams often need to draw on knowledge, experience and expertise that exists outside the boundary of the team. While the scope of the effort and techniques used in Agile user acceptance testing (AUAT) can impact the number of people and teams that will be involved with Agile user acceptance testing, typically there are a four fairly stable set of capabilities that actively participate in acceptance testing.

  1. The Agile Team – The team (or teams) is always actively engaged in AUAT. AUAT is not a single event, but rather is integrated directly into every step of the product life cycle. Acceptance test cases are a significant part of requirements. Techniques such as Acceptance Test Driven Development require whole team involvement.
  2. Product Owner/Product Management – The product owner is the focal point for AUAT activities. The product owner acts as a conduit for business knowledge and needs into the team. As efforts scale up to require more than a single team or for external software products, product management teams are often needed to convey the interrelationships between features, stories and teams.
  3. Subject Matter Experts/Real users – Subject matter experts (SMEs) know the ins and outs of the product, market or other area of knowledge. Involving SMEs to frame acceptance test or to review solutions evolve provides the team with a ready pool of knowledge that by definition they don’t have. Product owners or product management identify, organize and bring subject matter expertise to the team.
  4. Test Professionals/Test Coaches – AUAT is real testing, therefore everyone that is involved in writing and automating acceptance test cases, creating test environments and executing acceptance testing needs to understand. Test coaches (possibly test architects, also) are very useful to help everyone involved in AUAT regardless of technique to test effectively.

Over the years, who participated in user acceptance testing was as varied as the reason people said they were doing acceptance testing. Sometimes development teams would “perform” acceptance testing as proxy for the users. Other times software would be thrown over the wall and SMEs and other business users would do something that approximated testing. AUAT takes a different approach and builds it directly into the product development flow. Integrating UAT into whole the flow of developing requires that even the most cross-functional team access a whole cavalcade of roles inside and outside of the team to ensure that AUAT reduces the chance of doing the wrong thing and at the same time reduces the chance of doing the right thing wrong.