We defined Agile user acceptance testing as a process that confirms that the output of a project meets the business’ needs and requirements. We described the incorporation of acceptance testing into a project’s Agile processes. A full understanding of Agile user acceptance testing requires a deeper understanding of what acceptance testing is, who is involved and the benefits.
Acceptance testing is known under many different monikers, including functional tests (older, from xP) to customer tests or customer acceptance tests. The use of the term customer reflects the inclusion of the product owner in the Scrum process. Mike Cohn dubbed acceptance tests as conditions of satisfaction, which brings the focus of acceptance tests to the needs of the customer/product owner. Acceptance tests provide a mechanism to verify and validate whether the delivered business functionality works based on the requirements and needs of the application’s stakeholders. This means the acceptance tests need to validate the delivered functionality and how that the functionality interacts the broader system environment. Acceptance testing is generally black box testing, in other words, acceptance tests do not require an understanding of the structure of the code. The tester provides an input and receives an expected result, the person interpreting the test results does not care about how the result is generated.
In perfect world, product owners would craft the acceptance test cases as part of the user story. The acceptance test cases reflect the requirements for the story and in many cases are the single version of the requirements. In this perfect scenario, the product owners would craft the acceptance cases and the developers would leverage them to build the production and then to prove completion. Because not all product owners will write acceptance tests, a more typical scenario is that a business analyst (or someone with those skills) will frame the acceptance test cases and then socialize them with the product owner and other developers. In many cases, I have leveraged a BA (singularly or as part of the Three Amigos) even when the product owner was willing to write the acceptance tests in order to ensure quality and consistency. A BA, because of their training in requirements development, will understand the need for format and granularity consistency. I generally do not use developers or user interface designers in this role, as their acceptance test cases tend to emulate integration tests rather than acceptance cases.
In Agile, capturing acceptance test cases as part of developing user stories provides the team with a significant number of benefits. The first benefit is that the process generates closer collaboration between the product owner and the team. Closer collaboration generates better requirements. A second benefit is that that the acceptance criteria creates a contract for the team to determine when a story is complete at a functional level. The overall definition of done defines completion at the process and standards level, while the acceptance tests provides the definition of completion at the story level.
The bottom line reason for a product owner or the Three Amigos to define acceptance test is increase the probability that a team will develop solutions that meet the organization’s needs and that what gets delivered actually works correctly the first time. Higher customer satisfaction and fewer delivered defects is the best means to know whether your acceptance test process is on track.
February 4, 2014 at 10:56 pm
When I’m presented with the opportunity, I prefer to treat acceptance testing as the testing one should do with the system inserted its operational environment (or a close facsimile). For the customer, they are less worried about the system complying to its specification and much more interested to see the desired benefits realised. For example, I’m willing to bet that the T5 baggage handling system, MYKI ticketing system and the HealthCare.gov site all passed their “acceptance testing”, but clearly they all failed to deliver the intended business value when they went operational. That’s not only a failure of project governance, but also a lack of adequate testing to truly validate the system is fit for purpose.
February 4, 2014 at 11:53 pm
I agree that final acceptance testing should be done as close to the production environment as possible. That however does not mean that earlier rounds of acceptance testing can not be done in QA like environments.
February 5, 2014 at 3:06 am
There are certainly many benefits in a “test early, test often” approach, especially when done on a representative environment. However, while acceptance testing of this form can be claimed as “necessary” for Agile teams (frequent feedback, reduction of waste, minimisation of delays, etc), I think that it is not “sufficient”; we really need to have each product increment exposed to its operational environment to confirm that we are indeed Done and business value has been delivered (without detrimental consequences for the customer’s organisation). Although acceptance criteria “create a contract” and hence help protect the team from unreasonable demands, this is really a secondary concern as “our highest priority is to satisfy the customer” and that is achieved by bringing about the changes they desire, e.g. increase revenue, capture new markets, reduce overheads, etc. Many systems that get built, although meeting all their acceptance criteria, either don’t bring about the change desired or make matters worse (e.g. an increased load on a call centre, delays in the processing chain, loss of customers, increased stock holdings, etc).
February 5, 2014 at 4:36 am
[…] Agile User Acceptance Testing: What, Who and Why… Written by: Thomas Cagley […]
February 5, 2014 at 4:48 am
I guess Acceptance Testing (synonym for Functional Testing in AGILE) is different from UAT. From my understanding and what we follow (of course different organisations follow different termonologies and practises), Acceptance testing is done by internal testing team and UAT is done by customer and supported by internal resources (dev and test). We generally have one sprint before release for regression and UAT.
February 5, 2014 at 12:13 pm
Having spent more than a few years responding to problems in production that we were not able to “see” wen testing due to differences between the production environment and our best test environment, I understand the need to test as close to production as possible. Unfortunately in complex environments that is not always financially possible (in one life I did credit card processing).
Rumadak, I am not sure that I think hardening sprint / iterations are always the answer (can be). Why do you separate acceptance testing and UAT? What do you get from the seperation?
February 5, 2014 at 11:56 pm
[…] Agile user acceptance testing (AUAT) confirms that the output of a project meets the business’ needs and requirements. The concept of acceptance testing early and often is almost inarguable, whether you are using Agile or any other method. AUAT generates early customer feedback, which increases customer satisfaction and reduces the potential for delivering defects. The problem is that implementing an effective and efficient AUAT isn’t always easy. Several complicating factors include: […]
February 6, 2014 at 11:58 pm
[…] For example, Rumadak commented on the entry, Agile User Acceptance Testing: What, Who and Why, that “We generally have one sprint before release for regression and UAT.” I see three primary reasons for leveraging a hardening sprint. In these special cases planning […]
February 10, 2014 at 7:50 pm
[…] has been completed for the release. Many organizations couple the Demo/Review with a more formal user acceptance test (UAT) step, doing the demonstration then UAT. If you wait until just prior to a release to generate […]
September 21, 2014 at 9:11 pm
[…] 309 features our essay on Agile user acceptance testing. Agile user acceptance testing (AUAT) confirms that the output of a project meets the business’ needs and requirements. The […]
September 28, 2014 at 9:11 pm
[…] Process and Measurement Cast number 309 features our essay on Agile user acceptance testing. Agile user acceptance testing (AUAT) confirms that the output of a project meets the business’ needs and requirements. The […]
March 27, 2019 at 4:52 pm
candidiase tratamento caseiro bicarbonato
Agile User Acceptance Testing: What, Who and Why . . . | Software Process and Measurement
April 28, 2020 at 9:09 am
PEP GUARDIOLA has put Ben Gibson on his wanted list for the summer.