Agile User Acceptance Testing (AUAT) at the team level, focuses on proving that the functionality developed to solve a specific user story meets the user’s needs. Typically stories are part of a larger “whole,” and to truly prove that a business problem has been solved, acceptance testing needs to be performed as stories are assembled into features and features into applications/systems.
Individual teams accept user stories into sprints, if they are using time boxes as in Scrum. Stories should follow the guidelines found in the INVEST mnemonic coined by Bill Wake to generate a kernel of functionality that can be delivered. Because user stories are very granular, they often do not satisfy the overall business needs of the stakeholders. Product owners and other stakeholders generally want features. During backlog grooming features are broken down from epics into stories, then are developed and then assembled to satisfy the business need. A typical feature requires multiple stories (a one-to-many relationship). Two basic scenarios can be used to highlight the need to scale from story-level AUAT to feature- and system-level acceptance testing
Scenario One: Each Story Can Stand Alone
The simplest scenario would be the situation in which a feature is just the sum of the individual stories. This means that each independent story can be assembled and that no further acceptance testing is required. In this scenario, meeting the story-level acceptance criteria would satisfy the feature-level acceptance criteria and the system-level acceptance criteria. At best, this scenario is rare.
Scenario Two: Features Represent More Than The Sum of Parts
Features are often represent more than the sum of the individual stories. Even the relatively simple scenarios can be more than a sum of their parts. For example, consider a feature for maintaining a customer on an applications. Stories would include adding a customer, modifying a customer, deleting a customer and inquiring on a customer. The acceptance criteria for the feature would more than likely include criteria that the functionality in each story needs to work smoothly together or meet a performance standard all of which requires running an acceptance test at the feature level. Non-functional requirements are often reflected in overarching acceptance criteria captured at the feature level or system level. These overarching criteria require performing AUAT at the feature and system level.
The discussion of executing a feature- or system-level acceptance test often generates hot debate. The debate is less about the need to get acceptance and generate feedback at the feature or system level, but more about when this type of test should be done. Deciding on “when” is often a reflection on whether the organization and teams have adopted a few critical Agile techniques.
- Integrated code base – All teams should be building and committing to a single code base.
- Continuous builds (or at least daily) – The single code base should be re-built as code is committed (or at least daily) and validated.
- Team synchronization – All teams working together toward a common goal (SAFe calls this an Agile release train) should begin and end their sprints at the same time.
A solution I have used for teams that meet these criteria is to coodinate the feature acceptance test through the Scrum of Scrums as the second to last official activity of each synchronized sprint (prior to the retrospective(s)). The feature AUAT requires team and stakeholder participation so that everyone can agree that the criteria is met or not met. All of these activities assume that acceptance criteria were developed for each feature as it was added the backlog and that overall system acceptance criteria was crafted in the team charter at the beginning of the overall effort. This ensures that delivery of functionality can move forward to release (if planned) without delays.
Where organizations have not addressed the three criteria, often the response is to implement a “hardening” (also known as development plus one, test after or just a testing sprint), so that the system can be assembled and tested as a whole. Problems found after stories are accepted generally require reopening stories and re-planning. Also if work has gone forward and is being built on potentially bad code, significant rework can be required. My strong advice is to spend the time and money needed to implement the three criteria; therefore removing this need for hardening sprints.
Scaling AUAT to features that require more than a single story, team or sprint to complete is not as simple looking at each story’s acceptance criteria. Features and the overall system will have their own acceptance criteria. Scaling is facilitated by addressing the technical aspects of Agile and synchronizing activities, however these are only prerequisites to building layers of AUAT into the product development cycle.
Note – We have left a number of hanging issues, such as who should be involved in AUAT and if a story is truly independent does it require higher levels of AUAT? We will address these in the future. Are there other aspects of AUAT that you believe we should be address on this blog?