Acceptance Testing is rarely just one type of testing.

Many practitioners see Agile acceptance testing as focused solely on the business facing functionality. This is a misunderstanding; acceptance testing is more varied. The body of knowledge that supports the International Software Testing Qualifications Board’s testing certifications deconstructs acceptance testing into four categories:  (more…)


Don’t just throw it over the wall.

A user acceptance test (AUAT) is a process that confirms that what is produced meets the business needs and requirements. User acceptance testing is a form of validation done by executing the product. For a software product, UAT is performed by executing the code in the user’s environment and comparing the outcome to the users needs. In the classic V-model of testing, acceptance testing is the final proof that the requirements have been transformed into functional code. Agile frameworks have incorporated the steps for acceptance testing numerous ways. Not all of the patterns that have been adopted for Agile User Acceptance Testing (AUAT) are effective, and in some cases they are anti-effective. (more…)

Scaling up, up, up!

Scaling up, up, up!

Agile User Acceptance Testing (AUAT) at the team level, focuses on proving that the functionality developed to solve a specific user story meets the user’s needs. Typically stories are part of a larger “whole,” and to truly prove that a business problem has been solved, acceptance testing needs to be performed as stories are assembled into features and features into applications/systems.

Individual teams accept user stories into sprints, if they are using time boxes as in Scrum. Stories should follow the guidelines found in the INVEST mnemonic coined by Bill Wake to generate a kernel of functionality that can be delivered. Because user stories are very granular, they often do not satisfy the overall business needs of the stakeholders. Product owners and other stakeholders generally want features. During backlog grooming features are broken down from epics into stories, then are developed and then assembled to satisfy the business need. A typical feature requires multiple stories (a one-to-many relationship). Two basic scenarios can be used to highlight the need to scale from story-level AUAT to feature- and system-level acceptance testing

Scenario One: Each Story Can Stand Alone

The simplest scenario would be the situation in which a feature is just the sum of the individual stories. This means that each independent story can be assembled and that no further acceptance testing is required. In this scenario, meeting the story-level acceptance criteria would satisfy the feature-level acceptance criteria and the system-level acceptance criteria. At best, this scenario is rare.

Scenario Two: Features Represent More Than The Sum of Parts

Features are often represent more than the sum of the individual stories. Even the relatively simple scenarios can be more than a sum of their parts.  For example, consider a feature for maintaining a customer on an applications.  Stories would include adding a customer, modifying a customer, deleting a customer and inquiring on a customer. The acceptance criteria for the feature would more than likely include criteria that the functionality in each story  needs to work smoothly together or meet a performance standard all of which requires running an acceptance test at the feature level. Non-functional requirements are often reflected in overarching acceptance criteria captured at the feature level or system level. These overarching criteria require performing AUAT at the feature and system level.

The discussion of executing a feature- or system-level acceptance test often generates hot debate. The debate is less about the need to get acceptance and generate feedback at the feature or system level, but more about when this type of test should be done. Deciding on “when” is often a reflection on whether the organization and teams have adopted a few critical Agile techniques.

  1. Integrated code base – All teams should be building and committing to a single code base.
  2. Continuous builds (or at least daily) – The single code base should be re-built as code is committed (or at least daily) and validated.
  3. Team synchronization – All teams working together toward a common goal (SAFe calls this an Agile release train) should begin and end their sprints at the same time.

A solution I have used for teams that meet these criteria is to coodinate the feature acceptance test through the Scrum of Scrums as the second to last official activity of each synchronized sprint (prior to the retrospective(s)). The feature AUAT requires team and stakeholder participation so that everyone can agree that the criteria is met or not met. All of these activities assume that acceptance criteria were developed for each feature as it was added the backlog and that overall system acceptance criteria was crafted in the team charter at the beginning of the overall effort. This ensures that delivery of functionality can move forward to release (if planned) without delays.

Where organizations have not addressed the three criteria, often the response is to implement a “hardening” (also known as development plus one, test after or just a testing sprint), so that the system can be assembled and tested as a whole. Problems found after stories are accepted generally require reopening stories and re-planning. Also if work has gone forward and is being built on potentially bad code, significant rework can be required. My strong advice is to spend the time and money needed to implement the three criteria; therefore removing this need for hardening sprints.

Scaling AUAT to features that require more than a single story, team or sprint to complete is not as simple looking at  each story’s acceptance criteria. Features and the overall system will have their own acceptance criteria. Scaling is facilitated by addressing the technical aspects of Agile and synchronizing activities, however these are only prerequisites to building layers of AUAT into the product development cycle.

Note – We have left a number of hanging issues, such as who should be involved in AUAT and if a story is truly independent does it require higher levels of AUAT? We will address these in the future. Are there other aspects of AUAT that you believe we should be address on this blog?

In goes the money and out comes the soda? It is a test!

In goes the money and out comes the soda? It is a test!

Acceptance testing is necessity when developing any product. My brother, the homebuilder, includes acceptance testing through out the building process. His process includes planned and unplanned walkthroughs and backlog reviews with his clients as the house is built. He has even developed checklists for clients that have never had a custom home built. The process culminates with a final walk through to ensure the homeowner is happy. The process of user acceptance testing in Agile development has many similarities, including: participation by users, building UAT into how the teams and teams-of-teams work and testing user acceptance throughout the product development life cycle.

Acceptance testing is a type of black box testing. The tester knows the inputs and has an expected result in mind, but the window into how the input is transformed is opaque. An example of a black box test for a soda machine would be putting money into a soda machine, pressing the selection button and getting the correct frosty beverage. The tester does not need to be aware of all the steps between hitting the selector and receiving the drink. The story-level AUAT can be incorporated into the the day-to-day activity of an Agile team. Incorporating AUAT activities includes:

  1. Adding the requirement for the development of acceptance tests into the definition of ready to develop. (This will be bookended by the definition of done)
  2. Ensuring that the product owner or a well-regarded subject matter expert for the business participate in defining the acceptance criteria for stories and features.
  3. Reviewing acceptance criteria as part of the story grooming process.
  4. Using Acceptance Test Driven Development (ATDD) or other Test First Development methods. ATDD builds collaboration between the developers, testers and the business into the process by writing acceptance tests before developers begin coding.
  5. Incorporating the satisfaction of the acceptance criteria into the definition of done.
  6. Leveraging the classic Agile demo lead by the product owner to stakeholders performed at the end of each sprint. Completed (done) stories are demonstrated and stakeholders interact with them to make sure their needs are being addressed and to solicit feedback.
  7. Performing a final AUAT step using a soft roll-out or first use technique to generate feedback to collect final user feedback in a truly production environment. One of the most common problems all tests have is that they are executed in an environment that closely mirrors production. The word close is generally the issue, and until the code is run in a true production environment what exactly will happen is unknown. The concept of first use feedback borders on one the more problematic approaches that of throwing code over the wall and testing in production. This should never be the first time acceptance, integration or performance is tested, but rather treated as a mechanism to broaden the pool of feedback available to the team.

In a scaled Agile project acceptance testing at the story level is a step in a larger process of planning and actions. This process typically starts by developing acceptance criteria for features and epics which are then groomed and decomposed into stories.  Once the stories are developed and combined  a final acceptance test at the system or application level is needed to ensure what has been developed works as a whole package and meets the users needs.

Are there other techniques that you use to implement AUAT at the team level?

In the next blog entry we will address ideas for scaling AUAT.

Balloon glows require more expertise than a single team!

Balloon glows require more expertise than a single team!


Over the years I have heard many reasons for performing some form of user acceptance testing. Some of those reasons are somewhat humorous, such as “UAT is on the checklist, therefore we have to do it” while some are profound, such as reducing risk of production failures and lack of acceptance. Regardless of the reason acceptance testing does not happen by magic, someone has to plan and execute acceptance testing.  Even in the most automated environment acceptance testing requires a personal touch and in Agile, acceptance testing is a group affair.

The agile literature and pundits talk a great deal about the need for Agile teams to be cross functional. A cross-functional Agile team should include all of the relevant functional and technical expertise needed to deliver the stories they have committed to delivering. Occasionally this idea is taken too far and teams believe they can’t or don’t need to reach beyond their boundaries for knowledge or expertise. This perception is rarely true. Agile teams often need to draw on knowledge, experience and expertise that exists outside the boundary of the team. While the scope of the effort and techniques used in Agile user acceptance testing (AUAT) can impact the number of people and teams that will be involved with Agile user acceptance testing, typically there are a four fairly stable set of capabilities that actively participate in acceptance testing.

  1. The Agile Team – The team (or teams) is always actively engaged in AUAT. AUAT is not a single event, but rather is integrated directly into every step of the product life cycle. Acceptance test cases are a significant part of requirements. Techniques such as Acceptance Test Driven Development require whole team involvement.
  2. Product Owner/Product Management – The product owner is the focal point for AUAT activities. The product owner acts as a conduit for business knowledge and needs into the team. As efforts scale up to require more than a single team or for external software products, product management teams are often needed to convey the interrelationships between features, stories and teams.
  3. Subject Matter Experts/Real users – Subject matter experts (SMEs) know the ins and outs of the product, market or other area of knowledge. Involving SMEs to frame acceptance test or to review solutions evolve provides the team with a ready pool of knowledge that by definition they don’t have. Product owners or product management identify, organize and bring subject matter expertise to the team.
  4. Test Professionals/Test Coaches – AUAT is real testing, therefore everyone that is involved in writing and automating acceptance test cases, creating test environments and executing acceptance testing needs to understand. Test coaches (possibly test architects, also) are very useful to help everyone involved in AUAT regardless of technique to test effectively.

Over the years, who participated in user acceptance testing was as varied as the reason people said they were doing acceptance testing. Sometimes development teams would “perform” acceptance testing as proxy for the users. Other times software would be thrown over the wall and SMEs and other business users would do something that approximated testing. AUAT takes a different approach and builds it directly into the product development flow. Integrating UAT into whole the flow of developing requires that even the most cross-functional team access a whole cavalcade of roles inside and outside of the team to ensure that AUAT reduces the chance of doing the wrong thing and at the same time reduces the chance of doing the right thing wrong.

Agile re-defines acceptance testing as a “formal description of the behavior of a software product[1].”

Agile re-defines acceptance testing as a “formal description of the behavior of a software product[1].”

User acceptance test (UAT) is a process that confirms that the output of a project meets the business needs and requirements. Classically, UAT would happen at the end of a project or release. Agile spreads UAT across the entire product development life cycle re-defining acceptance testing as a “formal description of the behavior of a software product[1].” By redefining acceptance testing as a description of what the software does (or is supposed to do) that can be proved (the testing part), Agile makes acceptance more important than ever by making it integral across the entire Agile life cycle.

Agile begins the acceptance testing process as requirements are being discovered. Acceptance tests are developed as part of the of the requirements life cycle in an Agile project because acceptance test cases are a form of requirements in their own right. The acceptance tests are part of the overall requirements, adding depth and granularity to the brevity of the classic user story format (persona, goal, benefit). Just like user stories, there is often a hierarchy of granularity from an epic to a user story. The acceptance tests that describe a feature or epic need to be decomposed in lock step with the decomposition of features and epics into user stories. Institutionalizing the process of generating acceptance tests at the feature and epic level and then breaking the stories and acceptance test cases down as part of grooming is a mechanism to synchronize scaled projects (we will dive into greater detail on this topic in a later entry).

As stories are accepted into sprints and development begins, acceptance test cases become a form of executable specifications. Because the acceptance test describes what the user wants the system to do, then the functionality of the code can be compared to the expected outcome of the acceptance test case to guide the developer.

When development of user stories is done the acceptance test cases provide a final feedback step to prove completion. The output of acceptance testing is a reflection of functional testing that can be replicated as part of the demo process. Typically, acceptance test cases are written by users (often Product Owners or subject matter experts) and reflect what the system is supposed to do for the business. Ultimately, it provides proof to the user community that the team (or teams) are delivering what is expected.

As one sprint follows another, the acceptance test cases from earlier sprints are often recast as functional regression tests cases in later sprints.

Agile user acceptance testing is a direct reflection of functional specifications that guide coding, provide basis for demos and finally, ensure that later changes don’t break functions that were develop and accepted in earlier sprints. UAT in an Agile project is more rigorous and timely than the classic end of project UAT found in waterfall projects.

[1] http://guide.agilealliance.org/guide/acceptance.html, September 2015

Listen to the Software Process and Measurement Cast 309

Software Process and Measurement Cast number 309 features our essay on Agile user acceptance testing. Agile user acceptance testing (AUAT) confirms that the output of a project meets the business’ needs and requirements. The concept of acceptance testing early and often is almost inarguable, whether you are using Agile or any other method. AUAT generates early customer feedback, which increases customer satisfaction and reduces the potential for delivering defects. While implementing an effective and efficient AUAT isn’t always easy it most certainly is possible!

The essay begins:

The classic definition of a user acceptance test (UAT) is a process that confirms that the output of a project meets the business needs and requirements. UAT in an Agile project generally is more rigorous and timely than the classic end of project UAT found in waterfall projects. In waterfall projects, the UAT is usually the last step in the development process.  The problem with that classic scenario is that significant defects are found late in the process, or worse, the business discovers that what is being delivered isn’t exactly what they wanted. Agile projects provide a number of opportunities to interject UAT activities throughout the process, starting with the development of user stories, to the sprint reviews and demos, and finally the UAT sprints at the end of a release.  Each level provides a platform for active learning and feedback from the business.

Listen to the rest of the essay, here! —- this will be a link.


SPaMCAST 310 features our interview with Michael Burrows. This is Michael’s second visit to the Software Process and Measurement Cast.  In this visit we discussed his new book, Kanban from the Inside.  The book lays out why Kanban is a management method built on a set of values rather than just a set of techniques. The argument is made that Kanban leads to better outcomes for projects, managers, organizations and customers!

Buy and read the book before the interview!

Upcoming Events

DCG Webinars:

Agile Risk Management – It Is Still Important! October 24, 2014 11:230 EDT

Has the adoption of Agile techniques magically erased risk from software projects? Or, have we just changed how we recognize and manage risk?  Or, more frighteningly, by changing the project environment through adopting Agile techniques, have we tricked ourselves into thinking that risk has been abolished?

Upcoming Conferences:

I will be presenting at the International Conference on Software Quality and Test Management in San Diego, CA on October 1.  I have a great discount code!!!! Contact me if you are interested.

I will be presenting at the North East Quality Council 60th Conference October 21st and 22nd in Springfield, MA.

More on all of these great events in the near future! I look forward to seeing all SPaMCAST readers and listeners that attend these great events!

The Software Process and Measurement Cast has a sponsor.

As many you know I do at least one webinar for the IT Metrics and Productivity Institute (ITMPI) every year. The ITMPI provides a great service to the IT profession. ITMPI’s mission is to pull together the expertise and educational efforts of the world’s leading IT thought leaders and to create a single online destination where IT practitioners and executives can meet all of their educational and professional development needs. The ITMPI offers a premium membership that gives members unlimited free access to 400 PDU accredited webinar recordings, and waives the PDU processing fees on all live and recorded webinars. The Software Process and Measurement Cast some support if you sign up here. All the revenue our sponsorship generates goes for bandwidth, hosting and new cool equipment to create more and better content for you. Support the SPaMCAST and learn from the ITMPI.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.

You have to know where you are going BEFORE you get on the flight.

You have to know where you are going BEFORE you get on the flight.

All forms of user acceptance testing reduce project risk. Agile user acceptance testing just does it better. I could stop here and this would a short entry today, but I guess I’ll go ahead and make my point.

User acceptance testing (UAT) is most effective in reducing two forms of risk. The first is the risk of doing the wrong thing. In user acceptance testing, users interact with the software (or product) and make sure that it accomplishes what they need. Simply put, if the project team builds something that does not accomplish what the project stakeholders require, no business value is delivered. The second major form of risk that UAT targets is the risk of doing the right thing the wrong way. Assuming that the team has solved the proper business problem the solution needs to be delivered in a manner that can be used. Functionality with a poor UX or form factors might be technically correct, but is less valuable than it could be.

Classic user acceptance testing is done at or very near the end of the project and is based on the project requirements. The classic testing V-Model shows the progress from requirements to code and the linkages between deliverables and the tests that are based on those deliverables. The problem in the process that is inferred from the V-Model is twofold. The first is the feedback delay that is caused by waiting until the end of the project in order to test for acceptance. Ultimately the act of testing reduces the risk of delivering both the wrong thing and the right thing done wrong. The risk is dealt with by finding problems and rework at the end of the project, which if there are deadlines can cause secondary problems.  Pressure and stress can lead to shortcuts, technical debt and worse – defects that impact production. The second problem is the implied separation of the user acceptance test cases from the requirements. Organizations that consider user acceptance tests cases as a separate deliverable from the requirements typically forgo the knowledge that integrating the two documents provide.  In some organizations proxies, such as independent test groups (not real users), develop user acceptance test cases based on their interpretation of the requirements. This further foregoes possible feedback until the tests are executed, but at least the final user acceptance test provides an avenue to connect the requirements, test cases and functionality through the execution of acceptance testing, albeit late in the process .

Agile user acceptance testing builds on the risk reduction properties of the classic UAT. Acceptance criteria are written as part of a user story directly linking what is needed with how it will be proved. Linking the story and the acceptance criteria creates an unavoidable feedback loop that reduces both the risk of building the wrong thing or the right thing wrong. The combination of a definition of done that includes acceptance testing and Agile’s short time boxes that deliver potentially shippable functionality means that acceptance testing is performed as part of each sprint. The test-early-and-often mantra generates feedback that reduces risk earlier in the project. Finally, given that acceptance criteria are usually executed over and over as the project evolves, acceptance criteria are subject to heavy scrutiny. This helps make sure they are testing the right concepts in the software.

User acceptance testing reduces project risk. Regardless of whether you are doing Agile, lean, waterfall or something in-between, you should be doing UAT. However if you want to be truly effective and efficient, Agile user acceptance testing is far superior to other methods.

In need of some stabilization...

In need of some stabilization…

One of the central tenants of Scrum is the principle of delivering potentially shippable, functional software at the end of every sprint. This is an extension of the Agile Manifesto’s principle of delivering working software frequently. One generally accepted interpretation of this principle is that the software in production should be implementable at the end of the sprint. In order for this to happen, stories must be coded, tested (in various ways) and the broader user community needs to be convinced that it will work. The second most common interpretation of the principle focuses on the word potentially in the phrase ‘potentially shippable software’. Stressing potentially helps teams and organizations rationalize building a backlog of finished work that is released periodically or running software through final overall set of acceptance tests.

The solution some organizations adopt when they need a final set of tests is to use a hybrid practice called a hardening sprint. For example, Rumadak commented on the entry, Agile User Acceptance Testing: What, Who and Why, that “We generally have one sprint before release for regression and UAT.” I see three primary reasons for leveraging a hardening sprint. In these special cases planning and executing a hardening sprint makes sense due to legal or environmental constraints. They are:

  1. Some teams are contractually mandated to do a final acceptance test before production turnover. This scenario typically occurs in arm’s length outsourcing projects or in situations where you are purchasing commercial, off-the-shelf software.  The goal in this scenario is ensure that the organization buying the software is getting what they paid for. Note: In some highly regulated environments a final acceptance test is required.
  2. Some highly complex environments make integration and acceptance testing in a production-like environment difficult. Therefore waiting until the end of a project to do final integration and acceptance testing may be only option (in the short term). For example, when I worked in the credit card processing world, where projects sometimes needed a full test of the file maintenance cycle, including user interaction, which took roughly 36 to 48 hours and generally occurred on weekends due to the complexity of the technical environment and the resources that had to be available to run the tests.  In that case, running a daily, weekly or even bi-weekly cycle would have been prohibitive.
  3. Some types of testing require the overall system to be effective.  For example, overall stress and security testing not only requires as close to a production-like environment as possible, but also needs all of the functionality that will be delivered to actually test.

In a hardening sprint the team will run tests and will fix anything that breaks. A hardening sprint is not a backdoor to continue developing new stories or functions. The use of hardening sprints for either acceptance testing or integration testing does not preclude running integration and acceptance tests as part of every sprint.  The discipline of writing acceptance tests as part of every story and using those tests as criteria to determine whether a story is complete makes sense. Running acceptance tests every sprint makes sense even if a final overall hardening sprint is required.  Running acceptance tests as early as possible means teams will find problems that effect customer usability earlier, reducing the possibility of significant rework and  project  delays that might occur if we just wait for the hardening sprint.

ACUT isn't all sunshine and flowers.

AUAT isn’t all sunshine and flowers.

Agile user acceptance testing (AUAT) confirms that the output of a project meets the business’ needs and requirements. The concept of acceptance testing early and often is almost inarguable, whether you are using Agile or any other method. AUAT generates early customer feedback, which increases customer satisfaction and reduces the potential for delivering defects. The problem is that implementing an effective and efficient AUAT isn’t always easy. Several complicating factors include:

  1. Having the right technical environment. Pithythoughts strongly argued in his comments (read them here) that acceptance testing without exposure to the true production environment is insufficient to really determine whether the work is “done and that business value has been delivered.” I believe that no seasoned developer or IT pro would argue, however it rarely happens except for new products or for relatively self-contained applications.  For example, consider the expense of having an environment that mirrors production for SAP on top of multiple test and development environments. Environmental compromises are made, and each level of abstraction away from the production mirror increases the risk that what seems ‘acceptable’ might not be in the real world.
  2. Acceptance test cases can be fragile. One of the central tenants in Agile projects is to embrace change. In many cases that change will need to be reflected in the acceptance test cases and criteria. Given that acceptance tests are known for suffering from false positive test failures (they are perceived to be true when they are not) they need to be carefully evaluated as the project evolves or the feedback the team gets from Agile acceptance testing can be counterproductive. Acceptance test cases need to be refactored just like code.
  3. Acceptance Test Driven Development (ATDD) can encourage big upfront design rather than emergent design favored in Agile projects. The development of acceptance criteria can lead the product owner (or other stakeholders) involved in developing the test cases to focus on “how” business value will be delivered technically rather than on the functionality or the “what” will be delivered. Fixing the design too early any project increases the likelihood of rework, and many times is the reason for rejecting change later in the project.  An emergent design allows teams to design only what is needed and then to learn from implementing and demonstrating that design.  Acceptance testing is part of the feedback loop needed to create an evolving design.
  4. Some product owners are unable or unwilling to write acceptance criteria and tests. Many times, product owners have day jobs in addition to their role as the voice to the business, which makes it harder to participate in an Agile team.  In rare cases, some product owners look at the activity of writing Agile user acceptance test cases as being beneath them. I once heard a “product owner” state that writing test cases was an IT job or maybe a clerk’s job – certainly not something he should do. Regardless of whether you are using Agile or waterfall, user acceptance testing is a critical step towards implementation. I find that in these scenarios, generating the proper level of product owner involvement takes a significant amount of time and effort. In the short run, partner a business analyst with the product owner to shepherd the creation of acceptance criteria and then pursue a long term coaching opportunity to change the product owner’s attitude.

Agile user acceptance testing is a method of pursuing feedback from the business early and often by focusing the product owner (and by extension the rest of the business) on considering and then documenting the proof they will need to accept what the project will deliver. AUAT is not easy nor is it free.  There are numerous issues that need to be dealt with to make sure Agile acceptance testing is done correctly.  Without an environment that is as close to production it will be difficult to interpret the results.  Just because you capture acceptance criteria once does not mean you are done, acceptance criteria need to be continually review and refactored just like code.  If the process you are using to generate the acceptance criteria mean that you need to develop the solution too early, you need to step back and take the process more incrementally. Finally remember the word user in the Agile user acceptance testing otherwise you are just doing developer-based functionality testing.  Agile user acceptance testing is not all sunshine and flowers, but the outcome is generally worth the hassle.