How to Conduct Effective User Acceptance Testing

Posted on December 15, 2020 By Kalleid Blog

According to a report from Forbes, 25 percent of technology projects fail outright; 20 to 25 percent don’t show any return on investment; and as much as 50 percent need massive reworking by the time they’re finished.

Why is this the case?  It is a complex question that likely comes with complex answers.  Technical failures, scope creep, platform and performance issues, and the like all happen rampantly in enterprise IT organizations.

An often-overlooked aspect of successful IT project execution may hold the key to reducing risk and delivering value for the business.  IT risk and failures in IT projects often result from inadequate user-focused testing.  While designed to mitigate risk, inadequate user testing can actually hide critical gaps that you pay for later, such as rolling out systems that poorly suit the business needs.

There are many different types of testing that can occur during the software development lifecycle (SDLC) to verify that a software system is functioning properly prior to release. Some of these include:

• Functional testing – validates each function of the software system against the functional requirements
• User interface testing – verifies that the graphical user interface looks and works as expected across multiple browsers and devices
• Performance testing – validates that response times and overall effectiveness of the application is within accepted parameters
• Integration testing – verifies that the interfaces between different modules within an application are working properly
• Usability (aka user experience) testing – confirms that the application is user friendly
• Database testing – verifies that the application stores and handles data in a consistent, safe manner
• Security testing – ensures the application restricts actions and permissions wherever necessary
• Regression testing – ensures that aspects of a system that were not directly involved in a change (e.g., update, bug fix, maintenance, patch, enhancement, etc.) are unaffected by the changes and still functioning as intended
• User acceptance testing (UAT) – verifies that the software performs in a manner that is acceptable to the user.

This last type of testing (UAT) is critical, yet some organizations make the mistake of not doing it at all, or simply having users repeat the functional testing that was done by the development team. UAT should not be taken lightly, as it serves as an important safeguard to ensure developers have properly interpreted and implemented the functional requirements. Organizations which skip UAT testing, or do not conduct it properly, often realize its importance only after suffering financial and operational losses and post release rework.

In this blog, we discuss best practices for conducting UAT and writing UAT scripts.

What is User Acceptance Testing and Why is it Important?

Most of the software testing described in the previous section is done by the development team to find and fix bugs, and also to ensure that the software meets the functional requirements that were developed in collaboration with the business. Being technical people, developers design and validate the product from a technical perspective. As a result, there is always the chance that the developers misinterpret the requirements, or that the requirements were poorly defined and do not capture the essence of the customer’s needs.

Unlike functional testing done by the development team, UAT is not focused on finding bugs in the software (although any bugs found are certainly noted and communicated back to the developers). As the last phase of the software testing process, the purpose of UAT is for the business that requested the software to perform a final check to confirm that the developers have properly interpreted and implemented the functional requirements prior to system rollout. In other words, UAT testing helps to make sure the system does what is intended from a user perspective.

The user-centric approach of user acceptance testing makes it one of the most important aspects of testing or validating a computer system. One of the biggest mistakes a company can make is to rush through or skimp on UAT. As users are not involved in the software development, they bring a fresh mind to software testing, which can help to clarify requirements for the software and also unearth additional requirements that the business sponsor may not have thought of prior to seeing the system. UAT can also help the business sponsor discover problems in aspects of the application that were not tested in the development environment. Like many other aspects of software development, UAT is fundamentally about making sure the requirements are right the first time.

User Acceptance Testing Best Practices

UAT is performed at a critical juncture in the project delivery cycle, and it is therefore essential to plan and execute it successfully. Key best practices for successful UAT testing include:

Create a good UAT team – The first thing that you need to do to develop your testing plan is identify all system users and their different roles in the system. The UAT testing team should ideally be composed of users in every group or area of an organization so that every user role can be tested. That said, testing every role in a large company with a large application is usually not feasible. In this case, it is important to perform a risk-based assessment to identify and prioritize those roles involved in critical business objectives for testing. Key roles to consider:

• UAT testers – testers should be familiar with the new requirements that the system is intended to fulfill and have the time set aside to conduct a thorough testing process.
• Business owner – the business owner is responsible for guiding the UAT owner (i.e., project manager) in testing for the requirements. This person also owns the responsibility for managing any change control items that come out of UAT.
• UAT owner (i.e., project manager)– the UAT owner is responsible for updating the business sponsor on the status of the tests, collaborating with them on decisions, and managing the work for the testers. The UAT owner should ideally have good knowledge of the software system being tested.

Create a UAT plan – A testing plan should be developed that includes all the relevant information necessary for conducting UAT – dates, testers, testing environment, timeframe for building the testing environment, scope of testing, testing strategy (e.g., in person or self-paced testing), communication protocols, team roles and responsibilities, acceptance criteria, templates, etc. This plan should ride alongside the requirements solicitation and refinement process. As part of agile deployments, light UATs can be used to ensure acceptance of interim builds.

Create a proper testing environment – UAT should be conducted in a physical space that allows software QA team members, the project manager, and users to work through the test cases together. Additionally, UAT should be conducted in a virtual environment that matches the exact configuration of the production environment (e.g., a virtual laboratory environment) so that users can test real-world cases.

Create good documentation. Proper documentation is essential for UAT. In addition to the UAT plan, documentation for UAT should include:

• Test outcomes – testers should be provided with a template used to document all outcomes, comments, issues and statuses during UAT.
• Requirements for sign-off – the business owner should define and document a set of requirements for moving from UAT to production environment. Test cases should be designated as high, medium and low based on their impact to the business. Issues found in anything other than a low impact test case should be corrected prior to going live with the system.
• Change control process – a standard process should be documented and followed for any issues that need to be corrected before go-live.

Writing Effective UAT Scripts

Effective test scripts are the cornerstone of successful UAT. Before test scripts can be written, however, detailed user stories and acceptance criteria need to be developed, as these provide the foundational information necessary to write the scripts.

Develop user stories that tie back to business needs. To keep users at the center of the conversation, a set of real-world use cases (user stories) covering all essential user roles should be a part of the requirements development process. A central feature of Agile software development frameworks, user stories are deliberately simplified versions of requirements that clearly articulate how a specific software feature delivers value to the end user. User stories are usually expressed in the following format:

As a < type of user >, I want < some outcome> so that < some reason >

Here’s an example of a user story: As an analytical chemist, I want to be able to capture purification data and spectra in my electronic laboratory notebook so I can record the characterization data for the samples and share the information with colleagues.

Define Acceptance Criteria (AC). As a user story does not contain enough information to develop effective UAT scripts, a set of concise acceptance criteria should be developed for each user story.  Acceptance criteria provide the details of the user story that will form the basis of writing the UAT scripts. User stories define requirements for a software feature or functionality, whereas AC define the conditions that have to be met by the system for the user story or the requirement to be satisfied.

Each user story may have up to 3 or 4 acceptance criteria associated with it which help define the scope of the user story. In order to make sure all scenarios for a given user story are captured (including error scenarios), AC are best developed collaboratively along with the user stories in a Requirements Workshop attended by business analysts, business stakeholders, UAT owner, and members of the software QA team.

The agile approach to software development typically uses a scenario-oriented approach for describing acceptance criteria in a Given/When/Then format:

• GIVEN (the context/initial conditions)
• WHEN (an action performed)
• THEN (the expected result should occur)

Acceptance criteria will be used during UAT to determine if a particular feature meets user requirements and is performing at an acceptable level for business users. AC must be written in a way that allows users to grade them on a pass/fail basis, with the collective grades of all the AC in UAT informing the business decision on whether to accept and deploy the system.

Write Effective UAT scripts. Now that user stories and associated acceptance criteria have been defined, it is time to write the test scripts. Authors of the testing scripts may rely on functional specifications to build the content, but this is shortsighted. The UAT test scripts, unlike functional test scripts, should be written in concert with the users. The users should ensure that the scripts provide adequate coverage of the requirements and that the steps are executed in a way that reflects the business process.

Too often, UAT scripts are not reviewed by the business users prior to execution, resulting in errors or issues that could have been prevented through proper training or refactoring the scripts. If users are not engaged in reviewing the UAT script at all, they may execute them, only to find that the way the system was configured or designed does not suit the business process. By the time you reach the stage of test execution, it may be too late to address changes, or worse, your testing results can derail the deployment altogether.

A few additional tips for writing effective UAT scripts include:

• Make test cases easy to do – Put yourself in the shoes of the tester when writing test scripts and make sure the scripts are concise and clear.
• Remove bias from your statements – Using biased language can affect the results of the test. Consider the difference between “now see if you can find …” and “now find …” The former phrasing implies the task will be difficult, and this may convince the user that it is.
• Build your scripts at the same time you are designing user stories and acceptance criteria – it’s a good idea to be write the scripts as soon as possible in the SDLC so you don’t end up having to build a massive test database right before UAT is conducted.
• Dry run or practice the scripts with real users prior to formal execution. This is a critical step for ensuring your scripts are realistic and provide adequate coverage of the business process.

Conclusion

User acceptance testing is one of the most important aspects of testing or validating a computer system. In UAT, users test specific features of a software application from a user’s perspective to verify they satisfy business needs. Given this user-centric approach, UAT often uncovers missed issues and helps to clarify requirements for software applications.

In this blog, we discussed best practices for both conducting UAT and writing UAT scripts. Given the business and operational risks involved in releasing a new or updated application into your organizational workflows, any time and resources spent on an effective UAT process prior to system go-live is a wise investment. While software developers prefer to identify and fix problems in earlier phases of testing, finding issues in UAT is still much better (i.e., cheaper) than after the software has gone live.

by Kalleid