CS223: Software Engineering Lecture 31: Acceptance Testing

Recap Incremental approach Top Down approach Bottom up approach Big bang and Sandwich approach System Testing

Objective After completing this lecture students will be able to o Perform acceptance testing of their software

Acceptance testing Acceptance testing consists of o Comparing a software system to its initial requirements and to the current needs of its end-users o In the case of a contracted program, to the original contract It is a crucial step that decides the fate of a software system. Its visible outcome provides an important quality indication for the customer to determine whether to accept or reject the software product

Goals of Acceptance Testing Verify the man-machine interactions Validate the required functionality of the system Verify that the system operates within the specified constraints Check the system’s external interfaces

6 Types of Acceptance Testing Acceptance testing is a formal testing conducted to determine whether a system satisfies its acceptance criteria There are two categories of acceptance testing: o User Acceptance Testing (UAT)  It is conducted by the customer  To ensure that system satisfies the contractual acceptance criteria before being signed-off as meeting user needs. o Business Acceptance Testing (BAT)  It is undertaken within the development organization of the supplier  To ensure that the system will eventually pass the user acceptance testing.

Acceptance Criteria o Functional Correctness and Completeness o Accuracy o Data Integrity o Data Conversion o Backup and Recovery o Competitive Edge o Usability o Performance o Start-up Time o Stress o Reliability and Availability o Maintainability and Serviceability o Robustness o Timeliness o Confidentiality and Availability o Compliance o Installability and Upgradability o Scalability o Documentation 7 The acceptance criteria are defined on the basis of the following attributes:

Functional Correctness and Completeness Does the system do what we want it to do? All the features which are described in the requirements specification must be present in the delivered system. How to show the functional correctness of a system? Requirement traceability matrix

Accuracy Does the system provide correct results? Measures the extent to which a computed value stays close to the expected value False positive, false negative

Data Integrity Data integrity mechanisms detect changes in a data set. Preservation of the data while it is transmitted or stored

Data Conversion How can we undo a conversion and roll back to the earlier database version(s) if necessary? How much human involvement is needed to validate the conversion results? How are the current data being used and how will the converted data be used? Will the data conversion software conduct integrity checking as well?

12 Selection of Acceptance Criteria The acceptance criteria discussed are too many and very general The customer needs to select a subset of the quality attributes The quality attributes are prioritize them to specific situation IBM used the quality attribute list CUPRIMDS for their products – Capability, Usability, Performance, Reliability, Installation, Maintenance, Documentation, and Service Ultimately, the acceptance criteria must be related to the business goals of the customer’s organization

13 Acceptance Test Plan

Acceptance Test Execution The acceptance test cases are divided into two subgroups o The first subgroup consists of basic test cases, and o The second consists of test cases that are more complex to execute The acceptance tests are executed in two phases o First phase, the test cases from the basic test group are executed o If the test results are satisfactory then the second phase, in which the complex test cases are executed, is taken up. o In addition to the basic test cases, a subset of the system-level test cases are executed by the acceptance test engineers to independently confirm the test results 14

Activities in Acceptance Testing Acceptance test execution activity includes the following detailed actions: o The developers train the customer on the usage of the system o The developers and the customer co-ordinate the fixing of any problem discovered during acceptance testing o The developers and the customer resolve the issues arising out of any acceptance criteria discrepancy

Acceptance Test Execution 16 The acceptance test engineer may create o An Acceptance Criteria Change (ACC) document o To communicate the deficiency in the acceptance criteria  To the supplier Given to the supplier’s marketing department through the on-site system test engineers

Acceptance Test Report The acceptance test activities are designed to reach at a conclusion: o Accept the system as delivered o Accept the system after the requested modifications have been made o Do not accept the system Intermediate decisions are made before making the final decision. o Continuation of acceptance testing if the results of the first phase of acceptance testing is not promising o Changes be made to the system before acceptance testing can proceed to the next phase The acceptance team prepares a test report on a daily basis 17

18 Acceptance Test Status Report

19 Acceptance Test Summary Report Uniquely identifies the report What acceptance testing activities took place Difference between planned and executed testing Total number of test cases executed, Number of passing test cases, Number of failing test cases, Identifies all the defects, Summarizes the acceptance criteria to be changed The deviations of the acceptance criteria that are captured in the ACC during the acceptance testing are discussed. Accept as delivered Accept after the modifications Do not accept the system

20 Acceptance Testing in eXtreme Programming In XP framework the user stories are used as acceptance criteria The user stories are written by the customer as things that the system needs to do for them Several acceptance tests are created to verify the user story has been correctly implemented The customer is responsible for verifying the correctness of the acceptance tests and reviewing the test results A story is incomplete until it passes its associated acceptance tests Ideally, acceptance tests should be automated, either using the unit testing framework, before coding The acceptance tests take on the role of regression tests

State-of-Practice of Acceptance Testing 1. Study the requirements document, 2. Obtain help from a domain expert, 3. Develop a set of test cases, and 4. Demonstrate to the customer, by using the test cases, that the software indeed possesses the required functionality as formally or informally specified.

Other Approaches: FSM Requirement Specification Requirement Language Processor (RLP) Augmented Finite State Machine (FSM) Test Plan Generator (TPG) Test Scripts Automatic Test Execution (ATE) Test Results

Other Approaches: FSM

Scenario Analysis

Tools It runs automated acceptance tests written in a behavior- driven development (BDD) style

Tools Protractor is an end-to-end test framework for AngularJS applications. Protractor runs tests against your application running in a real browser, interacting with it as a user would. AngularJS extends HTML with new attributes.

Tools Selenium WebDriver Selenium is a portable software testing framework for web applications. WebDriver is the name of the key interface against which tests should be written

Tools Jasmine is an open source testing framework for JavaScript It is heavily influenced by other unit testing frameworks

Thank you Next Lecture: Software Maintenance