Requirements-Based Testing

Assignments Read Ostrand and Balcer paper on the “category partition method”, available on the Course Handouts page, for Monday, October 23.

Review: Test Adequacy Criteria Adequacy criterion: A rule that lets us judge the sufficiency of a set of test data for a piece of software. Adequacy criteria lead to systematic, rigorous methodologies for testing software.

Review: Some Adequacy Criteria Requirements coverage Statement coverage Branch coverage Condition coverage Path coverage Boundary value coverage Dataflow coverage

Review: Some Adequacy Criteria Requirements coverage Statement coverage Branch coverage Condition coverage Path coverage Boundary value coverage Dataflow coverage

Requirements Coverage Definition Requirements coverage: Given system S with requirements R(S), create a test suite T such that for each requirement r in R(S), T contains at least one test case that verifies that S behaves correctly relative to r.

Simple Requirements Coverage: Requirements for SQRT Startup: User types “sqrt” to invoke the program The prompt “Enter number:” is displayed and the system waits for input If user types “sqrt [arg]” the prompt is not displayed, the system proceeds at 2. Action on input: If input >= 0 and input < MAXINT, square root of input is displayed and program terminates If input < 0 and input > MININT system outputs “number must be positive” If input is a non-number, system says “enter a non-negative integer” If input < MININT or input > MAXINT output is unpredictable

Covering Requirements: Domain Partitioning When conditions are ranges, this partitions the space into one valid and two invalid sub-ranges (e.g. between 0 and MAXINT) When conditions are boolean, this partitions the space into one vallid and one invalid class (e.g. input is a number or a non-number) When conditions are sets, this partitions the space into one valid (in the set) and one invalid (not in the set), provided you expect each set member to be handled the same.

Covering Requirements: Boundary-Value Testing After partitioning the input domain into classes, test the program using input values not only “inside” the classes, but also at their boundaries. E.g., for sqrt, where requirement 2.b states “input >= 0 and input < MAXINT”, what boundaries could be considered?

Covering Requirements: Loop Testing Also consider requirements related to “loops” or iteration: addressing iteration implies considering “boundaries” related to numbers of repetitions. E.g., given the requirement: “for each record on the CallStack, free any registers associated with that record”, what boundaries could be considered, at least theoretically?

Requirements Coverage Process System Requirements Testing Requirements Test Cases

Testing Requirements Testing requirements are a list of program behaviors that we require be tested, derived from a program specification They make the different conditions that test cases must exercise explicit

Simple Requirements Coverage: Requirements for SQRT Startup: User types “sqrt” to invoke the program The prompt “Enter number:” is displayed and the system waits for input If user types “sqrt [arg]” the prompt is not displayed, the system proceeds at 2. Action on input: If input >= 0 and input < MAXINT, square root of input is displayed and program terminates If input < 0 and input > MININT system outputs “number must be positive” If input is a non-number, system says “enter a non-negative integer” If input < MININT or input > MAXINT output is unpredictable

Testing Requirements for SQRT Invoke SQRT with no argument Invoke SQRT with an argument Use a non-numeric argument Use an argument < 0 and > MININT Use an argument = 0 Use an argument > 0 and < MAXINT Use an argument > MAXINT Use an argument < MININT

Testing Requirements for SQRT with Traceability Invoke SQRT with no argument 1a, 1b Invoke SQRT with an argument 1c Use a non-numeric argument 2c Use an argument < 0 and > MININT 2b Use an argument = 0 2a, 2b Use an argument > 0 and < MAXINT 2a Use an argument > MAXINT 2d Use an argument < MININT 2d

Test Case for SQRT Test 1. PURPOSE: SETUP: Test the invocation of sqrt without arguments, and the operation of sqrt on an integer in the expected range. SETUP: None needed INPUTS and EXPECTED OUTPUTS: inputs expected output ------------------------------------------- “sqrt” “Enter number:” “4” “2” followed by command-line prompt TESTING REQUIREMENTS COVERED: 1 and 4

In the Absence of Requirements? From the Aristotle user manual: To invoke the Aristotle menu interface, type “aristotle source-file”, where source-file is an optional argument that names a C-language source file you wish to work with. The Aristotle top menu is displayed. You can exit the Aristotle menu interface from any menu by typing “x” and pressing the return key.

Requirements Coverage Ad hoc Documentation System Requirements Testing Requirements Test Cases

System Requirements Derived from Prior User Documentation 1 Invoking the system 1.a “aristotle [source-file] displays top menu 1.a.i source-file must be a valid C file 1.b “aristotle <CR>” displays top menu 2 Exiting the system 2.a “x <CR>” exits system from any menu From these, you can derive testing requirements.

Testing Requirements Derived from Prior System Requirements 1 Invoking the system 1.a Type “aristotle <filename>” for existing accessible C file 1.b Type “aristotle <filename>” for nonexisting file 1.c Type “aristotle <filename>” for existing but protected C file 1.d Type “aristotle <filename>” for a fortran file named .c 1.e Type “aristotle <CR>” 2 Exiting the system 2.a From each menu, try “x CR” 2.b From each menu, try “X CR” 2.c From each menu, try “C CR” for C not an option From these, you can construct test cases.

Test Plans A test plan is a “specification” for the testing process for a given system. A test plan contains: Information on the test process being used (black box, white box, etc.) Testing requirements Test cases that meet these requirements, including their inputs, outputs, and traceability information

Test Plans Test plans add rigour to testing, supporting reuse, assessment of success, replication, and documentation -- especially when coupled with meaningful testing methodologies or test adequacy criteria But at what cost? Test plans may be partially implemented via test automation.

Requirements Coverage Ad hoc Documentation System Requirements Test Plan Testing Requirements Test Cases