1. Requirements-Based Testing

2. Assignments
Read Ostrand and Balcer paper on the “category partition method”, available on the Course Handouts page, for Monday, October 23.

3. Review: Test Adequacy Criteria
Adequacy criterion: A rule that lets us judge the sufficiency of a set of test data for a piece of software.
Adequacy criteria lead to systematic, rigorous methodologies for testing software.

4. Review: Some Adequacy Criteria
Requirements coverage
Statement coverage
Branch coverage
Condition coverage
Path coverage
Boundary value coverage
Dataflow coverage

5. Review: Some Adequacy Criteria
Requirements coverage
Statement coverage
Branch coverage
Condition coverage
Path coverage
Boundary value coverage
Dataflow coverage

6. Requirements Coverage Definition
Requirements coverage: Given system S with requirements R(S), create a test suite T such that for each requirement r in R(S), T contains at least one test case that verifies that S behaves correctly relative to r.

7. Simple Requirements Coverage: Requirements for SQRT
Startup:
User types “sqrt” to invoke the program
The prompt “Enter number:” is displayed and the system waits for input
If user types “sqrt [arg]” the prompt is not displayed, the system proceeds at 2.
Action on input:
If input >= 0 and input < MAXINT, square root of input is displayed and program terminates
If input < 0 and input > MININT system outputs “number must be positive”
If input is a non-number, system says “enter a non-negative integer”
If input < MININT or input > MAXINT output is unpredictable

8. Covering Requirements: Domain Partitioning
When conditions are ranges, this partitions the space into one valid and two invalid sub-ranges (e.g. between 0 and MAXINT)
When conditions are boolean, this partitions the space into one vallid and one invalid class (e.g. input is a number or a non-number)
When conditions are sets, this partitions the space into one valid (in the set) and one invalid (not in the set), provided you expect each set member to be handled the same.

9. Covering Requirements: Boundary-Value Testing
After partitioning the input domain into classes, test the program using input values not only “inside” the classes, but also at their boundaries.
E.g., for sqrt, where requirement 2.b states “input >= 0 and input < MAXINT”, what boundaries could be considered?

10. Covering Requirements: Loop Testing
Also consider requirements related to “loops” or iteration: addressing iteration implies considering “boundaries” related to numbers of repetitions.
E.g., given the requirement: “for each record on the CallStack, free any registers associated with that record”, what boundaries could be considered, at least theoretically?

11. Requirements Coverage Process
System
Requirements
Testing
Requirements
Test
Cases

12. Testing Requirements
Testing requirements are a list of program behaviors that we require be tested, derived from a program specification
They make the different conditions that test cases must exercise explicit

13. Simple Requirements Coverage: Requirements for SQRT
Startup:
User types “sqrt” to invoke the program
The prompt “Enter number:” is displayed and the system waits for input
If user types “sqrt [arg]” the prompt is not displayed, the system proceeds at 2.
Action on input:
If input >= 0 and input < MAXINT, square root of input is displayed and program terminates
If input < 0 and input > MININT system outputs “number must be positive”
If input is a non-number, system says “enter a non-negative integer”
If input < MININT or input > MAXINT output is unpredictable

14. Testing Requirements for SQRT
Invoke SQRT with no argument
Invoke SQRT with an argument
Use a non-numeric argument
Use an argument < 0 and > MININT
Use an argument = 0
Use an argument > 0 and < MAXINT
Use an argument > MAXINT
Use an argument < MININT

15. Testing Requirements for SQRT with Traceability
Invoke SQRT with no argument a, 1b
Invoke SQRT with an argument c
Use a non-numeric argument c
Use an argument < 0 and > MININT b
Use an argument = a, 2b
Use an argument > 0 and < MAXINT a
Use an argument > MAXINT d
Use an argument < MININT d

16. Test Case for SQRT Test 1. PURPOSE: SETUP:
Test the invocation of sqrt without arguments, and the operation of sqrt on an integer in the expected range.
SETUP:
None needed
INPUTS and EXPECTED OUTPUTS:
inputs expected output
“sqrt” “Enter number:”
“4” “2” followed by command-line prompt
TESTING REQUIREMENTS COVERED:
1 and 4

17. In the Absence of Requirements?
From the Aristotle user manual:
To invoke the Aristotle menu interface, type “aristotle source-file”, where source-file is an optional argument that names a C-language source file you wish to work with. The Aristotle top menu is displayed. You can exit the Aristotle menu interface from any menu by typing “x” and pressing the return key.

18. Requirements Coverage
Ad hoc
Documentation
System
Requirements
Testing
Requirements
Test
Cases

19. System Requirements Derived from Prior User Documentation
1 Invoking the system
1.a “aristotle [source-file] displays top menu
1.a.i source-file must be a valid C file
1.b “aristotle <CR>” displays top menu
2 Exiting the system
2.a “x <CR>” exits system from any menu
From these, you can derive testing requirements.

20. Testing Requirements Derived from Prior System Requirements
1 Invoking the system
1.a Type “aristotle <filename>” for existing accessible C file
1.b Type “aristotle <filename>” for nonexisting file
1.c Type “aristotle <filename>” for existing but protected C file
1.d Type “aristotle <filename>” for a fortran file named .c
1.e Type “aristotle <CR>”
2 Exiting the system
2.a From each menu, try “x CR”
2.b From each menu, try “X CR”
2.c From each menu, try “C CR” for C not an option
From these, you can construct test cases.

21. Test Plans
A test plan is a “specification” for the testing process for a given system.
A test plan contains:
Information on the test process being used (black box, white box, etc.)
Testing requirements
Test cases that meet these requirements, including their inputs, outputs, and traceability information

22. Test Plans
Test plans add rigour to testing, supporting reuse, assessment of success, replication, and documentation -- especially when coupled with meaningful testing methodologies or test adequacy criteria
But at what cost?
Test plans may be partially implemented via test automation.

23. Requirements Coverage
Ad hoc
Documentation
System
Requirements
Test Plan
Testing
Requirements
Test
Cases