1. CS5123 Software Validation and Quality Assurance
Lecture 02
Overview of Software Testing

2. Approach to remove bugs
Testing
Feed input to software and run it to see whether its behavior is as expected
Limitations
Impossible to cover all cases
Test oracles (what is expected)
Static checking
Identify specific problems (e.g., memory leak) in the software by scanning the code or all possible paths
Limited problem types
False positives 2

3. Approach to remove bugs
Formal Proof
Formally prove that the program implements the specification
Limitations
Difficult to have a formal specification
The proof cost a lot of human efforts
Inspection
Manually review the code to detect faults
Limitations:
Hard to evaluate
Sometime hard to get progress 3

4. Answer is testing, why?
“50% of my employees are testers, and the rest spends 50% of their time testing”
---- Bill Gates, 1995
More reliable than inspection, relatively cheap
Actually in the old days, when testing is expensive, inspection was the major answer
You get what you pay (linear rewards)
Compared to other 3 approaches
Inspection, static checking, formal proof 4

5. Testing: Concepts 5 Test case Test oracle Test suite Test script
Test driver
Test coverage 5

6. Testing: Concepts 6 Test case Test oracle
An execution of the software with a given list of input values
Include:
Input values, sometimes fed in different steps
Expected outputs
Test oracle
The expected outputs of software by feeding in a list of input values
A part of test cases
Hardest problem in auto-testing: test oracle problem 6

7. Testing: Concepts: Example7

8. Testing: Concepts 8 Test suite Test Script A collection of test cases
Usually these test cases share similar pre-conditions and configuration
Usually can be run together in sequence
Different test suites for different purposes
Smoke test, Certain platforms, Certain feature, performance, …
Test Script
A script to run a sequence of test cases or a test suite automatically 8

9. Testing: Concepts 9 Test Driver Test Coverage
A software framework that can load a collection of test cases or a test suite
It can usually handle the configuration and comparison between expected outputs and actual outputs
Test Coverage
A measurement to evaluate how well the testing is done
The measure can be based on multiple elements
Code
Input combinations
Specifications 9

10. Granularity of Testing: V-model10

11. Granularity of testing
Unit / Module Testing
Test of a single module
Integration Testing
Test the interaction between modules
System Testing
Test the system as a whole, by developers on test cases
Acceptance Testing
Validate the system against user requirements, by customers with no formal test cases 11

12. Stage of software Testing
Development-time testing
Unit testing, Integration Testing
Before-release testing
System testing, Acceptance Testing
User testing
Actual usage -> field bugs & patches 12

13. Types of testing by how they are designed
White box testing
The tester knows everything about the implementation
They knows where the bugs are more probably be
They can exercise paths in the code
Black box testing
The tester are just like normal users
They just try to cover input space and corner cases 13

14. Black Box Testing: General Guidelines
Divide value range and cover each part
Cover boundary values
Try to reach all error messages
Try to trigger potential exceptions
Feed invalid inputs
wrong formats, too long, too short, empty, …
Try combinations of all above
Repeat same and use different inputs for many times if the input is a sequence 14

15. Black Box Testing Techniques
Boundary value testing
Boundary value analysis
Robustness testing
Worst case testing
Equivalence class testing
Decision table based testing 15

16. Boundary Value Analysis
Errors tend to occur near the extreme values of an input variables
Boundary value analysis focuses on the boundary of the input space to identity test cases
Boundary value analysis selects input variable values at their
Minimum
Just above the minimum
A nominal value
Just below the maximum
Maximum 16

17. Example of Boundary Value Analysis
Assume a program accepting two inputs y1 and y2, such that a < y1< b and c < y2 < d . . . . 17

18. Single Fault Assumption for Boundary Value Analysis
Boundary value analysis is also augmented by the single fault assumption principle
“Failures occur rarely as the result of the simultaneous occurrence of two (or more) faults”
In this respect, boundary value analysis test cases can be obtained by holding the values of all but one variable at their nominal values, and letting that variable assume its extreme values 18

19. Generalization of Boundary Value Analysis
The basic boundary value analysis can be generalized in two ways:
By the number of variables - (4n +1) test cases for n variables
By the kinds of ranges of variables : map to order
Strings
Sequences
Complex Data Structures, e. g., trees 19

20. Application Scenario of Boundary Value Analysis
Several independent variables that represent bounded physical quantities
No consideration of the function of the program, nor of the semantic meaning of the variables
Good news: We can distinguish between physical and other types of variables 20

21. Robustness Testing 21 A simple extension of boundary value analysis
In addition to the five boundary value analysis values of variables, we add values slightly greater that the maximum (max+) and a value slightly less than the minimum (min-)
The main value of robustness testing is to force attention on exception handling 21

22. Example of Robustness Testing. . . … . . . 22
UTSA CS3773

23. Worst Case Testing
No single fault assumption: error happens when more than one variable has an extreme value
Considering n inputs in boundary analysis, we take the Cartesian product of the five values for 1, 2, 3, … n variables
We can have 5n test cases for n input variables
The more interactions on inputs -> more on worse case testing
Input partitions: Length & Width vs. Length & price 23

24. Example of Worst Case Testing 24

25. Equivalence Class Testing
Divide the value range of an input to a number of subsets
Subsets are disjoint
The union of the subset if the value range
Values in one subset does not make difference for the software concerned
Water temp in a car: <32, 32 – 212, >=212
Normal colors vs. Metallic colors 25

26. Example of Equivalence Class Testing26

27. Equivalence Class Testing
The use of equivalence class testing has two motivations:
Sense of complete testing
The entire set is represented provides a form of completeness
Avoid redundancy
The disjointness assures a form of non-redundancy
Note
Also check boundaries
Combinations of inputs also follows the rule: more interaction -> more combinations 27

28. Equivalent Class for non-numeric inputs
Feature extraction
For string and structure inputs
Split the possible value set with a certain feature
Example:
String passwd => {contains space}, {no space}
It is possible to extract multiple features from one input
String name => {capitalized first letter}, {not}
=> {contains space}, {not}
=> {length >10}, {2-10}, {1}, {0}
One test case may cover multiple features 28

29. Decision Table
Make it easy to observe that all possible conditions are accounted for
Decision tables can be used for:
Specifying complex program logic
Generating test cases with oracles 29

30. Example of Decision Table
Conditions
Printer does not print Y N
A red light is flashing
Printer is unrecognized
Actions
Check the power cable X
Check the printer-computer cable
Ensure printer software is installed
Check/replace ink
Check for paper jam
Printer Troubleshooting 30

31. Decision Table Usage
The use of the decision-table model is applicable when:
Specification is given or can be converted to a decision table
The order in which the predicates are evaluated does not affect the interpretation of resulting action
Note:
Decision table needs not cover all combinations 31

32. White Box Testing: General Guidelines
Try to cover all branches
Study the relationship between input value and branch logic
Test more on complex modules
Measure complexities of modules by code size, number of branches and loops, number of calls and recursions 32

33. White Box Testing: Techniques
More difficult than black box testing
Seldom done manually
Automatic support
Symbolic execution
Complexity measurement and Defect prediction 33

34. Review: Test overview
Test is the practical choice: the best affordable approach
Concepts: test case, test oracle, test suite, test driver, test script, test coverage
Granularity: unit, integration, system, acceptance
Type by design principle: black-box, white-box
Black-box-testing: boundary, equivalence, decision table
White-box-testing: branch coverage, complexity 34