1. Verification and Testing

2. Assignment
Read Ghezzi Chapter 6 through the end of 6.3

3. Methods for Verifying Software
Testing
Inspections and Reviews
Static checking
Performance analysis
Reliability analysis
Symbolic execution
Model checking
Formal proofs
User studies

4. Methods for Verifying Qualities
Testing
Inspections and Reviews
Static checking
Performance analysis
Reliability analysis
Symbolic execution
Model checking
Formal proofs
User studies

5. Motivation
The National Institute for Standards and Technology (NIST) reported (2003)
Software faults cost $59.5b/year to US market
$22.2b could be saved with relatively simple testing practices – and with an expected reduction of
Number of faults
Development costs
Time to market
Maintenance costs

6. Testing -- Definition
The process of dynamically exercising a program, to determine whether it meets specific quality requirements such as conformance to specifications, performance, reliability, or robustness.
We’ll concentrate here on conformance to specifications.

7. Testing Basics
P is a program, D is the domain of inputs, R is the range of outputs, S is the specification for P
P is correct for t in D if P(t) = S(t)
A t in D is called a test case
A finite subset T of D is called a test suite or test set
Testing is all about selecting and applying T, and checking the results P t D R S

8. Testing Objectives (Informal)
Tests are intended to uncover faults in operating software
Good test cases have high probabilities of finding an as yet undiscovered fault
Successful tests cause program failures, i.e. find undiscovered faults
Successful test suites expose lots of faults
Cost limits the set of test cases that can be developed, so any statement about testing objectives should involve cost-effectiveness

9. Testing Terminology
Failure: externally visible incorrect program behavior
Fault: incorrect portions of code (may involve missing code as well as incorrect code)
Error: something bad a programmer did
Bug: informal term for fault/failure
Debugging: given a failure, find the fault that causes it
Testing: find failures
Oracle: “device” or “procedure” for determining correctness of output

10. Testing Versus Static Checking
Testing (dynamic)
“Run and see”
Run tests to see whether failures occur
Underlying each failure there are one or more faults to be fixed
Static checking
No program execution
Examples:
Inspections
Walkthroughs
Static Analysis

11. Faults and Failures Consider the code: SQUARE(z) y = 2 * z print y
What’s the fault? Failure? Error?
Coincidental correctness: when a faulty program
happens to produce correct output on certain inputs
that do execute the faulty statement.

12. Faults and Failures Consider the code fragment: … if ( y >= 2 )
x = 1 / (2-y)
endif
print y
What’s the fault? Failure? Error?
Compare this to the prior fault for detection difficulty!

13. Types of Testing Unit (module) -- individual “units”
Integration – groups of units
System – software with hardware
Acceptance – am I satisfied?
Regression – after changes
Beta – at specific user sites
Where is the most time spent?

14. Software Development Phases & Testing
Develop test plan and system tests; perform technical review
Develop integration tests; perform technical review
Develop and run unit tests; perform technical review
Run integration tests
Run system tests
Run regression tests
Requirements Analysis Phase:
Design Phase:
Implementation Phase:
Integration Phase:
Maintenance Phase:

15. More Terminology
Test Case: set of input data (provided at a specific system state) and expected output
Test Suite: collection of test cases
Test Driver: program written to test a unit module
Test Stub: module written to allow testing of a higher level component
Test Harness: test environment to run programs with stubs and drivers and check the results
Test Plan: description of a testing process including overall approach and specific tests

16. The “Test Case Problem”: A More Formal TreatmentD R S
P is a program, D is the domain of inputs, R is the range of outputs, S is the specification for P
Choose test suite T subset D s.t. for each fault in P, there exists t member T such that P(t) ≠ S(t)
For every P and D such a test suite exists, but there is no algorithm for computing it.

17. What are the Implications of This?
What really is our goal, in testing? It isn’t “achieving correctness”
Find “as many faults as we can”?
Test “for as long as we can”?
Ensure that the probability of failure is sufficiently low?
Achieve “sufficient confidence” in the system?
How do we define and quantify these things?