1. Object-Oriented and Classical Software Engineering Fifth Edition, WCB/McGraw-Hill, Stephen R. Schach

2. CHAPTER 6
TESTING

3. Overview Quality issues Nonexecution-based testing
What should be tested?
Testing versus correctness proofs
Who should perform execution-based testing?
When testing stops

4. Testing Two types of testing Execution-based testing
Nonexecution-based testing

5. Testing (contd) “V & V” Verification Validation
Determine if the phase was completed correctly
Validation
Determine if the product as a whole satisfies its requirements

6. Testing (contd) Warning
“Verify” also used for all nonexecution-based testing

7. Software Quality Not “excellence”
Extent to which software satisfies its specifications
Software Quality Assurance (SQA)
Goes far beyond V & V
Managerial independence
development group
SQA group

8. Nonexecution-Based Testing
Underlying principles
We should not review our own work
Group synergy

9. Walkthroughs 4–6 members, chaired by SQA Preparation—lists of items
Inspection
Up to 2 hours
Detect, don’t correct
Document-driven, not participant-driven
Verbalization leads to fault finding
Performance appraisal

10. Inspections Five-stage process Overview
Preparation, aided by statistics of fault types
Inspection
Rework
Follow-up

11. Fault statistics Recorded by severity and fault type
Compare with previous products
What if there are a disproportionate number of faults in a module?
Carry forward fault statistics to the next phase

12. Statistics on Inspections
82% of all detected faults (IBM, 1976)
70% of all detected faults (IBM, 1978)
93% of all detected faults (IBM, 1986)
90% decrease in cost of detecting fault (Switching system, 1986)
4 major faults, 14 minor faults per 2 hours (JPL, 1990). Savings of $25,000 per inspection
Number of faults decreased exponentially by phase (JPL, 1992)
Warning
Fault statistics and performance appraisal

13. Metrics for Inspections
Fault density (e.g., faults per KLOC)
Fault detection rate (e.g., faults detected per hour)
By severity (major/minor), by phase
What does a 50% increase in the fault detection rate mean?

14. Execution-Based Testing
Definitions
Failure (incorrect behavior)
Fault (NOT “bug”)
Error (mistake made by programmer)
Nonsensical statement
“Testing is demonstration that faults are not present”

15. What is execution-based tested?
“The process of inferring certain behavioral properties of product based, in part, on results of executing product in known environment with selected inputs.”
Inference
Known environment
Selected inputs
But what should be tested?

16. Utility Does it meet user’s needs? Ease of use Useful functions
Cost-effectiveness

17. Reliability Frequency and criticality of failure
Mean time between failures
Mean time to repair
Mean time, cost to repair results of failure

18. Robustness Range of operating conditions
Possibility of unacceptable results with valid input
Effect of invalid input

19. Performance Extent to which space and time constraints are met
Real-time software

20. Correctness of specifications
Incorrect specification for a sort
Function trickSort which satisfies this specification:

21. Correctness of specifications (coptd)
Incorrect specification for a sort:
Corrected specification for the sort:

22. Correctness
NOT necessary
NOT sufficient

23. Correctness Proofs
Alternative to execution-based testing

24. Example of Correctness Proof
Code segment to be proven correct

25. Example of Correctness Proof (contd)
Flowchart of code segment

26. Example of Correctness Proof

27. Correctness Proof Case Study
Never prove a program correct without testing it as well

28. Episode 1 1969 — Naur Paper “Naur text-processing problem”
Given a text consisting of words separated by blank or by nl (new line) characters, convert it to line-by-line form in accordance with following rules:
(1) line breaks must be made only where given text has blank or nl ;
(2) each line is filled as far as possible, as long as
(3) no line will contain more than maxpos characters
Naur constructed a procedure (25 lines of Algol 60), and informally proved its correctness

29. Episode 2 1970 — Reviewer in Computing Reviews
The first word of the first line is preceded by blank unless the first word is exactly maxpos characters long

30. Episode 3 1971 — London finds 3 more faults Including:
The procedure does not terminate unless a word longer than maxpos characters is encountered

31. Episode 4 1975 — Goodenough and Gerhart find three further faults
Including:
The last word will not be output unless it is followed by blank or nl

32. Proofs and Software Engineering
Lesson:
Even if product is proved correct, it must STILL be tested.

33. Three Myths Software engineers do not have enough math for proofs
Proving is too expensive to be practical
Proving is too hard

34. Proofs and Software Engineering (contd)
Can we trust a theorem prover ?
How to find input–output specifications, loop invariants
What if the specifications are wrong?
Can never be sure that specifications or a verification system are correct

35. Proofs and Software Engineering (contd)
Correctness proofs are a vital software engineering tool, WHERE APPROPRIATE. If
Human lives are at stake
Indicated by cost/benefit analysis
Risk of not proving is too great
Also, informal proofs can improve the quality of the product
Assert statement

36. Who Performs Execution-Based Testing?
Testing is destructive
A successful test finds a fault
Solution
1. The programmer does informal testing
2. SQA does systematic testing
3. The programmer debugs the module
All test cases must be
Planned beforehand, including expected output
Retained afterwards

37. When Testing Can Stop
Only when the product has been irrevocably retired