Slide 6.1 © The McGraw-Hill Companies, 2002 Object-Oriented and Classical Software Engineering Fifth Edition, WCB/McGraw-Hill, 2002 Stephen R. Schach

Slide 6.2 © The McGraw-Hill Companies, 2002 CHAPTER 6 TESTING

Slide 6.3 © The McGraw-Hill Companies, 2002 Overview l Quality issues l Nonexecution-based testing l Execution-based testing l What should be tested? l Testing versus correctness proofs l Who should perform execution-based testing? l When testing stops

Slide 6.4 © The McGraw-Hill Companies, 2002 Testing l Two types of testing –Execution-based testing –Nonexecution-based testing

Slide 6.5 © The McGraw-Hill Companies, 2002 Testing (contd) l “V & V” –Verification »Determine if the phase was completed correctly –Validation »Determine if the product as a whole satisfies its requirements

Slide 6.6 © The McGraw-Hill Companies, 2002 Testing (contd) l Warning –“Verify” also used for all nonexecution-based testing

Slide 6.7 © The McGraw-Hill Companies, 2002 Software Quality l Not “excellence” l The extent to which software satisfies its specifications l Software Quality Assurance (SQA) –Goes far beyond V & V –Managerial independence »Development group »SQA group

Slide 6.8 © The McGraw-Hill Companies, 2002 Nonexecution-Based Testing l Underlying principles –We should not review our own work –Group synergy

Slide 6.9 © The McGraw-Hill Companies, 2002 Walkthroughs l 4–6 members, chaired by SQA l Preparation—lists of items l Inspection –Up to 2 hours –Detect, don’t correct –Document-driven, not participant-driven –Verbalization leads to fault finding –Performance appraisal

Slide 6.10 © The McGraw-Hill Companies, 2002 Inspections l Five-stage process –Overview –Preparation, aided by statistics of fault types –Inspection –Rework –Follow-up

Slide 6.11 © The McGraw-Hill Companies, 2002 Fault Statistics l Recorded by severity and fault type l Compare with previous products l What if there are a disproportionate number of faults in a module? l Carry forward fault statistics to the next phase

Slide 6.12 © The McGraw-Hill Companies, 2002 Statistics on Inspections l 82% of all detected faults (IBM, 1976) l 70% of all detected faults (IBM, 1978) l 93% of all detected faults (IBM, 1986) l 90% decrease in cost of detecting fault (Switching system, 1986) l 4 major faults, 14 minor faults per 2 hours (JPL, 1990). Savings of $25,000 per inspection l Number of faults decreased exponentially by phase (JPL, 1992) l Warning –Fault statistics and performance appraisal

Slide 6.13 © The McGraw-Hill Companies, 2002 Metrics for Inspections l Fault density (e.g., faults per KLOC) l Fault detection rate (e.g., faults detected per hour) l By severity (major/minor), by phase l What does a 50% increase in the fault detection rate mean?

Slide 6.14 © The McGraw-Hill Companies, 2002 Execution-Based Testing l Definitions –Failure (incorrect behavior) –Fault (NOT “bug”) –Error (mistake made by programmer) l Nonsensical statement –“Testing is a demonstration that faults are not present”

Slide 6.15 © The McGraw-Hill Companies, 2002 What Is Execution-Based Testing? l “The process of inferring certain behavioral properties of the product based, in part, on the results of executing the product in a known environment with selected inputs.” –Inference –Known environment –Selected inputs l But what should be tested? –Correctness (of course!) –What else?

Slide 6.16 © The McGraw-Hill Companies, 2002 Utility l Does it meet the user’s needs? –Ease of use –Useful functions –Cost effectiveness

Slide 6.17 © The McGraw-Hill Companies, 2002 Reliability l Frequency and criticality of failure –Mean time between failures –Mean time to repair –Mean time, cost to repair the results of a failure

Slide 6.18 © The McGraw-Hill Companies, 2002 Robustness l Range of operating conditions l Possibility of unacceptable results with valid input l Effect of invalid input

Slide 6.19 © The McGraw-Hill Companies, 2002 Performance l Extent to which space and time constraints are met l Real-time software

Slide 6.20 © The McGraw-Hill Companies, 2002 Correctness l A product is correct if it satisfies its specifications

Slide 6.21 © The McGraw-Hill Companies, 2002 Correctness of specifications l Incorrect specification for a sort: l Function trickSort which satisfies this specification:

Slide 6.22 © The McGraw-Hill Companies, 2002 Correctness of specifications (coptd) l Incorrect specification for a sort: l Corrected specification for the sort:

Slide 6.23 © The McGraw-Hill Companies, 2002 Correctness (contd) l Technically, correctness is –NOT necessary –NOT sufficient

Slide 6.24 © The McGraw-Hill Companies, 2002 Correctness Proofs l An alternative to execution-based testing

Slide 6.25 © The McGraw-Hill Companies, 2002 Example of Correctness Proof l Code segment to be proven correct

Slide 6.26 © The McGraw-Hill Companies, 2002 Example of Correctness Proof (contd) l Flowchart of code segment

Slide 6.27 © The McGraw-Hill Companies, 2002 Example of Correctness Proof

Slide 6.28 © The McGraw-Hill Companies, 2002 Correctness Proof Case Study l Never prove a program correct without testing it as well

Slide 6.29 © The McGraw-Hill Companies, 2002 Episode 1 l 1969 — Naur Paper l “Naur text-processing problem” Given a text consisting of words separated by a blank or by newline characters, convert it to line-by-line form in accordance with the following rules: (1) line breaks must be made only where the given text contains a blank or newline ; (2)each line is filled as far as possible, as long as (3)no line will contain more than maxpos characters l Naur constructed a procedure (25 lines of Algol 60), and informally proved its correctness

Slide 6.30 © The McGraw-Hill Companies, 2002 Episode 2 l 1970 — Reviewer in Computing Reviews –The first word of the first line is preceded by a blank unless the first word is exactly maxpos characters long

Slide 6.31 © The McGraw-Hill Companies, 2002 Episode 3 l 1971 — London finds 3 more faults l Including: –The procedure does not terminate unless a word longer than maxpos characters is encountered

Slide 6.32 © The McGraw-Hill Companies, 2002 Episode 4 l 1975 — Goodenough and Gerhart find three further faults l Including: –The last word will not be output unless it is followed by a blank or newline

Slide 6.33 © The McGraw-Hill Companies, 2002 Proofs and Software Engineering l Lesson: l Even if product is proved correct, it must STILL be tested.

Slide 6.34 © The McGraw-Hill Companies, 2002 Three Myths l Software engineers do not have enough mathematics for proofs l Proving is too expensive to be practical l Proving is too hard

Slide 6.35 © The McGraw-Hill Companies, 2002 l Can we trust a theorem prover ? l How to find input–output specifications, loop invariants l What if the specifications are wrong? l Can never be sure that specifications or a verification system are correct Proofs and Software Engineering (contd) void theoremProver() { print  This product is correct  ; }

Slide 6.36 © The McGraw-Hill Companies, 2002 Proofs and Software Engineering (contd) l Correctness proofs are a vital software engineering tool, WHERE APPROPRIATE. l If –Human lives are at stake –Indicated by cost/benefit analysis –The risk of not proving is too great l Also, informal proofs can improve the quality of the product –Assert statement

Slide 6.37 © The McGraw-Hill Companies, 2002 Who Performs Execution-Based Testing? l Programming is constructive l Testing is destructive l A successful test finds a fault l Solution –1.The programmer does informal testing –2.SQA does systematic testing –3.The programmer debugs the module l All test cases must be –Planned beforehand, including the expected output –Retained afterwards

Slide 6.38 © The McGraw-Hill Companies, 2002 When Testing Can Stop l Only when the product has been irrevocably retired