Slide 6.1 CHAPTER 6 TESTING

Slide 6.2 Overview l Quality issues l Nonexecution-based testing l Execution-based testing l What should be tested? l Testing versus correctness proofs l Who should perform execution-based testing? l When testing stops?

Slide 6.3 Testing l Testing is an integral component of software process and an activity that must be carried out throughout the life cycle l Two types of testing –Execution-based testing –Nonexecution-based testing

Slide 6.4 Testing (contd) l “V & V” –Verification »Determine if the phase was completed correctly »Performed at the end of each phase –Validation »Determine if the product as a whole satisfies its requirements »Performed just before the product is delivered to the client l In this book, testing simply denotes V & V

Slide 6.5 Software Quality l Quality implies “excellence” of some sort l In S/W, we do not strive for “excellence” but merely getting the software to function correctly is enough l That is, does the software satisfy its specifications? l Read Just In Case You Wanted to Know on page 138 l Software Quality Assurance (SQA) –Ensures that the product is correct –Performs testing at the end of each phase as well as at the end of the product development –Managerial independence is important »Development group »SQA group

Slide 6.6 Nonexecution-Based Testing l Underlying principles –One should not review his or her own work –Why not? l Nonexecution-based testing is usually referred to as review l There are two types of reviews 1.Walkthroughs 2.Inspections

Slide 6.7 Walkthroughs l Should consist of 4-6 members l For example, a specification walkthrough should include –Specification team manager & member –Client representative –Member that will perform the next phase (design team member) –SQA team member (should chair the walkthrough) l The material to be reviewed should be distributed in advance to the participants l Each reviewer should study the material and prepare two lists: –items that the reviewer does not understand –items that the reviewer believes incorrect

Slide 6.8 Walkthroughs (contd) l Walkthrough process –Usually no more than 2 hours –Detect and record faults – DO NOT correct –Two ways of conducting walkthroughs 1.Participant driven »Participants present their lists of unclear and incorrect items »The author must respond to each query 2.Document driven –The author walks the participants through the document –The reviewers interrupt whenever unclear or incorrect items are presented –The document-driven approach is usually more thorough (i.e., finds more faults)

Slide 6.9 Inspections l More detailed than walkthrough and has five formal steps: overview, preparation, inspection, rework and follow-up l Inspection team should consist of 4 members l For example, design inspection team includes –Moderator (inspection team leader), designer, implementer, tester

Slide 6.10 Inspection Steps 1. Overview –An overview of the document to be reviewed is given by one of the authors –At the end of the overview session, the document is distributed to the participants 2. Preparation –Participants try to understand the document in detail, aided by statistics of fault types –Participants prepare the lists of unclear/incorrect items as well 3. Inspection –An author walks through the document with the inspection team –Faults are detected and recorded – DO NOT correct here –Within one day, the moderator generates a written report containing faults detected during inspection

Slide 6.11 Inspection Steps (contd) 4. Rework –The author resolves all faults and problems noted in the written report 5. Follow-up –The rework is thoroughly checked –All fixes must be checked to ensure that no new faults have been introduced

Slide 6.12 Statistics on Inspections l 82% of all detected faults (IBM, 1976) l 70% of all detected faults (IBM, 1978) l 93% of all detected faults (IBM, 1986) l 90% decrease in cost of detecting fault (Switching system, 1986) l 4 major faults, 14 minor faults per 2 hours (JPL, 1990). Savings of $25,000 per inspection l Number of faults decreased exponentially by phase (JPL, 1992)

Slide 6.13 Metrics for Inspections l Fault density (e.g., faults per KLOC) l Fault detection rate (e.g., faults detected per hour) l Fault detection efficiency (e.g., the number of major and minor faults detected per person-hour)

Slide 6.14 Execution-Based Testing l Definitions –Fault is the IEEE Standard terminology for “bug” –Failure is the observed incorrect behavior of the product as a consequence of the fault –Error is the mistake made by programmer l “Programming testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence [Dijkstra, 1972]

Slide 6.15 What is execution-based testing? l “Execution-based testing is a process of inferring certain behavioral properties of a product based, in part, on the results of executing the product in known environment with selected inputs.” [Goodenough, 1979] –Inference (i.e., deriving logical conclusion) –Known environment –Selected inputs l What must be tested? => Behavioral properties must be tested –Correctness, utility, reliability, robustness, and performance

Slide 6.16 Utility l Utility is the extent to which a user’s needs are met when a correct product is used under conditions permitted by its specifications l That is, does it meet user’s needs? –Ease of use –Useful functions –Cost-effectiveness

Slide 6.17 Reliability l Reliability is a measure of the frequency and criticality of product failure –How often the product fails? (Mean time between failures, MTBF) –How long it takes to restore? (Mean time to restore, MTTR) –But often more important is how long it takes to repair the results of the failure?

Slide 6.18 Robustness l Robustness is a function of a number of factors such as –Range of operating conditions –Possibility of unacceptable results with valid input –Effect of invalid input

Slide 6.19 Performance l Extent to which space and time constraints are met l Real-time systems have hard time constraints

Slide 6.20 Correctness l A product is correct if it satisfies its output specifications l But what if the specifications themselves are incorrect?

Slide 6.21 Correctness of specifications l Incorrect specification for a sort l Function trickSort which satisfies this specification:

Slide 6.22 Correctness of specifications (contd) l Incorrect specification for a sort: l Corrected specification for the sort:

Slide 6.23 Correctness l NOT necessary l NOT sufficient

Slide 6.24 Correctness Proofs l Alternative to execution-based testing

Slide 6.25 Example of Correctness Proof l Code segment to be proven correct

Slide 6.26 Example of Correctness Proof (contd) l Flowchart of code segment

Slide 6.27 Example of Correctness Proof

Slide 6.28 Correctness Proof Case Study l Never prove a program correct without testing it as well

Slide 6.29 Episode 1 l 1969 — Naur Paper l “Naur text-processing problem” Given a text consisting of words separated by blank or by nl (new line) characters, convert it to line-by-line form in accordance with following rules: (1) line breaks must be made only where given text has blank or nl ; (2) each line is filled as far as possible, as long as (3) no line will contain more than maxpos characters l Naur constructed a procedure (25 lines of Algol 60), and informally proved its correctness

Slide 6.30 Episode 2 l 1970 — Reviewer in Computing Reviews –The first word of the first line is preceded by blank unless the first word is exactly maxpos characters long

Slide 6.31 Episode 3 l 1971 — London finds 3 more faults l Including: –The procedure does not terminate unless a word longer than maxpos characters is encountered

Slide 6.32 Episode 4 l 1975 — Goodenough and Gerhart find three further faults l Including: –The last word will not be output unless it is followed by blank or nl

Slide 6.33 Proofs and Software Engineering l Lesson: l Even if product is proved correct, it must STILL be tested.

Slide 6.34 Three Myths l Software engineers do not have enough math for proofs l Proving is too expensive to be practical l Proving is too hard

Slide 6.35 Proofs and Software Engineering (contd) l Can we trust a theorem prover? l How to find input–output specifications, loop invariants l What if the specifications are wrong? l Can never be sure that specifications or a verification system are correct

Slide 6.36 Proofs and Software Engineering (contd) l Correctness proofs are a vital software engineering tool, WHERE APPROPRIATE. l If –Human lives are at stake –Indicated by cost/benefit analysis –Risk of not proving is too great l Also, informal proofs can improve the quality of the product –Assert statement

Slide 6.37 Who Performs Execution-Based Testing? l Testing is destructive –A successful test finds a fault l Solution –1.The programmer does informal testing –2.SQA does systematic testing –3.The programmer debugs the module l All test cases must be –Planned beforehand, including expected output –Retained afterwards

Slide 6.38 When Testing Can Stop? l Only when the product has been irrevocably retired