Software Testing

Purpose: to Insure that software meets its design specifications, especially at the boundary conditions. “Boundary Conditions:” conditions where performance changes from acceptable to unacceptable. –“Boundary Conditions” may sometimes mean where performance changes from one level (e.g. “summer”) to another (e.g. “fall”).

Simplifying Assumptions Software either fails or it doesn’t; no “partial failures.” Software performance can be measured without ambiguity. (“I’m positive that the criminal was about average height and had either green or blue eyes and may have had a hat or coat, but maybe not.”)

Two Types of Errors Software fails, but we think it works. (Type 1 error) –The guilty man is acquitted. Software works, but we think it fails. (Type 2 error) –The innocent man is convicted

Two Types of Errors Type 1 and Type 2 errors are complementary: Reducing one may increase the other. Accepting everything eliminates Type 2 error (no good cases are rejected) but maximizes Type 1 error (all bad cases are accepted). Rejecting everything eliminates Type 1 error (no bad cases are accepted) but maximizes Type 2 error (all good cases are rejected).

Two Types of Errors Complete performance testing must address both types of errors. –Testing “out-of bounds” cases tests for Type 1 error (accepting an incorrect input). –Testing “in-bounds” cases tests for Type 2 error (rejecting a correct input).

Examples of Type 1 and Type 2 Errors Goal: Select the the attractive women from the following set {Jodie Foster, Gwyneth Paltrow, Tom Hanks, Harrison Ford}. –{Jodie Foster, Gwyneth Paltrow, Tom Hanks} contains a Type 1 error. –{Jodie Foster} contains a Type 2 error –{Jodie Foster, Tom Hanks} contains both types.

Type 1 and Type 2 Errors Acceptable Performance Unacceptable Performance Accepting this point is a Type 1 error. Rejecting this point is a Type 2 error.

Type 1 and Type 2 Errors Acceptable PerformanceUnacceptable Performance Accept Positive Rejecting either of these points is a Type 2 error. Accept Negative Accepting either of these points is a Type 1 error. Print positive integers, reject negative ones. Accept a negative integer Reject a positive one Reject Positive Reject Negative

Testing for Both Types of Error Type 1 and Type 2 errors are complementary. –Testing for Type 1 tells nothing about Type 2 –Testing for Type 2 tells nothing about Type 1 Both types must be tested for independently.

Testing Project 2a Required: Test program to insure it printed positive numbers, rejected negative ones. Testing with positive numbers tests only for Type 2 error: does program wrongly reject proper input. Testing with negative numbers tests only for Type 1 error: does program wrongly accept improper input.

What to Test? Goal: every predicate –if, while, for, do-while, switch Limit: must be observable –input and output must be observable; “You can’t test what you can’t see.”

How To Test Develop specific test cases with known answers. –If you don’t know the answer beforehand, you’re “experimenting,” not “testing.” Trace program logic to insure that every predicate is tested at least once. –Analysis of program flow using graph theory can give a minimum set of tests to insure complete coverage. (McCabe cyclomatic complexity) Test at “critical points” (where behavior changes.)

How To Test Test at “Critical Points” (where behavior changes.) Save testing procedure, data, and results to use after maintenance.