Verification and Testing

Assignment Read Ghezzi Chapter 6 through the end of 6.3

Methods for Verifying Software Testing Inspections and Reviews Static checking Performance analysis Reliability analysis Symbolic execution Model checking Formal proofs User studies

Methods for Verifying Qualities Testing Inspections and Reviews Static checking Performance analysis Reliability analysis Symbolic execution Model checking Formal proofs User studies

Motivation The National Institute for Standards and Technology (NIST) reported (2003) Software faults cost $59.5b/year to US market $22.2b could be saved with relatively simple testing practices – and with an expected reduction of Number of faults Development costs Time to market Maintenance costs

Testing -- Definition The process of dynamically exercising a program, to determine whether it meets specific quality requirements such as conformance to specifications, performance, reliability, or robustness. We’ll concentrate here on conformance to specifications.

Testing Basics P is a program, D is the domain of inputs, R is the range of outputs, S is the specification for P P is correct for t in D if P(t) = S(t) A t in D is called a test case A finite subset T of D is called a test suite or test set Testing is all about selecting and applying T, and checking the results P t D R S

Testing Objectives (Informal) Tests are intended to uncover faults in operating software Good test cases have high probabilities of finding an as yet undiscovered fault Successful tests cause program failures, i.e. find undiscovered faults Successful test suites expose lots of faults Cost limits the set of test cases that can be developed, so any statement about testing objectives should involve cost-effectiveness

Testing Terminology Failure: externally visible incorrect program behavior Fault: incorrect portions of code (may involve missing code as well as incorrect code) Error: something bad a programmer did Bug: informal term for fault/failure Debugging: given a failure, find the fault that causes it Testing: find failures Oracle: “device” or “procedure” for determining correctness of output

Testing Versus Static Checking Testing (dynamic) “Run and see” Run tests to see whether failures occur Underlying each failure there are one or more faults to be fixed Static checking No program execution Examples: Inspections Walkthroughs Static Analysis

Faults and Failures Consider the code: SQUARE(z) y = 2 * z print y What’s the fault? Failure? Error? Coincidental correctness: when a faulty program happens to produce correct output on certain inputs that do execute the faulty statement.

Faults and Failures Consider the code fragment: … if ( y >= 2 ) x = 1 / (2-y) endif print y What’s the fault? Failure? Error? Compare this to the prior fault for detection difficulty!

Types of Testing Unit (module) -- individual “units” Integration – groups of units System – software with hardware Acceptance – am I satisfied? Regression – after changes Beta – at specific user sites Where is the most time spent?

Software Development Phases & Testing Develop test plan and system tests; perform technical review Develop integration tests; perform technical review Develop and run unit tests; perform technical review Run integration tests Run system tests Run regression tests Requirements Analysis Phase: Design Phase: Implementation Phase: Integration Phase: Maintenance Phase:

More Terminology Test Case: set of input data (provided at a specific system state) and expected output Test Suite: collection of test cases Test Driver: program written to test a unit module Test Stub: module written to allow testing of a higher level component Test Harness: test environment to run programs with stubs and drivers and check the results Test Plan: description of a testing process including overall approach and specific tests

The “Test Case Problem”: A More Formal Treatment D R S P is a program, D is the domain of inputs, R is the range of outputs, S is the specification for P Choose test suite T subset D s.t. for each fault in P, there exists t member T such that P(t) ≠ S(t) For every P and D such a test suite exists, but there is no algorithm for computing it.

What are the Implications of This? What really is our goal, in testing? It isn’t “achieving correctness” Find “as many faults as we can”? Test “for as long as we can”? Ensure that the probability of failure is sufficiently low? Achieve “sufficient confidence” in the system? How do we define and quantify these things?