Testing -- Part II

Testing The role of testing is to: w Locate errors that can then be fixed to produce a more reliable product w Design tests that systematically uncover different classes of errors

The purpose of testing is to: Remove as many errors as possible from the product < protecting the user from loss of time, effort, and money < protecting the reputation of the software developer in the marketplace

Testing Recall < Testing cannot show the absence of errors; it can only show that errors are present! < As much as 50% of the development effort will be expended in testing

Purpose of testing phase w to affirm the quality of the product and indirectly the process w to find and eliminate errors w to validate that the software solves the original problem w to demonstrate that all specified functions are performed by the product w to estimate the operational reliability of the system

Testing In general testing is "bottom-up" But- Programming is "top-down"

Testing 1. Test small units in all possible ways to detect any errors 2. Combine them into units, testing both their detailed structure and their function 3. Concentrate on how software responds to the typical kind of operations that the user will require.

Phases of Testing 1. Unit testing 2. Integration testing 3. System testing 4. Regression testing

Testing: Error categories a) Those that merit immediate attention (they crash the system). b) Errors that need to be corrected before testing is complete but can be ignored during the immediate testing schedule. c) there may be errors that may be acceptable to the user when compared to the cost of correcting them at the moment--errors put off until future releases. d) errors that are non-reproducible

Testing = Error Location The importance of removing an error is proportional to its severity, the frequency with which it will occur, and the degree to which the customer will be aware of it.

Regression testing w Testing that occurs after software modifications are made. w When "corrections" are made the software must subjected to all of the tests it has passed to this point. WHY??

Regression testing can be very painful! Why?

Basic testing strategies w Structural testing – focuses on a UNIT of code. (Unit testing) w Functional testing – focuses on the interaction/interface between code components. (Integration testing) w Pragmatic testing – System testing Mimics the real world Tests the system as a whole.

Structural testing w Idea is to check all logical paths through a module w Commonly used during unit tests w View the software in terms of its detailed design and attempt to test it so that we certify the correct behavior of every statement, every decision or every path through the code

Testing Every statement, every decision or every path --- EVERY! ???

Structural Testing is - Very detail focused - Program-based testing - Tests for conformity to the detailed design - Microscopic view

Structural Testing: Test cases (data) are needed that test: w all logical conditions for both true and false values w loops should be executed over their entire range of values

Structural Testing: Test cases (data) are needed that test: w validity of internal data structures must be verified w everything should be checked at the boundaries w loops need to be tested at every execution, especially if there is a local variable that affects anything inside the loop

Structural Testing is never perfect! Structural testing cannot test all possible paths for all combinations of logic, because even a simple procedure there are too many paths -- still each paths should be executed at least one

Functional testing Focus is on INTERFACE between the units w Observe properties of a software until it conforms to specifications w Determine whether the overall purpose has been satisfied w Utilized during the integration tests

Functional testing w Macroscopic view w Focuses on functions specified for the product. Are functions implemented satisfactorily? w Specification-based testing w Tests the product for conformity at the specification level.

Functional testing How is this done? A set of inputs is supplied and the outputs are to be verified against the results expected under the terms of the specification

Pragmatic testing w Commonly used during systems testing w Attempts to model the actual usage that the software will encounter after it is released t the customer w Focuses on usability and reliability rather than universal correctness

General Goals of testing w Detection and elimination of errors w Prevention of additional errors during software modification w Validation of the product (Are we building the right product?) w Release of product on the scheduled date

Goals may be in conflict a) if we resolve to correct every error that testing uncovers, minor errors may lead to contract penalties or a lost market opportunity b) bowing to the schedule and hurrying testing to the point that we fail to detect and correct serious errors may incur other contract penalties or loss of reputation.

We need predetermined goals for the product being tested: How dependable must the product be? w medical software w word processing software w scheduling software

We need predetermined goals for the product being tested: How important is it to meet the schedule?

Testing w Software must be tested in a realistic setting w Software must be carefully and extensively tested in the same way that users are expected to use it

Alpha testing w Alpha testing occurs when the software has been written for a specific "client." w These systems are usually "revised" or "adapted" as part of the testing process until the client agrees that the system is an acceptable implementation of the system requirements.

Beta Testing: Broad Market Appeal Beta Testing involves delivering a system to a number of potential customers who agree to use the system. AND Report problems to developers After exposing the system to real-time real-use operation.

Beta Testing The System is then "adapted" and either beta tested again or put out for general sale.