Chapter 11: Testing The dynamic verification of the behavior of a program on a finite set of test cases, suitable selected from the usually infinite execution domain, against the expected behavior

Testing Fundamentals  Testing is the one step in software engineering process that could be viewed as destructive rather than constructive. A successful test is one that breaks the software.  A successful test is one that uncovers an as yet undiscovered defect.  Testing can not show the absence of defects, it can only show that software defects are present.  For most software exhaustive testing is not possible.

Testing  Unit testing White-box Code walkthroughs and inspections  Integration testing Bottom-up Top-down Sandwich Big Bang  Functional testing (black-box)  Performance testing  Acceptance testing  Installation testing

System testing  Functional testing  Performance testing  Acceptance testing  Installation testing

System Testing Process  Functional testing: does the integrated system perform as promised by the requirements specification?  Performance testing: are the non- functional requirements met?

Functional Testing  Test cases derived from requirements specification document Black box testing Independent testers Test both valid and invalid input and the success of the test is determined by the produced output  Equivalence partitioning Boundary values

Equivalence partitioning  Reduces the number of test cases to a necessary minimum and select the right test cases to cover all possible scenarios  Example: method accepts an int value for month (1..12) as a parameter

Performance Testing  Stress tests  Volume tests  Recovery tests  Security tests  Timing tests  Environmental tests  Quality tests  Maintenance tests  Documentation tests  Human factors (usability) tests

Acceptance Tests  Enable the customers and users to determine if the built system meets their needs and expectations  Written, conducted, and evaluated by the customers

Types of Acceptance Tests  Pilot test: install on experimental basis  Alpha test: in-house test  Beta test: customer pilot  Parallel testing: new system operates in parallel with old system

Installation test  Does the system run at the customer site(s)?

System Testing

Pop quiz  How does system testing differ from unit and integration testing?  How does unit testing differ from integration testing?

In a method, a variable did not get initialized properly. Which type of testing would most likely expose this defect? A. Unit testing B. Integration testing C. Functional testing D. Performance testing E. Acceptance testing F. Installation testing

A gas pump system is supposed to allow the user to choose whether or not a receipt is printed, but the print function has not been implemented. Which type of testing is most likely to expose this defect? A. Unit testing B. Integration testing C. Functional testing D. Performance testing E. Acceptance testing F. Installation testing

A configuration file used by the reporting subsystem is not placed in the correct directory in the customer's environment. Which type of testing is most likely to expose this defect? A. Unit testing B. Integration testing C. Function testing D. Performance testing E. Acceptance testing F. Installation testing

The customer is unhappy with the number of screens that must be traversed before getting to the parts list screen, a screen accessed frequently when using the system. Which type of testing is most likely to expose this defect? A. Unit testing B. Integration testing C. Function testing D. Performance testing E. Acceptance testing F. Installation testing

Short answer question  What is the difference between verification and validation testing?

Test Documentation  Test plan: describes system and plan for testing all functions and characteristics  Test case specification: details each test and defines criteria for evaluating each feature  Test incident report: results of each test  Test report summary: lists all failures from the tests that need to be investigated

Test Plan (part of 480 QA plan)  Define the subsystems to be tested (and not tested)  Describe the process to follow for unit, integration, system, and acceptance testing  List the test cases  Plan the test schedule

Test Documentation

Defect Tracking Form

Regression Testing  Identifies new faults that may have been introduced as current ones are being corrected  Verifies that a new version or release still performs the same functions in the same manner as an older version or release

Quality Assurance concerns  Software reliability: operating without failure under given condition for a given time interval  Software availability: operating successfully according to specification at a given point in time  Software maintainability: for a given condition of use, a maintenance activity can be carried out within stated time interval, procedures and resources

Testing Safety-Critical Systems  Recognize that testing cannot remove all faults or risks  Assume that every mistake users can make will be made Do not assume that low-probability, high- impact events will not happen  Emphasize requirements definition, testing, code and specification reviews, and configuration control Cleanroom testing

Different Levels of Failure Severity  Catastrophic: causes death or system loss  Critical: causes severe injury or major system damage  Marginal: causes minor injury or minor system damage  Minor: causes no injury or system damage

One-minute quiz  What is meant by regression testing?  Is regression testing used for verification or validation?