1. jpf@fe.up.pt www.fe.up.pt/~jpf
TQS - Teste e Qualidade de Software (Software Testing and Quality) Introduction To Software Testing Concepts
João Pascoal Faria

2. Software testing
Software testing consists of the dynamic (1) verification of the behavior of a program on a finite (2) set of test cases, suitably selected (3) from the usually infinite executions domain, against the specified expected (4) behavior [source: SWEBOK]
(1) testing always implies executing the program on some inputs
(2) for even simple programs, too many test cases are theoretically possible that exhaustive testing is infeasible
trade-off between limited resources and schedules and inherently unlimited test requirements
(3) see test case design techniques later on how to select the test cases
(4) it must be possible to decide whether the observed outcomes of the program are acceptable or not
the pass/fail decision is commonly referred to as the oracle problem

3. Purpose of software testing
The main purpose of software testing is to find defects
"The goal of a software tester is to find bugs, find them as early as possible, and make sure that they get fixed" (source: Ron Patton)
The best test cases are the ones with higher probability of revealing defects / bugs
Testing does not prove the absence of defects!

4. Test cases
Test case - Inputs to test the system and the expected outputs from these inputs if the system operates correctly
Inputs may include an initial state of the system
Outputs may include a final state of the system
When test cases are executed, the system is provided with the specified inputs and the actual outputs are compared with the outputs expected
Example for a calculator:
(input) should give 8 (output)

5. Types of testing Level of detail or phase Accessibility
system
integration
unit
Accessibility
(test case design strategy/technique)
security
robustness
white box
black box
performance
usability
reliability
functional behaviour
focus here
Characteristics

6. Test levels or phases (1)
Unit testing
Testing of individual program units or components
Usually the responsibility of the component developer (except sometimes for critical systems)
Tests are based on experience, specifications and code
A principal goal is to detect functional and structural defects in the unit
Integration testing
Testing of groups of components integrated to create a sub-system
Usually the responsibility of an independent testing team (except sometimes in small projects)
Tests are based on a system specification (technical specifications, designs)
A principal goal is to detect defects that occur on the interfaces of units and their common behavior

7. Test levels or phases (2)
System testing
Testing the system as a whole
Usually the responsibility of an independent testing team
Tests are usually based on a requirements document (functional requirements/specifications and quality requirements)
A principal goal is to evaluate attributes such as usability, reliability and performance (assuming unit and integration testing have been performed)
Acceptance testing
Usually the responsibility of the customer
Tests are based on a requirements specification or a user manual
A principal goal is to check if the product meets customer requirements and expectations
(source: I. Sommerville)

8. Test levels and the extended V-model of software development
Execute acceptance tests
Specify Requirements
Execute
system tests
System/acceptance
test plan & test cases review/audit
Requirements review
Specify/Design Code System/acceptance tests
Design
Execute integration tests
Integration test plan & test cases
review/audit
Design review
Specify/Design Code Integration tests
Code
Execute
unit tests
Unit
test plan & test cases review/audit
Code reviews
(source: I. Burnstein, pg.15)
Specify/Design Code
Unit tests

9. Typical tests and reviews
(source: "Software Project Survival Guide", Steve McConnell)

10. Test harness Auxiliary code developed to support testing Test drivers
call the target code
simulate calling units or a user
where test procedures and test cases are coded (for automatic test case execution) or a user interface is created (for manual test case execution)
Test stubs
simulate called units
simulate modules/units/systems called by the target code

11. Good practices Write the test cases before the software to be tested
applies to any level: unit, integration or system
helps getting insight into the requirements
Code the test cases
because of the frequent need for regression testing (repetition of testing each time the software is modified)

12. References and further reading
Practical Software Testing, Ilene Burnstein, Springer-Verlag, 2003
Software Testing, Ron Patton, SAMS, 2001
Testing Computer Software,  2nd Edition, Cem Kaner, Jack Falk, Hung Nguyen, John Wiley & Sons, 1999
Software Engineering, Ian Sommerville, 6th Edition, Addison-Wesley, 2000
Guide to the Software Engineering Body of Knowledge (SWEBOK), IEEE Computer Society,