1. Based on slides created by Ian Sommerville & Gary Kimura
Lecture 10 – Testing
Sampath Jayarathna
Cal Poly Pomona
Based on slides created by Ian Sommerville & Gary Kimura

2. Topics covered Development testing Test-driven development
Release testing
User testing

3. Program testing
Testing is intended to show that a program does what it is intended to do and to discover program defects before it is put into use.
When you test software, you execute a program using artificial data.
You check the results of the test run for errors, anomalies or information about the program’s non-functional attributes.
Can reveal the presence of errors NOT their absence.
Testing is part of a more general verification and validation process, which also includes static validation techniques.

4. Testing Goals: Verification and Validation
Software testing is part of a broader group of activities called verification and validation that are involved in software quality assurance
Verification (Are the algorithms coded correctly?)
The set of activities that ensure that software correctly implements a specific function or algorithm
"Are we building the product right”.
The software should conform to its specification.
Validation (Does it meet user requirements?)
The set of activities that ensure that the software that has been built is traceable to customer requirements
"Are we building the right product”.
The software should do what the user really requires.

5. A Strategy for Testing Conventional Software
Code
Design
Requirements
System Engineering
Unit Testing
Integration Testing
Validation Testing
System Testing
Broader scope
Narrow to
Abstract to
concrete

6. Organizing for Software Testing
Testing should aim at "breaking" the software
Common misconceptions
The developer of software should do no testing at all
The software should be given to a secret team of testers who will test it unmercifully
The testers get involved with the project only when the testing steps are about to begin
Reality: Independent test group
Removes the inherent problems associated with letting the builder test the software that has been built
Removes the conflict of interest that may otherwise be present
Works closely with the software developer during analysis and design to ensure that thorough testing occurs

7. Software inspections
These involve people examining the source representation with the aim of discovering anomalies and defects.
Inspections not require execution of a system so may be used before implementation.
They may be applied to any representation of the system (requirements, design, configuration data, test data, etc.).
They have been shown to be an effective technique for discovering program errors.
During testing, errors can mask (hide) other errors. Because inspection is a static process, you don’t have to be concerned with interactions between errors.
Inspection can also consider broader quality attributes of a program, such as compliance with standards, portability and maintainability.

8. Inspections and testing

9. Stages of testing
Development testing, where the system is tested during development to discover bugs and defects.
Release testing, where a separate testing team test a complete version of the system before it is released to users.
User testing, where users or potential users of a system test the system in their own environment.

10. Development testing
Development testing includes all testing activities that are carried out by the team developing the system.
Unit testing
Concentrates on each component/function of the software as implemented in the source code
Integration testing
Focuses on the design and construction of the software architecture
Validation testing
Requirements are validated against the constructed software
System testing
The software and other system elements are tested as a whole

11. Unit testing
Unit testing is the process of testing individual components in isolation.
It is a defect testing process.
Units may be:
Individual functions or methods within an object
Object classes with several attributes and methods
Composite components with defined interfaces used to access their functionality.

12. Two Unit Testing Techniques
White-box testing
Knowing the internal workings of a product, test that all internal operations are performed according to specifications and all internal components have been exercised
Involves tests that concentrate on close examination of procedural detail
Logical paths through the software are tested
Test cases exercise specific sets of conditions and loops
Black-box testing
Knowing the specified function that a product has been designed to perform, test to see if that function is fully operational and error free
Includes tests that are conducted at the software interface
Not concerned with internal logical structure of the software

13. White-box Testing
Uses the control structure part of component-level design to derive the test cases
These test cases
Guarantee that all independent paths within a module have been exercised at least once
Exercise all logical decisions on their true and false sides
Execute all loops at their boundaries and within their operational bounds
Exercise internal data structures to ensure their validity
“Bugs lurk in corners and congregate at boundaries”

14. Basis Path Testing White-box testing technique proposed by Tom McCabe
Enables the test case designer to derive a logical complexity measure of a procedural design
Uses this measure as a guide for defining a basis set of execution paths
Test cases derived to exercise the basis set are guaranteed to execute every statement in the program at least one time during testing

15. Flow Graph Notation
A circle in a graph represents a node, which stands for a sequence of one or more procedural statements
A node containing a simple conditional expression is referred to as a predicate node
Each compound condition in a conditional expression containing one or more Boolean operators (e.g., and, or) is represented by a separate predicate node
A predicate node has two edges leading out from it (True and False)
An edge, or a link, is a an arrow representing flow of control in a specific direction
An edge must start and terminate at a node
An edge does not intersect or cross over another edge
Areas bounded by a set of edges and nodes are called regions
When counting regions, include the area outside the graph as a region, too

16. Flow Graph Example FLOW CHART FLOW GRAPH R4 R3 R2 R1 1 1 2 2 3 3 6 4 6R4 1 1 2 2 R3 3 3 6 4 6 4 R2 7 8 5 7 R1 8 5 9 9 11 10 11 10

17. Independent Program Paths
Defined as a path through the program from the start node until the end node that introduces at least one new set of processing statements or a new condition (i.e., new nodes)
Must move along at least one edge that has not been traversed before by a previous path
Basis set for flow graph on previous slide
Path 1:
Path 2:
Path 3:
Path 4:
The number of paths in the basis set is determined by the cyclomatic complexity

18. Cyclomatic Complexity
Provides a quantitative measure of the logical complexity of a program
Defines the number of independent paths in the basis set
Provides an upper bound for the number of tests that must be conducted to ensure all statements have been executed at least once
Can be computed three ways
The number of regions
V(G) = E – N + 2, where E is the number of edges and N is the number of nodes in graph G
V(G) = P + 1, where P is the number of predicate nodes in the flow graph G
Results in the following equations for the example flow graph
Number of regions = 4
V(G) = 14 edges – 12 nodes + 2 = 4
V(G) = 3 predicate nodes + 1 = 4
V (G) is the maximum number of independent paths in the graph

19. Deriving the Basis Set and Test Cases
Using the design or code as a foundation, draw a corresponding flow graph
Determine the cyclomatic complexity of the resultant flow graph
Determine a basis set of linearly independent paths
Prepare test cases that will force execution of each path in the basis set
Cyclomatic complexity can be calculated manually if the program is small. Automated tools need to be used if the program is very complex as this involves more flow graphs. Based on complexity number, team can conclude on the actions that need to be taken for measure.
OCLint - Static code analyzer for C and Related Languages
devMetrics - Analyzing metrics for C# projects
Reflector Add In - Code metrics for .NET assemblies
GMetrics - Find metrics in Java related applications
NDepends - Metrics in Java applications

20. A Second Flow Graph Example4 5 6 8 7 10 9 16 11 17 13 12 14 15

21. A Sample Diagram and Analyze3 4 6 7 9 12 13 10 15 16 18 20 21

22. Loop Testing - General
A white-box testing technique that focuses exclusively on the validity of loop constructs
Testing occurs by varying the loop boundary values
Examples: for (i = 0; i < MAX_INDEX; i++) while (currentTemp >= MINIMUM_TEMPERATURE)

23. Testing of Simple Loops
Skip the loop entirely
Only one pass through the loop
Two passes through the loop
m passes through the loop, where m < n
n –1, n, n + 1 passes through the loop
‘n’ is the maximum number of allowable passes through the loop

24. Testing of Nested Loops
Start at the innermost loop; set all other loops to minimum values
Conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration parameter values; add other tests for out-of-range or excluded values
Work outward, conducting tests for the next loop, but keeping all other outer loops at minimum values and other nested loops to “typical” values
Continue until all loops have been tested

25. Black-box Testing
Complements white-box testing by uncovering different classes of errors
Focuses on the functional requirements and the information domain of the software
Used during the later stages of testing after white box testing has been performed
The tester identifies a set of input conditions that will fully exercise all functional requirements for a program

26. Black-box Testing Categories
Incorrect or missing functions
Interface errors
Errors in data structures or external data base access
Behavior or performance errors
Initialization and termination errors

27. Equivalence Partitioning
A black-box testing method that divides the input domain of a program into classes of data from which test cases are derived
An ideal test case single-handedly uncovers a complete class of errors, thereby reducing the total number of test cases that must be developed
Test case design is based on an evaluation of equivalence classes for an input condition
An equivalence class represents a set of valid or invalid states for input conditions
From each equivalence class, test cases are selected so that the largest number of attributes of an equivalence class are exercise at once

28. Equivalence partitions

29. Guidelines for Defining Equivalence Classes
If an input condition specifies a range, one valid and two invalid equivalence classes are defined
Input range: 1 – 10 Eq classes: {1..10}, {x < 1}, {x > 10}
If an input condition requires a specific value, one valid and two invalid equivalence classes are defined
Input value: 250 Eq classes: {250}, {x < 250}, {x > 250}
If an input condition specifies a member of a set, one valid and one invalid equivalence class are defined
Input set: {-2.5, 7.3, 8.4} Eq classes: {-2.5, 7.3, 8.4}, {any other x}
If an input condition is a Boolean value, one valid and one invalid class are define
Input: {true condition} Eq classes: {true condition}, {false condition}

30. Automated testing
Whenever possible, unit testing should be automated so that tests are run and checked without manual intervention.
In automated unit testing, you make use of a test automation framework (such as JUnit) to write and run your program tests.
Unit testing frameworks provide generic test classes that you extend to create specific test cases. They can then run all of the tests that you have implemented and report, often through some GUI, on the success of otherwise of the tests.

31. Test-driven development
Test-driven development (TDD) is an approach to program development in which you inter-leave testing and code development.
Tests are written before code and ‘passing’ the tests is the critical driver of development.
You develop code incrementally, along with a test for that increment. You don’t move on to the next increment until the code that you have developed passes its test.
TDD was introduced as part of agile methods such as Extreme Programming. However, it can also be used in plan- driven development processes.

32. Test-driven development

33. Regression testing
Regression testing is testing the system to check that changes have not ‘broken’ previously working code.
In a manual testing process, regression testing is expensive but, with automated testing, it is simple and straightforward. All tests are rerun every time a change is made to the program.
Tests must run ‘successfully’ before the change is committed.

34. Release testing
Release testing is the process of testing a particular release of a system that is intended for use outside of the development team.
The primary goal of the release testing process is to convince the supplier of the system that it is good enough for use.
Release testing, therefore, has to show that the system delivers its specified functionality, performance and dependability, and that it does not fail during normal use.
Release testing is usually a black-box testing process where tests are only derived from the system specification.

35. Release testing and system testing
Release testing is a form of system testing.
Important differences:
A separate team that has not been involved in the system development, should be responsible for release testing.
System testing by the development team should focus on discovering bugs in the system (defect testing). The objective of release testing is to check that the system meets its requirements and is good enough for external use (validation testing).

36. User testing
User or customer testing is a stage in the testing process in which users or customers provide input and advice on system testing.
User testing is essential, even when comprehensive system and release testing have been carried out.
The reason for this is that influences from the user’s working environment have a major effect on the reliability, performance, usability and robustness of a system. These cannot be replicated in a testing environment.

37. Types of user testing Alpha testing Beta testing Acceptance testing
Users of the software work with the development team to test the software at the developer’s site.
Beta testing
A release of the software is made available to users to allow them to experiment and to raise problems that they discover with the system developers.
Acceptance testing
Customers test a system to decide whether or not it is ready to be accepted from the system developers and deployed in the customer environment. Primarily for custom systems.

38. Agile methods and acceptance testing
In agile methods, the user/customer is part of the development team and is responsible for making decisions on the acceptability of the system.
Tests are defined by the user/customer and are integrated with other tests in that they are run automatically when changes are made.
There is no separate acceptance testing process.
Main problem here is whether or not the embedded user is ‘typical’ and can represent the interests of all system stakeholders.