1. By Touseeftahir@ciitlahore.edu.pk
Software Testing By

2. Content Essence Terminology Classification Debugging IEEE Standards
Unit, System …
BlackBox, WhiteBox
Debugging
IEEE Standards

3. Definition
Glen Myers
Testing is the process of executing a program with the intent of finding errors

4. Objective explained Paul Jorgensen
Testing is obviously concerned with errors, faults, failures and incidents. A test is the act of exercising software with test cases with an objective of
Finding failure
Demonstrate correct execution

5. Principles All tests should be traceable to customer requirements
Tests should be planned long before testing begins
The Pareto principle applies to Testing
Typically, 80% of the errors come from 20% of the modules
Testing should begin ‘‘in the small’’ and progress towards “in the large”
Exhaustive Testing is not possible, but,
if time permits, conduct multiple failure mode testing
Test plans must receive independent review
Independent testing should mean that those doing the testing have an independent reporting path to upper management. They should not report to project management.
Those doing independent testing need to be involved in oversight of the software development process, or they will not be knowledgeable enough to do a thorough testing job.
The greater the criticality and reliability demands of the software, the greater the need for independent testing and V&V.

6. Testability
The ease with which a computer program can be tested .

7. Characteristics for Testability
Operability
the better it works , the more efficiently it can be tested
The system has few bugs
No bugs block the execution of tests
The product evolves in functional stages

8. Characteristics for Testability
Observability
what you see is what you test
Distinct output for each input
System states and variables visible during execution
Past system states and variables are visible
All factors affecting the output are visible
Incorrect output is easily identified
Internal errors are automatically detected and reported

9. Characteristics for Testability
Controllability
the better we can control the software , the more testing can be automated
All possible outputs can be generated through some combination of input
All code is executable through some combination of input
Input and Output formats are consistent and structured
All sequences of task interaction can be generated
Tests can be conveniently specified and reproduced

10. Characteristics for Testability
Decomposability
By controlling the scope of testing , isolate(separate) problems and perform smarter retesting
The Software system is built from independent modules
Software modules can be tested independently
While this is very important, it does not obviate(prevent) the need for integration testing

11. Characteristics for Testability
Simplicity
the less there is to test , the more quickly we can test it
Functional simplicity
Structural simplicity
Code simplicity

12. Characteristics for Testability
the fewer the changes , the fewer disruptions to testing
Changes are infrequent
Changes are controlled
Changes do not invalidate existing tests
The software recovers well from failures
Mention NASA new mission control center problem where they made a change and missed a side effect. The result was that the computers went into a saturation mode and had to be taken off line and replace with the old ones.
del----chg--fly
| |
---chg failure

13. Characteristics for Testability
Understandability
the more information we have , the smarter we will test
The design is well understood
Dependencies between internal, external and shared components well understood
Changes to design are well communicated
Technical documentation is instantly accessible, well- organized, specific and accurate

14. Terminology Error Fault Represents mistakes made by people
Is result of error. May be categorized as
Fault of Commission – we enter something into representation that is incorrect
Fault of Omission – Designer can make error of omission, the resulting fault is that something is missing that should have been present in the representation

15. Cont… Failure Incident Test case Occurs when fault executes.
Behavior of fault. An incident is the symptom(s) associated with a failure that alerts user to the occurrence of a failure
Test case
Associated with program behavior. It carries set of input and list of expected output

16. A Testing Life Cycle Fix Error Requirement Fault Specs Resolution
Design
Fault
Isolation
Error
Fault
Coding
Fault
Classification
Fault
incident
Testing

17. Cont… Verification Validation
Process of determining whether output of one phase of development conforms to its previous phase.
The software process is following software development procedures and standards.
Validation
Process of determining whether a fully developed system conforms to its SRS document

18. Verification versus Validation
Verification is concerned with phase containment of errors
Validation is concerned about the final product to be error free

19. Relationship – program behaviors
Specified
(expected)
Behavior
Programmed
(observed)
Fault Of
Omission
Commission
Correct portion

20. Classification of Test
There are two levels of classification
One distinguishes at granularity level
Unit level
System level
Integration level
Other classification is based on methodologies
Black box (Functional) Testing
White box (Structural) Testing

21. Relationship – Testing wrt Behavior
Program Behaviors
Specified
(expected)
Behavior
Programmed
(observed)
Test Cases
(Verified behavior) 8 7 5 6 1 4 3 2

22. Cont… 2, 5 1, 4 3, 7 Specified behavior that are not tested
Specified behavior that are tested
3, 7
Test cases corresponding to unspecified behavior

23. Cont… 2, 6 1, 3 4, 7 Programmed behavior that are not tested
Programmed behavior that are tested
4, 7
Test cases corresponding to un-programmed behaviors

24. Inferences
If there are specified behaviors for which there are no test cases, the testing is incomplete
If there are test cases that correspond to unspecified behaviors
Either such test cases are unwarranted
Specification is deficient (also implies that testers should participate in specification and design reviews)

25. Test methodologies Functional (Black box) inspects specified behavior
Structural (White box) inspects programmed behavior

26. Functional Test cases
Specified
Programmed
Test
Cases

27. Structural Test cases
Specified
Programmed
Test
Cases

28. When to use what Few set of guidelines available
A logical approach could be
Prepare functional test cases as part of specification. However they could be used only after unit and/or system is available.
Preparation of Structural test cases could be part of implementation/code phase.
Unit, Integration and System testing are performed in order.

29. Unit testing – essence Applicable to modular design
Unit testing inspects individual modules
Locate error in smaller region
In an integrated system, it may not be easier to determine which module has caused fault
Reduces debugging efforts

30. Test cases and Test suites
Test case is a triplet [I, S, O] where
I is input data
S is state of system at which data will be input
O is the expected output
Test suite is set of all test cases
Test cases are not randomly selected. Instead even they need to be designed.

31. Need for designing test cases
Almost every non-trivial system has an extremely large input data domain thereby making exhaustive testing impractical
If randomly selected then test case may loose significance since it may expose an already detected error by some other test case

32. Design of test cases
Number of test cases do not determine the effectiveness
To detect error in following code
if(x>y) max = x; else max = x;
{(x=3, y=2); (x=2, y=3)} will suffice
{(x=3, y=2); (x=4, y=3); (x=5, y = 1)} will falter
Each test case should detect different errors

33. Black box testing Equivalence class partitioning
Boundary value analysis
Comparison testing
Orthogonal array testing
Decision Table based testing

34. Equivalence Class Partitioning
Input values to a program are partitioned into equivalence classes.
Partitioning is done such that:
program behaves in similar ways to every input value belonging to an equivalence class.

35. Why define equivalence classes?
Test the code with just one representative value from each equivalence class:
as good as testing using any other values from the equivalence classes.

36. Equivalence Class Partitioning
How do you determine the equivalence classes?
examine the input data.
few general guidelines for determining the equivalence classes can be given

37. Equivalence Class Partitioning
If the input data to the program is specified by a range of values:
e.g. numbers between 1 to 5000.
one valid and two invalid equivalence classes are defined.
5000 1

38. Equivalence Class Partitioning
If input is an enumerated set of values:
e.g. {a,b,c}
one equivalence class for valid input values
another equivalence class for invalid input values should be defined.

39. Example
A program reads an input value in the range of 1 and 5000:
computes the square root of the input number
SQRT

40. Example (cont.) There are three equivalence classes: 5000 1
the set of smaller values than 1 i.e. 0 and negative values,
set of integers in the range of 1 and 5000,
integers larger than 5000.
5000 1

41. Example (cont.) The test suite must include: 5000 1
representatives from each of the three equivalence classes:
a possible test suite can be: {-5,500,6000}.
5000 1

42. Boundary Value Analysis
Some typical programming errors occur:
at boundaries of equivalence classes
might be purely due to psychological factors.
Programmers often fail to see:
special processing required at the boundaries of equivalence classes.

43. Boundary Value Analysis
Programmers may improperly use < instead of <=
Boundary value analysis:
select test cases at the boundaries of different equivalence classes.

44. Example
For a function that computes the square root of an integer in the range of 1 and 5000:
test cases must include the values: {0,1,5000,5001}.
5000 1

45. White-Box Testing Statement coverage Branch coverage Path coverage
Condition coverage
Mutation testing
Data flow-based testing

46. Statement Coverage Statement coverage methodology: The principal idea:
design test cases so that every statement in a program is executed at least once.
The principal idea:
unless a statement is executed, we have no way of knowing if an error exists in that statement

47. Statement coverage criterion
Observing that a statement behaves properly for one input value:
no guarantee that it will behave correctly for all input values.

48. Example Euclid's GCD Algorithm int f1(int x, int y){ while (x != y){
if (x>y) then
x=x-y;
else y=y-x; }
return x; }

49. Euclid's GCD computation algorithm
By choosing the test set
{(x=3,y=3),(x=4,y=3), (x=3,y=4)}
all statements are executed at least once.

50. Branch Coverage Test cases are designed such that:
different branch conditions is given true and false values in turn.
Branch testing guarantees statement coverage:
a stronger testing compared to the statement coverage-based testing.

51. Example Test cases for branch coverage can be:
{(x=3,y=3), (x=4,y=3), (x=3,y=4)}

52. Condition Coverage Test cases are designed such that: Example
each component of a composite conditional expression given both true and false values.
Example
Consider the conditional expression ((c1.and.c2).or.c3):
Each of c1, c2, and c3 are exercised at least once i.e. given true and false values.

53. Branch testing
Branch testing is the simplest condition testing strategy
compound conditions appearing in different branch statements are given true and false values.

54. Branch testing Condition testing Branch testing
stronger testing than branch testing:
Branch testing
stronger than statement coverage testing.

55. Condition coverage Consider a Boolean expression having n components:
for condition coverage we require 2n test cases.
practical only if n (the number of component conditions) is small.

56. Path Coverage Design test cases such that: Defined in terms of
all linearly independent paths in the program are executed at least once.
Defined in terms of
control flow graph (CFG) of a program.

57. Control flow graph (CFG)
A control flow graph (CFG) describes:
the sequence in which different instructions of a program get executed.
the way control flows through the program.

58. How to draw Control flow graph?
Number all the statements of a program.
Numbered statements:
represent nodes of the control flow graph.
An edge from one node to another node exists:
if execution of the statement representing the first node can result in transfer of control to the other node.

59. Example int f1(int x,int y){ while (x != y){ if (x>y) then x=x-y;
else y=y-x; }
return x; }

60. Example Control Flow Graph1 2 3 4 5 6

61. Path A path through a program:
A node and edge sequence from the starting node to a terminal node of the control flow graph.
There may be several terminal nodes for program.

62. Independent path Any path through the program:
introducing at least one new node that is not included in any other independent paths.
It may be straight forward to identify linearly independent paths of simple programs. However For complicated programs it is not so easy to determine the number of independent paths.

63. McCabe's cyclomatic metric
An upper bound:
for the number of linearly independent paths of a program
Provides a practical way of determining:
the maximum number of linearly independent paths in a program.

64. McCabe's cyclomatic metric
Given a control flow graph G, cyclomatic complexity V(G):
V(G)= E-N+2
N is the number of nodes in G
E is the number of edges in G

65. Example
Cyclomatic complexity = 7 – = 3.

66. Cyclomatic complexity
Another way of computing cyclomatic complexity:
determine number of bounded areas in the graph
Any region enclosed by a nodes and edge sequence.
V(G) = Total number of bounded areas + 1

67. Example From a visual examination of the CFG:
the number of bounded areas is 2.
cyclomatic complexity = 2+1=3.

68. Cyclomatic complexity
McCabe's metric provides:
a quantitative measure of estimating testing difficulty
Amenable to automation
Intuitively,
number of bounded areas increases with the number of decision nodes and loops.

69. Cyclomatic complexity
The cyclomatic complexity of a program provides:
a lower bound on the number of test cases to be designed
to guarantee coverage of all linearly independent paths.

70. Cyclomatic complexity
Defines the number of independent paths in a program.
Provides a lower bound:
for the number of test cases for path coverage.
only gives an indication of the minimum number of test cases required.

71. Path testing
The tester proposes initial set of test data using his experience and judgment.

72. Path testing
A testing tool such as dynamic program analyzer, then may be used:
to indicate which parts of the program have been tested
the output of the dynamic analysis used to guide the tester in selecting additional test cases.

73. Derivation of Test Cases
Draw control flow graph.
Determine V(G).
Determine the set of linearly independent paths.
Prepare test cases:
to force execution along each path

74. Example Control Flow Graph1 2 3 4 5 6

75. Derivation of Test Cases
Number of independent paths: 4
1, test case (x=1, y=1)
1, 2, 3, 5, 1, 6 test case(x=1, y=2)
1, 2, 4, 5, 1, 6 test case(x=2, y=1)

76. An interesting application of cyclomatic complexity
Relationship exists between:
McCabe's metric
the number of errors existing in the code,
the time required to find and correct the errors.

77. Cyclomatic complexity
Cyclomatic complexity of a program:
also indicates the psychological complexity of a program.
difficulty level of understanding the program.

78. Cyclomatic complexity
From maintenance perspective,
limit cyclomatic complexity
of modules to some reasonable value.
Good software development organizations:
restrict cyclomatic complexity of functions to a maximum of ten or so.

79. Mutation Testing The software is first tested:
using an initial testing method based on white-box strategies we already discussed.
After the initial testing is complete,
mutation testing is taken up.
The idea behind mutation testing:
make a few arbitrary small changes to a program at a time.

80. Mutation Testing Each time the program is changed,
it is called a mutated program
the change is called a mutant.

81. Mutation Testing A mutated program:
tested against the full test suite of the program.
If there exists at least one test case in the test suite for which:
a mutant gives an incorrect result, then the mutant is said to be dead.

82. Mutation Testing If a mutant remains alive:
even after all test cases have been exhausted, the test suite is enhanced to kill the mutant.
The process of generation and killing of mutants:
can be automated by predefining a set of primitive changes that can be applied to the program.

83. Mutation Testing The primitive changes can be:
altering an arithmetic operator,
changing the value of a constant,
changing a data type, etc.

84. Mutation Testing A major disadvantage of mutation testing:
computationally very expensive,
a large number of possible mutants can be generated.

85. Debugging Once errors are identified:
it is necessary identify the precise location of the errors and to fix them.
Each debugging approach has its own advantages and disadvantages:
each is useful in appropriate circumstances.

86. Brute-force method This is the most common method of debugging:
least efficient method.
program is loaded with print statements
print the intermediate values
hope that some of printed values will help identify the error.

87. Symbolic Debugger Brute force approach becomes more systematic:
with the use of a symbolic debugger,
symbolic debuggers get their name for historical reasons
early debuggers let you only see values from a program dump:
determine which variable it corresponds to.

88. Symbolic Debugger Using a symbolic debugger:
values of different variables can be easily checked and modified
single stepping to execute one instruction at a time
break points and watch points can be set to test the values of variables.

89. Backtracking This is a fairly common approach.
Beginning at the statement where an error symptom has been observed:
source code is traced backwards until the error is discovered.

90. Example int main(){ int i,j,s; i=1;
while(i<=10){ s=s+i; i++; j=j++;}
printf(“%d”,s); }

91. Backtracking
Unfortunately, as the number of source lines to be traced back increases,
the number of potential backward paths increases
becomes unmanageably large for complex programs.

92. Cause-elimination method
Determine a list of causes:
which could possibly have contributed to the error symptom.
tests are conducted to eliminate each.
A related technique of identifying error by examining error symptoms:
software fault tree analysis.

93. Program Slicing This technique is similar to back tracking.
However, the search space is reduced by defining slices.
A slice is defined for a particular variable at a particular statement:
set of source lines preceding this statement which can influence the value of the variable.

94. Example int main(){ int i,s; i=1; s=1; while(i<=10){ s=s+i; i++;}
printf(“%d”,s);
printf(“%d”,i); }

95. Debugging Guidelines
Debugging usually requires a thorough understanding of the program design.
Debugging may sometimes require full redesign of the system.
A common mistake novice programmers often make:
not fixing the error but the error symptoms.

96. Debugging Guidelines Be aware of the possibility:
an error correction may introduce new errors.
After every round of error-fixing:
regression testing must be carried out.

97. Program Analysis Tools
An automated tool:
takes program source code as input
produces reports regarding several important characteristics of the program,
such as size, complexity, adequacy of commenting, adherence to programming standards, etc.

98. Program Analysis Tools
Some program analysis tools:
produce reports regarding the adequacy of the test cases.
There are essentially two categories of program analysis tools:
Static analysis tools
Dynamic analysis tools

99. Static Analysis Tools Static analysis tools:
assess properties of a program without executing it.
Analyze the source code
provide analytical conclusions.

100. Static Analysis Tools Whether coding standards have been adhered to?
Commenting is adequate?
Programming errors such as:
Un-initialized variables
mismatch between actual and formal parameters.
Variables declared but never used, etc.

101. Static Analysis Tools
Code walk through and inspection can also be considered as static analysis methods:
however, the term static program analysis is generally used for automated analysis tools.

102. Dynamic Analysis Tools
Dynamic program analysis tools require the program to be executed:
its behaviour recorded.
Produce reports such as adequacy of test cases.

103. Integration testing
After different modules of a system have been coded and unit tested:
modules are integrated in steps according to an integration plan
partially integrated system is tested at each integration step.

104. System Testing System testing involves:
validating a fully developed system against its requirements.

105. Integration Testing
Develop the integration plan by examining the structure chart :
big bang approach
top-down approach
bottom-up approach
mixed approach

106. Example Structured Design
root
rms
Valid-numbers
rms
Valid-numbers
Get-good-data
Compute-solution
Display-solution
Validate
-data
Get-data

107. Big bang Integration Testing
Big bang approach is the simplest integration testing approach:
all the modules are simply put together and tested.
this technique is used only for very small systems.

108. Big bang Integration Testing
Main problems with this approach:
if an error is found:
it is very difficult to localize the error
the error may potentially belong to any of the modules being integrated.
debugging errors found during big bang integration testing are very expensive to fix.

109. Bottom-up Integration Testing
Integrate and test the bottom level modules first.
A disadvantage of bottom-up testing:
when the system is made up of a large number of small subsystems.
This extreme case corresponds to the big bang approach.

110. Top-down integration testing
Top-down integration testing starts with the main routine:
and one or two subordinate routines in the system.
After the top-level 'skeleton’ has been tested:
immediate subordinate modules of the 'skeleton’ are combined with it and tested.

111. Mixed integration testing
Mixed (or sandwiched) integration testing:
uses both top-down and bottom-up testing approaches.
Most common approach

112. Integration Testing In top-down approach: In bottom-up approach:
testing waits till all top-level modules are coded and unit tested.
In bottom-up approach:
testing can start only after bottom level modules are ready.

113. Phased versus Incremental Integration Testing
Integration can be incremental or phased.
In incremental integration testing,
only one new module is added to the partial system each time.

114. Phased versus Incremental Integration Testing
In phased integration,
a group of related modules are added to the partially integrated system each time.
Big-bang testing:
a degenerate case of the phased integration testing.

115. Phased versus Incremental Integration Testing
Phased integration requires less number of integration steps:
compared to the incremental integration approach.
However, when failures are detected,
it is easier to debug if using incremental testing
since errors are very likely to be in the newly integrated module.

116. System Testing There are three main kinds of system testing:
Alpha Testing
Beta Testing
Acceptance Testing

117. Alpha Testing
System testing is carried out by the test team within the developing organization.

118. Beta Testing
System testing performed by a select group of friendly customers.

119. Acceptance Testing System testing performed by the customer himself:
to determine whether the system should be accepted or rejected.

120. Stress Testing Stress testing (aka endurance testing):
impose abnormal input to stress the capabilities of the software.
Input data volume, input data rate, processing time, utilization of memory, etc. are tested beyond the designed capacity.

121. Performance Testing Addresses non-functional requirements.
May sometimes involve testing hardware and software together.
There are several categories of performance testing.

122. Stress testing Evaluates system performance Stress testing
when stressed for short periods of time.
Stress testing
also known as endurance testing.

123. Stress testing Stress tests are black box tests:
designed to impose a range of abnormal and even illegal input conditions
so as to stress the capabilities of the software.

124. Stress Testing
If the requirements is to handle a specified number of users, or devices:
stress testing evaluates system performance when all users or devices are busy simultaneously.

125. Stress Testing
If an operating system is supposed to support 15 multiprogrammed jobs,
the system is stressed by attempting to run 15 or more jobs simultaneously.
A real-time system might be tested
to determine the effect of simultaneous arrival of several high-priority interrupts.

126. Stress Testing
Stress testing usually involves an element of time or size,
such as the number of records transferred per unit time,
the maximum number of users active at any time, input data size, etc.
Therefore stress testing may not be applicable to many types of systems.

127. Volume Testing Addresses handling large amounts of data in the system:
whether data structures (e.g. queues, stacks, arrays, etc.) are large enough to handle all possible situations
Fields, records, and files are stressed to check if their size can accommodate all possible data volumes.

128. Configuration Testing
Analyze system behaviour:
in various hardware and software configurations specified in the requirements
sometimes systems are built in various configurations for different users
for instance, a minimal system may serve a single user,
other configurations for additional users.

129. Compatibility Testing
These tests are needed when the system interfaces with other systems:
check whether the interface functions as required.

130. Compatibility testing Example
If a system is to communicate with a large database system to retrieve information:
a compatibility test examines speed and accuracy of retrieval.

131. Recovery Testing These tests check response to:
presence of faults or to the loss of data, power, devices, or services
subject system to loss of resources
check if the system recovers properly.

132. Maintenance Testing Diagnostic tools and procedures:
help find source of problems.
It may be required to supply
memory maps
diagnostic programs
traces of transactions,
circuit diagrams, etc.

133. Maintenance Testing Verify that:
all required artefacts for maintenance exist
they function properly

134. Documentation tests
Check that required documents exist and are consistent:
user guides,
maintenance guides,
technical documents

135. Documentation tests Sometimes requirements specify:
format and audience of specific documents
documents are evaluated for compliance

136. Usability tests All aspects of user interfaces are tested:
Display screens
messages
report formats
navigation and selection problems

137. Environmental test
These tests check the system’s ability to perform at the installation site.
Requirements might include tolerance for
heat
humidity
chemical presence
portability
electrical or magnetic fields
disruption of power, etc.

138. Test Summary Report Generated towards the end of testing phase.
Covers each subsystem:
a summary of tests which have been applied to the subsystem.

139. Test Summary Report Specifies:
how many tests have been applied to a subsystem,
how many tests have been successful,
how many have been unsuccessful, and the degree to which they have been unsuccessful,
e.g. whether a test was an outright failure
or whether some expected results of the test were actually observed.

140. Regression Testing
Does not belong to either unit test, integration test, or system test.
In stead, it is a separate dimension to these three forms of testing.

141. Regression testing Regression testing is the running of test suite:
after each change to the system or after each bug fix
ensures that no new bug has been introduced due to the change or the bug fix.

142. Regression testing Regression tests assure:
the new system’s performance is at least as good as the old system
always used during phased system development.

143. How many errors are still remaining?
Seed the code with some known errors:
artificial errors are introduced into the program.
Check how many of the seeded errors are detected during testing.

144. Error Seeding Let: N be the total number of errors in the system
n of these errors be found by testing.
S be the total number of seeded errors,
s of the seeded errors be found during testing.

145. Error Seeding n/N = s/S N = S n/s
remaining defects: N - n = n ((S - s)/ s)

146. Example 100 errors were introduced.
90 of these errors were found during testing
50 other errors were also found.
Remaining errors= (100-90)/90 = 6

147. Error Seeding
The kind of seeded errors should match closely with existing errors:
However, it is difficult to predict the types of errors that exist.
Categories of remaining errors:
can be estimated by analyzing historical data from similar projects.

148. IEEE Standard 829 - 1998 Test plan identifier Introduction Test Items
Features to be tested
Features not to be tested
Approach
Item pass/fail criteria
Suspension criteria and resumption requirements

149. Cont… Test deliverables Testing tasks Environment needs
Responsibilities
Staffing and training needs
Risk and contingencies
Approvals

150. References Software Testing, A craftsman’s approach
Paul Jorgensen
Fundamental of Software Engineering
Rajib Mall
Software Engineering, A practitioner’s approach
Roger Pressman