1. Software Testing
COM 3220
11/28/2018
Testing/Spring 98

2. Three meanings of bug
error: mistake made by a developer. Mostly located in people’s head.
fault: an error may lead to one or more faults. Faults are located in text of program.
failure: execution of faulty code may lead to one or more failures. A failure occurs when there is a difference between the results of the correct and incorrect programs.
11/28/2018
Testing/Spring 98

3. Failure detection Compare actual output to expected output.
Expected output is from specification.
Specification: any external, independent description of the program, including user documentation.
Are often incomplete, incorrect, ambiguous or contradictory. Specification may be wrong, not the program!
11/28/2018
Testing/Spring 98

4. Motivation Derive tests from both the specification and the program.
Derivation is done by ”predicting” likely programmer errors or likely program faults.
Use general rules, e.g., always test boundary conditions.
11/28/2018
Testing/Spring 98

5. Motivation Check for faults of omission: missed special cases.
Most common type of fault according to a study by Glass.
Experienced testers have a catalog of programming cliches and associated errors available. See Test Requirement Catalog (low-level omissions).
11/28/2018
Testing/Spring 98

6. Motivation
First requirement of test design: Be methodical. Three stages:
Finding clues
sources for test requirements
Expanding them into test requirements
useful sets of inputs that should be considered
Writing test specifications
exact inputs and expected outputs
11/28/2018
Testing/Spring 98

7. Clues
What needs testing? Collect from specification, program, bug reports, etc.
Create a checklist.
11/28/2018
Testing/Spring 98

8. Test requirements Create a test requirement catalog 11/28/2018
Testing/Spring 98

9. Test specifications Describes input and exact expected output.
11/28/2018
Testing/Spring 98

10. Supplementary code inspections
Some faults that testing is poor at detecting.
11/28/2018
Testing/Spring 98

11. Test implementation Avoid having to write a lot of support code.
It is better to test larger subsystems because less support code needs to be written.
Individual routines are exercised more.
Testing the tests: test coverage as a crude measure.
During test design do not pay attention to coverage criteria.
11/28/2018
Testing/Spring 98

12. Test implementation
During test design do not pay attention to coverage criteria. Test requirements from other sources should do that anyway.
Complete subsystem testing will usually result in high coverage.
Treat missed branches as clues about weaknesses in the test design.
11/28/2018
Testing/Spring 98

13. Subsystem Specification Subsystem Code Catalogued Past Experience
Clues and
Test Requirements
Program and
Specification
Changes
Coverage
Test Specifications
Bug Reports
Implemented
Tests
11/28/2018
Testing/Spring 98

14. Application Graph algorithms:
Depth-first traversal
Finding all paths satisfying some restrictions.
Happens to be be a subsystem of Demeter/Java.
You don’t have to know anything about Demeter. You will learn the minimal things you need.
11/28/2018
Testing/Spring 98

15. Use Java to write testing code
You will need to write some Java code for testing.
11/28/2018
Testing/Spring 98

16. Subsystem Specification Subsystem Code Catalogued Past Experience
Part of Demeter/Java
Graph traversal
Subsystem
Specification
Subsystem
Code
Catalogued
Past Experience
Clues and
Test Requirements
Program and
Specification
Changes
Coverage
Test Specifications
Use Java/Scope
Bug Reports
Implemented
Tests
11/28/2018
Testing/Spring 98

17. What we want to test
Given a directed acyclic graph G (no multi-edges), traverse all paths from A via B to C.
Given a directed acyclic graph G (no multiedges), traverse all paths from A bypassing B to C.
11/28/2018
Testing/Spring 98

18. Notation for describing graphs
A = B C D. // node A has three successors
B = E. // node B has only one successor
E = . // E has no successor
This information is put into a file program.cd.
Two files program.beh are given. Contains the traversal specification. Counts visits of C.
11/28/2018
Testing/Spring 98

19. How to call the program demjava test
The program will print the paths it traversed and print how often it visits C.
11/28/2018
Testing/Spring 98

20. Clue list: from A via B to C
What does program do if there is no path from A via B to C?
What if A or B or C do not appear in the graph.
Check that paths from A to C not going through B are excluded: paths of length 1, 2 or 3.
11/28/2018
Testing/Spring 98

21. Clue list: From A bypassing B to C
What does program do if there is no path from A bypassing B to C?
What if A or B or C do not appear in the graph. Is it ok if B does not appear?
Check that paths from A to C going through B are excluded: paths of length 1, 2 or 3.
11/28/2018
Testing/Spring 98

22. Test specifications: From A via B to C
A=C B X.
B=C X.
C=.
X=C.
A=C B.
B=C.
C=. A A
A=B
B=C.
C=. A B B B C C X C
2 visits
1 visit
1 visit
11/28/2018
Testing/Spring 98

23. Test specifications: From A via B to C
A=C B X Y.
Y=B.
B=C X.
C=.
X=C. A Y B C X
4 visits
11/28/2018
Testing/Spring 98

24. Test specifications: From A bypassing B to C
A=C B X Y.
Y=B.
B=C X.
C=.
X=C. A Y B C X
2 visits
11/28/2018
Testing/Spring 98

25. Fundamental Assumptions of Subsystem Testing
Most errors are not very creative. Methodological checklist-based approaches will have a high payoff.
Faults of omission, those caused by a failure to anticipate special cases, are the most important and most difficult type.
Specification faults, especially omissions, are more dangerous than code faults.
11/28/2018
Testing/Spring 98

26. Fundamental Assumptions of Subsystem Testing
At every stage of testing, mistakes are inevitable. Later stages should compensate for them.
Code coverage is a good approximate measure of test quality. Must be used with extreme care.
11/28/2018
Testing/Spring 98

27. A summary of subsystem testing
Build the test requirement checklist
Find clues
Expand clues into test requirements
Design the tests
Combine requirements into tests
Check tests for common testing mistakes
Supplement testing with code inspections
11/28/2018
Testing/Spring 98

28. A summary of subsystem testing
Implement test support code
Implement tests
Evaluate and improve tests
use code coverage tool
find undertested or missing clues
find more test requirements
write more test requirements
11/28/2018
Testing/Spring 98

29. 11/28/2018
Testing/Spring 98

30. Test coverage tool
For example: For each traversal, which fraction of traversal methods are used?
How often is each adaptive method called?
Define global counters in Main class.
Use aspect language to instrument code. Generate code.
Testing tool development.
11/28/2018
Testing/Spring 98

31. Course ideas Advanced OO systems develops testing tools for
testing class?
Test UML graphical editor.
11/28/2018
Testing/Spring 98

32. Test strategies
a systematic method used to select and/or generate tests to be included in a test suite.
effective: likely to reveal bugs
Kinds
behavioral = black-box = functional
structural = white-box = glass-box testing
hybrid
11/28/2018
Testing/Spring 98

33. Testing strategies behavioral = black-box = functional
based on requirements
structural = white-box = glass-box testing
based on program (coverages)
hybrid
use combination
11/28/2018
Testing/Spring 98

34. Classification of bugs
unit/component bugs
integration bugs
system bugs
11/28/2018
Testing/Spring 98

35. Generic Testing Principles
Define the graph
Design node-cover tests (tests that confirm that the nodes are there)
Design edge-cover tests (that confirm all required links and no more)
Design loop tests
Beizer 2.5
11/28/2018
Testing/Spring 98

36. Generic Testing Principles: Example
Define the graph
UML class diagram
Design node-cover tests (tests that confirm that the nodes are there)
Build at least one object of each class
Design edge-cover tests (that confirm all required links)
use each inheritance edge and association
Beizer 2.5
11/28/2018
Testing/Spring 98

37. Generic Testing Principles: Example
Define the graph
Finite state machine
Design node-cover tests (tests that confirm that the nodes are there)
Use each state at least once
Design edge-cover tests (that confirm all required links)
use each state transition at least once
Beizer 2.5
11/28/2018
Testing/Spring 98

38. 11/28/2018
Testing/Spring 98

39. Quality factors Correctness Maintainability Portability
conform to specification
Maintainability
ease with which software can be changed
corrective: error fixing
adaptive: requirement changes MAJORITY
perfective: improve system
Portability
11/28/2018
Testing/Spring 98

40. Quality factors Testability Usability
how easy to test? Are requirements clear?
Usability
effort required to learn and operate system
Reliability: mean-time between failures
Efficiency: use of resources
Integrity, Security
11/28/2018
Testing/Spring 98

41. Quality factors Reusability Interoperability
Write Quality Manual to address those issues
11/28/2018
Testing/Spring 98

42. ISO 9000 Series of Standards (5 years old)
How can customers judge the competence of a software developer?
Adopted by 130 countries.
ISO 9001: Quality Systems - Model for Quality Assurance in Design, Development, Production, Installation and Servicing. (general design)
11/28/2018
Testing/Spring 98

43. ISO 9000 Series of Standards (5 years old)
ISO Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software.
ISO Quality Management and Quality System Elements
11/28/2018
Testing/Spring 98

44. Automatic Verification of Industrial Designs
Based on two papers in: Workshop on Industrial-Strength Formal Specification Techniques, 1995, Boca Raton, Florida, IEEE Computer Society
Automatic Verification of Industrial Designs, pages 88-96
Timing Analysis of Industrial Real-Time Systems, pages
11/28/2018
Testing/Spring 98

45. Successful formal methods in industry
Formal methods are mathematical techniques that have been used in the specification and verification of computer systems.
Want to know: Are we building the product correctly? (Different from: are we building the right product).
11/28/2018
Testing/Spring 98

46. Formal methods
Many different specification languages and proof techniques.
Some are difficult to apply since computers are not good at proving theorems (they need a lot of human help)
Exception: Symbolic Model Checking: Fast, based on OBDD techniques (Ordered Binary Decision Diagrams).
11/28/2018
Testing/Spring 98

47. Symbolic Model Checking
Determine correctness of finite state systems.
Developed at CMU by Clarke/Emerson
Specifications are written as formulas in a propositional temporal logic.
Temporal logic: expressing ordering of events without introducing time explicitly
11/28/2018
Testing/Spring 98

48. Temporal Logic
A kind of modal logic. Origins in Aristotle and medieval logicians. Studied many modes of truth.
Modal logic includes propositional logic. Embellished with operators to achieve greater expressiveness.
A particular temporal logic: CTL (Computation Tree Logic)
11/28/2018
Testing/Spring 98

49. Computation Tree Logic
Used to express properties that will be verified
Computation trees are derived from the state transition graphs
State transition graphs unwound into an infinite tree rooted at initial state
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

50. Computation Tree Logic
CTL formulas built from
atomic propositions, where each proposition corresponds to a variable in the model
Boolean connectives
temporal operators. Two parts
path quantifier (A, E)
temporal operator (F,G,X,U)
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

51. Computation Tree Logic
Paths in tree represent all possible computations in model.
CTL formulas refer to the computation tree
Campos/Clarke/Marrero/Minea
page 97
If the signal req is high then eventually ack will also be high
11/28/2018
Testing/Spring 98

52. Computation Tree Logic
path quantifier (A, E)
A: true for all paths from a given state
E: true for some paths from a given state
temporal operator (F,G,X,U)
F ( holds sometime in the future) is true of a path if there exists a state in the path that satisfies .
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

53. Computation Tree Logic
temporal operator (F,G,X,U)
F ( holds sometime in the future) is true of a path if there exists a state in the path that satisfies .
Example: EF(started and not ready): It is possible to get to a state where started holds but ready does not hold.
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

54. Computation Tree Logic
temporal operator (F,G,X,U)
G ( holds globally) is true of a path if  holds for all states in the path.
Example: AG(req implies AF ack). It is always the case that if the signal req is high then eventually ack will also be high.
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

55. Computation Tree Logic
temporal operator (F,G,X,U)
X ( holds in the next state) means that  is true in the next state.
 U ( holds until  holds) is satisfied by a path if  is true in some state in the path, and in all preceding states,  holds.
Example: AG(send implies A[send U recv]). It is always the case that if send occurs, then eventually recv is true, and until that time, send must remain true.
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

56. Computation Tree Logic
Example: AG EF restart: From any state it is possible to get to the restart state.
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

57. Computation Tree Logic
Examples: Dark circle indicates that a specification  is true in corresponding state. Light means false.
Campos/Clarke/Marrero/Minea
page 97
AF
AG
EG
11/28/2018
Testing/Spring 98

58. Computation Tree Logic
Model to be verified: Finite state machine. (S,I,R) where S is the set of all possible states, I the set of initial states, R a binary relation on S which defines the possible transitions.
Can verify systems with more than states (1995).
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

59. Computation Tree Logic: Railway Interlocking Control
Simple Interlocking Model C 4
Avoid derailments
and train crashes B
Campos/Clarke/Marrero/Minea
page 97 2 A 5 3
Track sections: 2,3,4,5
Control Signals: A,B,C
11/28/2018
Testing/Spring 98

60. Computation Tree Logic: Railway Interlocking Control
Simple Interlocking Model
Inputs
2T 0 no train in 2
1 2 occupied by train
or broken C 4 B
Campos/Clarke/Marrero/Minea
page 97 2 A 5 3
Track sections: 2,3,4,5
Control Signals: A,B,C
11/28/2018
Testing/Spring 98

61. Computation Tree Logic: Railway Interlocking Control
Simple Interlocking Model
SPEC
AG!(SignalA=1 and
SignalB=1)
SignalC=1)
AG(2T=0 implies
AX SignalA=0) C 4 B
Campos/Clarke/Marrero/Minea
page 97 2 A 5 3
Track sections: 2,3,4,5 (0: unoccupied)
Control Signals: A,B,C(0:red, 1:green)
11/28/2018
Testing/Spring 98

62. Output from checker
Specification AG(SignalA=1 and …) is false as demonstrated by the following execution sequence
state 1.1
state 1.2 …
Gives counterexample if there is one.
11/28/2018
Testing/Spring 98

63. Computation Tree Logic: Implementation: BDDs
Binary Decision Diagrams
A canonical representation for Boolean formulas (canonical = in simplest or standard form).
Invented by Randal Bryant, now at CMU.
Similar to a binary decision tree, but structure is a dag rather than a tree. Allows nodes and substructures to be shared.
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

64. Computation Tree Logic: Implementation: BDDs
Binary Decision Diagrams
a b c d result
1 1 1 a 1
What is
Boolean
formula? b c
Campos/Clarke/Marrero/Minea
page 97 1 1 d 1
All paths to 1 1
11/28/2018
Testing/Spring 98

65. Computation Tree Logic: Implementation: BDDs
Binary Decision Diagrams a 1
Given a variable ordering, the BDD
for a formula is unique.
There are efficient algorithms to
compute the BDD for not f and f or g
given the BDD of f and g. b c
Campos/Clarke/Marrero/Minea
page 97 1 1 d 1 1
11/28/2018
Testing/Spring 98

66. Computation Tree Logic: Implementation: BDDs
Binary Decision Diagrams a 1
For the purpose of model checking
also need to compute BDD of restricted
formulas. Bryant describes an algorithm
for computing the BDD of a restricted
formula such as f, where v=0. b c
Campos/Clarke/Marrero/Minea
page 97 1 1 d 1 1
11/28/2018
Testing/Spring 98

67. Computation Tree Logic: Implementation: BDDs
Binary Decision Diagrams: All Boolean formulas are represented by BDDs. BDDs built in a bottom-up manner.
The set of atomic formulas is precisely the set of state variables. (BDD for an atomic variable = one BDD variable)
Formulas are built from atomic formulas using Boolean connectives. Allows CTL formulas.
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

68. Symbolic Model Checking
Determine correctness of finite state systems.
Specifications are written as formulas in a propositional temporal logic.
Models to be checked are represented by state transition graphs
Verification is accomplished by an efficient breadth-first search.
11/28/2018
Testing/Spring 98

69. Symbolic Model Checking
View transition system as model of logic.
Verify whether specifications are satisfied for model.
Advantages:
completely automatic
provides counterexamples (execution trace which shows why formula is not true)
verify partially specified systems
11/28/2018
Testing/Spring 98

70. Symbolic Model Checking
Model checkers achieve great efficiency through the use of symbolic implementation techniques
represent states and transitions through Boolean formulas in BDD form
11/28/2018
Testing/Spring 98

71. Symbolic Model Checking
Representing the Model
Labeled state-transition graph M.
Use BDDs to represent graph and check whether formula holds.
Behavior determined by variables V
11/28/2018
Testing/Spring 98

72. Symbolic Model Checking
Representing the Model
Behavior determined by variables V
current state
V’ = Second copy of variables
next state
11/28/2018
Testing/Spring 98

73. Symbolic Model Checking
Representing the Model: Relationship between variables in the current state and the next states is written as a formula using V and V’. Boolean formula N representing transition relation. Covert to BDD.
11/28/2018
Testing/Spring 98

74. Computation Tree Logicb b s1 s2 a b b a a b b b
Campos/Clarke/Marrero/Minea
page 97
State transition graph and corresponding computation tree
Paths in tree represent all possible computations
11/28/2018
Testing/Spring 98

75. Computation Tree Logic
Used to express properties that will be verified
Computation trees are derived from the state transition graphs
State transition graphs unwound into an infinite tree rooted at initial state
Campos/Clarke/Marrero/Minea
page 97
11/28/2018
Testing/Spring 98

76. Exercise
Design a finite state machine with start state s and final state t and prove that for all transitions from s to t any encounter of state y is preceded by encountering first state x.
Run your model and specification with the model checker on the CMU model checking home page.
11/28/2018
Testing/Spring 98