1. Dr. Eng. Amr T. Abdel-Hamid
NETW 703
Protocol Design
Testing Basics
Dr. Eng. Amr T. Abdel-Hamid
Spring 2017

2. Program Testing or, Program Simulation
What is software testing?
Running a program
In order to find faults
a.k.a. defects
a.k.a. errors
a.k.a. flaws
a.k.a. faults
a.k.a. BUGS
This enables us to
Increase our confidence that the program has high quality and low risk
Because we can never be sure we caught all bugs

3. What isn’t software testing?
Purely static analysis: examining a program’s source code or binary in order to find bugs, but not executing the program
Verification & Validation: Based on building a model and checking it.

4. Why Testing?
Ideally: we prove code correct, using formal mathematical techniques (with a computer, not chalk)
Extremely difficult: for some trivial programs (100 lines) and many small (5K lines) programs
Simply not practical to prove correctness in most cases – often not even for safety or mission critical code

5. Unit, Integration, System Testing
Stages of testing
Unit testing is the first phase, done by developers of modules
Integration testing combines unit tested modules and tests how they interact
System testing tests a whole program to make sure it meets requirements
“Design testing” is testing prototypes or very abstract models before implementation – seldom mentioned, but when possible it can save time.
Model checking may be possible at this stage
Functional Testing tests a program from a “user’s” perspective – does it do what it should? Opposed to unit testing, which often proceeds from the perspective of other parts of the program

6. Faults, Errors, and Failures
Fault: a static flaw in a program
What we usually think of as “a bug”
Error: a bad program state that results from a fault
Not every fault always produces an error
Failure: an observable incorrect behavior of a program as a result of an error
Not every error ever becomes visible

7. To Expose a Fault with a Test
Reachability: the test much actually reach and execute the location of the fault
Infection: the fault must actually corrupt the program state (produce an error)
Propagation: the error must persist and cause an incorrect output – a failure

8. Example 1 Find the fault int findLast (int a[], int n, int x) {
// Returns index of last element in a
// equal to x, or -1 if no such.
// n is length of a
int i;
for (i = n-1; i > 0; i--) {
if (a[i] = x)
return i; }
return -1;
Find the fault

9. Example 1 (cont.) Here’s a test case: a = {} n = 0 x = 2
int findLast (int a[], int n, int x) {
// Returns index of last element in a
// equal to x, or -1 if no such.
// n is length of a
int i;
for (i = n-1; i > 0; i--) {
if (a[i] = x)
return i; }
return -1;
Here’s a test case:
a = {} n = 0 x = 2
Everything is ok, as
we did not reach the
bug

10. Example 1 (cont.) Here’s another: a = {3, 9, 4} n = 3 x = 2
int findLast (int a[], int n, int x) {
// Returns index of last element in a
// equal to x, or -1 if no such.
// n is length of a
int i;
for (i = n-1; i > 0; i--) {
if (a[i] = x)
return i; }
return -1;
Here’s another:
a = {3, 9, 4} n = 3 x = 2
Reaches the fault Infects state with error But no failure

11. Example 1 (cont.) And finally: a = {2, 9, 4} n = 3 x = 2
int findLast (int a[], int n, int x) {
// Returns index of last element in a
// equal to x, or -1 if no such.
// n is length of a
int i;
for (i = n-1; i > 0; i--) {
if (a[i] = x)
return i; }
return -1;
And finally:
a = {2, 9, 4} n = 3 x = 2
Reaches the fault Infects state with error And fails – returns -1
instead of 0

12. Controllability and Observability
Goals for a test case:
Reach a fault
Produce an error
Make the error visible as a failure
In order to make this easy the program must be controllable and observable
Controllability:
How easy it is to drive the program where we want to go
Observability:
How easy it is to tell what the program is doing

13. Black Box Testing Black box testing
Treats a program or system as a black box
i.e. testing that does not look at source code or internal structure of the system
Send a program a stream of inputs, observe the outputs, decide if the system passed or failed the test

14. How do we test finite state machines? Let’s say we have
Conformance Testing
How do we test finite state machines?
Let’s say we have
Known FSM A
Know all states and transitions
Unknown FSM B (same alphabet)
Can only perform experiments
How do we tell if A = B?
Known as the conformance testing or equivalence testing problem d a a b a c

15. Conformance Testing Exhaustive tests can be very expensive
In general, we cannot computationally afford to perform complete testing
We will always face the risk of missing errors
Even when we reduce our problem to the simplest model
The complexity of testing full equivalence to a reference model is simply too high
Exhaustion is exhausting

16. Protocol Conformance Testing
To confirm if an implementation conforms to its standard
Issue1: preparation of conformance tests in coverage of all design aspects
Issue2: time required to run test should not be unacceptably long
Two main limitations
Controllability: the IUT cannot be directly put into a desired state, usually requiring several additional state transitions
Observability: prevents the external tester from directly observing the state of the IUT, which is critical for a test to detect errors
Formal conformance testing techniques based on FSM
Generate a set of input sequences that will force the FSM implementation to undergo all specified transitions
Black box approach: only the outputs generated by the IUT (upon receipt of inputs) are observable to the external tester

17. Fault Model for FSM
Output fault: the machine provides an output different from the one specified by the output function
Transfer fault: the machine enters a different state than that specified by the transfer function
Transfer faults with additional states: number of states of the system is increased by the presence of faults, additional states is used to model certain types of errors
Additional or missing transitions: one basic assumption is that the FSM is deterministic and completely defined (fully specified). So the faults occur when it turns out to be non-deterministic and/or incompletely (partially) specified

18. FIFO input queues Fault Models
Ordering fault: FIFO ordering is not preserved, or in case of multiple input queues, some input event enters a wrong input queue.
Maximum length fault: the maximum length implemented is less than the one specified, or if an input event gets lost while queue is not overflow.
Flow control fault: errors of ordering or of loss occur, in case the number of submitted input events overflows the maximum queue length specified
Some other definitions & assumptions
Deterministic FSM: predictable behavior in a given state for a given input
Strongly connected: for each state pair (si, sj) there is a transition path going from si to sj , I.e. each state can be reached from any other state
Fully specified: form each state it has a transition for each input symbol. Otherwise partially specified
Minimal: the number of states of M is less than or equal to the number of states of any equivalent machine

19. Transition Level Approach
The methods for protocol conformance test sequence generation
Produce a test sequence which checks the correctness of each transition of the FSM implementation
By no means exhaustive, I.e. no guarantee to exhibit correct behavior given every possible input sequence.
The intent is to design a test sequence which guarantees “beyond a reasonable doubt”
Three basic steps for checking a transition:
Step 1: The FSM implementation is put into state si (e.g. reset+transfer)
Difficulty in realizing this is due to the limited controllability of the implementation
Step 2: Input ak is applied and the output is checked to verify that it is ol, as expected;
Step 3: The new state of the FSM implementation is checked to verify that it is sj, as expected
Difficulty in verifying this is due to the limited observability of the implementation
19/25

20. T-Method: Transition Tour Method
For a given FSM S, a transition tour is a sequence which takes the FSM S from the initial state s0, traverses every transition at least once, and returns to the initial state s0.
Straightforward and simple scheme
New state of the FSM is not checked
Detects all output errors
There is no guarantee that all transfer errors can be detected
The problem of generating a minimum-cost test sequence using the transition tour method is equivalent to the so-called “Chinese Postman” problem in graph theory
First studied by Chinese mathematician Kuan Mei-Ko (管梅谷) in 1962
20/25

21. T-Method EX1 Imp. 1 Specification Transition Tour: b b b a a a
y y x x y x
Imp. 2

22. Ex. 2 Transition Tour Sequence: r, a, r, c, r, c, a, b, r, c, a, b, b,
r, c, b, a, a
r, c, a, c, b, a, c, a,

23. Outline
Generate a Transition Tour Sequence for the specification below:

24. D-Method: Distinguishing Sequences
A sequence of inputs is a distinguishing sequence (DS) for an FSM S, if the output sequence produced by the FSM S in response to the input sequence is distinct for each initial state
A DS is used as a state identification sequence
A DS is a useful tool for achieving Step 3 in checking the new state
Fault detection power
Detects all output errors
Detects all transfer errors
Two severe drawbacks
In practice, very few FSMs actually possess a DS
Even if an FSM does have a DS, the upper bound on the length of the DS will be too large to be useful in general
The requirement is too strong (leading to W- & U- methods…)

25. D-Method Ex. 1

26. Ex. 1
The test cases ( -sequences) are:
state 1:
r,a,b,b
r,b,b,b
state 2:
r,b,a,b,b
r,b,b,b,b
state 3:
r,a,a,b,b
r,a,b,b,b
Test case structure corresponding to 3-steps:
preamble, tested transition, state identification
Transfer sequence (Preamble): the minimum cost (shortest path) input sequence taking FSM from one state to another.

27. Ex. 3

28. Ex. 4
Find DS and -sequences

29. W-Method: Characterizing Sequences
For FSMs that do not possess a DS, W-Method defines partial DS each of which distinguishes a state si from a subset of the remaining states instead of from every state of the FSM
The states of the FSM are first partitioned into blocks which can be distinguished by observing the sequence of outputs produced by a sequence of inputs
Each block is subsequently partitioned into distinguishable sub-blocks, and so on, until each block consists of exactly one state
Main idea is to iteratively find a DS for each subset
To identify a state (for step 3)
Applying an input sequence
Returning to the state via a transfer sequence
Applying a second input sequence, and so on
The complete set of such input sequences for an FSM is called the characterizing set
Attach each CS in the set to the end of each transfer sequence

30. Ex. 1
For the input sequence Acs1 = A,A, the response is identical for states 2 and 3 (01), but is distinct from that for states 0(00), 1(11), 4(10).
Another input sequence Acs2 = B is distinct for states 2(0) and 3(1).
Acs1 is required to identify states 0, 1, 4, and two input sequences
Acs1 and Acs2, along with appropriate transfer sequences, are required to identify states 2 and 3.
W = {AA, B}

31. Ex2
Find W

32. U-Method: Unique Input/Output Sequences
An I/O behavior that is not exhibited by any other state of the FSM
Answer the question of “is the implementation currently in state x?”
Advantages against DS & CS
Cost is never more than DS and in practice is usually much less (shorter)
Nearly all FSMs have UIO sequences for each state
DS – same for all states; UIO sequence – normally different for each state
To check state s by using UIO sequence of s
Apply input part of UIO, compare output sequence with the expected one
If the same, then the FSM is in the state s; otherwise, not in the state s
If not in state s, no information about the identity of the actual state s’

33. Ex. 1

35. Analysis Fault Testing Coverage
Fault coverage for D-, W-, and U-methods is better than of T-method
Fault coverage for D-, W-, and U-methods are the same
All of these four methods assume minimal, strongly connected and fully specified Mealy FSM model of protocol entities
On average, T-method produces the shortest test sequence, W-method the longest. D- and U- methods generate test sequence of comparable lengths
T-method test sequences are able to detect output faults but not transition
D-, W-, and U-methods are capable of detecting all kinds of faults and give the same performance.
U-method attracts more and more attentions and there are several approaches based on the basic idea with some improvements

36. White Box Testing Opens up the box!
(also known as glass box, clear box, or structural testing)
Use source code (or other structure beyond the input/output spec.) to design test cases
Brings us to the idea of
coverage

37. Coverage Measures
In general, used to measure the quality of a test suite
Even in cases where the suite was designed for some other purpose (such as testing lots of different use scenarios)
Not always a very good measure of suite quality, but “better than nothing”
We “open the box” in white box testing partly in order to look at (and design tests to achieve) coverage

38. Coverage
Literature of software testing is primarily concerned with various notions of coverage
Ammann and Offutt identify four basic kinds of coverage:
Graph coverage
Logic coverage
Input space partitioning
Syntax-based coverage

39. Graph Coverage
Cover all the nodes, edges, or paths of some graph related to the program
Examples:
Statement coverage
Branch coverage
Path coverage
Data flow coverage
Model-based testing coverage
Many more – most common kind of
coverage, by far

40. Statement/Basic Block Coverage
if (x < y) {
y = 0;
x = x + 1; }
else
x = y;
Statement coverage: Cover every node of these graphs 4 1 2 3
x >= y
x < y
x = y
y = 0
x = x + 1 3 1 2
x >= y
x < y
y = 0
x = x + 1
if (x < y) {
y = 0;
x = x + 1; }
Treat as one node because if one statement executes the other must also execute (code is a basic block)

41. Branch Coverage if (x < y){
y = 0;
x = x + 1; }
else
x = y;
Branch coverage vs. statement coverage: Same for if-then-else 4 1 2 3
x >= y
x < y
x = y
y = 0
x = x + 1 3 1 2
x >= y
x < y
y = 0
x = x + 1
if (x < y) {
y = 0;
x = x + 1; }
But consider this if-then structure. For branch coverage can’t just cover all nodes, but must cover all edges – get to node 3 both after 2 and without executing 2!

42. Path Coverage
How many paths through this code are there? Need one test case for each to get path coverage
if (x < y) {
y = 0;
x = x + 1; }
else
x = y; 4 1 2 3
x >= y
x < y
x = y
y = 0
x = x + 1
To get statement and branch coverage, we only need two test cases:
and 6 4 5
x >= y
x < y
y = 0
x = x + 1
Path coverage needs two more:
In general: exponential in the number of conditional branches!

43. Logic Coverage What if, instead of: ((a <= b) && !G) || (x >= y)
if (x < y) {
y = 0;
x = x + 1; } 1
((a>b) || G)) && (x < y)
y = 0
x = x + 1 2
((a <= b) && !G) || (x >= y) 3
we have:
if (((a>b) || G)) && (x < y)) {
y = 0;
x = x + 1; }
Now, branch coverage will guarantee that we cover all the edges, but does not guarantee we will do so for all the different logical reasons
We want to test the logic of the guard of the if statement

44. Testing “for” Coverage
Never seek to improve coverage just for the sake of increasing coverage
Coverage is not the goal
Finding failures that expose faults is the goal
No amount of coverage will prove that the program cannot fail
“Program testing can be used to show the presence of bugs, but never to show their absence!” – E. Dijkstra, Notes On Structured Programming