1. Testing Basics Dr. Eng. Amr T. Abdel-Hamid NETW 703 Winter 2010 Protocol Design

2. Dr. Amr Talaat Protocol Engineering Program Testing or, Program Simulation  What is software testing?  Running a program  In order to find faults  a.k.a. defects  a.k.a. errors  a.k.a. flaws  a.k.a. faults  a.k.a. BUGS  This enables us to  Increase our confidence that the program has high quality and low risk  Because we can never be sure we caught all bugs

3. Dr. Amr Talaat Protocol Engineering Testing  What isn’t software testing?  Purely static analysis: examining a program’s s ource code or binary in order to find bugs, but n ot executing the program  Verification & Validation: Based on building a m odel and checking it.

4. Dr. Amr Talaat Protocol Engineering Why Testing?  Ideally: we prove code correct, using formal mathematical techniques (with a computer, not chalk)  Extremely difficult: for some trivial programs (100 lines) and many small (5K lines) programs  Simply not practical to prove correctness in most cases – often not even for safety or mission critical code

5. Dr. Amr Talaat Protocol Engineering Unit, Integration, System Testing  Stages of testing  Unit testing is the first phase, done by developers of m odules  Integration testing combines unit tested modules and t ests how they interact  System testing tests a whole program to make sure it meets requirements  “Design testing” is testing prototypes or very abstract models befo re implementation – seldom mentioned, but when possible it can s ave time.  Model checking may be possible at this stage  Functional Testing tests a program from a “user’s” perspective – does it do what it should? Opposed to unit testing, which often proceeds from the perspective of other parts of the program

6. Dr. Amr Talaat Protocol Engineering Faults, Errors, and Failures  Fault: a static flaw in a program  What we usually think of as “a bug”  Error: a bad program state that results from a fault  Not every fault always produces an error  Failure: an observable incorrect behavior of a program as a result of an error  Not every error ever becomes visible

7. Dr. Amr Talaat Protocol Engineering To Expose a Fault with a Test  Reachability: the test much actually reach and execute th e location of the fault  Infection: the fault must actually corrupt the program stat e (produce an error)  Propagation: the error must persist and cause an incorre ct output – a failure

8. Dr. Amr Talaat Protocol Engineering Example 1 int findLast (int a[], int n, int x) { // Returns index of last element in a // equal to x, or -1 if no such. // n is length of a int i; for (i = n-1; i > 0; i--) { if (a[i] = x) return i; } return -1; } Find the fault

9. Dr. Amr Talaat Protocol Engineering Example 1 (cont.) int findLast (int a[], int n, int x) { // Returns index of last element in a // equal to x, or -1 if no such. // n is length of a int i; for (i = n-1; i > 0; i--) { if (a[i] = x) return i; } return -1; } Here’s a test case: a = {} n = 0 x = 2 Everything is ok, as we did not reach the bug

10. Dr. Amr Talaat Protocol Engineering Example 1 (cont.) int findLast (int a[], int n, int x) { // Returns index of last element in a // equal to x, or -1 if no such. // n is length of a int i; for (i = n-1; i > 0; i--) { if (a[i] = x) return i; } return -1; } Here’s another: a = {3, 9, 4} n = 3 x = 2 Reaches the fault Infects state with error But no failure

11. Dr. Amr Talaat Protocol Engineering Example 1 (cont.) int findLast (int a[], int n, int x) { // Returns index of last element in a // equal to x, or -1 if no such. // n is length of a int i; for (i = n-1; i > 0; i--) { if (a[i] = x) return i; } return -1; } And finally: a = {2, 9, 4} n = 3 x = 2 Reaches the fault Infects state with error And fails – returns -1 instead of 0

12. Dr. Amr Talaat Protocol Engineering Controllability and Observability  Goals for a test case:  Reach a fault  Produce an error  Make the error visible as a failure  In order to make this easy the program must be controllable and observable  Controllability:  How easy it is to drive the program where we want to go  Observability:  How easy it is to tell what the program is doing

13. Dr. Amr Talaat Protocol Engineering Black Box Testing  Black box testing  Treats a program or system as a black box  i.e. testing that does not look at source code or internal structure of the system  Send a program a stream of inputs, observe the outputs, decide if the system passed or failed the test

14. Dr. Amr Talaat Protocol Engineering Conformance Testing  How do we test finite state machines?  Let’s say we have  Known FSM A  Know all states and transitions  Unknown FSM B (same alphabet)  Can only perform experiments  How do we tell if A = B?  Known as the conformance testing or equivalence testing problem a c a b a d

15. Dr. Amr Talaat Protocol Engineering Conformance Testing  Exhaustive tests can be very expensive  In general, we cannot computationally afford to perform complete testing  We will always face the risk of missing errors  Even when we reduce our problem to the simplest mod el  The complexity of testing full equivalence to a refere nce model is simply too high  Exhaustion is exhausting

16. Dr. Amr Talaat Protocol Engineering Protocol Conformance Testing  To confirm if an implementation conforms to its standard  Issue1: preparation of conformance tests in coverage of all design aspects  Issue2: time required to run test should not be unacceptably long  Two main limitations  Controllability: the IUT cannot be directly put into a desired sta te, usually requiring several additional state transitions  Observability: prevents the external tester from directly observi ng the state of the IUT, which is critical for a test to detect errors  Formal conformance testing techniques based on FSM  Generate a set of input sequences that will force the FSM impleme ntation to undergo all specified transitions  Black box approach: only the outputs generated by the IUT (upon r eceipt of inputs) are observable to the external tester

17. Dr. Amr Talaat Protocol Engineering Fault Model for FSM  Output fault: the machine provides an output different from the one specified by the output function  Transfer fault: the machine enters a different state than that sp ecified by the transfer function  Transfer faults with additional states: number of states of the system is increased by the presence of faults, additional states i s used to model certain types of errors  Additional or missing transitions: one basic assumption is th at the FSM is deterministic and completely defined (fully specifi ed). So the faults occur when it turns out to be non-deterministi c and/or incompletely (partially) specified

18. Dr. Amr Talaat Protocol Engineering FIFO input queues Fault Models  Ordering fault: FIFO ordering is not preserved, or in case of multipl e input queues, some input event enters a wrong input queue.  Maximum length fault: the maximum length implemented is less th an the one specified, or if an input event gets lost while queue is no t overflow.  Flow control fault: errors of ordering or of loss occur, in case the n umber of submitted input events overflows the maximum queue len gth specified  Some other definitions & assumptions  Deterministic FSM: predictable behavior in a given state for a give n input  Strongly connected: for each state pair (si, sj) there is a transition path going from si to sj, I.e. each state can be reached from any ot her state  Fully specified: form each state it has a transition for each input sy mbol. Otherwise partially specified  Minimal: the number of states of M is less than or equal to the num ber of states of any equivalent machine

19. Dr. Amr Talaat Protocol Engineering 19/25 Transition Level Approach  The methods for protocol conformance test sequence generation  Produce a test sequence which checks the correctness of each tran sition of the FSM implementation  By no means exhaustive, I.e. no guarantee to exhibit correct behavi or given every possible input sequence.  The intent is to design a test sequence which guarantees “beyond a reasonable doubt”  Three basic steps for checking a transition:  Step 1: The FSM implementation is put into state s i (e.g. reset+tr ansfer)  Difficulty in realizing this is due to the limited controllability of the implementation  Step 2: Input a k is applied and the output is checked to verify tha t it is o l, as expected;  Step 3: The new state of the FSM implementation is checked to verify that it is sj, as expected  Difficulty in verifying this is due to the limited observability of t he implementation

20. Dr. Amr Talaat Protocol Engineering 20/25 T-Method: Transition Tour Met hod  For a given FSM S, a transition tour is a sequence which take s the FSM S from the initial state s 0, traverses every transition a t least once, and returns to the initial state s 0.  Straightforward and simple scheme  New state of the FSM is not checked  Detects all output errors  There is no guarantee that all transfer errors can be detected  The problem of generating a minimum-cost test sequence using the transition tour method is equivalent to the so-called “Chines e Postman” problem in graph theory  First studied by Chinese mathematician Kuan Mei-Ko ( 管梅谷 ) i n 1962

21. Dr. Amr Talaat Protocol Engineering T-Method EX1 Specificatio n Transition T our: b b b a a a y y x x y x Imp. 1 Imp. 2

22. Dr. Amr Talaat Protocol Engineering Ex. 2 Transition Tour Sequence: r, a, r, c, r, c, a, b, r, c, a, b, b, r, c, b, a, a r, c, a, c, b, a, c, a,

23. Dr. Amr Talaat Protocol Engineering Outline  Generate a Transition Tour Sequence for the specification below:

24. Dr. Amr Talaat Protocol Engineering D-Method: Distinguishing Sequences  A sequence of inputs is a distinguishing sequence (DS) for an F SM S, if the output sequence produced by the FSM S in respon se to the input sequence is distinct for each initial state  A DS is used as a state identification sequence  A DS is a useful tool for achieving Step 3 in checking the new s tate  Fault detection power  Detects all output errors  Detects all transfer errors  Two severe drawbacks  In practice, very few FSMs actually possess a DS  Even if an FSM does have a DS, the upper bound on the len gth of the DS will be too large to be useful in general  The requirement is too strong (leading to W- & U- methods…)

25. Dr. Amr Talaat Protocol Engineering D-Method Ex. 1

26. Dr. Amr Talaat Protocol Engineering Ex. 1  The test cases (  -sequences) a re:  state 1:  r,a,b,b  r,b,b,b  state 2:  r,b,a,b,b  r,b,b,b,b Test case structure correspon ding to 3-steps: preamble, tested transition, st ate identification Transfer sequence (Preamble) : the minimum cost (shortest path) input sequence taking F SM from one state to another. state 3:  r,a,a,b,b  r,a,b,b,b

27. Dr. Amr Talaat Protocol Engineering Ex. 3

28. Dr. Amr Talaat Protocol Engineering Ex. 4  Find DS and  -sequences

29. Dr. Amr Talaat Protocol Engineering W-Method: Characterizing Sequences  For FSMs that do not possess a DS, W-Method defines partial DS each of which distinguishes a state si from a subset of the re maining states instead of from every state of the FSM  The states of the FSM are first partitioned into blocks which can be distinguished by observing the sequence of outputs produce d by a sequence of inputs  Each block is subsequently partitioned into distinguishable sub- blocks, and so on, until each block consists of exactly one state  Main idea is to iteratively find a DS for each subset  To identify a state (for step 3)  Applying an input sequence  Returning to the state via a transfer sequence  Applying a second input sequence, and so on  The complete set of such input sequences for an FSM is called the characterizing set  Attach each CS in the set to the end of each transfer sequen ce

30. Dr. Amr Talaat Protocol Engineering Ex. 1  For the input sequence Acs1 = A,A, t he response is identical for states 2 a nd 3 (01), but is distinct from that for s tates 0(00), 1(11), 4(10).  Another input sequence Acs2 = B is d istinct for states 2(0) and 3(1).  Acs1 is required to identify states 0, 1, 4, and two input sequences  Acs1 and Acs2, along with appropriat e transfer sequences, are required to identify states 2 and 3. W = {AA, B}

31. Dr. Amr Talaat Protocol Engineering Ex2  Find W

32. Dr. Amr Talaat Protocol Engineering U-Method: Unique Input/Output Sequences  An I/O behavior that is not exhibited by any other state of the FSM  Answer the question of “is the implementation currently in state x?”  Advantages against DS & CS  Cost is never more than DS and in practice is usually much less (sh orter)  Nearly all FSMs have UIO sequences for each state  DS – same for all states; UIO sequence – normally different for eac h state  To check state s by using UIO sequence of s  Apply input part of UIO, compare output sequence with the expecte d one  If the same, then the FSM is in the state s; otherwise, not in the stat e s  If not in state s, no information about the identity of the actual state s’

33. Dr. Amr Talaat Protocol Engineering Ex. 1

34. Dr. Amr Talaat Protocol Engineering

35. Dr. Amr Talaat Protocol Engineering Analysis  Fault Testing Coverage  Fault coverage for D-, W-, and U-methods is better than of T-met hod  Fault coverage for D-, W-, and U-methods are the same  All of these four methods assume minimal, strongly connected and fu lly specified Mealy FSM model of protocol entities  On average, T-method produces the shortest test sequence, W-meth od the longest. D- and U- methods generate test sequence of compa rable lengths  T-method test sequences are able to detect output faults but not tran sition  D-, W-, and U-methods are capable of detecting all kinds of faults an d give the same performance.  U-method attracts more and more attentions and there are several a pproaches based on the basic idea with some improvements

36. Dr. Amr Talaat Protocol Engineering White Box Testing  Opens up the box!  (also known as glass box, clear box, or structural testin g)  Use source code (or other structure beyond the input/o utput spec.) to design test cases  Brings us to the idea of coverage

37. Dr. Amr Talaat Protocol Engineering Coverage Measures  In general, used to measure the quality of a test suite  Even in cases where the suite was designed for some o ther purpose (such as testing lots of different use scena rios)  Not always a very good measure of suite quality, but “b etter than nothing”  We “open the box” in white box testing partly in order to look at (and design tests to achieve) coverage

38. Dr. Amr Talaat Protocol Engineering Coverage  Literature of software testing is primarily concerned with va rious notions of coverage  Ammann and Offutt identify four basic kinds of coverage:  Graph coverage  Logic coverage  Input space partitioning  Syntax-based coverage

39. Dr. Amr Talaat Protocol Engineering Graph Coverage Cover all the nodes, edges, or paths of some graph relate d to the program Examples: Statement coverage Branch coverage Path coverage Data flow coverage Model-based testing coverage Many more – most common kind of coverage, by far

40. Dr. Amr Talaat Protocol Engineering Statement/Basic Block Coverage if (x < y) { y = 0; x = x + 1; } else { x = y; } 4 1 23 x >= yx < y x = y y = 0 x = x + 1 if (x < y) { y = 0; x = x + 1; } 3 1 2 x >= y x < y y = 0 x = x + 1 Statement coverage: Cover every node of these graphs Treat as one node because if one statement executes the other must also execute (code is a basic block)

41. Dr. Amr Talaat Protocol Engineering Branch Coverage if (x < y) { y = 0; x = x + 1; } else { x = y; } 4 1 23 x >= yx < y x = y y = 0 x = x + 1 if (x < y) { y = 0; x = x + 1; } 3 1 2 x >= y x < y y = 0 x = x + 1 Branch coverage vs. statement coverage: Same for if-then-else But consider this if-then structure. For branch coverage can’t just cover all nodes, but must cover all edges – get to node 3 both after 2 and without executing 2!

42. Dr. Amr Talaat Protocol Engineering Path Coverage if (x < y) { y = 0; x = x + 1; } else { x = y; } if (x < y) { y = 0; x = x + 1; } 4 1 23 x >= yx < y x = y y = 0 x = x + 1 6 4 5 x >= y x < y y = 0 x = x + 1 How many paths through this code are there? Need one test case for each to get path coverage To get statement and branch coverage, we only need two test cases: 1 2 4 5 6 and 1 3 4 6 Path coverage needs two more: 1 2 4 5 6 1 3 4 6 1 2 4 6 1 3 4 5 6 In general: exponential in the number of conditional branches!

43. Dr. Amr Talaat Protocol Engineering Logic Coverage if (((a>b) || G)) && (x < y)) { y = 0; x = x + 1; } 312 ((a = y) ((a>b) || G)) && (x < y) y = 0 x = x + 1 if (x < y) { y = 0; x = x + 1; } What if, instead of: we have: Now, branch coverage will guarantee that we cover all the edges, but does not guarantee we will do so for all the different logical reasons We want to test the logic of the guard of the if statement

44. Dr. Amr Talaat Protocol Engineering Testing “for” Coverage  Never seek to improve coverage just for the sake of increasing coverage  Coverage is not the goal  Finding failures that expose faults is the goal  No amount of coverage will prove that the progr am cannot fail “Program testing can be used to show the presence of bugs, but never to show their absence!” – E. Dijkstra, Notes On Structured Programming