1. Testing Interfaces in Multi-Program Systems ECEN5043 Software Engineering of Multiprogram Systems University of Colorado, Boulder

2. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 2 Topics Computational Models Implications of nondeterminism Model Testing –Class Testing extended –UML for state diagrams, concurrency, components –Timing and Time-outs -- UML and testing –Threads & synchronization

3. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 3 Topics (cont. 2) Dynamic Testing –Definition of component –Component Interactions Identifying -- nonprimitive classes Specifying –Testing Interactions Collections Collaborations One-way pass along Interaction between Testing & Design

4. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 4 Topics (cont. 3) Sampling Test Cases -- OATS revisited Testing Off-The-Shelf Components Protocol Testing Test Patterns -- McGregor & Sykes; Binder Testing Interactions at the System Level Testing Exceptions

5. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 5 Computational Models Sequential processing -- default –Model testing: straightforward inspections –Dynamic testing: see course #1 Concurrency –Multiple things that appear to be happening at the same time. –Central issue -- nondeterminism of interleaved statement execution –Preliminary thoughts on implications for testing?

6. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 6 Computational Models (cont.) Distributed –Characterized by: multiple processes to support a flexible architecture in which the number of participating objects can change –Can be distributed across multiple processes on the same machine -- we’re postponing this to course #3 when we focus on distributed architectures (on one or more processors).

7. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 7 Implications of Nondeterminism Nondeterminism –Difficult (impossible) to replicate a test run when the software contains multiple threads or concurrent processes –Ordering determined by the scheduler –Changes in programs NOT associated with the system under test can affect the order in which threads of the system under test are executed –If a failure is encountered and the defect is isolated and repaired, the fact that the program passes test execution is... ?

8. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 8 Nondeterminism and Static Testing More thorough model testing at the class level –Design (and review) should investigate whether there is a need an appropriate provision for synchronization During model testing, –Ensure test cases require execution of critical sections –Consider impact of different orderings with respect to starvation mutual exclusion deadlock synchronization

9. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 9 But how?? Three kinds of model testing 1.Is the code for one process correct by itself (are the sequential statements correct) assuming its interactions are correct 2.If a propos, inspect with multiple copies of same process in parallel 3.For shared data, different processes, have copies of each For the critical sections –“What would starvation require in this context?” Is it possible? –Similarly for mutual exclusion, deadlock, and necessary synchronization

10. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 10 Testing Multi-program Systems Methods -- review the class testing described in previous course –state transition coverage –code coverage –pre & post condition coverage Extend notion of state diagram to that of interacting components (programs/processes) Extend state transition coverage similarly –Recall notion of state transition pairs coverage

11. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 11 Sample State Transition Diagram - Gomaa

12. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 12 Time-Outs for Testing It’s possible that a process/program will make a request of another process that never responds The requester must be able to detect that and abandon the request Design should explicitly define correct behavior for –when the request is answered –and also when it is not –the behavior may be very different in those two cases. Model testing should verify these Dynamic testing should also.

13. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 13 Expressing Time on Interaction Diagrams Interaction diagrams are collaboration diagrams and sequence diagrams time expression -- resolves to a relative or absolute value of time when evaluated timing mark -- time-related name or label on a message timing constraint -- condition that must be satisfied with regard to time. –usually contains a time expression –may also contain a timing mark

14. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 14 Sample from Collaboration Diagram Reminder: A sequence or collaboration diagram focuses on the time ordering of the messages that go back and forth between objects. ;Login Page ;Account v:validateLogin(userID, password) {v.executionTime < 3 sec} v is the timing mark. The timing mark makes it easier to refer to that message elsewhere on the sequence diagram. The expression between { } is the constraint.

15. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 15 Types of Timing balking message -- synchronous message but the sending object gives up on the message if the receiving object is not ready to accept it. –timing constraint {wait = 0} timeout message -- synchronous message but the sender waits only for a specified period for the receiver to get ready to accept the message –timing constraint {wait = 50 ms}

16. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 16 Threads & Concurrent Processes Working definition: a thread is a unit of computation that can be scheduled During design and design review, should explicitly consider the number of threads –Increasing it Can simplify certain algorithms/techniques Increase the risk of sequencing problems –Reducing it Reduces sequencing problems Makes software more rigid and maybe inefficient

17. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 17 Concurrent Processes, Synchronization, & Notation During design inspections, verify presence of the helpful clauses in the UML state diagram Some languages provide a language keyword to automatically add the mechanism to prevent simultaneous access -- be sure it works as expected –In Java, the keyword synchronized is used on the signature of a method to specify the need for a synchronization mechanism (see later slide) C++ requires explicit structures that each individual developer must construct In C++, creation of an instance of a monitor object indicates the location at which synchronization is needed

18. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 18 Java sample public synchronized int get( ) { while (available == false) { try { wait ( ); } catch (InterruptedException e) { } } available = false; notifyAll( ); return contents; } public synchronized void put (int value) { while (available == true) { try { wait ( ); } catch (InterruptedException e) { } } contents = value; available = true; notify ( ); } Example from The Java Tutorial, Addison Wesley, Lesson 15, Threads of Control, pp. 285-325.

19. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 19 Dynamic Testing Concerns

20. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 20 Collections & Collaborators Collections: Classes that maintain associations with instances of other classes but never actually interact with those instances Hold the objects they are handed and return them in specific orders or find them based on specific criteria Test with techniques for primitive classes Collaborators: may be addressed directly (name) or by a pointer or a reference dynamic type of object may be different from static type associated Pointers & references are polymorphic Bound to an instance of any number of classes pre & post conditions refer to states or values of collaborating objects

21. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 21 Testing Collection Classes Test driver –create instances passed as parameters in messages to collection being tested –Ensure those instances are correctly built into and removed from the collection. Address capacity limitations –Test behavior under circumstances where it cannot allocate memory to add the new item to itself If defensive, use negative tests also –If the collection class uses a structure (array, etc.), include test cases to fill it completely and attempt to add one more. Test sequences of operations

22. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 22 Collaborators at Program Interfaces Collaborating class –Post condition in a class’ interface refers to the state of an instance and/or specifies that some attribute of that object is used or modified We assume operations are specified by pre-, post condition and class invariants Need to know if defensive design or design by contract –Changes how senders and receivers interact

23. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 23 Nondeterminism & Dynamic Testing There is no perfect way to do this but the following will help: Begin dynamic testing with as “clean” a machine as possible –document the environment –identify apps that must run while items-under-test are running –add basic set of typical unrelated apps For each test case (set of programs/ thread-generating processes) –Describe any modifications made to this “standard environment” –Note the order in which processes are started –Include a debugger that allows verification of order in which threads are created, executed, and deleted

24. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 24 Orthogonal array testing applied to OO Orthogonal array testing is a sampling technique to limit the explosion of test cases Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions An orthogonal array is an array of values in which each column represents a factor. A factor is a variable in an experiment; in our case, a specific class family in the software sys Each variable takes on a certain set of values called levels; in our case, a specific class Each cell has an instance of the class, that is, a specific state of an object

25. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 25 OATS – orthogonal array testing system The factors (columns) are combined pair-wise rather than representing all possible combinations of the levels for the factors. Suppose we have three factors – A, B, C – and each has three levels – 1, 2, and 3 How many possible combinations of these values are there? How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice? factor=class family; level=class; cell=instance

26. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 26 arrayfactor: class family factor: class family level: specific class an instance of the specific class level: specific class an instance of the specific class level: specific class an instance of the specific class a specific state

27. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 27 Excerpt from Fig. 6.16, M&S, 2 1 x 3 7 123456 1111111 2112222 3113333 4121122 5122233 6123311 7131213 8132321 9133132 10211332 11212113

28. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 28 OATS – uses a balanced design Every level of a factor appears exactly the same number of times as every other level of that factor Think of the rows of a table as test cases (In example, 18 of the possible 27 are not being conducted. This is a systematic way of reducing the number of test cases. If we decide to add more later, we know which ones have not been run. Also logical – most errors are between pairs of objects rather than among several objects

29. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 29 OATS adequacy criteria Ability to vary how completely the software under test is covered Exhaustive – all combinations of all factors Minimal – only interactions between the base classes from each hierarchy are tested Random – tester haphazardly selects cases from several of the classes; not statistically random Representative – uniform sample ensures every class is tested to some level Weighted representative – adds cases to the representative approach based on relative importance or risk associated with the class

30. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 30 Testing off-the-shelf components – Investigate the limits of whatever specification for it that is available –Create one if none is available –Developers will need it in order to properly use the component Begin with extreme boundary values -- even incorrect Do stress tests next If worth continuing –Treat as a class interacting with rest of your product –There is usually a main class that presents itself to the user or other products. Base tests on that class.

31. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 31 Protocol testing – Protocols: –Sequences of method invocations* –Identify them by combining a method whose post condition enables the precondition of another method –Easier to see from a state diagram than from written pre- and post conditions. Investigates whether the implementation of a class satisfies its specification. Abstract to state diagram with interacting components

32. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 32 Protocol Interaction Test Suite Consider as the “class under test” the component’s class that presents an interface to the other components Each protocol represents a life cycle for objects from that class in combination with instances of classes in the interacting components. –Begin with initial states of a given two objects as denoted in the state diagrams of the two corresponding classes –A sequence of states for each object –Ending with the terminal states –A test case takes the two objects through one complete sequence of methods

33. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 33 Protocol between two components The sequence of messages exchanged between them. A component should advertise –the services it provides (through its interface) –what it requires (through its preconditions) Should also describe the sequence in which these interactions are expected One set of test cases defined from the protocols should exercise the integration of two components through the complete protocol.

34. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 34 Example of protocol between two components Note: can be described using sequence diagram “user” component sends asynchronous message to the “hardware” component to start the action Eventually hw that services multiple clients sends an asynchronous message back that the action has been started “User” wants hw to stop -- similar exchange No notational convention but these four occur as a group although 2 for one component and 2 for the other –Tester class plays role of the other component in a protocol –Java -- Tester object can pass itself as a parameter to the component under test and then invoke the services specified in the protocol

35. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 35 Example: Timer Protocol – diagram on next slide Trace through the state diagram to create a life cycle test. Create the object Send one or more attach (...) messages Followed by enable ( ), disable ( ), and delete ( ) This is a simple example. A life cycle test case provides a test of the object in the ways that it will interact with its client objects When the object is one that interfaces with client objects in another program, this provides a realistic test sequence.

36. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 36 McGregor & Sykes Timer DisabledEnabled Timing notify( ) attach(TimerObserver) enable( ) disable( ) detach(TimerObserver) isEnabled( ) limit( )size( ) notify()/observers ->forAll(notify()) delete ( )

37. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 37 Test Patterns – Test patterns are design patterns for test software. –The pattern description explains the context within which the pattern should be considered –Provides a set of forces that guide a trade-off analysis –Explains how to construct the objects. When developer uses a certain design pattern, tester can use a certain test pattern to structure the test code. Binder gives several examples in some detail.

38. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 38 Test Design Patterns Presented in Binder Test Design Patterns are for designing test suites Robert Binder presents test design patterns for over 30 architectures

39. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 39 Exceptions An alternative route for returning from a method that does not necessarily move to the next statement after the method is invoked. –The exceptional return value is an object and can be arbitrarily complex –Where the exception is thrown varies based on the depth of the aggregation hierarchy

40. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 40 Testing Exceptions -- class Class level testing: verify that each of the methods does throw the exceptions it claims to throw in the appropriate circumstance. –Coverage criteria – class throws every exception contained in the specifications of the methods –At least one test case per exception –Handled in normal class testing because each potential throw should be a postcondition clause

41. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 41 Role of test driver of exceptions -- class Test driver sets up conditions to create exception event. –Provides try block inside that is a stimulus that invokes the method that throws the specific exception. –Exception caught by test driver verifies correct exception Exceptions are objects and belong to classes –catch statements can use typing system to verify that it is of the correct typeS

42. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 42 Testing Exceptions – integration Integration level -- test they are being –thrown at correct time –caught at the correct place During inspection of system-level design model –every user-defined exception instantiated should be traced to an object that will catch the exception using the sequence diagram for the scenario that throws the exception Exception handling can be model tested early

43. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 43 Testing interactions at system level Components can be so complex it’s easier to test them in the context of the application itself Which interactions are tested at system level? –Those that can also be verified at the system level –That is, can see the direct results of the tests –There must be a direct relationship between user interface and the ability to view test results By these criteria, many multi-program interactions must be tested as class vs. program or class vs. class because program vs. program interactions may not be visible at the system level.

44. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 44 Books McGregor & Sykes, A Practical Guide to Testing Object-Oriented Software, Addison Wesley, 2001. ISBN 0-201-32564-0 Scott, Kendall, UML Explained, Addison Wesley, 2001. ISBN 0-201-72182-1 Binder, Robert V., Testing Object-Oriented Systems: Models, Patterns, and Tools, Addison Wesley 2000. ISBN 0-201-80938-9

45. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 45 Orthogonal array testing applied to OO Orthogonal array testing is a sampling technique to limit the explosion of test cases Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions An orthogonal array is an array of values in which each column represents a factor.

46. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 46 Orthogonal array testing applied to OO An orthogonal array is an array of values in which each column represents a factor. A factor is –a variable in an experiment; in our case, a specific class family in the software system –Or states of that class family Each variable takes on a certain set of values called levels (rows); in our case, specific classes or states of those classes Each cell is an instance of the class, that is, a specific object or a particular state

47. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 47 OATS – orthogonal array testing system The factors (columns) are combined pair-wise rather than representing all possible combinations of the levels for the factors. Suppose we have three factors – A, B, C – and each has three levels How many possible combinations of these values are there? 3 x 3 x 3 How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice? factor=class family; level=class; cell=instance

48. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 48 How many pair-wise combinations? ABC 1113 2122 3131 4212 5221 6233 7311 8323 9332

49. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 49 Mapping the terms of OATS to OO sw arrayfactor: class family factor: class family state level: specific class an instance of the specific class A state of that instance level: specific class an instance of the specific class A state of that instance level: specific class an instance of the specific class A state of that instance A state of that nstance

50. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 50 OATS – uses a balanced design Every level of a factor appears exactly the same number of times as every other level of that factor Think of the rows of a table as test cases This is a systematic way of reducing the number of test cases. If we decide to add more later, we know which ones have not been run. Also logical – most errors are between pairs of objects rather than among several objects

51. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 51 Look at elevator example (M&S, p. 230) Want to test interactions between elevator controller, elevator, and wall button-panel Suppose: –Elevator can be basic, hi-speed, or express –Elevator Controller can control basic, hi-speed, or express –Wall button-panel can be one-button panel or two-button panel Elevator states: Moving to a floor, stopped, idle Elevator Controller states: idle, scheduling, dispatching Wall button-panel states?

52. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 52 Steps from M & S, p. 230-232 1.Identify all factors (class families in this case) 2.Determine levels for each factor (states) 3.Locate a standard orthogonal array that fits the problem 1.We have 2 class families that each have 3 levels and three states 2.We have one class family that has _a_ levels and _b_ states  That’s how many possible test cases?  3 x 3 x 3 x 3 x a x b

53. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 53 Excerpt from Fig. 6.16, M&S, 2 1 x 3 7 123456 1111111 2112222 3113333 4121122 5122233 6123311 7131213 8132321 9133132 10211332 11212113

54. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 54 Need a way to map Fig 6.16 to this ElevatorElevator State Elev CtrlElev Ctrl State Wall Button Panl Wall Button PnlStte 1 2 3 4 5 6 7 8

55. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 55 Mapping classes and states The class family with two levels (Wall Button Panel) maps to Column 1 Its states map to Column 2 Column 3 can have a class family with three levels; either elevator or elevator controller will do; –Map Elevator Controller family to column 3 –Map its states to column 4 What’s left? –Map Elevator family to column 5 –Map its states to column 6

56. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 56 Mapping levels – Wall Button Panel For Wall Button Panel –1 = 1-button panel –2 = 2-button panel One-button Panel states (column 2) –1 = off –2 = on For two-button panel (column 2) –1 = both off –2 = 1 requesting –3 = both requesting

57. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 57 Mapping levels – Elevator Controller Column 3 – elevator controller class family –1 = basic –2 = hi-speed –3 = express Column 4 – elevator controller class states –1 = idle –2 = scheduling –3 = dispatching

58. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 58 Mapping levels -- Elevator Column 5 = elevator class family –1 = basic –2 = hi-speed –3 = express Column 6 = elevator class states –1 = Moving to a floor –2 = stopped –3 = idle

59. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 59 Label Fig 6.16 with headers that “fit” Wall- Button Panel Wall- Button Pnl stte Elev Ctrlr Elev Ctrlr State ElevatorElevator State 1111111 2112222 3113333 4121122 5122233 6123311 7131213 8132321 9133132

60. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 60 When mapped to all of Fig 6.16 Each row becomes a test case Decode the level numbers for a row in the array back to the individual lists for each factor In Fig 6.16, there are actually two more columns. –We can ignore them because we don’t need them. –Or we may decide we can include one more class family and its states in the future

61. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 61 What about mismatches? When there is a difference in the number of levels, we can use a column that matches or exceeds the maximum What about the Wall-Button Panel at level 1 where there is only one button? –That only has two states but there can be a 3 in column 2 associated with the 1 in the column 1 –We interpret the 3 as if it were either 1 or 2. What do you want to emphasize in the testing? Arbitrary assignment High frequency High risk

62. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 62 Other what-ifs? See M & S, p. 232

63. April 17, 2007ECEN5043 University of Colorado Testing Interfaces 63 OATS adequacy criteria Ability to vary how completely the software under test is covered Exhaustive – all combinations of all factors Minimal – only interactions between the base classes from each hierarchy are tested Random – tester haphazardly selects cases from several of the classes; not statistically random Representative – uniform sample ensures every class is tested to some level Weighted representative – adds cases to the representative approach based on relative importance or risk associated with the class