1. Class Testing ECEN5033 University of Colorado

2. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 2 Overview Class Testing Testing Interactions between objects Testing Class Hierarchies

3. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 3 Basics – How to test a single class Class Testing How to Test a Class Aspects of Class Testing How to Construct Test Cases When is a class test suite adequate? How to Construct a Test Driver Testing Interactions between objects Testing Class Hierarchies

4. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 4 Definition of class testing Verifying implementation of a class = verifying the specification for that class If so, each of the instances should behave properly. Assumption: The class in question has a complete and correct specification that has been tested earlier in the context of models Spec is expressed in natural language or as a state transition diagram

5. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 5 Ways to test a class Code can be tested effectively by inspection (preferable when construction of a test driver is too difficult) execution of test cases (lends itself to easy regression testing later on) Remember: When you test a class, you are really: creating instances of that class and testing the behavior of those instances.

6. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 6 Aspects of Class Testing Decide test independently test as a component of a larger part of the system How decide – combination of the following: Role of the class in the system – degree of risk Complexity of the class Amount of effort associated with developing a test driver Sometimes, needs so many collaborators, makes more sense to test in a cluster

7. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 7 Who Tests? How much? Who – class usually tested by its developer Understands class’ spec Familiar with the code Test driver can be used by the developer to debug code while writing it Perpetuates misunderstanding – needs inspection at model stage to head that off What – ensure that the code for a class exactly meets the requirements – no more, no less

8. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 8 When are tests written? When – A test plan that identifies test cases should be developed soon after a class is fully specified and ready for coding. Especially if developer is the class tester Why? (What’s the danger in developer alone writing and reviewing test cases for the class?) When again? Iterative development – the driver and the test cases will be available to supplement or change as the class is enhanced or modified

9. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 9 How can you test “just a class”? How – Create a test driver that creates instances of the class sets up suitable environment around those instances to run a test case sends one or more messages to an instance as specified by a test case checks the outcome based on a reply value, changes to the instance, or parameters to the message deletes any instances it creates if responsible for storage allocation

10. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 10 How do we test static data in a class? How – continued: Static data members or operations testing required they belong to the class itself rather than to each instance of the class the class can be treated as an object If the behavior of the instances of a class is based on the values of class-level attributes test cases for testing the class-level attributes must be considered as an extension of the state of the instances

11. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 11 How MUCH testing is done at this level? How much – Adequacy of testing measured in terms of how much of the spec has been tested how much of the implementation has been tested Want to test operations and state transitions in many combinations Objects maintain state. “State” affects the meaning of operations.

12. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 12 Constructing test cases - identification Identification of test cases Should be made from the spec that has been reviewed rather than from the implementation which may embody developer’s misunderstandings Best: develop from spec and augment to test boundaries introduced by the implementation If no spec exists: create one from the code and verify it with the developer

13. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 13 Constructing test cases – identification 2 Identify requirements for test cases for all possible combinations of situations in which a precondition can hold post conditions can be achieved Create test cases for those requirements specific input values – typical and boundary determine correct outputs eliminate conditions that are not meaningful Add test cases to show what happens when a precondition is violated

14. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 14 Constructing Test Cases – from STD’s State Transition Diagrams They show behavior associated with instances of a class Each transition represents a requirement for one or more test cases

15. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 15 Constructing test cases from STD’s 2 Suppose 6 transitions between states Plus 1 constructor and 2 destructors That makes 9 requirements Select representative values Select boundary values on each side of a transition If the transition is guarded, select boundary values for the guard condition Boundary values are based on the range of attribute values associated with a state

16. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 16 Top-level Statechart for Elevator Control

17. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 17 Adequacy of a Class Test Suite Ideally – exhaustively test each class Practically – impossible or too hard Worth it to exhaustively test some classes with high risk Measures of adequacy to increase confidence that we have tested enough state-based coverage constraint-based coverage code-based coverage

18. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 18 State-based coverage How many of the transitions in a state transition diagram are covered by the test suite? “Covered” = touched at least once May reveal each transition was covered test values do not adequately cover value ranges If test cases were generated correctly from a state transition diagram with typical values and good boundary values, the test cases will achieve adequate state-based coverage If test cases generated from pre- and post conditions, then it is useful to check them against the state transition diagram to ensure each transition is covered.

19. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 19 State-based coverage: object interaction Note how operations interact w.r.t. transitions Current State Input or event ActionOutputNext State SaE1O1Sc SbE2O2Sc E3Sd Test cases for the transition from Sc to Sd may work if Sc was reached from Sa but not if Sc was reached from Sb. “State” is a function of history.

20. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 20 State-based coverage: transition pairs Concerning problem on previous page: Check that the test cases cover all pairs of transitions in the state transition diagram. In previous table, create test cases to test: SaSc and ScSd SbSc and ScSd

21. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 21 Statechart for Elevator Control

22. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 22 Hierarchical statechart for Elevator Control

23. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 23 Portion enlarged

24. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 24 Constraint-based coverage How many pairs of pre- and post conditions have been covered? Using the technique described earlier for generating test cases from pre and post conditions, if one test case is generated to satisfy each requirement, then the test suite meets this measure

25. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 25 Constraint-Based coverage: object interaction For each operation opn that is not an accessor operation, identify the operations op1, op2, etc. for which their preconditions are met when the post conditions for opn hold. That is, post condition(opn) satisfies precondition(op1), etc. Then execute test cases for operation sequences opn-op1, opn-op2, etc.

26. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 26 Code-based coverage How much of the code that implements the class is executed across all test cases in the suite? Minimum: Ensure that every line was executed at least once by the time all test cases (for that class) have completed execution. Commercial tools Add test cases to the suite if some lines have not been reached. Might still be inadequate – interaction between methods may fail to exercise interactions between methods Measure coverage for operation sequences

27. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 27 Test Driver Construction

28. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 28 Testing Interactions Class Testing Testing Interactions between objects Identifying & specifying object interactions Testing object interactions Collection classes Collaborator classes Testing and Design Approach Sampling COTS testing Patterns Testing exceptions Testing Class Hierarchies

29. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 29 Testing Interactions with other classes An OO program is made of a collection of objects that collaborate to solve a problem. Correct collaboration is necessary for program correctness. In this section, we assume interactions are sequential, not concurrent (that’s a subject for next semester) Focus: Ensure that messaging occurs correctly between objects whose classes have already been tested separately. Can happen embedded in application or in special environment using a test harness

30. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 30 What is an object interaction? Object interaction is: request by a sender object to a receiver object to perform one of the receiver’s operations all of the processing performed by the receiver to complete the request Includes messages between an object and its components an object and other objects with which it is associated

31. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 31 Object interaction impact During the processing of any single method-invocation on a receiving object Multiple object interactions can occur Want to consider impact of these on the internal state of the receiving object those objects with which it has an association Impact may be “no change” changes in attribute values in 1 or more objects involved state changes in 1 or more objects including state changes of creation or deletion of objects

32. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 32 Primitive and Non-Primitive Classes Interactions implied in a class spec where there are references to other objects Primitive class can be instantiated and used without needing to create any other instance Relatively few primitive classes in a program Since an OO program should model the objects in a problem and all the relationships between them, non-primitive classes are common & essential Non-primitive classes require the use of other objects in some or all of their operations Identify the classes of those objects

33. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 33 Identifying Interactions Identify interactions by association relationships in the class diagram regular associations aggregation composition Associations translate to class interfaces by: public operation names 1 or more classes as the type of a formal parameter public operation names classes as the type of a return value A class method creates an instance of another class or refers to a global instance of some class

34. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 34 Recognizing collaborators Collaborators may be addressed directly, e.g. using a variable name by a pointer or a reference dynamic type of the object may be different from the static type associated with the pointer pointers and references are polymorphic, bound to an instance of any number of classes Pre- and post conditions for operations in the public interface typically refer to states and/or specific attribute values of any collaborating objects

35. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 35 Collection and collaboration Collection class maintains associations with instances of other classes but never actually interacts with those instances that is, never requests a service Collaborating class Class with more extensive interactions Collection classes are less common

36. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 36 Testing Collection Classes What collection classes do store references to these objects, representing one-to-many relationships between objects in a program create instances of these objects delete instances of these objects Specification refers to other objects does not refer to values computed based on those objects

37. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 37 Identifying Collaborating Classes Non-primitive classes that are not collection classes Use other objects in 1 or more operations as part of their implementation When a post condition of an operation in a class’ interface refers to the state of an instance of an object and/or specifies that some attribute of that object is used or modified, that class is a collaborating class.

38. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 38 Choosing testing “chunk” size Number of potential collaborations can grow too large very quickly During class test: test composing object’s interactions with its composed attributes As successive layers of aggregation are integrated, test the interaction between an object and its associated objects Defect visibility issue – If too large a chunk is chosen, intermediate results may be incorrect but not seen at the level of test-result verification The more complex the objects, the fewer one should integrate prior to a round of testing

39. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 39 What factors increase object complexity? number of parameters for each method number of methods number of state attributes in each object Trying to test a chunk that is too complex can result in defects that are hidden from the tests.

40. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 40 What are the implications of defensive and contract design approaches when testing interactions?

41. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 41 Specifying interactions Remainder of slides assume Operations defined by a class are specified by preconditions, post conditions, and class invariants contract design approach

42. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 42 Testing Collection Classes Collection classes are testing in ways similar to primitive classes Test driver creates instances passed as parameters in messages to a collection Test cases ensure that those instances are incorporated into and removed from the collection address limitations placed on collection capacity Test sequences of operations – the way modifier operations on a single object interact with one another What additional test cases might be needed in a defensive approach?

43. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 43 Testing Collaborator Classes More complex than testing collection or primitive For example, if certain objects must be passed as parameters to a constructor, we can not test the constructor or the instance that requests the services of the constructor without ensuring there are instances of the objects to be passed as parameters

44. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 44 Contract vs. defensive approach & testing Contract More responsibility on human designer Less class-level testing due to fewer paths due to less error-checking code MORE interaction testing required to ensure human designer complied with client side of contract using precondition constraints – are preconditions in receiver met by sender “illegal” for test cases to violate preconditions Defensive More responsibility on error-checking code More class-level testing due to increased paths due to more code Once the class-level error- checking code is tested, the interaction testing is simplified since there is no need to test if preconditions are met Appropriate to violate preconditions to see if receiver catches that

45. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 45 Sampling Exhaustive testing: every possible test case covering every combination of values Not reasonable much of the time Want to select the ones that will find the faults in which we are the most interested Without prior information, random selection is as good as we can do A sample is a subset of a population (e.g. all possible test cases) that has been selected based on some probability distribution

46. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 46 Use profile Gave a way to associate each use of the system with a frequency These can be ranked by frequency Ranks can be turned into probabilities The higher the frequency of use, the larger the probability of selection

47. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 47 Operational profile – stratified sample Stratified sample is a set of samples in which each sample represents a specific subpopulation Example Select test cases that exercise each component of the architecture Divide a population of tests into subsets so that a subset contains all of the tests that exercise a specific component. Sample on each subset independent of the others Need a basis for stratifying

48. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 48 A basis for stratification of test cases Use the actors from the use cases Select a sample of test cases from the uses of each actor in the use cases Each actor uses some subset of the possible uses with some frequency; rank by frequency Stratifying the test case samples by each actors provides a way to increase the reliability of system Running selected tests uses the system the way it will be used in typical situations; finds defects likely to be found in typical use gives largest possible increase in reliability with least effort

49. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 49 Test data generation for a range of values Generate test data first using the specification as basis Select values based on boundary values just within and just on the boundaries if contract just outside the boundaries also if defensive Select values within intervals by sampling Consider a random function over range 0.. N such as int (random( ) * N) which generates a pseudo random value between 0 and 1 according to a uniform distribution. +: many values will be tested over many iterations Need to log actual values used in case of failure so that the failed test case can be re-created. A randomly chosen value causing failure is explicitly added to test suite and used to test the repaired software

50. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 50 Systematic sampling Want to be able to increase level of coverage systematically. We use boundary values coupled with sampling over ranges We never omit boundary value test data

51. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 51 Issues about sampling for interaction testing Sampling issues Objects passed as parameters Different states that can cause two objects from the same class to behave differently A class family as a subset of a larger family Possibility of combinatorial explosion in the number of test configurations

52. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 52 Puck Example from McGregor & Sykes Movable Sprite Stationary Sprite Sprite PaddlePuck BrickWallCeilingFloorPaddle LeftWallRightWall collideWith(self) Explosion of test cases...

53. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 53 Parameter sampling to test interaction Consider two abstract classes that interact where class B receives an instance of class A as a parameter Suppose B-method’s precondition places no restriction on the parameter How do we determine the population from which to sample objects to pass as this parameter? Suppose A is a base (abstract) class in a very large class family (a set of classes related by inheritance) an object from any one of the classes in the family can be substituted for its parameter This is the set from which to sample for possible parameters

54. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 54 Impact of inheritance hierarchy on s.t.d.’s Each member of the family may have different states that can cause two objects from the same class to behave differently. In the example, Puck and Wall have interesting differences in their states. In families of classes, the state transition tables are related along the lines of the inheritance hierarchy. as we look down the inheritance hierarchy, there will be the same number of states or more states in the derived class as in the base class should cover the states defined for each class, emphasizing the new states added at that level in inheritance hierarchy.

55. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 55 Class families as subsets of others If a class family is a subset of a larger family, after the tests are designed, they can be applied to any of the classes in the family, assuming the substitution principle has been followed during design*

56. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 56 Orthogonal array testing applied to OO Orthogonal array testing is a sampling technique to limit the explosion of test cases Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions An orthogonal array is an array of values in which each column represents a factor. A factor is a variable in an experiment; in our case, a specific class family in the software sys Each variable takes on a certain set of values called levels; in our case, a specific class Each cell has an instance of the class, that is, a specific state of an object

57. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 57 OATS – orthogonal array testing system The factors (columns) are combined pair-wise rather than representing all possible combinations of the levels for the factors. Suppose we have three factors – A, B, C – and each has three levels – 1, 2, and 3 How many possible combinations of these values are there? How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice? factor=class family; level=class; cell=instance

58. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 58 arrayfactor: class family factor: class family level: specific class an instance of the specific class level: specific class an instance of the specific class level: specific class an instance of the specific class a specific state

59. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 59 OATS – uses a balanced design Every level of a factor appears exactly the same number of times as every other level of that factor Think of the rows of a table as test cases (In example, 18 of the possible 27 are not being conducted. This is a systematic way of reducing the number of test cases. If we decide to add more later, we know which ones have not been run. Also logical – most errors are between pairs of objects rather than among several objects See handout for example

60. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 60 OATS adequacy criteria Ability to vary how completely the software under test is covered Exhaustive – all combinations of all factors Minimal – only interactions between the base classes from each hierarchy are tested Random – tester haphazardly selects cases from several of the classes; not statistically random Representative – uniform sample ensures every class is tested to some level Weighted representative – adds cases to the representative approach based on relative importance or risk associated with the class

61. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 61 Other testing – see McGregor & Sykes Testing off-the-shelf components – treat as a class interacting with rest of your product Protocol testing – investigates whether the implementation of a class satisfies its specification. Various protocols the object participates in can be inferred from the pre- and post conditions for individual operations. Test Patterns – design patterns for test software. explains the context of consideration, provides a set of forces that guide a trade-off analysis, explains how to construct the objects. When developer uses a certain design pattern, tester can use a certain test pattern to structure the test code. Testing Exceptions

62. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 62 Testing Exceptions Class level testing: verify that each of the methods does throw the exceptions it claims to throw in the appropriate circumstance. Coverage criteria – class throws every exception contained in the specifications of the methods At least one test case per exception Test driver sets up conditions to result in exception event. Exceptions are objects and belong to classes so catch statements can verify that it is of the correct type Integration level: being thrown at correct time in correct place Many important points covered in book

63. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 63 Testing interactions at system level Components can be so complex it’s easier to test them in the context of the application itself Which interactions are tested at system level? Those that can also be verified at the system level That is, can see the direct results of the tests There is a direct relationship between user interface and the ability to view test results

64. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 64 Testing Class Hierarchies McGregor & Sykes have 20 pages on this Inheritance provides powerful analysis & design tool Also provides powerful testing tool If inheritance applied during design in accordance with the substitution principle There is an inheritance relationship that holds for test suites. Test suites for subclasses can be derived from the test suite for their parent classes Analyzing changes between subclass & parent, can decide what test cases need to be added, rerun, modified, and not run