1. Testing Techniques  Equivalence class analysis  Cause effect graphs  All pair testing

2. Equivalence classes

5. Triangle Problem Read 3 integer values in range [1..200]. These 3 values represent the length of the sides of a triangle. The programme displays a message which establishes that the triangle is isosceles, equilateral or scalene.

6. Triangle Problem we are interested in 4 questions: – Is it a triangle? – Is it an isosceles? – Is it a scalene? – Is it an equilateral? We may define the input test data by defining the equivalence class through “viewing” the 4 output groups: – input sides do not form a triangle – input sides form an isosceles triangle – input sides form a scalene triangle – input sides form an equilateral triangle

7. Valid Classes

8. Non Valid Classes

9. Could you find other non valid classes ?

10. Cause effect graph

11. Methodology

12. Triangle example

13. Triangle Problem

14. Cause Effect Graph

15. Decision Table

16. Test cases

17. All pairs testing Based on orthogonal Array Drastically reduces number of test cases A combination of worst cases and equivalence class testing Well supported in the agile methods community Supported by commercial and non- commercial products

18. What Does All Pairs Do? Input is a set of equivalence classes for each variable. Sometimes the equivalence classes are those used in Boundary Value testing (min, min+, nominal, max-, max) Output is a set of (partial) test cases that approximate an orthogonal array, plus some pairing information. Why partial? No expected outputs are provided. natural enough, but a tester still needs to do this step.

19. All Pairs Assumptions Variables have clear equivalence classes. Variables are independent. Failures are the result of the interaction of a pair of variable values.

20. Example Suppose that we have three parameters. For each parameter, there are two possible values. – Values are : A, B for parameter 1. J, K for parameter 2. Y, Z for parameter 3. Degree of interaction coverage is 2. – We want to cover all potential 2-way interactions among parameter values.

21. Set of all possible test configurations AJY AKZ AKY AJZ KYB BZJ BYJ KZB There are 2 3 = 8 possible test configurations. Three parameters, two values for each.

22. Set of all possible degree 2 interaction elements Coverage measure: – Percentage of interaction elements covered. There are ( )  2 2 = 12 possible interaction elements. 3 2 AJ BK BJ AK AY AZ BZ BY JY KZ KY JZ

23. Test configurations as sets of interactions AJY AJ AY JY One test configuration... … covers 3 possible interaction elements.

24. …using a subset of all test configurations. Interaction test coverage goal Goal: cover all interaction elements… AJY KYB AKZ AKY AJZ BZJ BYJ KZB AJ BK BJ AK AY AZ BZ BY JY KZ KY JZ

25. AJY KYB AKZ AKY AJZ BZJ BYJ KZB AJ BK BJ AK AY AZ BZ BY JY KZ KY JZ Selection of test configurations for coverage of interaction elements Interaction elementsTest configurations AJAYJY AJY Degree 2 coverage: 3 / 12 = 25% Degree 3 coverage: 1 / 8 = 12.5%

26. AK AK AJY KYB AKZ AKY AJZ BZJ BYJ KZB AJ BK BJ AY AZ BZ BY JY KZ KY JZ Selection of test configurations for coverage of interaction elements Interaction elementsTest configurations AKZ AZKZ Degree 2 coverage: 6 / 12 = 50% Degree 3 coverage: 2 / 8 = 25%

27. AJY KYB AKZ AKY AJZ BZJ BYJ KZB AJY AKZ AJ BK BJ AK AY AZ BZ BY JY KZ KY JZ Selection of test configurations for coverage of interaction elements Interaction elementsTest configurations BZJ BJBZJZ Degree 2 coverage: 9 / 12 = 75% Degree 3 coverage: 3 / 8 = 37.5%

28. AJY KYB AKZ AKY AJZ BZJ BYJ KZB AJY AKZ BZJ AJ BK BJ AK AY AZ BZ BY JY KZ KY JZ Selection of test configurations for coverage of interaction elements Interaction elementsTest configurations BKBYKY KYB Degree 2 coverage: 12 / 12 = 100% Degree 3 coverage: 4 / 8 = 50%

29. Choosing the degree of coverage In one experiment, covering 2 way interactions resulted in the following average code coverage: – 93% block coverage. – 83% decision coverage. – 76% c-use coverage. – 73% p-use coverage. – Source: Cohen, et al, “The combinatorial design approach to automatic test generation”, IEEE Software, Sept. 1996. Another experience report investigating interactions among 2-4 components: – Dunietz, et al, “Applying design of experiments to software testing”, Proc. Of ICSE ‘97.

30. Online Mortgage Application (Thanks, Bernie Berger, STAREast) Region tier Property Credit Residence LTVNIVNAV Refinance Closing Cost Intro Rate Bank Emp

31. Twelve variables, with varying numbers of values, have 7 x 6 x 6 x 5 x 3 x 3 x 2 x 2 x 2 x 2 x 2 x 2 = 725,760 combinations of values. “All Pairs does it in 50.” (Bernie Berger, STAREast2003) Total number of combinations

32. 1 2 3 4 47 48 49 50 NY NJ NY Other L M L M H+1 H H+2 H+1 1 fam 2 fam 1 fam ~Coop 2 fam 3 fam Condo A+ A A- B B ~B ~A- <B Pri Vac Inv Pri ~Inv ~Vac ~Pri ~Vac 80% 90% 100% 90% ~80% ~100% ~80% ~90% Yes No Yes No ~No ~Yes ~No Yes No Yes ~No Yes No Yes No ~Yes ~No ~Yes Cust Bank Cust ~Cust Yes No Yes No ~Yes ~No Yes No Yes ~Yes ~No 66 54 46 1 1 1 2 Selected All Pairs Test Cases (note fault resolution!)

33. Mortgage Application Observations The example uses values, not equivalence classes. Values are independent, System level testing (apparently). The efficiency comes at the expense of fault isolation. There are 2 C 12 = 12!/(2! x (12 - 2)!) = 66 pairs in the first row of the table. If that test case fails, which of the 66 pairs caused the failure? NB: excellent candidate for regression testing

34. What if the fault involves more than just a pair of variables? pCn 1 12   2^  1  4095 pCn  n! p!(n  p)! Number of combinations of n things taken p at a time is Worst case (for first test case):