1. ICT Testing Product Lines of Industrial Size: Advancements in Combinatorial Interaction Testing Martin Fagereng Johansen PhD Thesis Defense, 2013-11-05

2. ICT INDUSTRIAL MOTIVATION 2

3. ICT TOMRA's Reverse Vending Machines Finale's Financial Reporting Systems ABB's Configurable Safety Module Eclipse IDEs – Free and Open Source 3

4. ICT About the Eclipse IDE Initiated and funded by IBM Widely used by software engineers to develop software Major competitor to Microsoft Visual Studio Many third-party extensions 4

5. ICT 5

6. Eclipse IDE – v3.7.0 (Indigo) – An Example of a Software Product Line 6 The problem: Can we gain confidence that any product will work?

7. ICT Which products are possible? 7 356,352 possibilities! → model its features and their relationships in a → feature model:

8. ICT Today: A Test Suite for Each Feature 8 http://archive.eclipse.org/eclipse/downloads/drops/R-3.7-201106131736/testResults.php http://wiki.eclipse.org/Eclipse/Testing

9. ICT OVERVIEW 9

10. ICT 10

11. ICT SAMPLING 11

12. ICT Faulty Features 12 Unit tests may find faults inside a single feature. n test suites required for a product line with n features. What about faulty cooperation between features? What if they interact incorrectly?

13. ICT Interaction Faults 13 2-wise interaction fault reproducible by including 2 specific features the others do not matter

14. ICT Interaction Faults 14 3-wise interaction fault reproducible by including 3 specific features the others do not matter

15. ICT Kuhn et al. 2004: Almost all bugs can be attributed to the interaction of a few features. Empirics Show: 15

16. ICT Only a few products are needed to cover all simple interactions. i.e. testing a few well-selected products might reveal almost all bugs Examples (2-wise testing): For the "e-shop product line" with 287 features: 21 products For the Linux kernel with almost 7,000 features: 480 products Covering Arrays 16

17. ICT 17 ?

18. ICT Configuring Feature Models Feature models can be solved by configuration: …or by satisfying the corresponding Boolean formula: R ∧ (A ⇒ R) ∧ (B ⇒ R) ∧ (C ⇒ A) ∧ (D ⇒ A) ∧ (C ∨ D) ∧ ¬(C ∧ D) ∧ (E ⇒ B) ∧ (F ⇒ B) ∧ (E ∨ F) ∧ (D ⇒ E) e.g. R = 1, A = 1, B = 1, C = 0, D = 1, E = 1, F = 0 The SAT problem. 18

19. ICT State of the art argument SAT is the classic NP-complete problem. worst-case analysis (Cook 1971) Configuring basic feature models i.e., finding a single product of a product line SPLE-SAT – Software Product Line Engineering Boolean SATisfiability Includes only feature models that occur in SPLE. Argument SPLE-SAT = SAT, and SAT is NP-complete i.e., SPLE-SAT is NP-complete i.e., SPLE-SAT is impractical (unless P=NP, due to Cobham's thesis) i.e. because sampling involves SPLE-SAT, sampling is impractical. 19

20. ICT Our Argument If SPLE-SAT is impractical: Configuring a feature model is impractical. i.e., testing product lines is of no concern. If we cannot find any products, why care about their quality? However, if we have a product line with products: Finding them were practical. We care about their quality. i.e., SPLE-SAT is practical. Also: If a feature model is too hard to configure then it cannot serve its purpose as an SPLE artifact. A customer cannot use it to customize a product to their needs. i.e., SPLE-SAT is practical. 20

21. ICT Empirical Investigation: SAT time 21 SPLE-SAT is very quick. Even for the largest models. E.g. The Linux Kernel Routinely configured by hand.

22. ICT Conclusions as Venn Diagrams State of the Art Conclusion: Our Conclusion: 22 SAT = SPLE-SAT Hard SAT

23. ICT 23 ?

24. ICT ? 24

25. ICT Sampling: Impractical in Practice 25

26. ICT A New, Efficient Algorithm: ICPL ICPL 26 2

27. ICT What makes ICPL quick? Based on a greedy polynomial time approximation algorithm (PTAS) for the set covering problem (SCP) Chvátal's algorithm (Chvátal 1979) We know SPLE-SAT is quick. Strategically run SPLE-SAT often and infer as much as possible. Utilize modern hardware. large amounts of memory (128 GB) truly parallel processing (64 concurrent executions) Separate out data-parallel sub-algorithms. ++ 27

28. ICT Comparison 28 State of the art: Our new algorithm (ICPL):

29. ICT Comparison CaseFeaturesClauses in Constraint tSotA (s)ICPL (s)Improvement in speed Violet1019032443286.9% Eshop287222366598.6% Ecos1,2442,7682?185? FreeBSD1,39617,3522?240? Linux6,888187,1932?33,702? 29

30. ICT 30 ?

31. ICT 31

32. ICT MARKET-FOCUSED SAMPLING 32

33. ICT Industrial Context TOMRA's Reverse Vending Machines: 33

34. ICT Feature Model of TOMRA RVM 34 435,808 possibilities!

35. ICT The 12 Products in Their Test-lab 35

36. ICT Full Sampling was Too Costly The problem Too many test-products Their Need: Optimize the selection of 12 products. Our answer: 1. Model the market situation. 2. Select the most relevant products according to that model. 36

37. ICT Our Model of the Market Situation: "Weighted sub-product lines" 37

38. ICT Better Coverage with 12 Products 38 t coverage All Inter- actions Interactions of market

39. ICT 39 ? ?

40. ICT Interactions 40 TestCSV succeeds for both. CSV works with and without Web Tools. Does CSV work without GEF? Does CSV work with CDT? Etc… With a 3-wise covering array, we get a few products with: With a 2-wise covering array, we get a few products with:

41. ICT 41 What Eclipse Tests Today:2-Wise Covering Array:

42. ICT 42 Test Results – Pair-wise Testing

43. ICT Possible Causes Two (or more) features … access the same resource have overlapping GUI elements SWTBot tests have dependencies that interact wrongly wait for each other (deadlock) +++ 43

44. ICT Potential Faults Found using Existing Test Cases Strategic application of existing tests revealed potential faults. Relatively inexpensive to apply. Raises confidence on success. Such a large scale, fully reproducible and documented application of a product line testing technique is not found in the existing literature. 44

45. ICT Also Applied to the ABB-case 45 Two bugs identified with 5 test cases.

46. ICT 46 ? ?

47. ICT 47 SPLCATool

48. ICT Future Work Further empirical study of faults in software product lines. Complete application to the Eclipse IDE With test cases for all features; it is possible today! A good source of further empirics. A good basis of further improvements. Even quicker algorithms for covering array generation. Less memory usage. Higher degree of parallelism. Improved test allocation. Based on specification, model or implementation. Based on meta-data such as versions. 48

49. ICT Summary SPLE-SAT was investigated. Realistic feature models are readily configurable. Encourages the investigation into faster algorithms. A fast algorithm for sampling. Enables the use of sampling for product line testing. Theory and algorithms for market-focused sampling. One approach for automatic allocation of test cases. Enables the production of a test report from (1) an implementation, (2) a test case collection and (3) feature model. An automatic and scalable technique for software product line testing supported by free, open source tooling. SPLCATool 49

50. ICT 50