1. M Global Software Group 1 Motorola Internal Use Only Better Software Quality at a Lower Cost: Testing to Eliminate Software Black Holes Isaac (Haim) Levendel, Director Systems Engineering Technology Center, Global Software Group

2. M 2 Motorola Internal Use Only Central theme Process to accelerate software quality improvement by maximizing the impact of software testing, given: –a limited testing budget –limited time We generally cannot afford full testing coverage How do we manage risk?

3. M Global Software Group 3 Motorola Internal Use Only Traditional theory of software reliability: bugs are homogeneously distributed

4. M Global Software Group 4 Motorola Internal Use Only Implications of the traditional theory of software reliability 1.Development data can be used to predict field readiness that’s good! But is the prediction correct? 2.Full coverage testing is necessary to improve the software that’s bad!

5. M Global Software Group 5 Motorola Internal Use Only Humans deal with complexity in an iterative way We use successive approximation methods The lack of complete understanding of the correct design space is the root cause of most software errors –These errors happen at all phases of the software manufacturing process At every point in time, a SW black hole is –An incomplete/incorrect area of the software

6. M Global Software Group 6 Motorola Internal Use Only Levendel’s Axiom “Bugs Abhor Loneliness” (They create software black holes)

7. M Global Software Group 7 Motorola Internal Use Only Implications of the black hole theory of software reliability 1.Development data can be used to predict field readiness that’s good! But is the prediction correct? 2.Full coverage testing is necessary to improve the software that’s bad!

8. M Global Software Group 8 Motorola Internal Use Only Disintegration of software black holes (4) Software black hole (1) (2) (3) (5) Acceptable?

9. M Global Software Group 9 Motorola Internal Use Only At the end of the process: the SW product variance has been reduced

10. M Global Software Group 10 Motorola Internal Use Only The software crafting phases Requirements Design Coding Unit Design Integration “Make believe” Requirements Design Coding Unit Design Integration Reality

11. M Global Software Group 11 Motorola Internal Use Only Requirements Design Software black holes are left behind anywhere in the crafting process Coding Unit Design Integration Each phase leaves SW black holes

12. M Global Software Group 12 Motorola Internal Use Only The black hole theory of software Supporting data and experience

13. M Global Software Group 13 Motorola Internal Use Only Software modules affected by changes for new features (Subsystem #1)

14. M Global Software Group 14 Motorola Internal Use Only Software modules affected by changes for new features (Subsystem #2)

15. M Global Software Group 15 Motorola Internal Use Only Number of new and modified lines of source code for each internal release (total of 34 internal releases)

16. M Global Software Group 16 Motorola Internal Use Only Number of software modules changed per fix for each internal release (total of 34 internal releases)

17. M Global Software Group 17 Motorola Internal Use Only Number of lines of software changed per fix for each internal release (total of 34 internal releases)

18. M Global Software Group 18 Motorola Internal Use Only Software field products will always have bugs Cost Per Bug Fix Reproducibility (1/MTBF) Time

19. M Global Software Group 19 Motorola Internal Use Only Identifying black holes

20. M Global Software Group 20 Motorola Internal Use Only Role of testing: help to identify black hole (4) Software black hole (1) (2) (3) (5) Acceptable?

21. M Global Software Group 21 Motorola Internal Use Only Advancing the release time by repairing a software black hole in one step B’ A = A black hole is found B = Field delivery Number of Tests t (or time elapsed) Cumulative No. of Failures 300 250 200 150 100 50 0 0 1,000 2,000 3,000 4,000 5,000 O 150 100 50 0 Climb to a “better” curve = The black hole is fixed O’ A’

22. M Global Software Group 22 Motorola Internal Use Only Testing to improve product quality Home on SW Black Hole Explore SW Black Hole Repair SW Black Hole (redesign SW module) Stop More SW black holes? Yes No Find bug Fix bug Execute test Retest fix Stop More Tests? Yes No

23. M Global Software Group 23 Motorola Internal Use Only How do we position the initial testing on software black holes? Desirable initial tests Not worth testing

24. M Global Software Group 24 Motorola Internal Use Only Information sources for guiding the testing toward potential software black holes Proc. Phase Info Info type Past releases Planning of current release Design/coding of current release Testing of current release Field data  Laboratory problems  SW change information  SW planning problems  TIME

25. M Global Software Group 25 Motorola Internal Use Only Correlation between trouble reports and fixes helps identify problem-prone software modules Trouble report 1 SW Module 1 Fix 1b Fix 1a SW Module 2 SW Module 3 SW Module 4 Trouble report 2 Fix 2a

26. M Global Software Group 26 Motorola Internal Use Only Code changes and new code may affect problem-prone software modules Code change request 1 SW Module 1 Change 1 SW Module 2 SW Module 3 SW Module 4 Code change request 2 Change 2

27. M Global Software Group 27 Motorola Internal Use Only White box testing leads to higher fail rate Trouble report 1 SW Module 1 Fix 1b Fix 1a SW Module 2 SW Module 3 SW Module 4 Trouble report 2 Fix 2a Test x fails Test y passes Test z fails

28. M Global Software Group 28 Motorola Internal Use Only Test program alerts Test programs cannot be fully designed one year ahead of time!!! –because one does not know in advance all the places where the functionality is going to break They must be adaptive!!! –Need to combine upfront planning with adaptive execution This includes regression tests

29. M Global Software Group 29 Motorola Internal Use Only Practical Testing Question Black Box Testing or White Box Testing?

30. M Global Software Group 30 Motorola Internal Use Only Practical Testing Question Black Box Testing or White Box Testing? Both!! Why? Black box testing reflects requirements White box testing reflects software black holes

31. M Global Software Group 31 Motorola Internal Use Only Who would want to test like that?

32. M Global Software Group 32 Motorola Internal Use Only 4 weeks of testing data

33. M Global Software Group 33 Motorola Internal Use Only Test program architecture & Test program metrics

34. M Global Software Group 34 Motorola Internal Use Only Managing complexity Complexity can be managed by decomposing large systems into a tree-like hierarchy of functions The depth and levels of the decomposition are arbitrary Functional Area Functional Package Functional Subset Functional Functional Functional Component Component Component

35. M Global Software Group 35 Motorola Internal Use Only How a test suite traverses functional components Test 1 Test 2 Component 1 Component 2 Component 3 Component 4 Component 5 Component 6

36. M Global Software Group 36 Motorola Internal Use Only Two key issues with components 1.The SW architecture is a good starting point for a decomposition into functional components 2.It is essential to characterize the mapping between test suites and functional components

37. M Global Software Group 37 Motorola Internal Use Only Implementing test programs: the combinatorial explosion problem Functional Area Functional Package Functional Subset Functional Component Functional Package Functional Subset Functional Component Functional Component Underlying Functional structure Unmanageable size of test program + Feature interactions

38. M Global Software Group 38 Motorola Internal Use Only Filtering test program scope to reduce cost Functional Area Functional Package Functional Subset Functional Component Functional Package Functional Subset Functional Component Functional Component Customer rollout plan Suspected vulnerabilities Underlying Functional structure Manageable Test program

39. M Global Software Group 39 Motorola Internal Use Only Collect failures on tests ( ) and components ( ) Test 1 Test 2 Component 1 Component 2 Component 3 Component 4 Component 5 Component 6

40. M Global Software Group 40 Motorola Internal Use Only Fail rates for typical project’s components Likely software black holes

41. M Global Software Group 41 Motorola Internal Use Only Create time and space repetitions FS 2 FS 3 FS n FS 1 FS 2 FS 3 FS n FS 1 FS 2 FS 3 FS n FS 1 FS 2 FS 3 FS n FS 1 Evaluation Evaluation Period 1 Period 2 Period 3 Period 4   Comparisons Time repetition Space repetition

42. M Global Software Group 42 Motorola Internal Use Only A reporting mechanism to motivate repairs of software black holes It is more economical to climb the quality curve in steps The reporting should motivate these step functions The solution: –Drive the testing to identify software black holes –Orient the test reporting to point at software black holes & recommend solutions

43. M Global Software Group 43 Motorola Internal Use Only Log the learning 1.The exercise of human judgment is indispensable 2.Data per se has NO meaning

44. M Global Software Group 44 Motorola Internal Use Only The key common sense question about testing in general Common Sense is not so common - Voltaire (1694-1778) How does one measure testing productivity? –In number of tests executed per $? or –In number of test failed per $?

45. M Global Software Group 45 Motorola Internal Use Only The key common sense question about testing in general Common Sense is not so common - Voltaire (1694-1778) How does one measure testing productivity? –In number of tests executed per $? or –In number of test failed per $ !

46. M Global Software Group 46 Motorola Internal Use Only (Dire) Conclusion If one accepts the software black hole hypothesis, then: the current reliability and availability technologies need to be rethought