1. CIS-74 Computer Software Quality Assurance Systematic Software Testing Chapter 11: Improving the Testing Process

2. Process Improvement Models 1. One produced by authors 2. CMM 3. ISO 4. TPI

3. Improving the Testing Process (Authors’ Model) Step 1: Baseline Current Practices Key terms: Baseline: a measurement of where a specific process is at any given point in time. Benchmark: a measurement of where a specific process is when compared to other companies or a model. (Also used to refer to software performance.)

4. Improving the Testing Process (Authors’ Model) Step 2: Develop Vision and Goals The point? Keep everyone focused on the same target. Great quote: “If you don’t know where you’re going, any road will get you there.” Key term: Escape: a defect that is not discovered by current evaluation activity, and hence, “escapes” to the next phase.

5. Improving the Testing Process (Authors’ Model) Step 3: Formulate/Prioritize Requirements The point? How to get from the Step-1 baseline to the Step-2 goals.

6. Improving the Testing Process (Authors’ Model) Step 4: Establish a Project The point? Ensuring that the process improvement effort will have sufficient resources to succeed.

7. Improving the Testing Process (Authors’ Model) Step 5: Develop a Plan The recommendation? Use the test plan template as a starting point if no project plan template is available in your organization.

8. Improving the Testing Process (Authors’ Model) Step 6: Introduce Change Incrementally The points? Trying to do too much too soon can backfire. It’s hard to measure the ROI on a specific improvement if multiple improvements are implemented at the same time.

9. Improving the Testing Process (Authors’ Model) Step 6: Introduce Change Incrementally (continued) Key terms: Influence leader: “One whose example is followed by his or her peers because the influence leader is perceived to be exceptionally innovative, authoritative, or technically astute.”

10. Improving the Testing Process (Authors’ Model) Step 6: Introduce Change Incrementally (continued) Adaptive Change - little potential for resistance Innovative Change - moderate potential for resistance Radically Innovative Change - high potential for resistance

11. Improving the Testing Process (Authors’ Model) Step 7: Measure Results The point? Determining if the project has met the requirements specified in Step 3.

12. Improving the Testing Process (Authors’ Model) Step 8: Return to Step 1 The point? Process improvement is a never- ending effort!

13. Authors’ Suggestions for Obtaining Buy-In 1.Ask the developers and testers to participate in creating the requirements. Key term: “Not Invented Here” syndrome, often abbreviated as NIH. 2.Effective Communications The testing staff must be confident that their recommendations will be taken seriously.

14. Authors’ Suggestions for Obtaining Buy-In (continued) 3.Metrics Provide metrics early on that make clear the benefits of the proposed change. 4.Develop Feedback Loops Developers & testers must have opportunity to provide feedback on what is and is not working.

15. Authors’ Suggestions for Obtaining Buy-In (continued) 5.Provide Training (on the process improvement process). 6.Pick a Champion and Sponsor. Champion: “An influence leader who’s willing to serve as the on-site oracle for the new process.” Sponsor: “A senior manager who can help fight for resources and get buy-in.”

16. Authors’ Suggestions for Obtaining Buy-In (continued) 7. Post-Project Reviews Can be used instead of first two steps of the authors’ suggested process improvement model. Purpose: Identify strengths to repeat on future projects. Identify opportunities for improvement.

17. ISO Certification International Organization for Standardization Federation of standardization bodies from 140 countries ISO not an acronym! From Greek word “isos” which means “equal” ISO 9000 standards for quality management system

18. ISO Certification (continued) ISO audits apply to entire organization, not just the testing group. Companies undergoing ISO audits are usually more open to process improvement activities. Cons of ISO Certification: old-fashioned, just a paper trail (in the minds of some).

19. Capability Maturity Model (CMM) Level 1: Initial Level 2: Repeatable Level 3: Defined Level 4: Managed Level 5: Optimizing

20. Capability Maturity Model (CMM) (continued) Level 1: Initial Process is ad hoc and undisciplined!

21. Capability Maturity Model (CMM) (continued) Level 2: Repeatable Focus is on project management.

22. Capability Maturity Model (CMM) (continued) Level 3: Defined Focus is on documenting processes.

23. Capability Maturity Model (CMM) (continued) Level 4: Managed Focus is on quantitative understanding and control of processes and products.

24. Capability Maturity Model (CMM) (continued) Level 5: Optimizing Focus is on continuous process improvement.

25. Capability Maturity Model (CMM) (continued) Pros: High visibility within software industry Cons: Doesn’t focus on developer, tester, OR customer, and hence, lacks their input Assumes correlation between level of process use and quality of product Too much process for small entrepreneurial organizations

26. Test Process Improvement (TPI) Model Developed in response to the CMM’s failure to address testing issues until Level 3 Breaks testing process into 20 key areas which need to be addressed by testing org Each key area must be rated as to its “Level of Maturity” (A-D, with D the highest).

27. Test Process Improvement (TPI) Model (continued) Not all of the 20 key areas have all four levels of maturity. Checkpoints: TPI questions which have to be answered in order to determine the level of maturity for a key area Improvement suggestions: Advice on how to facilitate the test improvement process

28. Test Process Improvement (TPI) Model (continued) Three labels for Level of Maturity--Controlled, Efficient, & Optimizing--are imprecise. True rating is based on where shading occurs within the TPI Assessment matrix. To really get into using TPI, one needs to read Test Process Improvement book by inventors of TPI.

29. Final Words from Authors Regardless which process improvement model your org uses, it’s necessary to continuously seek to improve your testing effectiveness!

30. Review Questions

31. What is the name for a measurement of an organization’s processes at a particular point in time?

32. A baseline

33. What is the name for a measurement of where an organization’s processes are compared to other companies or to a static model?

34. A benchmark

35. What is the term for a defect that is not discovered by the current evaluation activity and hence, is pushed on to the next phase?

36. An escape

37. What is the term for a person whose example is followed by his or her peers because s/he is perceived to be exceptionally innovative, authoritative, or technically astute?

38. An influence leader

39. Which type of change has the lowest potential resistance to change--innovative, adaptive, or radically innovative?

40. Adaptive

41. What is the term for an influence leader who’s willing to serve as the on-site oracle for a new process?

42. A champion

43. What is the term for a senior manager who can help fight for resources and get buy-in for process improvements?

44. A sponsor

45. What phrase mentioned in the book has as its acronym NIH? (NIH was not mentioned in the book.)

46. Not Invented Here

47. What is the full-length name of ISO?

48. International Organization for Standardization

49. How come the short name for the International Organization for Standardization is ISO rather than IOS?

50. ISO is derived from the Greek word isos, which means “equal.”

51. What is the name of the ISO series of standards used to establish a quality management system?

52. ISO 9000

53. What does CMM stand for?

54. Capability Maturity Model

55. How many levels are there within CMM?

56. Five

57. What are the five CMM levels in order?

58. Initial Repeatable Defined Managed Optimizing

59. At what CMM level are formal testing processes introduced?

60. Level 3 - Defined

61. At which CMM level is documentation the primary focus?

62. Level 3 - Defined

63. What are two disadvantages of the CMM?

64. 1.It does not focus on the customer. 2.It assumes a direct correlation between the level of process usage, and the quality of the product being developed.

65. What does TPI stand for?

66. Test Process Improvement

67. What shortcoming of the CMM was addressed by the TPI model?

68. The CMM’s failure to address testing issues until level 3.

69. How many key areas need to be addressed by a testing organization using TPI?

70. 20

71. How many maturity levels might each of the 20 key areas in the TPI have?

72. Four - A, B, C, or D

73. Which of the TPI maturity levels indicates the most mature process?

74. D

75. What is the term for a question from the TPI which must be answered in order to determine the maturity of each key area?

76. A checkpoint

77. What are the three levels of maturity (in order from lowest to highest) in the TPI model?

78. Controlled, Efficient, Optimizing

79. What is the difference between a blank TPI assessment form and one that shows actual status?

80. The latter has shaded-in cells to indicate the maturity levels (or lack thereof).

81. What values appear as column headers on a TPI assessment form?

82. 0 - 13

83. End of Chapter 11