1. SCAMPI Sampling Rules 1 Sampling the SCAMPI Sampling Rules or Trying to Explain the Unexplainable Pat O’Toole, PACT www.PACTCMMI.com May, 2012 www.PACTCMMI.com

2. SCAMPI Sampling Rules 2 Acknowledgments Terms like these are often used in the following material: CMMI ® Framework IDEAL SM Model Capability Maturity Model Integration SM and IDEAL SM are service marks of Carnegie Mellon University ® CMMI is registered in the U.S. Patent & Trademark Office

3. SCAMPI Sampling Rules 3 SCAMPI Class A v1.3  SCAMPI –S tandard C MMI A ppraisal M ethod for P rocess I mprovement  SCAMPI MDD –SCAMPI M ethod D efinition D ocument  250+ page process document for SCAMPI A appraisals  Specific source of angst –SCAMPI MDD v1.3 Section 1.1.4 –It’s only 7 pages, but do NOT drive or operate heavy equipment after reading it!

4. SCAMPI Sampling Rules 4 New Terminology with MDD v1.3  Basic Units  Sampling Factors  Subgroups  Coverage Rules.

5. SCAMPI Sampling Rules 5 New Terminology with MDD v1.3  Basic Units  Sampling Factors  Subgroups  Coverage Rules.

6. SCAMPI Sampling Rules 6 Basic Unit  For CMMI-DEV, “Basic Unit” is roughly equivalent to “Project” –However, “project” doesn’t work as well for the Services or Acquisition constellations –“Basic Unit” is much more generic –Therefore, for purposes of this presentation, I will use “PROJECT!”

7. SCAMPI Sampling Rules 7 New Terminology with MDD v1.3  Basic Units  Sampling Factors  Subgroups  Coverage Rules.

8. SCAMPI Sampling Rules 8 Sampling Factors – Why?  Project work may be performed in different ways depending on various characteristics –Small projects may be conducted differently than large ones –Project work conducted by your USA-based team may be done differently than your offshore development group –Scrum projects are run differently than Waterfall projects  Sampling factors ensure that the organizational scope provides a robust (and complete) view of the various ways work is performed by the organization.

9. SCAMPI Sampling Rules 9 Mandatory Candidate Sampling Factors  The Sampling Factors that MUST be considered: –Location (e.g., Minneapolis vs. Las Vegas) –Customer (e.g., DoD vs. commercial industry) –Size (e.g., small vs. medium vs. large vs. 4XL) –Organizational Structure (e.g., different divisions) –Type of Work (e.g., development vs. maintenance)  For each candidate, the question to be answered is: –Do different settings for this factor affect the way that work is performed in the organization to be appraised?  Yes – Relevant Sampling Factor  No – Irrelevant Sampling Factor.

10. SCAMPI Sampling Rules 10 Other Potential Sampling Factors  Other Relevant Sampling Factors may exist and, if so, must also be considered –Life Cycle (e.g., Scrum projects vs. Waterfall projects) –Funding Source (e.g., customer vs. internal funding) –Criticality (e.g., nice-to-have vs. life critical) –Etc.  Your organization may have unique characteristics that significantly affect how work is performed –These would be included as Relevant Sampling Factors in a SCAMPI A appraisal.

11. SCAMPI Sampling Rules 11 Example

12. SCAMPI Sampling Rules 12 New Terminology with MDD v1.3  Basic Units  Sampling Factors  Subgroups  Coverage Rules.

13. SCAMPI Sampling Rules 13 Subgroup  “Cluster” of projects that share common relevant sampling factor values and exhibit similar process implementations –If Customer, Size, and Type of Work are relevant sampling factors, how many POSSIBLE subgroups are there?  Customer: Internal / External  Size: Small / Medium / Large  Type of Work: Development / Maintenance  2 values for x 3 values for x 2 values for = 12 possible subgroups.

14. SCAMPI Sampling Rules 14 How Many ACTUAL Subgroups?

15. SCAMPI Sampling Rules 15 Number of projects in the given subgroup How Many Projects per Subgroup? Minimum # of projects from a given subgroup = Number of subgroups X Total number of projects

16. SCAMPI Sampling Rules 16 Back to Our Example… # of IMD = (6 x 2) / 15 = 0.8 => 1 # of ISM = (6 x 1) / 15 = 0.4 => 1 # of EMD = (6 x 5) / 15 = 2.0 => 2 # of ESM = (6 x 3) / 15 = 1.2 => 1 # of ELD = (6 x 2) / 15 = 0.8 => 1 # of EMM = (6 x 2) / 15 = 0.8 => 1.

17. SCAMPI Sampling Rules 17 What if… # of IMD = (6 x 200)/1500 = 0.8 => 1 # of ISM = (6 x 100)/1500 = 0.4 => 1 # of EMD = (6 x 500)/1500 = 2.0 => 2 # of ESM = (6 x 300)/1500 = 1.2 => 1 # of ELD = (6 x 200)/1500 = 0.8 => 1 # of EMM = (6 x 200)/1500 = 0.8 => 1

18. SCAMPI Sampling Rules 18 New Terminology with MDD v1.3  Basic Units  Sampling Factors  Subgroups  Coverage Rules.

19. SCAMPI Sampling Rules 19 Coverage Rules  Three flavors: –Coverage Rules for Process Areas –Coverage Rules for Basic Units (whoops! – Projects) –Coverage Rules for Support Functions.

20. SCAMPI Sampling Rules 20 Cover Rules for Process Areas  PA Coverage Rule 1 (in for a penny, in for a pound) –Evaluation of a given PA must cover all practices, both specific and generic –There is a wee bit of wiggle room here as the rule includes the words, “…for which they supply data”  PA Coverage Rule 2 (no need to keep pounding) –If a PA is performed in a consistent way across subgroups, evaluation of that PA may be consolidated –For example, if CM, DAR, PPQA, and RSKM are performed in the same manner across all projects, then they need not be evaluated for all 6 subgroups.

21. SCAMPI Sampling Rules 21 Project Coverage Rules  Project Coverage Rule 1 –For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area  This sampled project shall provide data for all process areas  Project Coverage Rule 2 –For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area  Project Coverage Rule 3 – For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area.

22. SCAMPI Sampling Rules 22 ATLAS #19a  Can you apply what you’ve learned? –Well, heard?  Let’s find out…  Work individually or together to answer the questions in ATLAS #19a –There is no need to provide comments.

23. SCAMPI Sampling Rules 23 Cover Rules for Process Areas  PA Coverage Rule 1 –Evaluation of a given PA must cover all practices, both specific and generic –HINT: Start with the question, “Which projects contribute to its fulfillment?”

24. SCAMPI Sampling Rules 24 Project Coverage Rules  Project Coverage Rule 1 –For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area  This sampled project shall provide data for all process areas  Project Coverage Rule 2 –For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area  Project Coverage Rule 3 – For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area.

25. SCAMPI Sampling Rules 25 ATLAS #19a Results  ATLAS #19a was distributed to all SCAMPI lead appraisers and only SCAMPI lead appraisers  61 SCAMPI lead appraisers responded –That’s about a 12% response rate –But it’s still more than “one expert’s opinion”  Let’s see how YOU did compared with those responsible for REALLY understanding this stuff…

26. SCAMPI Sampling Rules 26 Question 1 1.Is Process Area Coverage Rule 1 fulfilled? 36% Yes / 64% No Which project(s) contribute to its fulfillment: P1? 82% Yes / 17% No P2? 90% Yes / 10% No P3? 74% Yes / 24% No P4? 7% Yes / 93% No.

27. SCAMPI Sampling Rules 27 Question 2 2. Is Project (BU) Coverage Rule 1 fulfilled? “For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area” 34% Yes / 66% No Which project(s) contribute to its fulfillment: P1? 42% Yes / 56% No P2? 14% Yes / 86% No P3? 4% Yes / 96% No P4? 4% Yes / 96% No.

28. SCAMPI Sampling Rules 28 Question 3 3. Is Project (BU) Coverage Rule 2 fulfilled? “For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area” 51% Yes / 49% No Which project(s) contribute to its fulfillment: P1? 54% Yes / 46% No P2? 97% Yes / 3% No P3? 22% Yes / 78% No P4? 18% Yes / 82% No.

29. SCAMPI Sampling Rules 29 Question 4 4. Is Project (BU) Coverage Rule 3 fulfilled? “For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area.” 48% Yes / 52% No Which project(s) contribute to its fulfillment: P1? 94% Yes / 6% No P2? 98% Yes / 2% No P3? 93% Yes / 7% No P4? 25% Yes / 75% No.

30. SCAMPI Sampling Rules 30 Optical Illusion – 2 Images in One

31. SCAMPI Sampling Rules 31 Modified Coverage Rules - #1  PA Coverage–1: Objective evidence provided for a PA must address all practices that are part of that PA, both specific practices as well as all generic practices. The Basic Unit and Support Function coverage rules indicate the types of objective evidence that must be provided for each practice.  Project Coverage–1: For each subgroup, both artifacts and affirmations shall be provided for each practice for at least one project.  Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for each practice for at least one PA  Project Coverage–3: For all sampled projects in each subgroup either artifacts or affirmations shall be provided for each practice for at least one PA.

32. SCAMPI Sampling Rules 32 Modified Coverage Rules - #2  PA Coverage–1: Objective evidence provided for a PA must address all practices that are part of that PA, both specific practices as well as all generic practices. [No change]  Project Coverage–1: For each subgroup, some combination of both artifacts and affirmations shall be provided for at least one project.  Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, some combination of both artifacts and affirmations shall be provided for at least one PA  Project Coverage–3: For all sampled projects in each subgroup either artifacts and/or affirmations shall be provided for at least one PA.

33. SCAMPI Sampling Rules 33 Conclusion  The new sampling and coverage rules are intended to provide a truly representative set of projects that reflect how work is performed in the organization –There are some issues, however:  They only solve the “proportional problem,” not the “how much is enough” problem  There are multiple ways to interpret the rules as written, arriving at significantly different results  Based on ATLAS #19a, the SEI has committed to clarify, or even re-write the coverage rules  With respect to brain function, covering this topic is equivalent to drinking 6 beers; please drive carefully!

34. SCAMPI Sampling Rules 34 Questions??? Pat O’Toole Process Assessment, Consulting & Training (PACT) PACT.otoole@att.net www.PACTCMMI.com If anyone is interested in receiving my monthly “Do’s and Don’ts” articles, either: Give me a business card Send me an email.