1. Practical Uses of Software Measurement for Process Improvement January 10, 2007 - V1.0 Larry Dribin, Ph.D. larry@pearlstreetgroup.com 847-807-7390

2. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 2 Agenda  Software Measurement Programs  IT Measurement for Many Companies Today  Software Process Improvement Programs  Tailoring a IT Measurement Program to Support Process Improvement The goal of this presentation is to discuss how software measurement programs can be designed to better support process improvement. Lord Kelvin Comment: ‘When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…” Lord Kelvin, Popular Lectures and Addresses, 1889 Lord Kelvin Comment: ‘When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…” Lord Kelvin, Popular Lectures and Addresses, 1889

3. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 3 The Measurement Cycle Why Measure:  To Manage – Quality – Cycle Time – Productivity – Risk  To Communicate – Management – Peers  To Improve Performance (Process Improvement) “What gets measured gets done!” Measures Decisions Facts Actions Results

4. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 4 Software Measurement Programs Analysis & Decision Making Processes Measurement Storage Process Measurement Collection Processes Dashboard Production Process And Reporting Get Value! Make Decisions Original Processes & Organization Refine Processes, Activities And/or Organizations

5. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 5 IT Measurement for Many Companies Today  Gather and report on numerous measures (25 to over 100)  Typically focus is on financial and operational performance measures Example Financial Measures: – IT Budget – Plan vs. Actual – Hours spent on discretionary development vs. “maintenance” Example Operational Measures – Number of calls to the Help Desk – Number of calls answered by 1 st level support

6. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 6 Example Financial Measure

7. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 7 Example Operational Measure

8. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 8 Software Process Improvement Programs  There are many types of process improvement initiatives that IT Organizations are pursuing today: – CMMI – ITIL – CoBiT / SOX – TQM – QA Improvement – Testing Improvement – Agile Development  All of these initiatives require process measurement to be effective Strive for “Best Practices”, but require “GSP – Good Software Practices”. Similar to FDA requirement that Pharmaceutical companies follow “GMP – Good Manufacturing Practices”. Strive for “Best Practices”, but require “GSP – Good Software Practices”. Similar to FDA requirement that Pharmaceutical companies follow “GMP – Good Manufacturing Practices”.

9. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 9 Process Improvement Measures Original Processes Original Processes Process Progress Measures measure how well the process is being implemented: Value Tomorrow Value Today Restructured Processes Restructured Processes Outcome Measures measure the result of the improvement: (Financial, Operational, &/or Customer) Process improvement measures consist of both Outcome and Progress Metrics

10. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 10 IT Measurement and Software Process Improvement  The IT financial and operational measures described above are typically designed to assist IT management explain what IT has been doing for the business  This is important, because if business managers do not feel that they are getting value for their IT dollars, they: – Reduce IT Budgets – Reduce IT Headcount – Outsource IT As a result of this focus, many IT Measurement Programs are not designed to support process improvement.

11. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 11 CMMI Measurement Focus  Measurement & Analysis is a Key Process Area for the CMMI  Focus on is developing Measurement Programs: Collecting, storing, reporting and analyzing measurement results  Measurement is embedded in all Key Process Areas (KPA)  CMM Measurement Requirements for Requirement Management KPA Measurements are made and used to determine the status of the activities for managing requirements.  Examples of these measures include: – Status of each requirement – Change activity for each requirement – Cumulative number of changes to the requirements, including total number of changes proposed, open, approved and incorporated into the system baseline. These practices provide high level guidance when developing specific measures for process improvement

12. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 12 Designing Measurement Programs  Follow a similar approach to application development: – Gather Measurement Requirements – Analyze and Prioritize Measurement Requirements – Design a Measurement Solution – Develop the Measurement Solution What to collect How to collect Where to store When and How-Too Present and Report Measurements Results – Deploy Measurement Solution – Harvest the Benefits  Useful techniques: – Goal->Question->Metric to identify key metrics – Dashboards to organize the metrics for specific audiences – Balanced Score Cards to develop balanced dashboards linked to organizational goals

13. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 13 Goal->Question->Metric Technique Well known technique developed by Victor Basili to identify the measures needed for process improvement 1.Define Goals using the Balanced Score Card or other technique Example Goal: Let’s improve programmer productivity. 2.Ask Questions about goals to define the process improvement initiative in more detail Questions: How much does an average developer develop? 3.Questions lead to defining specific measures such as: Number of developers working in a month Time each developer spends working in the month Total number of Lines of Code (LOC) (or Function Points – FP) Produced by each developer per month Productivity = (#LOC or FP) / (# Hours Worked) -> Average Lines Of Code (LOC) or Function Points (FP) produced per developer per month

14. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 14 Example – Requirements Management Effectiveness  Goal: – Improve Requirements Management  Questions: – How many Requirement do we have? – Are these the right requirements for our problem? – Has each Requirement been approved by all key stakeholders? – Has the Requirement changed?  Potential Metrics: – Process Progress Measure: Percent of Project Teams following the new/improved process – Outcome Measures: Number of Requirements Status of each Requirement Number of changes to the requirements Number of Requirements Defects found during Requirements Validation Number of Requirements Defects found during Coding Number of Requirements Defects found during Testing

15. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 15 Example – Production Defects  Goal: – Reduce Application Production Defects  Questions: – What is a Defect? – What is an Application? – What Application did the Defect occur in?  Potential Measures – Process Progress Measures: The percent of Project Teams tracking Defects using standard defect classifications – Outcome Measures: Production Defects – defined as defects logged to the Help Desk by Application by Defect Type using standard defect classifications

16. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 16 Detailed Pareto of Trouble Tickets within Category

17. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 17 Example – Testing Metrics  Goal: – Measure Regression Test progress and when completed  Questions: – What is Regression Test Progress? – How do we know that we are done?  Potential Measures – Process Progress Measures: The percent of Project Teams tracking Regression Test Progress – Outcome Measures: Number of Regression Tests Planned Number of Regression Tests Completed Percent of Regression Tests that Completed Successfully Newly Opened Defects per Week Newly Closed Defects per Week Total Defects Open

18. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 18 Example – Regression Test Progress

19. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 19 Example - Regression Test Completed Complete when all Regression Tests have been executed, No. of Open Minor Defects < 20 and No Critical or Major Defects

20. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 20 Dashboards  Provide a holistic graphical view of multiple related measures displayed to enable stakeholders to see measurement interdependencies and take appropriate action  May have the same measure occur on different dashboards  Different Dashboards for different stakeholders – Directors of Development – Test Managers – Project Managers Project Management Dashboard Project Cost Work Completed # of Defects Director Management Dashboard Project Cost Work Completed # of Defects % Test Completed Test Management Dashboard Project Cost % Test Completed # of Defects

21. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 21 Example of What a Dashboard Can Show  Project starts out understaffed  PM is catching up via overtime and hiring additional resources  Project is getting back to plan  But is starting to run over budget because of the overtime

22. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 22 The Balanced Score Card and Process Improvement IT Measurement Programs should link to multiple organizational goals such as: Customer and Process Improvement views as well as the Financial and Operational reporting needs of IT Management How do the business owners see us? Financial View What must we excel at? Operational View How do customers see us? Customer View How do we improve, add value & stay competitive? Process Improvement View Organization The Balanced Score Card provides 4 key views and is an excellent technique for designing a Measurement Program that links to organizational goals. Source: Kaplan, R. & Norton, D., ”The Balanced Scorecard - Measures that Drive Performance”, HBR, January-February, 1992.

23. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 23 Sample Balanced Score Card for Project Manager  Financial: – Effort & Overtime – Earned Value - Effort  Customer: – Defect Rate – Earned Value – Schedule – Defect Priority  Operational: – Defects found by hours – Defects by cause – Turnover  Improvement: – Build Defects

24. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 24 Transactional System 1 Time Reporting Transactional System 3 Project Mgmt Metrics Program Architecture Transactional System 2 Defect Tracking Data Repository (Data Warehouse or Data Mart) Data Model Data Dictionary Metrics Catalog Company Balanced Score Card Departmental Dashboards Company Strategy & Goals

25. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 25 Summary Design your Software Measurement Program to collect measures that can be used for Process Improvement The result will be to increase the effectiveness of the measurement program itself and increase the effectiveness of software process improvement initiatives.

26. 01/10/2007 CSPIN f v1.0 - © Pearl Street Group 2007 26 Questions and Discussion Larry Dribin, Ph.D.  larry@pearlstreetgroup.comlarry@pearlstreetgroup.com 847-807-7390