6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 1 Planning for Measurement WM Software Process and Quality Measurement is a sampling process designed to tell us something about the universe in which we live that will enable us to predict the future in terms of the past through the establishment of principles of natural law. - Walter A. Shewhart [1931]

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 2 Overview ÙIdentify the process issues w.r.t. measurement ÙSelecting and defining measures ÙIntegrating measures with the software process

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 3 Identifying Process Issues Process stable? Clarify Business Goals Identify & prioritize issues Select & define measures Collect,verify & retain data Analyze process data Process Capable? Yes No Remove assignable causes New measures? Change Process No Continually improve Yes New issues? Yes New goals, strategy? Yes No Clarify Business Goals Identify & prioritize issues Select & define measures

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 4 Steps for Identifying Process Issues ÔClarify your business goals and objectives (increase function, reduce cost, reduce time to market, improve productivity) ÔIdentify the critical processes (e.g., requirements management, design, testing) ÔList the objectives for each critical process ÔList the potential problem areas associated with the processes ÔGroup the list of potential problems into common areas or topics

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 5 Generic Process Model Receives (inputs resources that are used or supplied) Consists of (activities, flow paths, agents) Consumes (consumables) Holds (internal artifacts, such as inventory and work in progress) Produces (products, by-products, effects) You can use process modeling and enactment tools (e.g. Endeavor, Process Weaver) to understand the process properties.

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 6 Determining what entities to model ÔGeneral: What determines product quality? What determines success? What do your customers want? ÔRisks: What could go wrong? What is not working well? What might signal early warnings? ÔTemporal: Where are the delays? How big is our backlog? Where is the backlog occurring? ÔCapability: What things can we control? What limits our capabilities? What limits our performance? Ask the following types of questions (use GQM - Goal- Question-Metric technique)

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 7 Selecting Process Performance Measures Process performance refers to the characteristic values we see when measuring attributes of products and services that come from a process. It can be measured in two ways:  by measuring attributes of products the process produces, and  by measuring attributes of the process itself Examples of measurable product attributes include: function, size, execution speed, module strength). Examples of process attributes include: effort expended, clock time or computer time to perform a task, the sizes and durations of work flows and backlogs, the number, types and sources of defects detected, repaired or reported. see pages 27 & 28

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 8 Criteria for Good Measures Ôshould relate closely to the issues under study (e.g. quality, resource consumption, or time elapsed) Ôshould have high information content Ôshould pass a reality test, does the measure really reflect what you need to know about your processes Ôshould permit easy and economical data collection Ôshould permit consistently collected, well-defined data Ôshould show measurable variation (measures that show no variation provide no information) Ôshould have diagnostic value -- what happened and what causes it

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 9 Defining Process Performance Measures ÔDifferent users of measurements have different needs (e.g. cyclomatic complexity) ÔDifferent organizations have established different practices. ÔUnambiguous communication of measurement results is inherently difficult (operational set-up is very complex) ÔStandard methods of communicating results don’t exist Once we have identified our measures, we must define them - naming them is not sufficient. Unfortunately the software industry has not been very successful at getting agreement on standard measures (e.g. KLOC). Why?

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 10 Well-Defined Measures  Communication - every user must understand precisely the measure used  Repeatability - someone else should be able to repeat the measurements in the same situation an get the same results  Traceability - the origins of the data are identified in terms of time, source, sequence, activity, product, status, environment, measurement tool used and collecting agent Deming introduced the notion of a well-defined measures having the following criteria

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 11 Comments of Well-Defined Measures  Repeatability is difficult … why?  Traceability is important in order to assess and improve the process … why?  Measures of performance can signal process instability so it is important that the context of the measurement be recorded.  Well-defined measures can be defined using frameworks and checklists as shown in Fig (p. 33) - produced by SEI.

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 12 Integrating Measures with the Software Process Must begin by analyzing the existing measurement activities. Questions like the following should be asked: –What data elements are required for a measure? –Which ones are collected now? –How are they collected? –Which processes provide the data? –How are the data stored and reported? Problem reporting is generally a good area to start your analysis. Third stage in measurement planning.

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 13 Sources for Problem Reporting Measures Product Synthesis Review & Inspections Testing Customer Support Problem Reporting Rev/Inspect Reporting Test Reporting Customer Problem Analysis and Corrective Actions Activity Specification Databases Problem Tracking Data

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 14 Diagnosis of Existing Measures  What existing measures and processes can be used to satisfy our data requirements?  What elements of our measurement definitions and practices must be changed or modified?  What new or additional processes are needed? Diagnosis means determining how well existing measures meet the needs of our goal-driven analysis, and proposing appropriate actions for 1) using the data, 2) adapting the data to our needs, 3) adapting our needs to the data, 4) obtaining critical missing data. Diagnostic questions might be: This leads to the preparation of a Measurement Action Plan

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 15 Measurement Evaluation Fig of Florac & Carleton provides an excellent table of factors and associated considerations that should be considered for measurement evaluation. It applies to both existing and new measures. If this form of evaluation is made prior to planning, the planning process will go very smoothly. Fig. 2.15

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 16 Preparing a Measurement Action Plan Define the data elements and scales to be used. Define the frequencies of collection and where in the process measurements will be made. Define timelines for moving measurement results from collection points to databases or users Create forms and procedures for collecting and recording data Define how data is to be stored and accessed. Define db admin function for the data. Determine who will collect and assess the data.

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 17 Preparing a Measurement Action Plan Define how the data will be analyzed and reported. Identify the supporting tools that must be developed or acquired to help automate and administer the process (Why are tools important?) Prepare a process guide for collecting the data. Part II

6/1/2015WM-001 Planning for Measurement - copyright Paul Sorenson slide 18 Summary Planning is critical to the successful execution of any process -- the measurement process is no exception. Measuring software begins with planning -- not just of what to measure but how the measures will be effectively used to support some major goal of the organization. Measurement planning involves identifying the process management issues, selecting and defining the product and process measures, and integrating the resulting measurement activities into the organization’s existing software processes.