Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version 7.09 SMU CSE 8314 Software Measurement and Quality Engineering Module 31 Methods of Observation Part 2 - Significance and Response (Displaying, Analyzing and Interpreting Data)

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version IntakeMeaningSignificanceResponse (*) Satir, Virginia et al. (references) The Observation Process (*) Previous Module This Module Other Modules

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Methods of Display and Interpretation (*) (*) Weinberg, Vol 2, chapters 4-8; Grady, chapters 2, 12 (references) Significance

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Methods of Display and Interpretation “Diagrams are Nothing; Diagramming is Everything” -- Weinberg, after Eisenhower (*) (*) Eisenhower, Dwight, “Plans are Nothing, Planning is Everything.”

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Methods of Diagramming  The way you present the data is associated with the message you want to convey  It is important to know what you want to communicate before selecting a way of graphing the data  An important objective of many types of diagrams is to show the relationships between different factors

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Methods to be Described  Fishbone Diagrams  Scatter Charts  Histograms  Pareto Charts  Run (Trend) Charts  Control Charts

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Fishbone (Ishikawa) Diagrams Project Late Resources Design Workstations Skills Motivation Errors People Complexity See previous module for more discussion

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Scatter Charts Purpose: –To show correlation (or lack of correlation) between two variables Method: –Plot two or more variables on an x-y or scatter chart

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Scatter Chart Example

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version A form of Linear Correlation V2 = a * V1 + b

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version A Non-Linear Correlation

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version No Apparent Correlation

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Further Categorization of Data  Sometimes the scatter chart hides a relationship because we did not segregate the data sufficiently well –e.g, suppose the last diagram showed fault density and weeks late on shipping –And suppose we had two kinds of projects: those that do inspections and walkthroughs and those that use only testing to identify defects –If we further segregate the data, we might see the following

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Segregated Scatter Chart

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Notes about Correlation  Correlation does NOT necessarily mean cause and effect  Correlation can come in many shapes

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Histogram Purpose: –To show the variance in a collection of data –Usually the data are expected to cluster about a mean Method: –Use a bar or column chart to show data values as a function of some variable –Can also show frequency of occurrence vs. data value

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Histogram Example Second Data Value vs. Core Index

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Histogram Example Unexpected Variation

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Histogram Example # of Occurrences vs. Core Index

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Low Variability about a Mean

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version High Variability about a Mean

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Skewed Data Target Value

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Uncontrolled Data Target Value

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Do Projects with Walkthroughs Ship On Time More Often?

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version No, but they have Fewer Defects

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Pareto Charts Purpose: –To identify the most significant cases –To highlight where to focus –To separate the significant few from the trivial many Method: –Sort data by vertical axis (“y”) value

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Example Days Lost due to Vacation

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Example No Apparent Discriminator Showing only the top of each bar helps show differences Sometimes, Pareto analysis shows that there is no significant difference

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Discussion  The previous charts are good at showing correlations between factors  And sometimes there is a genuine causal relationship between the factors  But the charts do not say much about the meaning of an observation –What does it mean to say that there are 5 defects per KLOC in the output? –Is this good, bad, typical, ???  For such purposes we need to show reference points in our data

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Reference Point Example Corporat e Goal

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Run Charts or Trend Charts  Purpose: –To show the variance over time –To show how much variation is normal –To help understand what constitutes normal variance and what constitutes exceptional data  Method: –Plot all data using a line chart and then compute and (optionally) plot the average as a separate line  Note that the average is based on current data, not past history

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Run Chart Example Data relative to Recent History

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Run Chart Example

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Run Chart with Moving Average

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Control Charts  Purpose: –To track performance –To know when a process or a machine is performing out of its normal range –To know when to take action  Method: –Show actual data vs. average and expected variation (control limits). –Very much like a run chart, but with control limits added and with average based on prior data rather than current data

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Typical Control Chart

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version More on Control Charts  This will be addressed in a later module on quantitative process management

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Recommended Reading  Weinberg, vol. 2, Chapter 5 -- Slip Charts (see references)

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Data Analysis Significance

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Data Analysis  Improper analysis can lead to wrong conclusions  Proper analysis is very hard, it requires: –Insight into the problem –Knowledge about software development –Knowledge about the application –Knowledge of the customer situation –Tracking down the real facts –Looking at the data in several ways  Telling the difference can be even harder!

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Example of Need for Proper Analysis

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Naive Conclusion Don’t inspect designs or code. Wait until code is done because it is cheaper to find and fix the defects while testing.

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Proper Analysis Shows...  Each phase detects different defects –Those introduced early may not be detected during code and test phase PhaseType XType YType Z RA PD DD C&T I&T

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Proper Analysis Also Shows...  Net cost for post-release defects is higher for those introduced in early stages Defect Correction Cost by Phase when Defect was Introduced

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version An Alternative Way to Evaluate Inspections and Walkthroughs  Track the net gain for inspections and walkthroughs D = defects detected per inspection T = time (staff hours) per inspection f = time (staff hours) to fix a defect after an inspection F = time (staff hours) to find and fix defects after release –F R = F for requirements defects –F D = F for design defects –etc. (i.e., subscript is phase of origin)

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Tracking the Gain Metric D * (F-f) - T = Gain for the inspection

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Analyzing the Impact of Defect Prevention Activities  Collect key post-release data –Incoming defects –Repair time & cost per defect (staff hours) –Calendar time per defect –Phase during which defect was introduced  Compare different projects to see impact

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Project 1 Phase% of DefectsAvg Fix IntroducedCost Req Des Cod Test Total Project 2 Phase% of DefectsAvg Fix IntroducedCost Req Des Cod Test Total Comparison of Two Projects  Project 1 invested more money in up front activities and ended up with a significantly lower net cost to fix defects.  We also need to understand the total number of defects.

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version What About Taking Action?  This module is primarily about interpreting things properly  Other modules address various ways of taking action on the basis of measurements Response

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version Summary  Drawing proper significance from data depends on methods of displaying the data and interpreting the results  Proper analysis avoids incorrect conclusions

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version References  Andersen, Bjorn and Tom Fagerhaug, Root Cause Analysis, ASQ Press,  Satir, Virginia et al., The Satir Model, Family Therapy and Beyond, Palo Alto, CA., Science and Behavior Books, ISBN:  Weinberg, Gerald M. Quality Software Management, Volume 2, First Order Measurement, Dorset House, New York, ISBN:

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M31 - Version END OF MODULE 31