1 Pop Quiz What is an Orthogonal Defect? Which is more precise: predictive models or management models? Why? Essay: Why are metrics and models an important element to managing the quality of software development? T/F An evolutionary prototype is evolved rather than thrown away

2 Software Quality Engineering CS410 Class 10 Quality Management Models

3 –Tools for helping to monitor and manage the quality of software when it is under development –Goal - provide early warning signs so that timely improvement actions can be planned and implemented –Models must cover entire development phase (front-end and back-end) to be useful –Some Reliability Models can be used as Quality Management Models

4 Quality Management Models Rayleigh Model Framework –Based on the two assumptions: 1. Defect rate observed during development is positively correlated with defect rate observed in the field 2. Given the same error injection rate, if more defects are found and removed early then fewer will remain in later stages (I.e. the field) –“Do it right the first time” It is important to manage quality throughout the entire development process

5 Quality Management Models 1. Best scenario is to prevent errors from being injected into the development process (and hence, into the product) 2. When errors are introduced, improve front-end processes (design reviews, code reviews, etc.) to remove as many of them, as early as possible. 3. Unit testing serves as the last chance to catch errors in the front-end process (the gatekeeper for front-end process escapes).

6 Quality Management Models The Rayleigh model is a good Quality Management model because it promotes defect prevention and early defect detection As an in-process tool, the data can indicate the quality direction of the current project Comparison to previous models would show whether the defect removal is more/less/same front-end loaded: same - quality should be similar to previous products more - quality should be better than previous products less - quality action should be taken

7 Quality Management Models Rayleigh curve shifts Left - more front-end loaded (early defect removal) Right - less front-end loaded Up - more error injection Down - less error injection (defect prevention) Goal - shift the curve left and down as much as possible Fig 9.2 p. 223

8 Quality Management Models Actions for shifting the curve –Left (early defect removal) focus on design reviews / code inspections –moderator training –inspection checklists –in-process measurements to track effectiveness mini-builds (prior to formal integration)

9 Quality Management Models –Actions for shifting the curve –Down (defect prevention) implementation of DPP use of CASE tools improved communications between interface owning groups

10 Quality Management Models Challenge - knowing exactly what the differences in the curve represent: –Lower defect removal rates (lower curve) could be less errors injected, or ineffective design reviews and code inspections –Higher defect removal (higher curve) could mean higher error injection or better design reviews and code inspections Additional indicators are needed to help interpret the data

11 Quality Management Models Quality of process metrics can be used as additional indicators, for example: –Inspection effort (expressed in hours) combined with defect rate Inspection Effort and Defect Rate when compared to historical data

12 Quality Management Models –Best case - high effort / low defects: Indicator that the design and/or code contained fewer defects, and that the design reviews and code inspections were effective, and better quality was ensured. –Good / Not Bad case - high effort / high defects: Indicator that error injection may be higher, but process was effective in detecting defects.

13 Quality Management Models –Unsure case - low effort / low defects: Not sure whether less defects were injected, or whether less time spent in design reviews and code inspections turned up less defects. –Worst case - low effort / high defects: Indicator of high error injection, but process was not effective in detecting defects.

14 Quality Management Models PTR Sub-model –Continuous integration of some product life cycles make parametric models difficult. –Spiral and Iterative models make it difficult to differentiate between front-end and back-end processes. –These life cycles sometimes have the concept of continuous integration. –Different Quality management models are required for this type of product development life cycle.

15 Quality Management Models PTR (Problem Tracking Reports) are a common tool used for managing the change process during development and testing. Non-parametric model that is a sub-model of the overall defect removal model. PTR model spreads over time the number of defects that are expected to be removed during formal testing so that precise tracking may be done.

16 Quality Management Models PTR model is a function of: 1. Planned or actual lines of code integrated over time. Can be estimated from historical data, design documents, and estimating tools 2. Expected overall PTR rate per KLOC. Can be estimated from historical data. 3. PTR-surfacing pattern after the code is integrated. Depends on testing activities and integration plans (eg. weekly integration - faster discovery / fix / integration cycle than bi-weekly or monthly)

17 Quality Management Models Deriving PTR Sub-model Curve 1. Determine code integration plan 2. Multiple expected PTR rates times expected KLOCs to derive expected PTRs per integration 3. Spread PTRs over time depending on PTR spread pattern and sum of number of PTRs 4. Update model when integration plan changes or actual integration data becomes available 5. Plot the curve and track current project in terms of months until GA (General Availability)

18 Quality Management Models Example PTR Sub-model Fig 9.7 p. 229 –Note - PTR sub-model is non-parametric and therefore cannot make projections. It is used for tracking current data. –Goal - compared to the model, if the actual curve (actual defect arrivals) increases, and peaks earlier, then it will decline faster relative to the GA date (I.e. less defects will be shipped to the field)

19 Quality Management Models PTR Arrival/Backlog Projection Model –Goal - determine whether the scheduled code-freeze date can be met without sacrificing quality. In other words - will the PTR arrival rate and PTR backlog decrease to the acceptable levels? –Other models do not answer this question: PTR Sub-model - cannot predict Reliability Growth Models - predictions are made too late

20 Quality Management Models Model uses a general linear approach, and assumes that relevant variables observed over time perform an adequate prediction. –Predictor Variables: Chronological Time - eg. weeks Time Lag Variables - PTR mean time to resolution Cumulative KLOC integrated - total of all integrations Significant activities - eg. component test, system test, etc. PTR Arrival = constant + f(Week, Week 2, Week 3, KLOC, KLOC 2, # Arrivals in preceding week, System Test) + e Example: Fig p. 235

21 Quality Management Models Reliability Growth Models –Models from previous products can be used to track the defect rates of the current product –To experience significant quality improvement, the current defect arrival rate must fall below the model curve –Comparing to a model allows for quality actions to be identified and implemented. –Models can be good for determining the end date of testing –Other models should be used in conjunction because the Reliability Growth Models do not focus on the front-end of the process

22 Quality Management Models Criteria for Model Evaluation –Most Important Criteria: Timeliness of quality indication –“Raises a red flag” –Allows more time to react and recover Scope of process coverage –Should address each phase –Should address quality of stage deliverables Capability –Ability for model to provide information for planning and managing software development

23 Quality Management Models In-Process Metrics and Reports –A defect tracking and reporting system and a set of related in-process metrics is an important element in implementing Quality Management Models. –Especially important for large projects. –Multiple teams –Metric data and feedback needs to be available to all levels, and all teams

24 Quality Management Models –Example - Using an Inspection Effort/Defect Rate report and matrix to access the rigor and effectiveness of the inspection process: Fig p. 240 and Fig p. 242 A tool to help measure process quality by tracking inspection effort and defects detected/removed

25 Quality Management Models –Example - Inspection Report showing defect origin Fig p. 245 “Among the defects found by this stage, what is the percentage that should have been found by the previous stage?” Does not require the total defect data like the DRE approach does. This approach can be used as an in-process quality management tool

26 Quality Management Models Orthogonal Defect Classification –Orthogonal - pertaining to, or composed of right angels (Webster’s II) –In context: A set of mutually independent cause categories –ODC - A method for in-process quality management based on defect cause (or type) analysis, where a distribution of defect types is associated with process phases

27 Quality Management Models Concept: By examining the distribution of defect types, one can tell which development phase the current project is at (or should be at). Eight Defect types: 1. Function (missing or incorrect functions) - Design Phase 2. Interface - low-level design 3. Checking - low-level design or code implementation

28 Quality Management Models Eight Defect types (cont.): 4. Assignment - Code Phase 5. Timing/Serialization - low-level design 6. Build/Package/Merge - library tools 7. Documentation - publications 8. Algorithms - low-level design

29 Quality Management Models Keys to ODC - look at the majority of defect types vs. where the project actually is. May be an indicator of an out of control condition. Defect Trigger - a condition that allows (forces) a defect to surface. Challenges for ODC: –What are defect triggers? –Can defects really be classified as orthogonal? –Overlap and interrelations between proposed defect types –Indirect method for accessing project progress

30 Quality Management Models Summary Quality Management Models, unlike reliability models, need to focus on: –Timeliness - for quality indications –Scope of coverage - all phases of development process –Capability - provides information (indicators and attributes) about quality Tracking and reporting systems and sets of related in-process metrics are needed to make Quality Management Models work. Defect cause analysis (eg. ODC) can lead to a better understanding of quality of the project.