Copyright USC-CSSE 1 Quality Management – Lessons of COQUALMO (COnstructive QUALity MOdel) A Software Defect Density Prediction Model AWBrown and Sunita Chulani, Ph.D. Presented by Marilee Wheaton

Copyright USC-CSSE 2 Outline Behavioral Underpinnings Hidden Factory Defect Types COQUALMO Framework The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) COQUALMO Integrated with COCOMO II

Copyright USC-CSSE 3 USC Modeling Methodology

Copyright USC-CSSE 4 Delphi Assessment Ask each expert the range for driver –Apply personal experience, –Look at completed projects, –Guess (WAG), Collect and share in a meeting: discuss why/how different people made their estimate Repeat until no changes Final values (for each parameter) Max=(Hmax + 4*AVEmax + Lmax)/6

Copyright USC-CSSE 5 Adding Project Data Effort Adjustment Multipliers (typical) –Linear Regression

Copyright USC-CSSE 6 Fig 11., Pg 170 SwCEwCII a posteiori Baysian Update Used to combine expert judgement with sampled data: spread of datasets weights combination

Copyright USC-CSSE 7 Outline Model Framework *The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) COQUALMO Integrated with COCOMO II

Copyright USC-CSSE 8 COQUALMO Model Framework

Copyright USC-CSSE 9 Software Devel. Process(es) (cont.) : the hidden factory

Copyright USC-CSSE 10 Outline Model Framework *The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) COQUALMO Integrated with COCOMO II

Copyright USC-CSSE 11 The Defect Introduction (DI) Sub-Model Software Size estimate Software product, process, computer and personnel attributes (subset of COCOMO II factors) Defect Introduction Sub-Model Number of non-trivial requirements, design and code defects introduced

Copyright USC-CSSE 12 A-Priori Expert-Judgment Based Code DI Ranges

Copyright USC-CSSE 13 DI Model Equations For each artifact j, Quality Adjustment Factor (QAF) Estimated Number of Defects Introduced =

Copyright USC-CSSE 14 Initial Data Analysis on the DI Model DIR = Defect Introduction Rate

Copyright USC-CSSE 15 Outline Model Framework The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results *The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) COQUALMO Integrated with COCOMO II

Copyright USC-CSSE 16 Defect Removal Sub-Model Defect removal activity levels Number of non-trivial requirements, design and coding defects introduced Number of residual defects/ unit of size Software Size Estimate The Defect Removal (DR) Sub-Model

Copyright USC-CSSE 17 Defect Removal Profiles 3 relatively orthogonal profiles –Automated Analysis –People Reviews –Execution Testing and Tools Each profile has 6 levels –Very Low, Low, Nominal, High, Very High, Extra High Very Low--removes the least number of defects Extra High--removes the most defects

Copyright USC-CSSE 18 Automated Analysis

Copyright USC-CSSE 19 Static [Module-Level Code] Analysis "Static code analysis is the analysis of computer software that is performed without actually executing programs built from that software (analysis performed on executing programs is known as dynamic analysis). In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code. "*softwaredynamic analysissource codeobject code *

Copyright USC-CSSE 20 Static [Module-Level Code] Analysis SWEBOK [sans references] "4.2. Quality Analysis and Evaluation Techniques Various tools and techniques can help ensure a software design’s quality. "Software design reviews : informal or semiformal, often group-based, techniques to verify and ensure the quality of design artifacts (for example, architecture reviews, design reviews and inspections, scenario-based techniques, requirements tracing) "Static analysis : formal or semiformal static (nonexecutable) analysis that can be used to evaluate a design (for example, fault-tree analysis or automated cross-checking) "Simulation and prototyping : dynamic techniques to evaluate a design (for example, performance simulation or feasibility prototype)" * Software Engineering Body of Knowledge. Alain Abran, et al., Swebok_Ironman_June_23_ 2004.pdf, pg 54

Copyright USC-CSSE 21 Peer Reviews

Copyright USC-CSSE 22 Syntactic Versus Semantic Checking Both sentences below are semantically correct, only one is semantically correct. A panda enters the bar, eats shoots and leaves. A panda enters the bar, eats, shoots and leaves.

Copyright USC-CSSE 23 Execution Testing and Tools

Copyright USC-CSSE 24 Technique Selection Guidance “Under specified conditions, …” Peer reviews are more effective than functional testing for faults of omission and incorrect specification (UMD, USC) Functional testing is more effective than reviews for faults concerning numerical approximations and control flow (UMD, USC)

Copyright USC-CSSE 25 Residual Defects Equation Estimated Number of Residual Defects DRes Est, j = Estimated No. of Residual Defects for the j th artifact C j = Calibration Constant for the j th artifact DI Est, j = Estimated No. of Defects Introduced for the j th artifact (output of DI Sub-Model) i = Defect Removal profile DRF ij = Defect Removal Fraction 3 i=1

Copyright USC-CSSE 26 Defect Densities from Expert- Judgment Calibrated COQUALMO

Copyright USC-CSSE 27 Validation of Defect Densities Average defect density using Jones’ data weighted by CMM maturity level distribution of 542 organizations is 13.9 defects/kSLOC Average defect density using COQUALMO is 14.3 defects/kSLOC Residual defect density from Jones’ data (.135* * *18.3) = 13.9 defects/kSLOC

Copyright USC-CSSE 28 An Independent Validation Study Aim: To validate expert-determined COQUALMO has correct trends in defect rates Sample Project: Size = kSLOC Actual Defect Density = 6 defects/kSLOC

Copyright USC-CSSE 29 Outline Model Framework The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) *COQUALMO Integrated with COCOMO II

Copyright USC-CSSE 30 Integrated COQUALMO