1. www.ischool.drexel.edu INFO 631 Prof. Glenn Booker Week 1 – Defect Analysis and Removal 1INFO631 Week 1

2. www.ischool.drexel.edu INFO631 Week 12 Motivation Much of software measurement is devoted to improving the quality of the product To do so, it helps to understand how, when, and why mistakes are made during the software life cycle, resulting in defects It is assumed that defects are queued for being fixed (removed) after they are detected

3. www.ischool.drexel.edu INFO631 Week 13 Defect Analysis Overview Defect analysis looks at: –When a defect was created (“injected”) –When a defect was found (“detection”) –What caused a defect (type of defect, and/or orthogonal defect classification) –How a defect was found (“triggers”) Yes, there are often many terms for the same activity!

4. www.ischool.drexel.edu INFO631 Week 14 Defect Creation (Injection) Defects may be created, detected, and removed during every phase of the software life cycle, including during maintenance –Most are created during requirements analysis, design, and coding –Testing and maintenance create relatively few defects

5. www.ischool.drexel.edu INFO631 Week 15 Waterfall Life Cycle Requirements Analysis High Level Design Low Level (Detailed) Design Coding Unit Testing Component (Integration) Testing System Testing Maintenance

6. www.ischool.drexel.edu INFO631 Week 16 Defect Detection Defects may be found: –During quality assurance activities between phases of the life cycle (such as during major reviews and inspections); and often –Within each phase (during preliminary reviews, peer reviews and testing activities) Here we mostly consider the former (between- phase activities)

7. www.ischool.drexel.edu INFO631 Week 17 Defect Detection Defects are found (discovered) by –Reviews and formal inspections For requirements analysis, high level design, low level design, and coding –Testing Unit testing Integration testing System testing Acceptance testing

8. www.ischool.drexel.edu INFO631 Week 18 Origin of Defects & Type What kind of defects are made in each life cycle phase? –Requirements - incorrect specification; missing requirements –High level design - design does not cover all requirements, or is inflexible –Low level design - mismatch between HLD and LLD; design does not cover all requirements

9. www.ischool.drexel.edu INFO631 Week 19 Origin of Defects & Type –Coding - code errors –Integration – interface, compatibility problems –Unit testing - bad fixes –Component testing - bad fixes –System testing - bad fixes –Acceptance testing - bad fixes –Maintenance (enhancement) - bad requirements, design, coding, testing

10. www.ischool.drexel.edu INFO631 Week 110 Defect Causal Analysis Regardless of when defects are created or discovered, they may be analyzed to determine their cause –Causes may range from the mundane to the esoteric –Once identified, causes should be recorded to support defect prevention activities

11. www.ischool.drexel.edu INFO631 Week 111 Types of Defect Causes Some causes may include: –Unclear requirements –Incorrect architecture –Incomplete design –Unfamiliarity with programming language –Typos –Inadequate understanding of interfaces –Inadequate understanding of standards (e.g. TCP/IP, ODBC, SQL, etc.)

12. www.ischool.drexel.edu INFO631 Week 112 Types of Defect Causes and: –Poor documentation for legacy system –Design based on outdated requirements –Code written to outdated design –Poor design for future expansion (e.g. hardwired constants, buried assumptions, etc.) –Conflicting requirements –Bad fix of a previous problem –…and many more

13. www.ischool.drexel.edu INFO631 Week 113 Orthogonal Defect Classification ODC is a formal type of defect causal analysis Classify defects by the type of defect, and the life cycle phase when it was created This particular scheme is still somewhat experimental, but the basic concept is widely accepted

14. www.ischool.drexel.edu INFO631 Week 114 ODC - Defect Type A possible set of definitions: –Function error–affects capability, user, product or hardware interfaces, global data structures –Assignment error – errors with initialization of control blocks or data structure –Interface error – errors in interacting with other components, modules, or or device drivers –Checking – errors in validating data and values

15. www.ischool.drexel.edu INFO631 Week 115 ODC - Defect Type –Timing/serialization – errors in management of shared and real-time resources –Build/package/merge – errors due to mistakes in library systems, change management, or version control –Documentation – errors in publications and maintenance notes –Algorithm – errors regarding efficiency or correctness that affect the task

16. www.ischool.drexel.edu INFO631 Week 116 Orthogonal Defect Classification TypePhase Caused FunctionalDesign InterfaceLow Level Design (LLD) CheckingLLD or Coding AssignmentCoding TimingLLD Build/packageLibrary tools DocumentationPublications AlgorithmLLD

17. www.ischool.drexel.edu INFO631 Week 117 Defect Trigger Is a condition that allows a defect to surface (become visible); examples include: Design conformance Logic or data flow incomplete Workload or stress (performance) Boundary conditions (extremes) Bug fix Recovery timing User code

18. www.ischool.drexel.edu INFO631 Week 118 Defect Removal Modeling Defect removal is critical to reducing (development or maintenance) cycle time and cost, and improving quality Michael Fagan and Capers Jones are noteworthy authors See, for example, “Fagan Inspections”

19. www.ischool.drexel.edu INFO631 Week 119 Defect Removal Model (DRM) Software development phase-based DRM covers –Defect injection into a phase –Defect removal during a phase –Defect carryover across phases –Effectiveness of defect removal within a phase

20. www.ischool.drexel.edu INFO631 Week 120 Defect Removal Model DRM is a quality management tool Provides insights into the defect removal process and where this process might be improved Used after a project is completed; provides post mortem information about that project If defect removal process is similar for a new project, then the DRM can be used to improve the defect removal process of that new project

21. www.ischool.drexel.edu INFO631 Week 121 Defect Removal Model Defects are created (injected) by incorrect requirements gathering, analysis, design, coding, or by bad fixes –Any given defect can be removed in the phase in which it was injected or in a later phase Remove defect means fix bug, redo specs and/or design as needed

22. www.ischool.drexel.edu INFO631 Week 122 Defect Removal Effectiveness Phase based defect detection activities –Requirements analysis & inspection –Design inspection –Code inspection, Build verification testing –Unit, string, integration, regression, system testing –Fix verification inspection & testing

23. www.ischool.drexel.edu INFO631 Week 123 Defect Removal Model Applies to each life cycle phase Life Cycle Phase Defects Injected (new mistakes) Net Defects from previous Phase Defect Detection (inspection) Known Defects Undetected Defects Fixed Defects Unfixed Defects + Bad Fix Defects Net Defects Bad Fix Defects

24. www.ischool.drexel.edu INFO631 Week 124 Defect Removal Effectiveness For any given phase of defect removal Defects present at removal operation –Defects found during removal operation + defects found later –May be determined ex post facto (after the fact), or using a statistical prediction model Removal Effectiveness = Defects found by removal operation Defects present at removal operation ________________________________ X 100

25. www.ischool.drexel.edu INFO631 Week 125 Defect Removal Effectiveness Defect removal effectiveness = (# of defects found by inspection) / (# of defects originally present) *100 Early detection percentage = (# of major inspection errors) / (# of major and minor errors) * 100

26. www.ischool.drexel.edu INFO631 Week 126 Defect Matrix Assumptions Defects are removed in the same life cycle phase when they are found No defects are knowingly left unfixed No bad fixes –Or at least they are blended into the number of defects created in that life cycle phase

27. www.ischool.drexel.edu INFO631 Week 127 Sample Defect Matrix— When Originated vs. When Found When Originated (injected, or created) When Found/FixedWhen Found/Fixed 1 00

28. www.ischool.drexel.edu INFO631 Week 128 Defect Removal Effectiveness Life cycle Phase(s) Defects passed from Previous life cycle phases Defects Created this Phase Defects Found & removed this Phase Defects passed to Next life cycle phase Next = (Previous + Created) – Found DRE = Found / (Previous + Created) * 100

29. www.ischool.drexel.edu INFO631 Week 129 High Level Design Effectiveness There are no requirements defects removed; 122 defects are passed to HLD High (Top) Level Design (I0) Inspection Effectiveness –Defects found and removed at I0: 730 –Defects existing on step entry (escapes from requirements phase: 122 –Defects injected in current phase: 859 –E(I0) = 730/(122+859) x 100 = 74%

30. www.ischool.drexel.edu INFO631 Week 130 Low Level Design Effectiveness Low Level Design (I1) Inspection Effectiveness –Defects found and removed at I1: 729 –Defects existing on step entry (escapes from requirements phase and I0): 122+859-730 = 251 –Defects injected in current phase: 939 –E(I1) = 729/(251+939) x 100 = 61%

31. www.ischool.drexel.edu INFO631 Week 131 Code Inspection Effectiveness Code Inspection (I2) Effectiveness –Defects found and removed at I2: 1095 –Defects present on step entry (escapes from requirements phase, I0, and I1): 251+939-729 = 461 –Defects injected in current phase: 1537 –E(I2)= 1095/(461+1537) x 100 = 55%

32. www.ischool.drexel.edu INFO631 Week 132 Unit Testing Effectiveness Unit Testing (UT) Effectiveness –Defects found and removed at UT: 332 –Defects existing on step entry (escapes from previous phases): 461+1537-1095 = 903 –Defects injected in current phase (bad fixes): 2 –E(UT) = 332/(903+2) x 100 = 37% Can follow the same pattern for the other testing phases and post-release

33. www.ischool.drexel.edu INFO631 Week 133 Summary Effectiveness Measures Overall Design & Coding Inspection Effectiveness –IE = (730+729+1095)/(122+859+939+1537) x 100 = 74% Overall Effectiveness of all Testing activities –TE = (332+387+111)/(903+2+4+1)x100 = 91% Overall Defect Removal Effectiveness of the development process (not including release) –DRE = (0+730+729+1095+332+387+111)/(122+859+ 939+1537+2+4+1) x 100 = 3384/3464*100 = 97.7%

34. www.ischool.drexel.edu INFO631 Week 134 Rayleigh Model The Rayleigh Model describes the number of defects which will be discovered, by development phase It’s a special case of the Weibull family of distributions, which we’ll cover later

35. www.ischool.drexel.edu INFO631 Week 135 Rayleigh Model Assumptions Defect rate observed during development process is positively correlated with defect rate in field Given the same defect injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages, leading to fewer defects in the field

36. www.ischool.drexel.edu INFO631 Week 136 Rayleigh Model Inspections I0 = High Level Design Inspection I1 = Low Level Design Inspection I2 = Code Inspection UT = Unit Testing CT = Component (Integration) Testing ST = System Testing GA = after General Availability (release or Fielding of the system)

37. www.ischool.drexel.edu INFO631 Week 137 Rayleigh Model Number of Defects Development Phase I0 I1 I2 UT CT ST GA Each bar represents the number of defects found during that life cycle phase’s inspection activity

38. www.ischool.drexel.edu INFO631 Week 138 Desired Rayleigh Curve Trends Effect of Early Defect Removal and Reducing Error Injection

39. www.ischool.drexel.edu INFO631 Week 139 Cost of Defect Removal by Phase Defect removal and rework is less costly the closer that the defects are found relative to the phase in which they are injected –Rework in the I0, I1, and I2 inspection levels can be 10 to 100 times less expensive than if it is done during formal testing –Reviews can reduce number of defects reaching testing phases by factor of 10 These reductions cut testing costs by 50-80%, even including the review costs

40. www.ischool.drexel.edu INFO631 Week 140 Cost Effectiveness of DRM Relative cost of fixing a problem found in design/coding, testing, or after release are: 1:20:82 (Remus, 1983) 1:13:92 (Kan, 1989) Cost of defect removal can be analyzed by inspection type, testing phase, defect severity, defect origin, etc.