1. www.ischool.drexel.edu INFO 636 Software Engineering Process I Prof. Glenn Booker Week 9 – Quality Management 1INFO636 Week 9

2. www.ischool.drexel.edu INFO636 Week 92 Software Quality Management The PSP focuses on managing defects to produce a quality product This, in turn, helps improve productivity –Productivity still falls as the size of a system increases, due to its increasing complexity To improve further, must improve your process quality

3. www.ischool.drexel.edu INFO636 Week 93 What is Quality? Crosby considered software quality to be “conformance to requirements” Meeting user needs is often key to determine quality We can distinguish between the quality of the product created, and the quality of the process used to create it

4. www.ischool.drexel.edu INFO636 Week 94 Product Quality A software product must provide the functions defined in its requirements, and must be a functioning application –Beyond that, how sensitive the user is to other aspects of usability, reliability, compatibility, etc. varies wildly –Poor quality products tend to have a lot of focus on testing to find defects

5. www.ischool.drexel.edu INFO636 Week 95 Process Quality Process quality focuses on meeting the developer’s needs A quality process supports creating good products –Process consistency and repeatability are generally critical –Efficiency and usability are less obvious objectives

6. www.ischool.drexel.edu INFO636 Week 96 Cost of Quality Quality is a big cost issue for software Each defect has costs for –detecting it –finding the source –determining a solution –fixing the problem (req’ts, design, and/or code) –reviewing the fix –testing the fix to see if it works –testing the fix to check for regression problems –updating documentation to reflect the fix

7. www.ischool.drexel.edu INFO636 Week 97 Cost of Quality The relative cost of finding and correcting a defect increases through the life cycle –During requirements or design phases, a defect has a relative cost of about 1 –Coding and testing phases, cost 10-40 –After release to the customer, cost 100-1000

8. www.ischool.drexel.edu INFO636 Week 98 Cost of Quality Many of you also noted that the time to fix defects found in testing or later is much higher than those found earlier (see pp. 276-7) PSP experience has shown that reviews are twice as efficient at finding defects as testing

9. www.ischool.drexel.edu INFO636 Week 99 Testing Schedules Many organizations focus on getting into system testing quickly, as a sign of progress However, more time in requirements and design would produce higher quality code which needs less testing time

10. www.ischool.drexel.edu INFO636 Week 910 Cost of Defect Removal Typical developers inject (create) about 100 defects per thousand LOC –About half are found by compiler –The rest need to be found by inspection or testing Using the PSP, about 70% of defects entering inspection are found

11. www.ischool.drexel.edu INFO636 Week 911 Cost of Defect Removal Inspection and review average about ½ hour per defect, whereas testing averages 8-10 hours per defect – so the cost benefit of inspection becomes clear

12. www.ischool.drexel.edu INFO636 Week 912 An Example Consider a 50,000 LOC project –About 5000 defects are initially created, but 2500 are found by the compiler (and are presumably fixed immediately) –Without reviews or inspections, those 2500 defects would take at least 2500*8 = 20,000 hours to find and fix using testing alone

13. www.ischool.drexel.edu INFO636 Week 913 An Example –Using inspections, we would expect to find 1750 of those defects, and fix them in ½ hour each for 875 hours –The remaining 750 defects would be found by testing, and fixed in 750*8=6000 hours Total defect removal time with inspection and test is 6,875 hours, versus 20,000 hours for testing alone –That saves $1.3M at $100/hr

14. www.ischool.drexel.edu INFO636 Week 914 Cost of (Low) Quality Poor quality can be measured to determine its impact on the project –Failure cost = the cost of finding and repairing failures –Appraisal cost = the cost of conducting reviews (called), and testing which finds no defects –Prevention cost = the cost of preventing the failures from occurring

15. www.ischool.drexel.edu INFO636 Week 915 Cost of Quality (COQ) For the PSP we simplify these to –Failure cost = time for compile and test –Appraisal cost = time for reviews and inspections –Prevention costs are low, since we have few projects to examine for defect trends Based on this, define COQ measures

16. www.ischool.drexel.edu INFO636 Week 916 COQ Measures –Failure COQ = 100*(compile time + test time)/(total devel time) –Appraisal COQ = 100*(design review time + code review time)/(total devel time) –Total COQ = Failure COQ + Appraisal COQ –Appraisal cost as % of total quality = 100*(appraisal COQ)/(total COQ) –A/FR ratio = Appraisal COQ/(Failure COQ)

17. www.ischool.drexel.edu INFO636 Week 917 COQ Measures The purpose of these measures is to help look for trends in quality over several projects –Hope to see higher ‘appraisal cost as % of total quality’ over time –This should lead to lower defect rates for later projects

18. www.ischool.drexel.edu INFO636 Week 918 Quality Strategy Important to keep product and process measures separate – both are important Avoid productivity measures by themselves – they can help with planning, and be compared to other measures, but don’t use them as a sole objective

19. www.ischool.drexel.edu INFO636 Week 919 Process Yield Total process yield is helpful – the percent of defects removed before test You can also evaluate yield for every life cycle phase, but only after hidden defects have been later discovered The life cycle is a filtering process –Each phase has some kind of step to help find and remove defects to keep from feeding them into the next phase

20. www.ischool.drexel.edu INFO636 Week 920 No One Answer The PSP is a personal process for many reasons – one is that there is no one best way to find and remove defects –Need to look for the best ways for you to improve your work –Then reevaluate your strategy periodically

21. www.ischool.drexel.edu INFO636 Week 921 Benchmarking It can be helpful to compare your product and process measures to other people’s, or to industry standards –For example, plot the A/FR ratio over time It should increase (p. 288) –Or look at A/FR versus the yield –Or Test defects per kLOC versus A/FR

22. www.ischool.drexel.edu INFO636 Week 922 Benchmarking Productivity can be compared to other measures, such as yield, to look for trends Yield acts like the filter part of the life cycle process –Each step’s yield reduces the number of defects to be found later, so a small improvement in yield can be significant

23. www.ischool.drexel.edu INFO636 Week 923 Injection and Removal The key to understanding your software development process is to balance the injection of new defects with activities to detect and remove them To achieve greater control, need to examine the factors which cause defects to occur

24. www.ischool.drexel.edu INFO636 Week 924 Defect Causes Education – didn’t know how to do it Communication – weren’t informed about something Oversight – forgot to do something Transcription – knew what to do, but made a mistake Process – the process misled your actions

25. www.ischool.drexel.edu INFO636 Week 925 Yield Management While the overall process yield doesn’t care if you have 80% review yield, and 40% test yield, or 40% review and 80% test, the costs and A/FR ratio of those situations are quite different This is why so much emphasis is placed on the quality of early reviews (see recommendations on pp. 298-300)

26. www.ischool.drexel.edu INFO636 Week 926 Defect Prevention Finding and fixing defects is purely defensive To be more proactive, need to prevent defects from being created Analyze the types of defects which occur frequently in your experience Determine the causes of common defect types (see slide 24 again)

27. www.ischool.drexel.edu INFO636 Week 927 Defect Prevention Then look for ways to prevent those defects from being made again, e.g. –Improve review checklists –Get additional instruction –Conduct more detailed design –Get independent reviews And change your processes accordingly

28. www.ischool.drexel.edu INFO636 Week 928 Defect Prevention The critical follow-up is to re-examine your defects later, and see if your changes helped or not Incidentally, defect prevention and continuous process improvement are the highest levels of process maturity –CMM and CMMI level 5 activities