1. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 1 4/19/2003 Day 4, Part 4 Software Quality Assurance

2. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 2 4/19/2003 Outline Introduction Software Quality Quality Control Quality Assurance Cost of Quality Analysis Summary

3. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 3 4/19/2003 Introduction

4. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 4 4/19/2003 SW Quality Assurance Typical Symptoms of a Problem We have a procedure to prevent that! Why did it happen? The procedure was not followed. Nobody even knew about it

5. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 5 4/19/2003 Software Quality Assurance Symptoms of Another Problem They’ve violated company policy and we are facing a lawsuit! Did they know about the policy? Who authorized what they did?

6. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 6 4/19/2003 Other Typical Symptoms of Quality Problems Upper management was sure embarrassed when the product failed at the customer demo Didn’t anyone tell them it wasn’t tested?

7. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 7 4/19/2003 Other Typical Symptoms of Quality Problems The customers are complaining that it doesn’t work as advertised. But we tested it for three weeks!

8. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 8 4/19/2003 Poor Quality Costs Money In a world-wide marketplace, higher quality means more profit Higher quality can mean lower cost (see advanced course) The costs associated with poor quality are often hard to quantify –Loss of customers –Low pride of workmanship –...

9. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 9 4/19/2003 Motivation of the Software Developer Pride in Workmanship and Skill Fear of Failure Quality Software

10. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 10 4/19/2003 Software Quality Assurance Purposes: –To provide management and developers with visibility into the process being followed and the products being built So they can manage the outcome –To provide a quality control mechanism

11. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 11 4/19/2003 Software Quality Assurance Typical Practices: –Reviews –Audits –Communication –Measurements –Independent confirmation of compliance Standards, requirements, procedures

12. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 12 4/19/2003 What Is Software Quality?

13. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 13 4/19/2003 There Are Many Definitions Crosby: Quality is Conformance to Requirements Juran: Quality is Fitness for Intended Use Webster: –(1) Quality is that which makes something what it is –(2) Quality is the Degree of Excellence Weinberg: Quality is value to someone

14. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 14 4/19/2003 The Essential Characteristics The product does what it is supposed to do The customer gets what they paid for

15. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 15 4/19/2003 External vs. Internal Quality External Quality: –Is determined by the factors whose presence or absence may be observed by the users –Is used as the ultimate criterion to judge the quality of a system Internal Quality: –Factors invisible to the end users –Help achieve the external quality

16. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 16 4/19/2003 External Quality Factors Correctness –Does the system conform to its specs? Robustness –Does the software system react appropriately to abnormal conditions? Extendibility –Ease of adapting the software product to changes in its specifications

17. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 17 4/19/2003 External Quality (continued) Compatibility –The ease with which the system can be combined with other systems Efficiency –The ability of the software to put as little demand on hardware as possible Portability –The ease of transferring software products to different hardware/software platforms

18. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 18 4/19/2003 External Quality (continued) Functionality –The extent of possibilities supported by the software system Timeliness & Economy –The ability of the software to be cost effectively developed and released in a timely manner

19. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 19 4/19/2003 External Quality (continued) Ease of use –The ease with which people with different backgrounds can learn to use the software and then effectively apply it to their benefit Verifiability –How easily and cost effectively can the correctness of the system be judged?

20. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 20 4/19/2003 Internal Quality Factors Modularity –The degree to which elements of the software are independent from each other Readability –How hard is it to read and understand the source code? Reusability –The ability of the elements of the software to be used in later systems

21. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 21 4/19/2003 Internal and External Quality Factors Some of the external quality factors have an equally strong impact on the internal quality –Timeliness & Economy –Compatibility –Portability –Extendibility

22. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 22 4/19/2003 Evolution of Software Quality Quality Control Quality Assurance Quality Engineering Today Introduced Here, Details in Next Session

23. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 23 4/19/2003 Quality Control

24. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 24 4/19/2003 Quality Control (QC) Goal: Keep Quality at an “acceptable level” by rejecting unacceptable products Requirements DevelopmentQC Inspection Pass Fail Standards of Quality

25. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 25 4/19/2003 Software Quality Control Criteria for QC Inspection –How many requirements are met? –How many tests have passed / failed? Problems (maybe) Unique to Software –Are the tests adequate? –Do the fixes inject more errors? –Developers may intentionally fail to test or avoid testing, to save time or because they are overconfident

26. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 26 4/19/2003 Testing

27. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 27 4/19/2003 Why Do We Test Software? To find defects in the software And, ultimately, to convince ourselves that most of the defects have been removed (when tests do not uncover defects) It is very easy in the stress of a project to focus on the second objective instead of the first.

28. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 28 4/19/2003 Sample Of An Ineffective Software Test This is based on a hardware test technique Why is it ineffective? “Run the software for 7 days and see whether it fails” Class discussion

29. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 29 4/19/2003 What Would Effective Software Tests Look Like? Test the different ways the software might fail Test under different operating conditions Test all of the requirements –Stated –Unstated Reflect actual customer use Others?... Class discussion

30. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 30 4/19/2003 What Would Effective Software Tests Do For Us? Provide programmers with information they can use to prevent errors –Where the errors happen –Where the errors come from Give management information it needs to rationally assess the risk in releasing the software Validate the software –Make sure it does what is should do

31. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 31 4/19/2003 Testing Strategies A good testing strategy provides a systematic way to select and generate tests to detect the faults that are most serious or most likely –Determines what needs to be tested –Decides whether to base tests on specs (Black Box Testing) or the structure of the software(Glass Box Testing) –Determines what test data should be used

32. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 32 4/19/2003 Testing Vocabulary Failure A particular case where the system does not meet its specification Fault or Defect or Bug The problem in the system that may cause a failure Error A conceptual mistake in the mind of the programmer that causes faults The mapping may not be 1-1

33. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 33 4/19/2003 Black Box (Behavioral) Testing Testing that verifies whether the system behaves in accordance with its specifications No information is needed about the structure of the system to generate test cases Typically, one creates a test case to cover each requirement in the system specification

34. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 34 4/19/2003 Characteristics of Black Box Testing Tends to find bugs that directly affect the end-users Particularly suitable for larger systems –where it is impossible to test every possible way the software could fail Tends to miss defects that stem from special cases or complex conditions

35. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 35 4/19/2003 Glass Box (Structural) Testing Verifies the system behavior against its design Generally suitable to test small portions of the system, where the testers understand the software’s structure Typically you develop test cases for each path through the software, each potential value of each parameter, etc. Knowing the structure you know what to test

36. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 36 4/19/2003 Comparing The Two Types of Testing Both techniques are applicable to testing at any level Each technique tends to have limitations that the other can compensate for The techniques used for quality control purposes are often hybrid

37. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 37 4/19/2003 Testing Techniques Different testing techniques may be used for either black box or white box testing, depending on the nature of the specification –Control Flow Testing –Loop Testing –Data Flow Testing –Transaction Flow Testing –Syntax Testing –Finite State Testing

38. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 38 4/19/2003 The Nature of Bugs (Faults) - I Unit/Component Bugs –In principle, these should not escape the unit test process If unit testing is performed –The cost of out-of-phase detection is generally much more than in-phase detection –But these kinds of bugs can be detected earlier, via code reviews and inspections

39. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 39 4/19/2003 The Nature of Bugs (Faults) - II Integration Bugs –Arise due to wrong assumptions about interfaces or interaction between otherwise correct components –The more the design is coupled (i.e., the less modular it is), the harder it is to find the cause of integration bugs –These bugs can be reduced by good interface design and effective management of interface changes

40. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 40 4/19/2003 The Nature of Bugs (Faults) - III System Testing –Qualitative in nature –Detects problems where all correctly cooperating components fail to work together to achieve a system specification The limited capability of humans to handle complexity is the major cause of bugs. A good software process can reduce bugs significantly The limited capability of humans to handle complexity is the major cause of bugs. A good software process can reduce bugs significantly

41. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 41 4/19/2003 Two Approaches to System Testing Coverage based testing –The test cases are generated to cover the whole system’s specifications with equal rigor Usage based testing –The test cases focus on more frequently used features –Ensures better reliability –Gains in schedule time

42. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 42 4/19/2003 DevelopTest ? Errors Injected Errors Reduced How Many Errors are Left? Quality Control is Not Enough Testing can detect the presence of errors, but not their absence

43. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 43 4/19/2003 The Fundamental Problem of Quality Control... It does not improve anything

44. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 44 4/19/2003 Other Problems with Quality Control Adversarial relationship between QC and developers Weak motive to improve Little help with improvement –You know it is defective, but not why

45. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 45 4/19/2003 Quality Assurance

46. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 46 4/19/2003 Quality Assurance Goal: To assure that quality is achieved In addition to the product, quality assurance also focuses on the process Ensures that any inadequacies in process & the standards are identified and brought to management’s attention

47. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 47 4/19/2003 Quality Assurance Looks at the Entire Process Requirements DevelopmentQC Inspection Pass Fail Standards of Quality Process and Design Standards

48. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 48 4/19/2003 Each Step of the Process Has an Effect on Final Defect Rate Process Step I = Defects Input O = Defects Output F = Defects Found & Fixed C = Defects Created O = I + C - F

49. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 49 4/19/2003 Typical Responsibilities of QA Review all development plans for completeness Participate as inspection moderators in design and code inspections Review a significant sample of all test results to determine adherence to standards Periodically audit SCM performance to determine adherence to standards

50. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 50 4/19/2003 At Each Step, QA must... Inspect the product for –Conformance with standards –Satisfaction of requirements Evaluate the process for –Conformance with standards –Opportunities to improve –Process defects that may cause problems later on Evaluate the support processes as well

51. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 51 4/19/2003 At Each Step, QA must... Monitor reviews, inspections, walkthroughs, etc. to see that they are accomplishing their objectives Trace back to prior steps when defects are found In addition to evaluating the development process, the QA procedures must also be tailored and revised for the needs of each new project

52. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 52 4/19/2003 Trace software requirements or specs to original system or customer requirements Inspect or evaluate software requirements against standards to see that they are –Complete –Correct –Consistent –Testable & –Understandable PROGRAM SAMPLE (IN1,IN2) INTEGAR IN1,IN2,A,B,C; FOR I = 1 STEP 3 UNTIL 99 DO IF IN1 < A*B THEN IN2 := C + A*B ELSE IN2 := C + IN1 ENDIF ENDDO ** THE NEXT PART WILL SORT DO FOREVER Ii := I + 1 TEMP := ARRAY[I] - ARRAY[I+1] IF TEMP < 0, SWAP (ARRAY, i) ENDLOOP CALL CHECKORDER(ARRAY) ** NEXT WE DO THE i/o CALL PRINT(ARRAY) CALL DEBUG (ARRAY, I) CASE I OF CALL SUB1 During Requirements Analysis...

53. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 53 4/19/2003 “Clean Room” Software Development Technique Motive: don’t let the bugs get in Method: filter them out from the beginning -- Define the requirements using a formal notation –Reduces ambiguity –Some consistency/correctness issues can be checked automatically -- Carry out rigid inspections Benefits: fewer problems later in the development process (e.g., reduced testing).

54. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 54 4/19/2003 Trace the design to the requirements specification (+ “derived requirements”) Evaluate the design against standards –Complexity –Cohesion –Understandability, etc. Evaluation of design effectiveness, correctness, consistency, completeness Evaluate plans for testing and test cases During Design...

55. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 55 4/19/2003 Trace code to design specifications Evaluate code against coding standards Code walkthroughs or inspections to check on coding mistakes Test coverage analysis –Do the tests address all requirements? (black box tests) –Do the tests adequately evaluate the design and its implementation? (white box tests) During the Coding Step...

56. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 56 4/19/2003 Test procedures and code analysis –Evaluate test code like any other code –Are the procedures documented? –Are the expected results documented? Test results analysis –Were the tests properly executed? –Were the results correct? At Module Testing...

57. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 57 4/19/2003 Two Philosophies of Testing Code Test a Little Review Thorough Test CodeReview Thorough Test

58. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 58 4/19/2003 Make sure that regression tests are performed –Retest all modules potentially affected by any change Do independent tests –Someone other than the person who developed the software Acceptance Tests for formal qualification During Integration...

59. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 59 4/19/2003 When do you Perform the Formal Qualification Test? Hardware Development Operating System Development Software Development System Integration System FQT Software FQT: Here or Here

60. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 60 4/19/2003 Planning for QA Each development project should have a documented Software Quality Assurance Plan (SQAP) –May be part of SW Development Plan The SQAP documents tasks to be performed, standards of verification, procedures and organizational structure There is an IEEE standard for SQAPs

61. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 61 4/19/2003 Pitfalls in implementing QA Lack of sufficiently skilled staff in QA –Software professionals are typically more interested in development jobs than QA QA management cannot effectively negotiate with the development team Senior Management commitment fades under schedule pressures –The developers stop caring about QA & action revolves around insignificant issues

62. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 62 4/19/2003 Pitfalls in Implementing QA (continued) QA organization operates without suitably documented and approved development standards and procedures Development team does not produce verifiable quality plans –Eventually results in a debate over which bugs are more important to fix rather than whether the product has sufficient quality or not

63. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 63 4/19/2003 Pitfalls in Implementing QA (continued) Disputes over who has responsibility for product quality and for performing QA functions –The software developers and managers are responsible for product quality –Quality assurance is a method to help them achieve quality through independent evaluation of their work (like a coach)

64. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 64 4/19/2003 Reporting Chain for QA Staff Except in the highest maturity organizations, it is strongly recommended that the QA staff have a separate reporting chain from the project team –This improves their objectivity –And assures that disputes are resolved at a level in the organization that has responsibility for the possible consequences

65. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 65 4/19/2003 Benefits Achieved by Quality Assurance over Quality Control Significant insight is gained into the process that can be used to gain long term benefits –Development can focus on specific weak points in the quality –Process improvements can be identified –Steps that are difficult to perform can be automated

66. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 66 4/19/2003 Problems with Quality Assurance Can still be adversarial (although perhaps less than with quality control) Motivation to fix the process is still weak Can be more costly than quality control It can be hard to show benefits until long after the product has been shipped

67. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 67 4/19/2003 Quality Engineering Goal: Build quality in as part of the software engineering process A philosophical change that relies on the professional pride of the software engineer Engineering staff define and execute the quality assurance tasks Team approach to quality, with rewards based on quality Quality professionals are more like coaches and educators than evaluators

68. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 68 4/19/2003 Quality Engineering Focuses on People As Well As Process Finding errors is good -- it keeps them from leaking through to the customer Problems result in process changes, not punishment of people Everyone appreciates that a competitive process is the way to remain a competitor Measurements are used so that decisions are based on fact (in addition to intuition) Independent tests are welcome -- they tell you if you are as good as you think you are

69. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 69 4/19/2003 Cost of Quality Analysis

70. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 70 4/19/2003 Quality (Fewer Defects; Customer Satisfaction) Productivity Cycle Time Customer Value Quality Improvements Cost Money How can we justify them?

71. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 71 4/19/2003 Managers and Technical Staff must be convinced that Quality problems are serious Poor quality costs them money Quality is worth fixing Quality can be fixed by proper techniques In Order to Justify Quality Improvements...

72. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 72 4/19/2003 The Fundamental Prejudice Quality improvement is a non-value- added activity –It costs overhead resources –The benefits are not necessarily real So we seek to avoid it Cost of Quality (COQ) Analysis is used to address this prejudice

73. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 73 4/19/2003 The First Question This is what management and technical staff will typically ask when approached with a proposal to improve quality “What does it cost to improve quality?”

74. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 74 4/19/2003 The Right Questions “What’s the payoff?” “What is the return on investment for quality improvements?

75. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 75 4/19/2003 Categorizing Quality-Related Costs Quality Related Costs Cost of Conformance Cost of Non- Conformance Prevention Costs Appraisal Costs Internal Failure Costs External Failure Costs

76. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 76 4/19/2003 Cost of Conformance Things that improve quality –Prevention costs - prevent poor quality Predictive metrics, training, root cause analysis, process improvement –Appraisal costs - detect poor quality Tests, inspections, audits

77. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 77 4/19/2003 Cost of Non-Conformance The price you pay when you fail to achieve quality –Internal failures Costs incurred before product shipment –External Failures Costs incurred after product shipment

78. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 78 4/19/2003 Root Cause Analysis Predictive Metrics Preventive Measures Invest in Prevention A Contemporary View of the Way to Achieve Quality Prevention Costs Evaluation and Appraisal Costs Failure Costs Invest Here to Save Here Tests Reviews Inspections Audits Evaluations Failures in the Field Loss of Customers Support Costs

79. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 79 4/19/2003 Example of Prevention During the Design Process: What process error resulted in this product problem? Inspections Audits Reviews } Defects in Design Root Cause Analysis Process Improvements Defects in Process Is the Product OK? Is the Process OK?

80. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 80 4/19/2003 Effects of Maturity on Costs SEI Maturity Level Cost as a Percent of Development Cost 10 20 30 40 50 60 1 2 34 5 Prevention Appraisal Internal Failures External Failures Total COQ As reported by Knox (see references)

81. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 81 4/19/2003 Analyzing the Cost of Quality A Process is –A collection of tasks, –Carried out in a particular order, –Producing particular artifacts. Each task can be analyzed for its value produced and its costs Alternatively, you can look at the costs and value of each artifact produced by the process

82. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 82 4/19/2003 Net Cost of Quality For a task whose purpose is to improve quality: Total Cost of Performing the Task minus Total Savings equals Net Cost of Quality negative Goal: net cost should be negative

83. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 83 4/19/2003 Analyzing the Net Cost of a Process 1) Identify and list all steps or tasks of the process –Document Each -- purpose, description, procedures, etc. –Include everything you do. Best if you watch and record actual behavior of practitioners.

84. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 84 4/19/2003 Analyzing the Net Cost of a Process 2) For each task, do a basic value added analysis: –Divide all tasks into Value Added and Non-Value Added –A task is “value added” if it meets three criteria: Changes the product The change is wanted by the customer It is done right the first time (is not rework)

85. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 85 4/19/2003 Analyzing the Net Cost of a Process 3) Divide into Five Categories: –Costs of Performing the Process (Value Added) –Prevention Costs (non value added, cost of quality) –Appraisal or Evaluation Costs (non value added, cost of quality) –Failure Response Costs (non value added, cost of quality) –Everything else - (unrelated costs, non- value added, non cost of quality)

86. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 86 4/19/2003 Categorizing Tasks and Subtasks Cost of Non Conformance FailureAppraisalPrevention Cost of Conformance Non Cost of QualityCost of Quality (all non-value-added) Design Development Fabrication Documentation Assembly Process Creation Upgrade Shipping Training Planning Simulation Modeling Consulting Qualifying Certifying Inspection Testing Audits Monitoring Measure- ment Verification Analysis Rework Service Modification Expediting Recall Correction Retest Error Analysis Non- Value Added Value Added Over- head Errors Ineff- icien- cies

87. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 87 4/19/2003 Analyzing the Net Cost of a Process 4) Determine Costs –Determine the cost in labor or other units for each task –Sum for all tasks

88. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 88 4/19/2003 Example - Cost of a Process Non- Value Added Cost of ConformanceCost of Non Conformance FailureAppraisalPreventionNon Cost of QualityCost of Quality TOTAL 600 100 450 95 300 450 400 275 200 150 3170 100% Program Design Design Reviews Code Code Reviews Unit Test Integration Test System Test Configuration Mgt Problem Solving Rework Retest 540 400 940 30% 60 120 4% 40 35 300 450 400 275 1500 45% 200 150 500 15% Value Added 60 50 110 3.5%

89. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 89 4/19/2003 What if You Don’t Know? Often at this point you have tasks you do not know how to handle: –Don’t know the cost –Don’t know if value added –Don’t know the true impact of the task Which leads us to step 5 - Defining a metrics program to help with this analysis

90. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 90 4/19/2003 Analyzing the Net Cost of a Process 5) For each task, determine: –Customer requirements (input & output) –Supplier requirements (input & output) –Measures that determine if requirements are met –Measures that determine why or why not –Measures that determine the cost of the task

91. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 91 4/19/2003 Analyzing the Net Cost of a Process 6) For each task: –Analyze the measurement requirements Is it measured now? Can it be measured? Should it be measured? –Determine the level of effort or cost to measure

92. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 92 4/19/2003 You Will Use the Measures to Determine... What tasks cost the most What tasks are worth improving What tasks should be eliminated

93. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 93 4/19/2003 Analyzing the Net Cost of a Process 7) Look for opportunities to improve –Tasks that are not value added but have high cost –Costs of non-conformance

94. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 94 4/19/2003 Comments The more mature the process, the more tasks in the prevention column and the fewer in the appraisal and failure columns Your process maturity may have a lot to do with what techniques work and what techniques do not work

95. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 95 4/19/2003 Comments (continued) By tracking these costs and looking at the impact of process changes, you can determine the cost and the benefit of changes Thus you can optimize the process

96. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 96 4/19/2003 More Comments If you track current costs –You open your eyes to the cost of non conformance If you track the effect of changes –You learn what works to improve your process

97. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 97 4/19/2003 Summary Quality Control focuses on the end product Quality Assurance focuses on the process Quality Engineering integrates with the Software development process Quality costs can be categorized into conformance and non-conformance Cost of Quality Analysis helps you evaluate the cost and benefit of quality improvement activities

98. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 98 4/19/2003 References Beizer, Boris, 1995, Black Box Testing. Humphrey, Watts, 1989, Managing the Software Process Knox, 1993, Raytheon studies reported by Houston and Keats, Software Quality Matters, vol 5, no 1 (Spring, 1997), U. of Texas SW Quality Institute Meyer, Bertrand, 1997, Object Oriented Software Construction

99. Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 99 4/19/2003 References Musa, John D, 1992, “The Operational Profile,” in Software Reliability Engineering: An overview. Weinberg, Gerald M., 1992, Quality Software Management, Volume 1, Systems Thinking, Dorset House, New York, ISBN 0- 932633-22-6.