Presentation is loading. Please wait.

Presentation is loading. Please wait.

Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 1 8/8/2004 Day 4, Part 4 Software Quality Assurance.

Similar presentations


Presentation on theme: "Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 1 8/8/2004 Day 4, Part 4 Software Quality Assurance."— Presentation transcript:

1 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 1 8/8/2004 Day 4, Part 4 Software Quality Assurance

2 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 2 8/8/2004 Outline Introduction Software Quality Quality Control Quality Assurance Cost of Quality Analysis Summary

3 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 3 8/8/2004 Introduction

4 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 4 8/8/2004 SW Quality Assurance Typical Symptoms of a Problem We have a procedure to prevent that! Why did it happen? The procedure was not followed. Nobody even knew about it

5 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 5 8/8/2004 Software Quality Assurance Symptoms of Another Problem They’ve violated company policy and we are facing a lawsuit! Did they know about the policy? Who authorized what they did?

6 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 6 8/8/2004 Other Typical Symptoms of Quality Problems Upper management was sure embarrassed when the product failed at the customer demo Didn’t anyone tell them it wasn’t tested?

7 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 7 8/8/2004 Other Typical Symptoms of Quality Problems The customers are complaining that it doesn’t work as advertised. But we tested it for three weeks!

8 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 8 8/8/2004 Poor Quality Costs Money In a world-wide marketplace, higher quality means more profit Higher quality can mean lower cost (see advanced course) The costs associated with poor quality are often hard to quantify –Loss of customers –Low pride of workmanship –...

9 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 9 8/8/2004 Motivation of the Software Developer Pride in Workmanship and Skill Fear of Failure Quality Software

10 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 10 8/8/2004 Software Quality Assurance Purposes: –To provide management and developers with visibility into the process being followed and the products being built So they can manage the outcome –To provide a quality control mechanism

11 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 11 8/8/2004 Software Quality Assurance Typical Practices: –Reviews –Audits –Communication –Measurements –Independent confirmation of compliance Standards, requirements, procedures

12 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 12 8/8/2004 What Is Software Quality?

13 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 13 8/8/2004 There Are Many Definitions Crosby: Quality is Conformance to Requirements Juran: Quality is Fitness for Intended Use Webster: –(1) Quality is that which makes something what it is –(2) Quality is the Degree of Excellence Weinberg: Quality is value to someone

14 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 14 8/8/2004 The Essential Characteristics The product does what it is supposed to do The customer gets what they paid for

15 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 15 8/8/2004 External vs. Internal Quality External Quality: –Is determined by the factors whose presence or absence may be observed by the users –Is used as the ultimate criterion to judge the quality of a system Internal Quality: –Factors invisible to the end users –Help achieve the external quality

16 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 16 8/8/2004 External Quality Factors Correctness –Does the system conform to its specs? Robustness –Does the software system react appropriately to abnormal conditions? Extendibility –Ease of adapting the software product to changes in its specifications

17 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 17 8/8/2004 External Quality (continued) Compatibility –The ease with which the system can be combined with other systems Efficiency –The ability of the software to put as little demand on hardware as possible Portability –The ease of transferring software products to different hardware/software platforms

18 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 18 8/8/2004 External Quality (continued) Functionality –The extent of possibilities supported by the software system Timeliness & Economy –The ability of the software to be cost effectively developed and released in a timely manner

19 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 19 8/8/2004 External Quality (continued) Ease of use –The ease with which people with different backgrounds can learn to use the software and then effectively apply it to their benefit Verifiability –How easily and cost effectively can the correctness of the system be judged?

20 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 20 8/8/2004 Internal Quality Factors Modularity –The degree to which elements of the software are independent from each other Readability –How hard is it to read and understand the source code? Reusability –The ability of the elements of the software to be used in later systems

21 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 21 8/8/2004 Internal and External Quality Factors Some of the external quality factors have an equally strong impact on the internal quality –Timeliness & Economy –Compatibility –Portability –Extendibility

22 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 22 8/8/2004 Evolution of Software Quality Quality Control Quality Assurance Quality Engineering Today Introduced Here, Details in Next Session

23 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 23 8/8/2004 Quality Control

24 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 24 8/8/2004 Quality Control (QC) Goal: Keep Quality at an “acceptable level” by rejecting unacceptable products Requirements DevelopmentQC Inspection Pass Fail Standards of Quality

25 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 25 8/8/2004 Software Quality Control Criteria for QC Inspection –How many requirements are met? –How many tests have passed / failed? Problems (maybe) Unique to Software –Are the tests adequate? –Do the fixes inject more errors? –Developers may intentionally fail to test or avoid testing, to save time or because they are overconfident

26 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 26 8/8/2004 Testing

27 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 27 8/8/2004 Why Do We Test Software? To find defects in the software And, ultimately, to convince ourselves that most of the defects have been removed (when tests do not uncover defects) It is very easy in the stress of a project to focus on the second objective instead of the first.

28 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 28 8/8/2004 Sample Of An Ineffective Software Test This is based on a hardware test technique Why is it ineffective? “Run the software for 7 days and see whether it fails” Class discussion

29 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 29 8/8/2004 What Would Effective Software Tests Look Like? Test the different ways the software might fail Test under different operating conditions Test all of the requirements –Stated –Unstated Reflect actual customer use Others?... Class discussion

30 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 30 8/8/2004 What Would Effective Software Tests Do For Us? Provide programmers with information they can use to prevent errors –Where the errors happen –Where the errors come from Give management information it needs to rationally assess the risk in releasing the software Validate the software –Make sure it does what is should do

31 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 31 8/8/2004 Testing Strategies A good testing strategy provides a systematic way to select and generate tests to detect the faults that are most serious or most likely –Determines what needs to be tested –Decides whether to base tests on specs (Black Box Testing) or the structure of the software(Glass Box Testing) –Determines what test data should be used

32 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 32 8/8/2004 Testing Vocabulary Failure A particular case where the system does not meet its specification Fault or Defect or Bug The problem in the system that may cause a failure Error A conceptual mistake in the mind of the programmer that causes faults The mapping may not be 1-1

33 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 33 8/8/2004 Black Box (Behavioral) Testing Testing that verifies whether the system behaves in accordance with its specifications No information is needed about the structure of the system to generate test cases Typically, one creates a test case to cover each requirement in the system specification

34 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 34 8/8/2004 Characteristics of Black Box Testing Tends to find bugs that directly affect the end-users Particularly suitable for larger systems –where it is impossible to test every possible way the software could fail Tends to miss defects that stem from special cases or complex conditions

35 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 35 8/8/2004 Glass Box (Structural) Testing Verifies the system behavior against its design Generally suitable to test small portions of the system, where the testers understand the software’s structure Typically you develop test cases for each path through the software, each potential value of each parameter, etc. Knowing the structure you know what to test

36 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 36 8/8/2004 Comparing The Two Types of Testing Both techniques are applicable to testing at any level Each technique tends to have limitations that the other can compensate for The techniques used for quality control purposes are often hybrid

37 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 37 8/8/2004 Testing Techniques Different testing techniques may be used for either black box or white box testing, depending on the nature of the specification –Control Flow Testing –Loop Testing –Data Flow Testing –Transaction Flow Testing –Syntax Testing –Finite State Testing

38 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 38 8/8/2004 The Nature of Bugs (Faults) - I Unit/Component Bugs –In principle, these should not escape the unit test process If unit testing is performed –The cost of out-of-phase detection is generally much more than in-phase detection –But these kinds of bugs can be detected earlier, via code reviews and inspections

39 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 39 8/8/2004 The Nature of Bugs (Faults) - II Integration Bugs –Arise due to wrong assumptions about interfaces or interaction between otherwise correct components –The more the design is coupled (i.e., the less modular it is), the harder it is to find the cause of integration bugs –These bugs can be reduced by good interface design and effective management of interface changes

40 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 40 8/8/2004 The Nature of Bugs (Faults) - III System Testing –Qualitative in nature –Detects problems where all correctly cooperating components fail to work together to achieve a system specification The limited capability of humans to handle complexity is the major cause of bugs. A good software process can reduce bugs significantly The limited capability of humans to handle complexity is the major cause of bugs. A good software process can reduce bugs significantly

41 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 41 8/8/2004 Two Approaches to System Testing Coverage based testing –The test cases are generated to cover the whole system’s specifications with equal rigor Usage based testing –The test cases focus on more frequently used features –Ensures better reliability –Gains in schedule time

42 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 42 8/8/2004 DevelopTest ? Errors Injected Errors Reduced How Many Errors are Left? Quality Control is Not Enough Testing can detect the presence of errors, but not their absence

43 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 43 8/8/2004 The Fundamental Problem of Quality Control... It does not improve anything

44 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 44 8/8/2004 Other Problems with Quality Control Adversarial relationship between QC and developers Weak motive to improve Little help with improvement –You know it is defective, but not why

45 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 45 8/8/2004 Quality Assurance

46 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 46 8/8/2004 Quality Assurance Goal: To assure that quality is achieved In addition to the product, quality assurance also focuses on the process Ensures that any inadequacies in process & the standards are identified and brought to management’s attention

47 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 47 8/8/2004 Quality Assurance Looks at the Entire Process Requirements DevelopmentQC Inspection Pass Fail Standards of Quality Process and Design Standards

48 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 48 8/8/2004 Each Step of the Process Has an Effect on Final Defect Rate Process Step I = Defects Input O = Defects Output F = Defects Found & Fixed C = Defects Created O = I + C - F

49 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 49 8/8/2004 Typical Responsibilities of QA Review all development plans for completeness Participate as inspection moderators in design and code inspections Review a significant sample of all test results to determine adherence to standards Periodically audit SCM performance to determine adherence to standards

50 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 50 8/8/2004 At Each Step, QA must... Inspect the product for –Conformance with standards –Satisfaction of requirements Evaluate the process for –Conformance with standards –Opportunities to improve –Process defects that may cause problems later on Evaluate the support processes as well

51 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 51 8/8/2004 At Each Step, QA must... Monitor reviews, inspections, walkthroughs, etc. to see that they are accomplishing their objectives Trace back to prior steps when defects are found In addition to evaluating the development process, the QA procedures must also be tailored and revised for the needs of each new project

52 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 52 8/8/2004 Trace software requirements or specs to original system or customer requirements Inspect or evaluate software requirements against standards to see that they are –Complete –Correct –Consistent –Testable & –Understandable PROGRAM SAMPLE (IN1,IN2) INTEGAR IN1,IN2,A,B,C; FOR I = 1 STEP 3 UNTIL 99 DO IF IN1 < A*B THEN IN2 := C + A*B ELSE IN2 := C + IN1 ENDIF ENDDO ** THE NEXT PART WILL SORT DO FOREVER Ii := I + 1 TEMP := ARRAY[I] - ARRAY[I+1] IF TEMP < 0, SWAP (ARRAY, i) ENDLOOP CALL CHECKORDER(ARRAY) ** NEXT WE DO THE i/o CALL PRINT(ARRAY) CALL DEBUG (ARRAY, I) CASE I OF CALL SUB1 During Requirements Analysis...

53 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 53 8/8/2004 “Clean Room” Software Development Technique Motive: don’t let the bugs get in Method: filter them out from the beginning -- Define the requirements using a formal notation –Reduces ambiguity –Some consistency/correctness issues can be checked automatically -- Carry out rigid inspections Benefits: fewer problems later in the development process (e.g., reduced testing).

54 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 54 8/8/2004 Trace the design to the requirements specification (+ “derived requirements”) Evaluate the design against standards –Complexity –Cohesion –Understandability, etc. Evaluation of design effectiveness, correctness, consistency, completeness Evaluate plans for testing and test cases During Design...

55 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 55 8/8/2004 Trace code to design specifications Evaluate code against coding standards Code walkthroughs or inspections to check on coding mistakes Test coverage analysis –Do the tests address all requirements? (black box tests) –Do the tests adequately evaluate the design and its implementation? (white box tests) During the Coding Step...

56 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 56 8/8/2004 Test procedures and code analysis –Evaluate test code like any other code –Are the procedures documented? –Are the expected results documented? Test results analysis –Were the tests properly executed? –Were the results correct? At Module Testing...

57 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 57 8/8/2004 Two Philosophies of Testing Code Test a Little Review Thorough Test CodeReview Thorough Test

58 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 58 8/8/2004 Make sure that regression tests are performed –Retest all modules potentially affected by any change Do independent tests –Someone other than the person who developed the software Acceptance Tests for formal qualification During Integration...

59 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 59 8/8/2004 When do you Perform the Formal Qualification Test? Hardware Development Operating System Development Software Development System Integration System FQT Software FQT: Here or Here

60 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 60 8/8/2004 Planning for QA Each development project should have a documented Software Quality Assurance Plan (SQAP) –May be part of SW Development Plan The SQAP documents tasks to be performed, standards of verification, procedures and organizational structure There is an IEEE standard for SQAPs

61 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 61 8/8/2004 Pitfalls in implementing QA Lack of sufficiently skilled staff in QA –Software professionals are typically more interested in development jobs than QA QA management cannot effectively negotiate with the development team Senior Management commitment fades under schedule pressures –The developers stop caring about QA & action revolves around insignificant issues

62 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 62 8/8/2004 Pitfalls in Implementing QA (continued) QA organization operates without suitably documented and approved development standards and procedures Development team does not produce verifiable quality plans –Eventually results in a debate over which bugs are more important to fix rather than whether the product has sufficient quality or not

63 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 63 8/8/2004 Pitfalls in Implementing QA (continued) Disputes over who has responsibility for product quality and for performing QA functions –The software developers and managers are responsible for product quality –Quality assurance is a method to help them achieve quality through independent evaluation of their work (like a coach)

64 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 64 8/8/2004 Reporting Chain for QA Staff Except in the highest maturity organizations, it is strongly recommended that the QA staff have a separate reporting chain from the project team –This improves their objectivity –And assures that disputes are resolved at a level in the organization that has responsibility for the possible consequences

65 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 65 8/8/2004 Benefits Achieved by Quality Assurance over Quality Control Significant insight is gained into the process that can be used to gain long term benefits –Development can focus on specific weak points in the quality –Process improvements can be identified –Steps that are difficult to perform can be automated

66 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 66 8/8/2004 Problems with Quality Assurance Can still be adversarial (although perhaps less than with quality control) Motivation to fix the process is still weak Can be more costly than quality control It can be hard to show benefits until long after the product has been shipped

67 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 67 8/8/2004 Quality Engineering Goal: Build quality in as part of the software engineering process A philosophical change that relies on the professional pride of the software engineer Engineering staff define and execute the quality assurance tasks Team approach to quality, with rewards based on quality Quality professionals are more like coaches and educators than evaluators

68 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 68 8/8/2004 Quality Engineering Focuses on People As Well As Process Finding errors is good -- it keeps them from leaking through to the customer Problems result in process changes, not punishment of people Everyone appreciates that a competitive process is the way to remain a competitor Measurements are used so that decisions are based on fact (in addition to intuition) Independent tests are welcome -- they tell you if you are as good as you think you are

69 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 69 8/8/2004 Cost of Quality Analysis

70 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 70 8/8/2004 Quality (Fewer Defects; Customer Satisfaction) Productivity Cycle Time Customer Value Quality Improvements Cost Money How can we justify them?

71 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 71 8/8/2004 Managers and Technical Staff must be convinced that Quality problems are serious Poor quality costs them money Quality is worth fixing Quality can be fixed by proper techniques In Order to Justify Quality Improvements...

72 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 72 8/8/2004 The Fundamental Prejudice Quality improvement is a non-value- added activity –It costs overhead resources –The benefits are not necessarily real So we seek to avoid it Cost of Quality (COQ) Analysis is used to address this prejudice

73 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 73 8/8/2004 The First Question This is what management and technical staff will typically ask when approached with a proposal to improve quality “What does it cost to improve quality?”

74 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 74 8/8/2004 The Right Questions “What’s the payoff?” “What is the return on investment for quality improvements?

75 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 75 8/8/2004 Categorizing Quality-Related Costs Quality Related Costs Cost of Conformance Cost of Non- Conformance Prevention Costs Appraisal Costs Internal Failure Costs External Failure Costs

76 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 76 8/8/2004 Cost of Conformance Things that improve quality –Prevention costs - prevent poor quality Predictive metrics, training, root cause analysis, process improvement –Appraisal costs - detect poor quality Tests, inspections, audits

77 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 77 8/8/2004 Cost of Non-Conformance The price you pay when you fail to achieve quality –Internal failures Costs incurred before product shipment –External Failures Costs incurred after product shipment

78 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 78 8/8/2004 Root Cause Analysis Predictive Metrics Preventive Measures Invest in Prevention A Contemporary View of the Way to Achieve Quality Prevention Costs Evaluation and Appraisal Costs Failure Costs Invest Here to Save Here Tests Reviews Inspections Audits Evaluations Failures in the Field Loss of Customers Support Costs

79 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 79 8/8/2004 Example of Prevention During the Design Process: What process error resulted in this product problem? Inspections Audits Reviews } Defects in Design Root Cause Analysis Process Improvements Defects in Process Is the Product OK? Is the Process OK?

80 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 80 8/8/2004 Effects of Maturity on Costs SEI Maturity Level Cost as a Percent of Development Cost 10 20 30 40 50 60 1 2 34 5 Prevention Appraisal Internal Failures External Failures Total COQ As reported by Knox (see references)

81 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 81 8/8/2004 Analyzing the Cost of Quality A Process is –A collection of tasks, –Carried out in a particular order, –Producing particular artifacts. Each task can be analyzed for its value produced and its costs Alternatively, you can look at the costs and value of each artifact produced by the process

82 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 82 8/8/2004 Net Cost of Quality For a task whose purpose is to improve quality: Total Cost of Performing the Task minus Total Savings equals Net Cost of Quality negative Goal: net cost should be negative

83 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 83 8/8/2004 Analyzing the Net Cost of a Process 1) Identify and list all steps or tasks of the process –Document Each -- purpose, description, procedures, etc. –Include everything you do. Best if you watch and record actual behavior of practitioners.

84 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 84 8/8/2004 Analyzing the Net Cost of a Process 2) For each task, do a basic value added analysis: –Divide all tasks into Value Added and Non-Value Added –A task is “value added” if it meets three criteria: Changes the product The change is wanted by the customer It is done right the first time (is not rework)

85 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 85 8/8/2004 Analyzing the Net Cost of a Process 3) Divide into Five Categories: –Costs of Performing the Process (Value Added) –Prevention Costs (non value added, cost of quality) –Appraisal or Evaluation Costs (non value added, cost of quality) –Failure Response Costs (non value added, cost of quality) –Everything else - (unrelated costs, non- value added, non cost of quality)

86 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 86 8/8/2004 Categorizing Tasks and Subtasks Cost of Non Conformance FailureAppraisalPrevention Cost of Conformance Non Cost of QualityCost of Quality (all non-value-added) Design Development Fabrication Documentation Assembly Process Creation Upgrade Shipping Training Planning Simulation Modeling Consulting Qualifying Certifying Inspection Testing Audits Monitoring Measure- ment Verification Analysis Rework Service Modification Expediting Recall Correction Retest Error Analysis Non- Value Added Value Added Over- head Errors Ineff- icien- cies

87 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 87 8/8/2004 Analyzing the Net Cost of a Process 4) Determine Costs –Determine the cost in labor or other units for each task –Sum for all tasks

88 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 88 8/8/2004 Example - Cost of a Process Non- Value Added Cost of ConformanceCost of Non Conformance FailureAppraisalPreventionNon Cost of QualityCost of Quality TOTAL 600 100 450 95 300 450 400 275 200 150 3170 100% Program Design Design Reviews Code Code Reviews Unit Test Integration Test System Test Configuration Mgt Problem Solving Rework Retest 540 400 940 30% 60 120 4% 40 35 300 450 400 275 1500 45% 200 150 500 15% Value Added 60 50 110 3.5%

89 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 89 8/8/2004 What if You Don’t Know? Often at this point you have tasks you do not know how to handle: –Don’t know the cost –Don’t know if value added –Don’t know the true impact of the task Which leads us to step 5 - Defining a metrics program to help with this analysis

90 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 90 8/8/2004 Analyzing the Net Cost of a Process 5) For each task, determine: –Customer requirements (input & output) –Supplier requirements (input & output) –Measures that determine if requirements are met –Measures that determine why or why not –Measures that determine the cost of the task

91 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 91 8/8/2004 Analyzing the Net Cost of a Process 6) For each task: –Analyze the measurement requirements Is it measured now? Can it be measured? Should it be measured? –Determine the level of effort or cost to measure

92 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 92 8/8/2004 You Will Use the Measures to Determine... What tasks cost the most What tasks are worth improving What tasks should be eliminated

93 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 93 8/8/2004 Analyzing the Net Cost of a Process 7) Look for opportunities to improve –Tasks that are not value added but have high cost –Costs of non-conformance

94 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 94 8/8/2004 Comments The more mature the process, the more tasks in the prevention column and the fewer in the appraisal and failure columns Your process maturity may have a lot to do with what techniques work and what techniques do not work

95 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 95 8/8/2004 Comments (continued) By tracking these costs and looking at the impact of process changes, you can determine the cost and the benefit of changes Thus you can optimize the process

96 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 96 8/8/2004 More Comments If you track current costs –You open your eyes to the cost of non conformance If you track the effect of changes –You learn what works to improve your process

97 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 97 8/8/2004 Summary Quality Control focuses on the end product Quality Assurance focuses on the process Quality Engineering integrates with the Software development process Quality costs can be categorized into conformance and non-conformance Cost of Quality Analysis helps you evaluate the cost and benefit of quality improvement activities

98 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 98 8/8/2004 References Beizer, Boris, 1995, Black Box Testing. Humphrey, Watts, 1989, Managing the Software Process Knox, 1993, Raytheon studies reported by Houston and Keats, Software Quality Matters, vol 5, no 1 (Spring, 1997), U. of Texas SW Quality Institute Meyer, Bertrand, 1997, Object Oriented Software Construction

99 Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 99 8/8/2004 References Musa, John D, 1992, “The Operational Profile,” in Software Reliability Engineering: An overview. Weinberg, Gerald M., 1992, Quality Software Management, Volume 1, Systems Thinking, Dorset House, New York, ISBN 0- 932633-22-6.


Download ppt "Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 1, Page 1 8/8/2004 Day 4, Part 4 Software Quality Assurance."

Similar presentations


Ads by Google