1. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 SMU CSE 8314 Software Measurement and Quality Engineering Module 13 Software Reliability Models - Part 1

2. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 2 Contents  Seeding and Tagging  Weibull Distribution  Rayleigh Distribution  IEEE SW Reliability Standard  Summary

3. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 Seeding and Tagging

4. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 4 Seeding and Tagging  Introduce a given number of errors into the software -- say E of them  Run standard tests, detecting D of them  Compute D/E = % of errors detected  Suppose D 2 = number of other errors detected  Then you assume the total number of errors in the software is D 2 *E/D

5. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 5 Example of Seeding and Tagging  200 defects found so far  Inject 20 defects  Find 12 of them  Therefore, assume total defects = 200 * 20 / 12 = 4000 / 12 = 333 => 333 - 200 = 133 defects remaining By performing this analysis from time to time, you can estimate your defect density over time.

6. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 Distributions Other than Exponential Weibull Distribution Rayleigh Distribution

7. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 7 Weibull Distribution  (shape parameter) allows reliability to increase or to decrease makes it equal to the exponential distribution (t)  =   t  -1 This is a useful model for trying to fit lots of different data sets, because it allows both increases and decreases.

8. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 8 Rayleigh Distribution: Same as Weibull with  = 2 (t)  =  t  -2 Some researchers have found that this distribution fits certain software cases.

9. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 IEEE Reliability Standards

10. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 10 The Real Need A way to manage reliability throughout the life cycle of the software  This is the basis for the IEEE software reliability standard: “IEEE STD 982.1 Software Reliability Metrics” & IEEE Guide 982.2 (also see: IEEE STD 730-1984 Software Quality Assurance)

11. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 11 IEEE Software Reliability Standard 39 Measures applicability (1) Dobbins, James H., “SW Reliability Management,” in Handbook of SW Quality Assurance, Chapter 19.

12. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 12 IEEE Standard Reliability Measures  Following are available for each measure –Description of use –Identification & definition of primitives –How it is implemented –How results are to be integrated –Special considerations –Special training or experience required –Specific example –Summary of benefits –Experience history –Published References

13. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 13 Factors Contributing to Selection of IEEE Measures  Ease of collection of primitive data  Relationship between results & reliability  Ease of interpretation of results  Usefulness of results in management of the aspect being measured  Need for measurements in each aspect of each life cycle phase  Ease of implementation  Cost of implementation

14. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 14 Three Basic Phases Predict Reliability Estimate Reliability Measure Reliability Start Coding Release Software Requirements Design Code Test Maintain Support

15. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 Some of the IEEE Measures Unless otherwise specified, all of these are measured during development to help you assess, predict, estimate and/or control defect levels in your software.

16. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 16 Fault Density Total Faults per 1000 Lines of Code Used to –Predict remaining faults –Assess testing sufficiency –Establish historical data You can also track this after releasing the software

17. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 17 Data Used to Measure Fault Density i = failure number n i = number of faults per failure ----- N T =  n i = total faults found n f = fault density

18. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 18 Fault Density Formula or n f = N T / KSLOC n f =  n i / KSLOC

19. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 19 Graph of Fault Density

20. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 20 Use of Fault Density  You can use fault density to estimate how close you are to finding all of the faults, and thus make decisions about whether to release software to the next development phase (or to the customer)  After software release, you can continue to measure (to see how well you estimated defects remaining)

21. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 21 Additional Data Used to Categorize Faults d i = date of failure S i = severity of failure CL i = class of failure C i = type of fault These can help you determine which faults to focus on You may measure density separately for different classes or types or severity levels

22. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 22 Faults Detected (per time period)  This can give you some idea of how well you are finding defects and thus how many defects are left in the software

23. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 23 Faults Detected Per Week

24. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 24 Defect Density (pre-release)  This is similar to fault density, but is usually normalized by the size of the software  It tells you how your software compares with other software, so you have an idea of whether yours meets your norms or expectations You can also track this after releasing the software

25. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 25 Defect Density Definitions (pre-release)  A defect is a problem found during an inspection of the design, requirements, code, etc. i = inspection number D i = total defects found in ith inspection n = number of inspections  Defect density is the total defects, normalized by the size of the software

26. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 26 Defect Density Equation (pre-release) n DD =  D i / KSLOC i=1

27. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 27 Using Defect Density  To use defect density, you should track how many you find before you release the product to the next phase  Over time, you can determine typical behaviors and thus meet goals or predict defect levels

28. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 28 Defect Density Requirements Analysis

29. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 29 Defect Density Software Design

30. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 30 Defect Density Code and Unit Test

31. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 31 Requirements Traceability Goals: –Identify missing requirements –Identify features that are not required  Also known as “gold plating” –Measure progress in design and coding phases

32. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 32 Requirements Traceability Equation RT = R1 / R2 * 100% Data needed to compute this: R1 = number of requirements traceable to specific design or code elements R2 = total number of requirements

33. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 33 Requirements Traceability Variants and Usage Additional variants: –Trace to test cases –Trace from code/design/tests to requirements Usage notes: –This type of measure can be hard to keep up with and of limited utility if requirements change frequently. It works best if requirements are stable and defined.

34. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 34 Defect Index (for each phase) Goal: –Provide an estimate of the relative correctness of the software Primitive Data: i = phase number N i = no of defects found in phase i Further Refinement: S i = # of serious defects found in phase i M i = # of medium defects found in phase i T i = # of trivial defects found in phase i

35. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 35 Weighting for Defect Importance W S (typically 10) = weighting factor for serious defects W M (typically 3) = weighting factor for medium defects W T (typically 1) = weighting factor for trivial defects

36. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 36 Phase Defect Index W S * S i W M * M i W T * T i Pi =______ + ______ + ______ N i N i N i

37. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 37 Use of Phase Defect Index  This indicates the defect level of each phase  It can be used to see if a given phase is generating too many defects and, if so, whether they are important ones

38. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 38 Overall Defect Index  i * P i DI =______ KSLOC

39. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 39 Software Maturity Index Goal: –Help determine if we are ready for delivery Primitive Data: M T = # of functions in current release F c = # of functions changed since prior release F a = # of functions added since prior release F d = # of functions deleted from prior release

40. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 40 Software Maturity Measure No 1: –For the most recently delivered software, count F c –For the next release, count F a, F d, M T M T - (F a + F c + F d ) MI =_______________ M T

41. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 41 Software Maturity Measure No 2: M T - F c SMI =_______ M T

42. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 42 Notes on IEEE Measures  These are indicators, not absolute measures  They must be calibrated to your data in order to be of most use  Despite the fact that they are over twenty years old, not very many practicing software organizations have used them –Cost –Not invented here –Fear of measurement

43. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 43 Summary  Simple models such as “seed and test” can give some concept of reliability  More complex distribution models may give better results if they match the behavior of the specific type of software  IEEE measures represent one set of possible metrics for estimating and managing reliability

44. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 44 References  IEEE STD 982.1, Software Reliability Metrics & IEEE Guide 982.2. New York, Institute of Electrical and Electronics Engineers, Inc.  Lyu, Michael R., Handbook of Software Reliability Engineering, IEEE, 1996, Catalog # RS00030. ISBN 0-07-039400-8.

45. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 45 END OF MODULE 13