Testing Metrics Software Reliability CSSE 376, Software Quality Assurance Rose-Hulman Institute of Technology April 5, 2007

Outline Testing Metrics An Important Metric: Reliability

Common Metrics Product Process KLOC Function Points #Bugs Staff hours thousands of lines of code need to remove comment lines? Function Points #Bugs Process Staff hours Tests planned Tests passed

Bug Density Measure #Bugs/KLOC Expect different densities at different stages of a project May categorize bugs by severity

Example Bug Density

Cartoon of the Day (1/3)

Using Bug Metrics Count bugs discovered during each phase of a project Compare to previous projects provides estimates of expected values at each phase -- could use to set milestone deviation of more than 20% from expected indicates need for investigation

Analysis of Bug Data Root cause analysis Trend analysis Search for explanations Might look at other process data (effort, experience of team, etc.) Trend analysis Make predictions from current data

Reliability

Failures vs. Faults Fault: Failures: developer-oriented 6 faults/1000 source lines Failures: customer-oriented 3 failures/1000 CPU hours

Calculating Reliability probability of failure-free operation for a specified time interval 0.82 for 8 CPU hours Failure Intensity number of observed failures within a specified time interval 3 failures/1000 CPU hours

Factors Influencing Reliability Fault removal by error correction (debugging) Fault introduction by error correction (unintended) by new feature development Operational profile

Operational Profile Probability of Use Functions

Example Function Usage Probability Distribution Interval Change 32% 0-31 Delete 14% 32-45 Insert 46% 46-91 Print 8% 92-99

Test Generation Test Random Numbers Test Cases 1 29, 11, 47, 52, 26, 94 C, C, I, I, C, P 2 62, 98, 39, 78, 82, 65 I, P, D, I, I, I 3 83, 32, 58, 41, 36, 17 I, D, I, D, D, C 4 36, 49, 96, 82, 20, 77 D, I, P, I, C, I

Test Compression Real use of a product involves repetitive operations different users invoke the same operations same user invokes the same operations on different days Redundant tests waste computer and personnel time Compression: when generating random tests, do not duplicate previous tests

Cartoon of the Day (2/3)

Cartoon of the Day (3/3)

Curve Fitting Reliability models focus on failure removal Use a random process to model the failure removal process

Execution Time Model Failure Intensity Execution Time Goal

Resource Constraints Early phase of a project Middle phase Late phase constrained by availability of developers (debuggers) Middle phase constrained by availability of testers Late phase constrained by availability of machines may run tests in parallel to increase number of tests per CPU hour

Adjusting for Calendar Time Estimate resource usage during each phase of the project Model calculates failure intensity in terms of execution time Model adjusts fault removal rate according to resource constraints

Calendar Time Component Constrained by debuggers Failure Intensity Constrained by testers Constrained by machines Goal Execution Time

Calculating CalendarTime/ExecutionTime Ratio 10 staff-hours to fix each failure 2 failures/CPU-hr That means it will take 10 * 2 = 20 staff-hrs per CPU-hr Suppose you have 5 developers Then you have 20 / 5 = 4 hrs per CPU-hr Each CPU-hr will take 4 calendar hours

Estimating Completion Establish a failure intensity objective Record execution times of failures Run model to estimate reliability Model reports estimated completion date Values are not absolute---within confidence bounds

Estimating Completion Failure Intensity Calendar Time Goal Ship Date

Acceptance Charts Bugs Reject Continue Testing Accept Time