1 Exponential Distribution and Reliability Growth Models Kan Ch 8 Steve Chenoweth, RHIT Right: Wait – I always thought “exponential growth” was like this!

Slides:



Advertisements
Similar presentations
Part II – TIME SERIES ANALYSIS C3 Exponential Smoothing Methods © Angel A. Juan & Carles Serrat - UPC 2007/2008.
Advertisements

Empirical Evaluation of Defect Projection Models for Widely-deployed Production Software Systems FSE 2004 Paul Li, Mary Shaw, Jim Herbsleb Institute for.
1 Defect Removal Effectiveness Kan ch 6 Steve Chenoweth, RHIT Left – Some defects are easier to remove than others. This is the cruise ship Costa Concordia,
SE 450 Software Processes & Product Metrics 1 Introduction to Quality Engineering.
G. Alonso, D. Kossmann Systems Group
HSRP 734: Advanced Statistical Methods July 24, 2008.
1 In-Process Metrics for Software Testing Kan Ch 10 Steve Chenoweth, RHIT Left – In materials testing, the goal always is to break it! That’s how you know.
COURSE: JUST 3900 INTRODUCTORY STATISTICS FOR CRIMINAL JUSTICE Instructor: Dr. John J. Kerbs, Associate Professor Joint Ph.D. in Social Work and Sociology.
Model Assessment, Selection and Averaging
1 Software Architecture CSSE 477: Week 5, Day 1 Statistical Modeling to Achieve Maintainability Steve Chenoweth Office Phone: (812) Cell: (937)
1 Software Reliability Growth Models Incorporating Fault Dependency with Various Debugging Time Lags Chin-Yu Huang, Chu-Ti Lin, Sy-Yen Kuo, Michael R.
SE 450 Software Processes & Product Metrics Reliability: An Introduction.
Swami NatarajanJune 17, 2015 RIT Software Engineering Reliability Engineering.
Introduction to Quality Engineering
SE 450 Software Processes & Product Metrics Software Metrics Overview.
1 Fundamentals of Reliability Engineering and Applications Dr. E. A. Elsayed Department of Industrial and Systems Engineering Rutgers University
Swami NatarajanJuly 14, 2015 RIT Software Engineering Reliability: Introduction.
Today Concepts underlying inferential statistics
1 Software Quality Metrics Ch 4 in Kan Steve Chenoweth, RHIT What do you measure?
1 Quality Management Models Kan Ch 9 Steve Chenoweth, RHIT Right – To keep in mind – For Kan, this is part of Total Quality Management.
WINTER 2012IE 368. FACILITY DESIGN AND OPERATIONS MANAGEMENT 1 IE 368: FACILITY DESIGN AND OPERATIONS MANAGEMENT Lecture Notes #3 Production System Design.
1 CHAPTER M4 Cost Behavior © 2007 Pearson Custom Publishing.
Elec471 Embedded Computer Systems Chapter 4, Probability and Statistics By Prof. Tim Johnson, PE Wentworth Institute of Technology Boston, MA Theory and.
Overview Software Quality Assurance Reliability and Availability
Kan Ch 7 Steve Chenoweth, RHIT
Item Response Theory. What’s wrong with the old approach? Classical test theory –Sample dependent –Parallel test form issue Comparing examinee scores.
Diane Stockton Trend analysis. Introduction Why do we want to look at trends over time? –To see how things have changed What is the information used for?
Software faults & reliability Presented by: Presented by: Pooja Jain Pooja Jain.
Pop Quiz How does fix response time and fix quality impact Customer Satisfaction? What is a Risk Exposure calculation? What’s a Scatter Diagram and why.
Software Reliability Model Deterministic Models Probabilistic Models Halstead’s software metric McCabe’s cyclomatic complexity metrix Error seeding Failure.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 24 Slide 1 Critical Systems Validation 1.
INFO 631 Prof. Glenn Booker Week 2 – Reliability Models and Customer Satisfaction 1INFO631 Week 2.
Inference for regression - Simple linear regression
1 1 Slide © 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole.
1 Measurement Theory Ch 3 in Kan Steve Chenoweth, RHIT.
Handouts Software Testing and Quality Assurance Theory and Practice Chapter 15 Software Reliability
1SAS 03/ GSFC/SATC- NSWC-DD System and Software Reliability Dolores R. Wallace SRS Technologies Software Assurance Technology Center
Managing Software Projects Analysis and Evaluation of Data - Reliable, Accurate, and Valid Data - Distribution of Data - Centrality and Dispersion - Data.
Dr. Richard Young Optronic Laboratories, Inc..  Uncertainty budgets are a growing requirement of measurements.  Multiple measurements are generally.
Verification & Validation
Inference for a Single Population Proportion (p).
Software Reliability SEG3202 N. El Kadri.
Modeling and Simulation CS 313
1 POP Quiz T/F Defect Removal Effectiveness and Defect Removal Models are not true Predictive Models Define DRE What is a Checklist? What is it for? What.
Value of information Marko Tainio Decision analysis and Risk Management course in Kuopio
Chapter 19: Quality Models and Measurements  Types of Quality Assessment Models  Data Requirements and Measurement  Comparing Quality Assessment Models.
Ch 9.5: Predator-Prey Systems In Section 9.4 we discussed a model of two species that interact by competing for a common food supply or other natural resource.
1 Pop Quiz What is an Orthogonal Defect? Which is more precise: predictive models or management models? Why? Essay: Why are metrics and models an important.
1 Chapter Two: Sampling Methods §know the reasons of sampling §use the table of random numbers §perform Simple Random, Systematic, Stratified, Cluster,
1. 2 Traditional Income Statement LO1: Prepare a contribution margin income statement.
Software Reliability Research Pankaj Jalote Professor, CSE, IIT Kanpur, India.
Chapter 10 Verification and Validation of Simulation Models
Statistics : Statistical Inference Krishna.V.Palem Kenneth and Audrey Kennedy Professor of Computing Department of Computer Science, Rice University 1.
Methods for Estimating Defects Catherine V. Stringfellow Mathematics and Computer Science Department New Mexico Highlands University October 20, 2000.
Sampling and estimation Petter Mostad
1 Software Quality Engineering. 2 Quality Management Models –Tools for helping to monitor and manage the quality of software when it is under development.
CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 1 SMU CSE 8314 /
Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 SMU CSE 8314 Software Measurement.
Main Title Slide Software Reliability Estimates/ Projections, Cumulative & Instantaneous - Dave Dwyer With help from: Ann Marie Neufelder, John D. Musa,
Chapter 7: The Distribution of Sample Means
Statistics for Business and Economics Module 1:Probability Theory and Statistical Inference Spring 2010 Lecture 4: Estimating parameters with confidence.
Chapter 15 Forecasting. Forecasting Methods n Forecasting methods can be classified as qualitative or quantitative. n Such methods are appropriate when.
Demand Forecasting Production and Operations Management Judit Uzonyi-Kecskés Ph.D. Student Department of Management and Corporate Economics Budapest University.
© Chinese University, CSE Dept. Software Engineering / Software Engineering Topic 1: Software Engineering: A Preview Your Name: ____________________.
Software Metrics and Reliability
Software Reliability Definition: The probability of failure-free operation of the software for a specified period of time in a specified environment.
Chapter 10 Verification and Validation of Simulation Models
Software Reliability Models.
Critical Systems Validation
Exponential Smoothing
Presentation transcript:

1 Exponential Distribution and Reliability Growth Models Kan Ch 8 Steve Chenoweth, RHIT Right: Wait – I always thought “exponential growth” was like this!

2 Reliability growth models Exponential … and more Rayleigh tries to model the whole lifecycle. These models, in contrast, are for formal testing phases. Worth doing because “Defect arrival or failure patterns during such testing are good indicators of the product’s reliability when it is used by customers.” During formal testing, the software gets more stable, and “reliability grows.”

3 What do we mean by “reliability”? More general than often used for large systems, where – – Reliability = MTBF, and – A “failure” is like a system crash. Instead, here “reliability” = absence of all kinds of measurable faults, including: – Little things not working, and – Features not meeting specs.

4 What is our “Exponential model”?

5 And, it works… Lots of places where this model is known to predict failures over time. Can estimate defect arrival rates or time between failures. Kan used for AS/400 studies. Easy to set up. A good place to start in modeling testing.

6 Kan’s AS/400 results

7 More complex models Try to fix problems with exponential model. Generally more complex to set up. May be harder to get their additional data.

8 Jelinski-Moranda model A time between failures model Assumes – – N software faults at start of testing. – Failures occur purely at random. – All faults contribute equally to cause a failure. – Fix time is negligible. – Fix for each failure is perfect. Thus, software product’s failure rate improves by the same amount at each fix.

9 Jelinski-Moranda model, cntd So, the “hazard function” – The instantaneous failure rate function – At time t i, which is between failures (i-1) and I, is – Z(t i ) =  [N – (i – 1)], where – N = number of defects at beginning of testing, and –  is a proportionality constant (not a function). As each fault is removed, the hazard function decreases in value, and the time between failures increases.

10 Even messier models Each of these tries to get rid of some possibly untenable assumption in the previous models. E.g., in the Goel-Okumoto Imperfect Debugging Model, – Defect fixes are allowed to be imperfect. – Z(t i ) = [N – p(i – 1)], where – N = the number of faults at the start of testing, – P = probability of imperfect debugging, and – = the failure rate per fault.

11 The Delayed S and Inflection S Models Allow for separate defect detection and defect isolation processes. The S-shaped curve that results allows for the delay between first failure observation and the time of reporting. – The S-curve results from this delay in reliability growth. The inflection model also allows for detection dependence – the more we find, the more undetectable failures become detectable.

12 A comparison of predictions

13 Criteria for Model Evaluation Predictive validity Capability Quality of assumptions Applicability Simplicity Bold = Kan’s preferences.

14 How to model reliability 1.Examine the data. 2.Select the model or several models. 3.Estimate the parameters for the model. 4.Obtain the fitted model, by substituting the parameters. 5.Perform a goodness-of-fit test and assess reasonableness of the model. 6.Make reliability predictions based on the fitted model.

15 Step 1 - Examine the data

16 Steps 2, 3, & 4 2. Picked Exponential Model. Had used Delayed S and Inflection S Models before, but, – In this data, Kan didn’t notice an increase-then- decrease pattern typical of those models! 3. Used two SAS processes to get constants K and. (See slide 4.) 4. Plugged those into the exponential model.

17 Step 5 Used Kolmogorov- Smirnov goodness-of-fit test, to validate the model:

18 Step 6 Kan used this model during 4 years worth of testing! “The projection from this model was very close tto the estimate from the Rayleigh model and to the actual field defect data.”

19 Test Compression Factor These tests work best when: – Project is large, – Defect arrival rates tend not to fluctuate much, – The system is not safety-critical (!), and – Environment-specific factors are taken into account. But, even being this cautious doesn’t mean you can extend the model into customer use (e.g., maintenance)!

20 What’s different about maintenance? During testing, the sole purpose is defect detection and removal. Test cases are maximized for defect detection. Defect arrivals – usually much higher. Customer, in contrast, takes time to find defects! Defect arrivals may be much more spread out. The difference between these = the “compression factor.”

21 Real-life comparisons

22 Estimating Total Defects Use reliability models and, Historical patterns for projecting into field use.

23 Fast vs Slow Ramp-up Large systems and O/S’s – slow (3 years) Typical apps – fast (2 years)

24 Conclusion on ramp-ups Use your own historical data to estimate for new products. Kan recommends a nonparametric method – Like 3-point moving average – To smooth the data.

25 Recommendations for small projects For Ch 7 - Rayleigh can work, but requires people with the modeling expertise. From Ch 6 – Use a test effectiveness metric, validate with prior projects. Then, For Ch 8 – Gather data, etc., as described on p 220. Pick a simple metric (in terms of both data needed and complexity to validate). Interval estimating is easier. Cross-validate results across projects, if comparable. Know your assumptions!