1 NASA OSMA SAS02 Software Reliability Modeling: Traditional and Non-Parametric Dolores R. Wallace Victor Laing SRS Information Services Software Assurance.

Slides:



Advertisements
Similar presentations
Design of Experiments Lecture I
Advertisements

Exercise 7.5 (p. 343) Consider the hotel occupancy data in Table 6.4 of Chapter 6 (p. 297)
Testing and Quality Assurance
Hypothesis Testing Steps in Hypothesis Testing:
CS 795 – Spring  “Software Systems are increasingly Situated in dynamic, mission critical settings ◦ Operational profile is dynamic, and depends.
1 The Role of the Revised IEEE Standard Dictionary of Measures of the Software Aspects of Dependability in Software Acquisition Dr. Norman F. Schneidewind.
Prediction of fault-proneness at early phase in object-oriented development Toshihiro Kamiya †, Shinji Kusumoto † and Katsuro Inoue †‡ † Osaka University.
Figures – Chapter 24.
Validation and Monitoring Measures of Accuracy Combining Forecasts Managing the Forecasting Process Monitoring & Control.
Chapter 12 - Forecasting Forecasting is important in the business decision-making process in which a current choice or decision has future implications:
1 Spreadsheet Modeling & Decision Analysis: A Practical Introduction to Management Science, 3e by Cliff Ragsdale.
Swami NatarajanJune 17, 2015 RIT Software Engineering Reliability Engineering.
Software Testing and QA Theory and Practice (Chapter 15: Software Reliability) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory and Practice.
Quantitative Trading Strategy based on Time Series Technical Analysis Group Member: Zhao Xia Jun Lorraine Wang Lu Xiao Zhang Le Yu.
Monté Carlo Simulation MGS 3100 – Chapter 9. Simulation Defined A computer-based model used to run experiments on a real system.  Typically done on a.
1 Software Testing Techniques CIS 375 Bruce R. Maxim UM-Dearborn.
Overview Software Quality Assurance Reliability and Availability
1 Prediction of Software Reliability Using Neural Network and Fuzzy Logic Professor David Rine Seminar Notes.
Pop Quiz How does fix response time and fix quality impact Customer Satisfaction? What is a Risk Exposure calculation? What’s a Scatter Diagram and why.
SENG521 (Fall SENG 521 Software Reliability & Testing Software Reliability Tools (Part 8a) Department of Electrical & Computer.
ECE 355: Software Engineering
Inference for regression - Simple linear regression
SAS 03/ GSFC/SATC-ERAU-DoC Fault Tree Analysis Application for Safety and Reliability Massood Towhidnejad Embry-Riddle University Dolores Wallace & Al.
Handouts Software Testing and Quality Assurance Theory and Practice Chapter 15 Software Reliability
1SAS 03/ GSFC/SATC- NSWC-DD System and Software Reliability Dolores R. Wallace SRS Technologies Software Assurance Technology Center
1 Using Excel to Implement Software Reliability Models Norman F. Schneidewind Naval Postgraduate School 2822 Racoon Trail, Pebble Beach, California, 93953,
597 APPLICATIONS OF PARAMETERIZATION OF VARIABLES FOR MONTE-CARLO RISK ANALYSIS Teaching Note (MS-Excel)
1 Software Quality CIS 375 Bruce R. Maxim UM-Dearborn.
Software Reliability SEG3202 N. El Kadri.
Chapter 6 : Software Metrics
Lecture 12 Statistical Inference (Estimation) Point and Interval estimation By Aziza Munir.
1 Spreadsheet Modeling & Decision Analysis: A Practical Introduction to Management Science, 3e by Cliff Ragsdale.
Software Measurement & Metrics
Product Metrics An overview. What are metrics? “ A quantitative measure of the degree to which a system, component, or process possesses a given attribute.”
1 Software Reliability Assurance for Real-time Systems Joel Henry, Ph.D. University of Montana NASA Software Assurance Symposium September 4, 2002.
Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Modelling Engin Deveci.
1 OO Metrics-Sept2001 Principal Components of Orthogonal Object-Oriented Metrics Victor Laing SRS Information Services Software Assurance Technology Center.
Ch. 1.  High-profile failures ◦ Therac 25 ◦ Denver Intl Airport ◦ Also, Patriot Missle.
1 Reliability-Sept2001 Software Reliability Through Hardware Reliability Dolores R. Wallace SRS Information Services Software Assurance Technology Center.
Time Series Analysis and Forecasting
Lecture 8 Simple Linear Regression (cont.). Section Objectives: Statistical model for linear regression Data for simple linear regression Estimation.
Chin-Yu Huang Department of Computer Science National Tsing Hua University Hsinchu, Taiwan Optimal Allocation of Testing-Resource Considering Cost, Reliability,
Maintenance Workload Forecasting
Software Metrics Cmpe 550 Fall Software Metrics.
Chapter 3: Software Project Management Metrics
Methods for Estimating Defects Catherine V. Stringfellow Mathematics and Computer Science Department New Mexico Highlands University October 20, 2000.
CSc 461/561 Information Systems Engineering Lecture 5 – Software Metrics.
Measurement and quality assessment Framework for product metrics – Measure, measurement, and metrics – Formulation, collection, analysis, interpretation,
Time Series Analysis and Forecasting. Introduction to Time Series Analysis A time-series is a set of observations on a quantitative variable collected.
Software Engineering1  Verification: The software should conform to its specification  Validation: The software should do what the user really requires.
1 NASA OSMA SAS02 Software Fault Tree Analysis Dolores R. Wallace SRS Information Services Software Assurance Technology Center
SENG521 (Fall SENG 521 Software Reliability & Testing Overview of Software Reliability Engineering Department of Electrical.
Object-Oriented (OO) estimation Martin Vigo Gabriel H. Lozano M.
1 SAS 04/ GSFC/SATC-NSWCDD System and Software Reliability Technical Presentation Naval Surface Warfare Center Dahlgren Division Software Assurance Technology.
Software Reliability Estimates/ Projections, Cumulative & Instantaneous Presented by Dave Dwyer With help from: Ann Marie Neufelder, John D. Musa, Martin.
Main Title Slide Software Reliability Estimates/ Projections, Cumulative & Instantaneous - Dave Dwyer With help from: Ann Marie Neufelder, John D. Musa,
T EST T OOLS U NIT VI This unit contains the overview of the test tools. Also prerequisites for applying these tools, tools selection and implementation.
 Galorath Incorporated 2004 AccuScope – A Tool For Relative Comparisons Lee Fischman & Kenneth Fong Galorath Incorporated SCEA 2004 Manhattan Beach,
CS223: Software Engineering Lecture 21: Unit Testing Metric.
Slide 1SATC June 2000 Dolores R. Wallace* NASA Goddard Space Flight Center Greenbelt, Maryland for the American Society.
Software Testing. SE, Testing, Hans van Vliet, © Nasty question  Suppose you are being asked to lead the team to test the software that controls.
Software Metrics and Reliability
Software Metrics 1.
Hardware & Software Reliability
Software Reliability Definition: The probability of failure-free operation of the software for a specified period of time in a specified environment.
Software Reliability PPT BY:Dr. R. Mall 7/5/2018.
Object-Oriented Metrics
Software Reliability Models.
Test Case Test case Describes an input Description and an expected output Description. Test case ID Section 1: Before execution Section 2: After execution.
Software Testing “If you can’t test it, you can’t design it”
Presentation transcript:

1 NASA OSMA SAS02 Software Reliability Modeling: Traditional and Non-Parametric Dolores R. Wallace Victor Laing SRS Information Services Software Assurance Technology Center dwallac, NASA OSMA SAS '02

2 NASA OSMA SAS02 The Problem Critical NASA systems must execute successfully for a specified time under specified conditions -- Reliability Most systems rely on software Hence, a means to measure software reliability is essential to determining readiness for operation Software reliability modeling provides one data point for reliability measurement

3 NASA OSMA SAS02 Software Reliability Modeling (SRM) – Traditional Captures hardware reliability engineering concepts Mathematically models behavior of a software system from failure data to predict reliability growth Invokes curve-fitting techniques to determine values of parameters used in the models Validates models with data with statistical analysis Using parametric values, predicts future measurements, e.g., –Mean time to failure –Total number faults remaining –Number faults at time t

4 NASA OSMA SAS02 Synopsis FY01 –Identify mathematics of hardware reliability not used in software –Identify differences between hardware, software affecting reliability measurement –Identify possible improvements FY02 –Demonstrate practicality of SRM at GSFC –Fault correction improvement – Schneidewind –Non-parametric model - Laing

5 NASA OSMA SAS02 SRM: Data Collection Resistance to data collection Data content –Accuracy of content –Dates of failure, correction –Calendar time not execution time –Activities/ phase when failures occur Data manipulation –Frequency counts –Interval size and length –Time between failure

6 NASA OSMA SAS02 IntervalCounter Sample had 35 weeks – simplified fault count

7 NASA OSMA SAS02 SMERFS^3 3-D OUTPUT

8 NASA OSMA SAS02 Practical Method SATC Services –SATC executes models and prepares analysis –SATC provides training and public domain tool Improvements –Recommendations to projects for data collection –IntervalCounter to simplify data manipulation

9 NASA OSMA SAS02 Fault Correction Adjustments Reliability growth occurs from fault correction Failure correction proportional to rate of failure detection Adjusted model with delay dT (based on queuing service) but same general form as faults detected at time T Process: use SMERFS Schneidewind model to get parameters; apply to revised model via spreadsheet Results –Show reliability growth due to fault correction –Predict stopping rules for testing

10 NASA OSMA SAS02 SMERFS^3 – Excel Approach* Best approach: combine SMERFS^3 with Excel. SRT provides model parameter estimation. Copy and paste parameters from SRT into spreadsheet. Excel extends capabilities of SRT by allowing user to provide equations, statistical analysis, and plots. * CASRE or other software reliability modeling tool may be used with EXCEL Recommended approach until the SRM tools incorporate this new model.

11 NASA OSMA SAS02 Non-parametric Reliability Modeling Hardware - Wears out over time - Increasing failure rate Software - Do not wear over time - Decreasing failure rate

12 NASA OSMA SAS02 Continued Hardware Reliability Modeling - “Large” independent random sampling - Model reliability - Make predictions Software Reliability Modeling - “Small” observed dependent sample (of size one?) - Not based on independent random sampling - Model reliability - Make predictions? Do we search for the silver bullet of SWR models?

13 NASA OSMA SAS02 Reliability Trending Hardware Reliability 100% Maximum 0% Minimum …  Time Software Reliability 100% Maximum 0% Minimum …  Time

14 NASA OSMA SAS02 Software Reliability Bounds 100% Maximum Estimated Bound Estimated Model 0% Minimum …  Time

15 NASA OSMA SAS02 Calculation of Estimated Models and Bounds Dynamic Metrics - Failure rate data - Problem reports Static Code Metrics - Traditional - Source Lines of Code (SLOC) - Cyclomatic Complexity (CC) - Comment Percentage (CP) - Object-Oriented - Coupling Between Objects (CBO) - Depth of Inheritance Tree (DIT) - Weighted Methods per Class (WMC)

16 NASA OSMA SAS02 Combining Dynamic and Static Metrics The Proportional Hazards Model (PHM) PHM Non-Parametric Component (Static) R(t|z) = {R 0 (t)} g(z) Parametric Component (Dynamic) - Where zβ = z 1 β 1 + z 2 β 2 + … + z p β p, β i ’s are unknown regression coefficients and z i ’s are static code metrics data

17 NASA OSMA SAS02 Tool Schema Input Data z = (z 1, z 2, … z p ) Database Observed Data Data Processing R(t|z) = {R 0 (t)} g(z) Weighted Average Raw Data Output Data Estimated Model Estimated Bound - Process Below Bounds Action - Corrective Action - Process Above Bounds - No Corrective Action

18 NASA OSMA SAS02 SUMMARY Software reliability modeling –Provides useful measurements for decisions –Does not require expert knowledge of the math! –Is relatively easy with use of software tools Fault correction improvement –Adapts model to be more like software –Demonstrates combined use of traditional SRM tools with spreadsheet technology Non-parametric modeling –New approach shows promise –Prototype to be expanded

19 NASA OSMA SAS02 AIAA Recommended Steps (specific to SRM) Characterizing the environment Determining test approach Selecting models Collecting data Estimating parameters Validating the models Performing analysis

20 NASA OSMA SAS02 Fault Correction Modeling Software reliability models focus on modeling and predicting failure occurrence –There has not been equal priority on modeling the fault correction process. Fault correction modeling and prediction support to –predict whether reliability goals have been achieved –develop stopping rules for testing –formulate test strategies –rationally allocate test resources.

21 NASA OSMA SAS02 Equations: Prediction and Comparison Worksheets Remaining Failures Predicted at Time t: r(t) = (  /  ) – X s,t Cumulative Number of Failures Detected at Time T: D(T) = (α/β)[1 – exp (-β ((T –s + 1)))] + X s-1 Cumulative Number of Failures Detected Over Life of Software T L : D(T L ) =  /  + X s-1 Equations developed by Dr. Norman Schneidewind, Naval Postgraduate School, Monterey, CA Time to Next Failure(s) Predicted at Time t