Achieving High Software Reliability Using a Faster, Easier and Cheaper Method NASA OSMA SAS '01 September 5-7, 2001 Taghi M. Khoshgoftaar The Software Measurement Analysis and Reliability Toolkit

Outline Introduction Overview of SMART Data Analysis and Modeling Features Current Utilization of SMART Case-Based Reasoning Empirical Study Conclusions NASA OSMA SAS '01 September 5-7, 2001

Introduction Necessity of an integrated tool for efficient empirical software quality research Commercial packages are available but expensive and don’t always match our exact needs In house development gives us availability, flexibility and possibility to evolve NASA OSMA SAS '01 September 5-7, 2001

Overview of SMART First version back in 1998 Current version 2.0 Written in Microsoft Visual C++ Runs on Microsoft Windows based platforms User friendly GUI NASA OSMA SAS '01 September 5-7, 2001

SMART GUI NASA OSMA SAS '01 September 5-7, 2001

SMART GUI NASA OSMA SAS '01 September 5-7, 2001

Features included in SMART Data management Multiple Linear Regression (MLR) Case-based reasoning (CBR), Case-based reasoning with two group clustering Case-based reasoning with three group clustering Module order modeling (MOM) NASA OSMA SAS '01 September 5-7, 2001

Organizational Flowchart NASA OSMA SAS '01 September 5-7, 2001

Current Utilization of SMART at the ESEL Laboratory Empirical research: – Comparative studies of software quality models – Case studies based on real world systems NASA OSMA SAS '01 September 5-7, 2001

Case Based Reasoning Based on automated reasoning processes Easy to use Results are easy to understand and to interpret Looks at past cases that are similar to the present case in an attempt to predict or classify an instance NASA OSMA SAS '01 September 5-7, 2001

Case Based Reasoning: Additional Advantages The ability to alert users when a new case is outside the bounds of current experience The ability to interpret the automated classification through the detailed description of the most similar case The ability to take advantage of new or revised information as it becomes available The ability for fast retrieval as the size of the library scales up NASA OSMA SAS '01 September 5-7, 2001

Case Based Reasoning Working hypothesis for software quality modeling: – Current cases that are in development will more than likely be fault-prone if past cases having similar attributes were fault-prone NASA OSMA SAS '01 September 5-7, 2001

Case Based Reasoning: Comparing the Cases Similar cases to a new module or nearest neighbors are determined by similarity functions: – Absolute Distance – Euclidean Distance – Mahalonobis Distance NASA OSMA SAS '01 September 5-7, 2001

Case Based Reasoning: Prediction Methods The value of the dependent variable is estimated using the values of the dependent variables of the nearest neighbors and a solution algorithm: – Unweighted Average – Inverse-Distance Weighted Average NASA OSMA SAS '01 September 5-7, 2001

Case Based Reasoning: Classification Methods Used to classify a software module into a particular class (fault-prone, not fault- prone). The types of classification methods include: – Majority Voting – Data Clustering NASA OSMA SAS '01 September 5-7, 2001

Case Study: System Description Two data sets were obtained from two large Windows-based applications used primarily for customizing the configuration of wireless products. The data sets were obtained from the initial release of these applications. The applications are written in C++, and they provide similar functionality. NASA OSMA SAS '01 September 5-7, 2001

Case Study: System Description NASA OSMA SAS '01 September 5-7, 2001

Case Study: Data Collection Effort Data collected by engineers over several months using the available information in: – Configuration Management Systems – Problem Reporting Systems

Case Study: Independent Variables NASA OSMA SAS '01 September 5-7, 2001

Case Study: Accuracy Evaluation Average Absolute Error: Average Relative Error: NASA OSMA SAS '01 September 5-7, 2001

Case Study: Prediction Results NASA OSMA SAS '01 September 5-7, 2001

Case Study: Classification Evaluation NASA OSMA SAS '01 September 5-7, 2001

Case Study: Classification Results Entire data set: Fit and Test data set: NASA OSMA SAS '01 September 5-7, 2001

Case Study: Return On Investment Classification using CBR NASA OSMA SAS '01 September 5-7, 2001

Conclusion A tool that matches our needs Used for our extensive empirical work Proved useful on large scale case study – Faster – Easier – Cheaper Ready for future enhancement NASA OSMA SAS '01 September 5-7, 2001

Reminder We will be presenting the tool on Friday, Please feel free to visit us! NASA OSMA SAS '01 September 5-7, 2001