1. Achieving High Software Reliability
OSMA Software Assurance Symposium 2002
Achieving High Software Reliability
The Software Measurement Analysis and Reliability Toolkit
&
Module-Order Modeling
Taghi M. Khoshgoftaar
Empirical Software Engineering Laboratory
Florida Atlantic University
Boca Raton, Florida USA
September 6, 2002

2. Overview
SMART: The Software Measurement Analysis and Reliability Toolkit
Module-Order Modeling
Investigating the impact of underlying prediction models on module-order models
Empirical case studies
Summary
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

3. SMART Case-Based Reasoning Module-Order Models
quantitative software quality prediction models: predicting faults, code churn, etc.
qualitative software classification (risk-based) models: two-group and three-group models
Module-Order Models
priority-based ranking of modules with respect to their software quality
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

4. SMART (GUI)
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

5. SMART (GUI)
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

6. Module-Order Models Why module-order models?
Classification models are not suitable from the business & improved cost-effective view points
same quality improvement resources applied to all modules predicted as high-risk or fault-prone
A priority-based software quality improvement is more suited for a cost-effective usage of available resources
inspecting the most fault-prone modules first
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

7. MOMs ...
Answers practical questions posed by project management, such as
which & how many modules to target for V&V?
what’s the best usage of available resources?
Different underlying quantitative software quality prediction models available
what is their impact on the performance of module-order models?
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

8. MOMs ... Components of a module-order model
underlying software quality prediction model
ranking of modules according to the predicted quality factor, and
procedure for evaluating accuracy and effectiveness of predicted ranking
Alberg diagrams: faults accounted-for by rankings
Performance diagrams: measuring accuracy of the predicted ranking with respect to actual (perfect) ranking
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

9. MOMs ...
Based on schedule & resources allocated for testing and V&V, determine a range of cutoff percentages that includes the management’s options for covering the last module (as per the ranking) to be inspected
Choose a set of representative cutoff percentages, ‘c’, from that range
for each c, determine the number of faults accounted for by the actual & predicted ranking
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

10. Case Study Example A large legacy telecommunications system
mission-critical software
written in a procedural language
software metrics from four system releases, with a few thousand modules in each release
fault data comprised of faults discovered during post unit testing, including system operations
24 product metrics & 4 execution metrics used
Release 1 used as fit data & others as test data
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

11. Fault Prediction Models
Rank-order based on average absolute error and average relative error of models
1. CART-LAD regression tree
2. Case-Based Reasoning (SMART)
3. Multiple Linear Regression
4. Artificial Neural Networks
5. CART-LS regression tree
6. S-PLUS regression tree
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

12. Results of MOMs Group 1 Group 2 (all available regression trees) CBR
MLR
ANNs
Group 2 (all available regression trees)
CART-LS
CART-LAD
S-PLUS
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

13. Results of MOMs ... Alberg Diagram for Group 1: Release 2
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

14. Results of MOMs ... Performance Diagram for Group 1: Release 2
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

15. Results of MOMs ... Alberg Diagram for Group 2: Release 2
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

16. Results of MOMs ... Performance Diagram for Group 2: Release 2
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

17. Results Summary
Group 1 models, i.e., CBR, ANN, & MLR had similar performances with respect to their module-order models
S-PLUS (Group 2) module-order model performed similar to CBR, ANN, & MLR
Though CART-LAD yielded best AAE and ARE values, it showed a relatively-lower performance as a module-order model
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

18. Results Summary ...
When used as a module-order model, CART-LS is better than CART-LAD
In contrast, with respect to AAE and ARE values CART-LAD is better than CART-LS
Overall, for this case study the CART-LS module-order model performed generally better than the other five models, i.e., CBR, CART-LAD, ANN, MLR, and S-PLUS
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

19. Results Summary ... Observing the effects of data characteristics
performance of MOMs is dependent on the system domain and the software application
Are AAE & ARE good performance metrics for selecting underlying prediction models for module-order modeling?
Selecting the prediction models based on AAE and ARE did not provide any conclusive insight into the performance of a module-order model
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

20. Conclusion
Software fault prediction and quality classification models by themselves may not be sufficient from the business and practical view points (return-on-investment)
Module-order modeling presents a more goal-oriented approach by predicting a priority-based ranking of modules with respect to software quality
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

21. Conclusion ...
Case studies investigating the impact of different underlying prediction models on module-order models
Completed the ready-to-use (stand alone) version of SMART, including its
requirements and specifications document
design, implementation, and integration document
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

22. Future Work
A resource-based approach for the selection and evaluation of software quality models
Developing models that provide an improved goal- and objective-oriented software quality assurance
lowering the expected cost of misclassification
improving the cost-benefit factor of models
a better focus on return-on-investment
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

23. Future Work ...
Applying the SMART technology to software metrics and fault data collected from a NASA software project
evaluating performance & benefits of SMART in the context of NASA software data
Incorporating SMART into a live NASA software project
demonstrating practical technology transfer
September 6, 2002
Empirical Software Engineering Laboratory
Florida Atlantic Univeristy

24. Achieving High Software Reliability
OSMA Software Assurance Symposium 2002
Achieving High Software Reliability
Thank You …
Taghi M. Khoshgoftaar
(561)
Empirical Software Engineering Laboratory
Florida Atlantic University
Boca Raton, Florida, USA
September 6, 2002