Completing the Loop: Linking Software Features to Failures 31 July 2003 Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Paul Garnett Keith Lesch

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. STATUS Contract Awarded - xx July 2003

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Background Significant effort at IV&V Facility –Data/Metrics –Issue Repositories Problem: Little consistency among the data

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Problem Statement Do software features predict software failures and defects? Existing NASA tool identifies system features (SIAT) Existing NASA database of system defects and failures (PITS)

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Need a Link! Input to machine learning tools Show how features can predict faults

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Objective Develop methods and tools Create a way to integrate –System Analysis Tools –Defect Tracking Tools

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Hypothesis Predict software faults!

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Scope Analysis Methods –Work with existing and future IV&V projects –Identify essential defect data –Analyze existing defect data to determine adequacy Orthogonal Defect Classification –Enhance analyses and work products –Greater capability for later project-independent analysis –Define additional fields (in PITS as required)

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Year 1 Defect classification –Addition of ODC fields –Facilitate implementation on selected projects LINKER tool –Develop prototype/Initial version –Testing using historical PITS data

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Link 5 Technologies IV&V Defect Tracking Supports ODC fields Detailed features Control/Data Flow Automatically learn predictors Requires linking PITS/SIAT Replace NASA keywords

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Uniqueness of the Research First attempts to: –Achieve fault predictions by linking software features to failures in IV&V –Apply ODC on repeatable basis to IV&V projects –Involve personnel who are – Integrally involved in PITS development & maintenance – Experienced in IV&V analysis and use of PITS

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Application IV&V projects to be chosen at start of research project Initial projects will pilot implementation of ODC ODC use will provide for greater data uniformity across IV&V projects Output from chosen initial projects will link data from PITS and SIAT for analysis

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Technology Transfer Potential Every NASA IV&V program may benefit Research results can be generalized across IV&V projects Other agencies using PITS and SIAT could apply this research to their projects May lead to project-specific, NASA, or industry- wide conclusions on software fault/failure predictors

Copyright © 2003, Mountain State Information Systems, Inc. All rights reserved. MOUNTAIN STATE INFORMATION SYSTEMS, INC. Future Efforts Get Started Years 2 and 3 Additional Software feature extractors Additional Defect Databases Additional Machine Learning Tools