1. SAS03/FaultPrediction1 NASA OSMA SAS ‘03 Software Requirements Analysis As Fault Predictor Dolores R. Wallace SRS Technologies Software Assurance Technology Center http://satc.gsfc.nasa.gov dwallac@pop300.nasa.gov William M. Wilson SRS Technologies/SATC wilsonw@pop300.nasa.gov

2. SAS03/FaultPrediction2 Software Requirements Analysis As Fault Predictor Presentation Outline 1.Research Rationale & Objective 2.Requested Project Data 3.Planned Approach 4.Data Acquisition & Analysis Obstacles 5.Results & Findings 6.Conclusions 7.Recommendations

3. SAS03/FaultPrediction3 Research Rationale & Objective Rationale The earlier in the lifecycle that software faults can be found the less expensive it is to correct them. The earliest opportunity to preclude software faults is during the development of the system's requirements. Objective To determine if software requirements analysis results can be used as a predictor of software faults.

4. SAS03/FaultPrediction4 Requested Project Data Requirements Specification –Accessible in electronic media in MS Word format for use with the ARM tool –Conforming to NASA-STD-2100 –Specification statements identified by hierarchical numbers

5. SAS03/FaultPrediction5 Requested Project Data – Cont. Verification Test Matrix –Requirements to Test or vice versa - OR - Traceability Matrix –Requirements to Design Elements –Design Element To Software Component - AND - Test Plans –Tests vs. Software Components.

6. SAS03/FaultPrediction6 Requested Project Data – Cont. Failure Reports/Data - Preferably in DDTS format –Including: test id, module id, CSCI id, release/build id, suspected cause of failure and resolution (e.g., actual cause of failure - what was fixed) Configuration Control Reports –Baseline item changed -e.g., Requirement, Design, Code –Reason/Source of change – e.g., RID, Failure Report, PR, etc.

7. SAS03/FaultPrediction7 Planned Approach 1.Use ARM tool to analyze projects’ software requirements. 2.Collected projects’ failure data in a DDTS database. 3.Organize the data to represent same components for each project. 4.Conduct correlation analysis. 5.Study results of step 4 to prove or disprove hypothesis.

8. SAS03/FaultPrediction8 ARM TOOL Reports Identifies and Counts: Size of Requirements Document –Lines of Text –Numbered Paragraphs –Number of Specification Statements –Number of Unique Specification Subjects Number of Specifications Containing –Weak, Optional and/or Incomplete Phrases. –Compound and/or Complex Statements. Requirements Document Structure –Depth and Distribution of Specifications

9. SAS03/FaultPrediction9 Data Acquisition & Analysis Obstacles Common Problems –No commonality or standardization of data or formats across project. –Current projects are reluctant to provide data to outside organizations. –Data from completed projects is incomplete, not in a usable form or is not accessible.

10. SAS03/FaultPrediction10 Data Acquisition & Analysis Obstacles Cont. Specific Problems –No project staff available to guide through the mass of documents in the collection –Documents locked on a WEB site –Documents in “.pdf” format –Variation of format and media within project –Complete set of data for a specific Build/Release not available –Data held by support contractor and released only on a “Project Need-To-Know” basis

11. SAS03/FaultPrediction11 Results & Findings Data provided by four projects –Projects A and B Major subsystems of operational information processing system –Projects C and D Flight instrumentation packages

12. SAS03/FaultPrediction12 Results & Findings Cont

13. SAS03/FaultPrediction13 Conclusions Data from many more projects is required to improve the possibility of finding a number of data sets with sufficient commonality to support analysis. Analysis approach needs to be expanded to include determining if failures attributed to documentation and design flaws are in reality related to deficiencies in requirements.

14. SAS03/FaultPrediction14 Recommendations Advocate and support: The use of NASA-STD-2100 documentation standard The use of a format to report problems and failures that is compatible with DDTS The creation and use of centralized Center repositories for data from completed projects.

15. SAS03/FaultPrediction15 Summary Lessons –Getting data requires strong, interested Civil servant advocate –Analyzing data requires tedious effort, even with automated tools Future –Analyze latest set of data –Attempt correlations for the 4 data sets