1. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova Preventing and Eliminating Software Faults through the Life Cycle PI: Katerina Goseva-Popstojanova Student: Margaret Hamill Lane Dept. Computer Science and Electrical Engineering West Virginia University, Morgantown, WV E-mail: Katerina.Goseva@mail.wvu.edu

2. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 22 Problem NASA spends time and effort to track problem reports/change data for every project. These repositories are rich, underused sources about the way software systems fail and the software faults that cause these failures. Our goal: Based on systematic and thorough analysis of the available empirical data, build quantitative and qualitative knowledge that contributes towards improvement of software quality by –preventing introduction of faults into the system –more efficiently eliminating them through the life cycle –compiling lessons learned & recommendations to be used by the pilot project and throughout the Agency

3. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 3 3 Approach Propose appropriate classification scheme Refine Identify the most frequent fault types & most frequent failure types Compile Lessons Learned & Recommendations document Compile a check list to support avoidance & elimination of different faults types Explore change tracking systems & automate data extraction Classify fault/failure data from the pilot project Identify common patterns and unusual dependencies Quantify multiple dimensions for each fault type

4. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 4 Pilot study: Basic facts A large NASA mission consisting of millions of lines of code –21 CSCIs, over 8,000 files Developed iteratively at two different locations –Each CSCI has multiple releases with added functionality We analyzed over 2,800 Software Change Requests (SCRs) collected in a period of almost 10 years CSCI id # of releases # of files #non conf SCRs non conf SCRs per file 112073921.89 22200240.12 3228790.03 42228160.07 52269130.05 62321220.07 72289770.27 822701070.40 92125210.17 103356270.08 113444430.10 123277120.04 1335991210.20 143280830.30 153811081.33 1631691350.80 1747472010.27 1845872110.36 1955522630.48 20713688610.63 2174151120.27 Total--807128580.35

5. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 5 Technical challenges Assuring data quality is an important step of any empirical research effort –Inaccurate data may lead to misleading observations –Both the IV&V team and the project team have been extremely valuable in helping us to understand the change tracking system, determine the meaning of different attributes, and verify the quality of the data The research approach and analysis techniques can be used by any project that tracks problem reports/change data –However, due to the lack of a unified change tracking system, some amount of unique work on exploration of the data format and automation of data extraction may be needed

6. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 6 Current capability Fault localization [SAS 07] –58-62% of failures map to faults in more than one file Sources of failures (i.e., type of faults) –Dominating fault types: requirements faults (33%), coding faults (33%), data problems (14%) Activities when the problem was discovered (e.g., inspection, testing, analysis, on-orbit) –Almost 50% discovered by analysis, only 3% on-orbit Severity –Around 8% of SCRs are safety critical, less than 1% on-orbit Lessons learned & recommendations for product and process improvements –Compiled a document for internal use based on our results and the feedback from the IV&V team and project team

7. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 7 Current capability: Identifying the most common fault types Most common fault types –Requirements faults: 33% –Coding faults: 33% –Data problems: 14% Early life cycle activities (requirements & design): 38.25% Late life cycle activities (coding, interface & integration): 48.57% This finding contradicts the common belief that majority of faults are introduced during early life cycle activities [Boehm et al.1975, Endres 1975, Basili et al.1984]

8. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 8 Current capability: Distribution of fault types Development & testing vs. On-orbit Only 3% of SCRs are on-orbit. Note the logarithmic scale of Y axes.

9. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 9 Current capability: Safety critical SCRs across fault types Only around 8% of all SCRs are safety critical. The highest percentage of safety critical SCRs comes from coding faults (3.60%), followed by requirement faults, design faults & data problems (3.35% total).

10. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 10 Based on our results and the feedback from the IV&V team and project team we compiled a document for internal use which summarizes the Lessons learned & Recommendations for product and process improvement –Prevent the introduction of faults and improve the effectiveness of detection Example: Increase effort spent on design and implementation of data repository used to share data between CSCIs –Improve the quality of the data & change tracking process Example: Ensure the changes to the software artifacts (e.g., requirements, code, etc) made to fix the problem are recorded and can be easily associated with a specific SCR Current capability: Lessons Learned / Recommendations

11. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 11 Planned capability [FY08-10] Classify faults and failures using several additional attributes Conduct more complex, multivariable analysis Continually update Lessons Learned & Recommendations for Improvement document Explore the best ways to prevent and eliminate most common faults and failures throughout the lifecycle; compile the results in a check list Increase awareness of our work so other projects within NASA can benefit from it

12. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 12 Relevance to NASA Our research is based on a large NASA mission as a pilot study –The research work is done in active collaboration with the IV&V team and the project team –To the best of our knowledge, this is the largest dataset considered so far in the published literature –Quantitative and qualitative results are regularly delivered and discussed with the project Based on the internal & external validity of our results, we believe that observed trends are intrinsic characteristics of large, complex software systems –Other similar projects across the Agency may benefit –Lessons Learned & Recommendations can be used proactively by newer projects

13. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 13 Acknowledgements We thank the following NASA civil servants and contractors for their valuable support! Jill Broadwater Pete Cerna Susan Creasy Randolph Copeland James Dalton Bryan Fritch Nick Guerra John Hinkle Lynda Kelsoe Debbie Miele Lisa Montgomery Don Ohi Chad Pokryzwa David Pruett Timothy Plew Scott Radabaugh