Philip Preissing and Jan Schulte 2009-06-19 Iterative Development of a Domain-Specific Fault Classification An Industrial Case Study Philip Preissing and Jan Schulte 2009-06-19

Table of Contents Introduction Related Work & Quality Characteristics WEMSE Workshop- 2009-06-19 Introduction RUAG Context Related Work & Quality Characteristics Development Process Case Study Discussion Summary

RUAG Aerospace Sweden AB WEMSE Workshop- 2009-06-19 Headquarter in Göteborg 360 employees Formerly SAAB Aerospace Highly reliable Satellite Equipment Computers Antennas Microwave Systems Projects Ariane 5 Herschel/Planck Galileo

Problems at RUAG High dependability & reliability ECSS Standards WEMSE Workshop- 2009-06-19 High dependability & reliability ECSS Standards Late faults several times more expensive  Software Testing accounts for 60% of Development Time Master thesis to develop an optimization framework for the Verification & Validation Activities (VAs)

Goals WEMSE Workshop- 2009-06-19 Measure defects Gain insight into VAs Fault-slippage between VAs Overlap in VAs  Fault classification(FC) to group similar types of defects Analyze problems in the process on a high level Simplify the measurement process

Example WEMSE Workshop- 2009-06-19

Related Work Schemes exist WEMSE Workshop- 2009-06-19 Schemes exist Orthogonal Defect Classification (IBM) Standard Classification for SW Anomalies (IEEE) Origins, Types & Modes (HP) FC in use at Ericsson Classifications need to be adapted (Case studies) Development processes Expert opinion Commit comments  Development process needed

Quality characteristics WEMSE Workshop- 2009-06-19 Classes should… … be at most 5-10 … describe the fault type … be orthogonal … be consistent … be complete … be applicable to every software artifact

Development Process Initial Fault Classification Iterative Refinement WEMSE Workshop- 2009-06-19 Initial Fault Classification Iterative Refinement Selection Classification Comparison & Discussion Quality Review

Case Study – Initial Fault Classification WEMSE Workshop- 2009-06-19 Class Assignment Understandability Threading/ Concurrency Function/Interface Hardware Interface Timing/ Performance Algorithms Robustness

Case Study – Iterative Refinement WEMSE Workshop- 2009-06-19 Select data source Code Inspection sheets Classification of faults Completely sure Uncertain Don‘t know Analysis Several iterations

Case Study – Quality Review WEMSE Workshop- 2009-06-19 Agreement Factor (Consistency) Initial: 0.30 Final: 0.71 Workshop Project Manager, Designer, Developer, Tester Only minor changes < 0.45 Inadequate agreement > 0.62 Good agreement > 0.78 Excellent agreement

Advantages & Drawbacks WEMSE Workshop- 2009-06-19 Analysis of all stages Low developer involvement Participants have limited understanding Validation first performed at the end

Lessons learned A posteriori analysis difficult WEMSE Workshop- 2009-06-19 A posteriori analysis difficult Documentation important Alternative data sources difficult Checkin comments Observation of developers Interviews

Summary Software Testing very expensive WEMSE Workshop- 2009-06-19 Software Testing very expensive FC necessary to analyse & improve process Process to develop domain-specific FC Incorporating existing schemes Iterative refinement by analyzing all steps Quality review Validation in industrial case study at RUAG

Thank you very much! Merci beaucoup ! Tack så mycket! Muchas gracias! WEMSE Workshop- 2009-06-19 Thank you very much! Merci beaucoup ! Tack så mycket! Muchas gracias! Muito obrigado! Vielen Dank! Moltes gràcies! Mille grazie!