1. Philip Preissing and Jan Schulte 2009-06-19
Iterative Development of a Domain-Specific Fault Classification An Industrial Case Study
Philip Preissing and Jan Schulte

2. Table of Contents Introduction Related Work & Quality Characteristics
WEMSE Workshop
Introduction
RUAG
Context
Related Work & Quality Characteristics
Development Process
Case Study
Discussion
Summary

3. RUAG Aerospace Sweden AB
WEMSE Workshop
Headquarter in Göteborg
360 employees
Formerly SAAB Aerospace
Highly reliable Satellite Equipment
Computers
Antennas
Microwave Systems
Projects
Ariane 5
Herschel/Planck
Galileo

4. Problems at RUAG High dependability & reliability ECSS Standards
WEMSE Workshop
High dependability & reliability
ECSS Standards
Late faults several times more expensive
 Software Testing accounts for 60% of Development Time
Master thesis to develop an optimization framework for the Verification & Validation Activities (VAs)

5. Goals
WEMSE Workshop
Measure defects
Gain insight into VAs
Fault-slippage between VAs
Overlap in VAs
 Fault classification(FC) to group similar types of defects
Analyze problems in the process on a high level
Simplify the measurement process

6. Example
WEMSE Workshop

7. Related Work Schemes exist
WEMSE Workshop
Schemes exist
Orthogonal Defect Classification (IBM)
Standard Classification for SW Anomalies (IEEE)
Origins, Types & Modes (HP)
FC in use at Ericsson
Classifications need to be adapted (Case studies)
Development processes
Expert opinion
Commit comments
 Development process needed

8. Quality characteristics
WEMSE Workshop
Classes should…
… be at most 5-10
… describe the fault type
… be orthogonal
… be consistent
… be complete
… be applicable to every software artifact

9. Development Process Initial Fault Classification Iterative Refinement
WEMSE Workshop
Initial Fault Classification
Iterative Refinement
Selection
Classification
Comparison & Discussion
Quality Review

10. Case Study – Initial Fault Classification
WEMSE Workshop
Class
Assignment
Understandability
Threading/ Concurrency
Function/Interface
Hardware Interface
Timing/ Performance
Algorithms
Robustness

11. Case Study – Iterative Refinement
WEMSE Workshop
Select data source
Code Inspection sheets
Classification of faults
Completely sure
Uncertain
Don‘t know
Analysis
Several iterations

12. Case Study – Quality Review
WEMSE Workshop
Agreement Factor (Consistency)
Initial: 0.30
Final: 0.71
Workshop
Project Manager, Designer, Developer, Tester
Only minor changes
< 0.45
Inadequate agreement
> 0.62
Good agreement
> 0.78
Excellent agreement

13. Advantages & Drawbacks
WEMSE Workshop
Analysis of all stages
Low developer involvement
Participants have limited understanding
Validation first performed at the end

14. Lessons learned A posteriori analysis difficult
WEMSE Workshop
A posteriori analysis difficult
Documentation important
Alternative data sources difficult
Checkin comments
Observation of developers
Interviews

15. Summary Software Testing very expensive
WEMSE Workshop
Software Testing very expensive
FC necessary to analyse & improve process
Process to develop domain-specific FC
Incorporating existing schemes
Iterative refinement by analyzing all steps
Quality review
Validation in industrial case study at RUAG

16. Thank you very much! Merci beaucoup ! Tack så mycket! Muchas gracias!
WEMSE Workshop
Thank you very much!
Merci beaucoup !
Tack så mycket!
Muchas gracias!
Muito obrigado!
Vielen Dank!
Moltes gràcies!
Mille grazie!