1. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong Fighting Against Software Defects CHEN Xinyu 2010-02-01

2. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 2 Outline  What software defect are  Why software defects occur  What can be done Detection Analysis Prevention  Conclusion Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 2

3. 3 Outline  What software defects are  Why software defects occur  What can be done Detection Analysis Prevention  Conclusion

4. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 4 1. What Software Defects Are A software defect is a deficiency in a software product that causes it to perform unexpectedly. Software Defect Number of Defects We need more information! Are we loosing control? http://www.research.ibm.com/softeng/ODC/ODCEG.HTM Minimizing the number of defects Software’s complexity and accelerated development schedules make avoiding defects difficult

5. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 5 2. Why Software Defects Occur  A defect in software results from some type of mistake Human error Systemic error in the development process  Guidelines and procedures  Domain-specific information  Organization information ……  Not every mistake leads to a defect, but almost all defects can be traced back to some type of mistake

6. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 6 Dynamics of Software Failure Production Problematic Specifications Cognitive Breakdowns Runtime Failures Runtime Errors  Requirements Software Defects Usability Issues Specifications Functionality Semantics … Programmer Knowledge Attention Strategies … Programming System Interfaces Information Notations … Program Algorithms Data structures Variables Classes Interfaces … A.J. Ko, B.A. Myers A frameworkand methodology for studying the causes of software errors in programming systems Journal of Visual Languages & Computing, 16, pp. 41–84, 2005.

7. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 7 3. What Can Be Done  Three Levels in Improving Software Quality Detection Prevention Analysis Look for trends and insights Proactively identify and eliminate potential defects Test quality into the software Learn from the past

8. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 8 A Bug's Life at Microsoft Product Code Run test case Enter bug report Sustained Engineering Transition Checklist Specify code fix Modify design Close bug Reactive bug Build workflow Resolve fixed bug Fix code Design OK? Design or code Triage Regress bug Close bug Resolve “Won’t Fix” Fail Pass Fix approved Investigation approved No or refine Proposed code fix Proposed design fix Code issue Design issue Not approved A. Page, K. Johnston, B. Rollison, How we test software at Microsoft Microsoft press, 2009

9. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 9 IBM: Orthogonal Defect Classification (ODC)  Defects are collected and analyzed in-process during an on-going software development  Classifying each defect into orthogonal (mutually exclusive) attributes Opener section: These are the attributes one can classify when you find a defect Closer section: These are the attributes one can classify when you know how the defect was fixed ODC Attributes Activity Trigger Impact Target Type Qualifier Source Age A.A. Shenvi Defect prevention with orthogonal defect classification ISEC’09, pp. 83-87, Feb. 2009

10. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 10 Missing Unclear Wrong Changed Better Way Mode (Why?) Mode (Why?) HP: Defect Origins, Types, and Modes Specifications/ Requirements Specifications/ Requirements Design Code Environmental Support Environmental Support Documentation Other Origin (Where?) Origin (Where?) Requirements or Specifications Functionality Other Requirements or Specifications Functionality Other HW Interface SW Interface User Interface Functional Description Other HW Interface SW Interface User Interface Functional Description Other Process (Interprocess) Communication Data Definition Module Design Logic Description Error Checking Standards Other Process (Interprocess) Communication Data Definition Module Design Logic Description Error Checking Standards Other Logic Computation Data Handling Module Interface / Implementation Standards Other Logic Computation Data Handling Module Interface / Implementation Standards Other Test SW Test HW Development Tools Integration SW Other Test SW Test HW Development Tools Integration SW Other Type (What?) Type (What?)

11. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 11 Defect Analysis Example

12. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 12 Root Cause Analysis  Identify where in the software development lifecycle a defect was introduced Identify where attention could be concentrated Misunderstood Requirements Misunderstood Requirements Missing/Incorrect Requirements/Design Missing/Incorrect Requirements/Design Scattered Requirements Scattered Requirements Implicit Requirements Not Addressed Implicit Requirements Not Addressed Domain Knowledge Domain Knowledge Functional Defects Functional Defects Fishbone or cause-and-effect diagram

13. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 13 Defect Prevention  A level 5 Key Process Area (KPA) in the Capability Maturity Model Integration (CMMI) A strategy applied to the software development life cycle that identifies root causes of defects and prevents them from recurring. Software Defect Prevention

14. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 14 Fan-out Phenomenon in Software Development Scenarios Specification Design Implementation Internal Testing Internal Testing Customer Testing Customer Testing Work product Time

15. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 15 Moving Quality Upstream Requirements Design & Implementation Test Maintenance Time Rate of Discovery Defect Prevention New Old 100X Increase in Cost of Removing Defects Late defect discovery results in significant correct and rework Software Defect Rate of Discovery vs. Time P. Narayana Software defect orevention - In a nutshell http://software.isixsigma.com/library/content/c030611a.asp

16. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 16 Failure Modes and Effects Analysis (FMEA)  Identify potential failure modes in a product design A failure mode is a type of failure that can occur in a product  Assess the risk of each potential failure  Implement appropriate actions to eliminate or mitigate those failure modes How can the system fail?What bad things will happen if the failure occurs? How many failures will the system experience?

17. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 17 FMEA Worksheet Process Step, Function or Task Potential Failure Mode Potential Effect(s) of Failure Potential Causes Current ControlImprovement Impact RankLikelihood Rank Risk Priority Owner Detectability Rank Fail to create log file dir Fatal error that stops setup Setup phase 5Dev oversight API failure No disk space Code review 3 2 2 Error message 2 30 20 Incorrect determine phase Fail to create log file dir Fatal error that stops setup Setup phase 5 Fail to create log file dir Fatal error that stops setup Setup phase 5 2 2 Try to reinitialize drive 5 Incorrect determine phase Unpredictable errors are generated 3 Incorrect use of API API failure 3 2 Stop (exit) setup Error message 1 218 10 Setup phase Setup phase Add to code review list Improve the API spec. Add disk space check Create more detailed API doc Improve the API spec. Dan John Sam Mary John Setup logging … Risk Priority Number (RPN) = Impact Severity  Occurrence Likelihood  Detectability

18. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 18 Fault Tree Analysis (FTA)  Starts with a failure  Focuses on deducing all the potential causes and their relationships using Boolean logic  FMEA and FTA are complementary techniques FMEA is used to identify potential failures FTA is then used to discover the causes of those failures

19. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 19 FTA Example

20. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 20 4. Conclusion  The causes of software defects Human errors Systemic errors  Three levels to improve software quality Defect detection  Testing Defect analysis  ODC  Defects origins, types, and modes  Root cause analysis Defect prevention  FMEA  FTA

21. Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 21 Reference  M. McDonald, R. Musson, R. Smith, The practical guide to defect prevention, Microsoft press, 2008 Thank You!