Software Defect Prevention Using Orthogonal Defect Classification Twin-SPIN January 6, 2005 Presented by: Megan Graham, ASQ CSQE megan@metagraham.net

Overview Defect prevention is a process used to improve software quality Orthogonal Defect Classification is a tool that characterizes defect data used in defect analysis

Agenda About Defect Prevention ODC Concepts Applying ODC to the DP Process Summary

Agenda About Defect Prevention ODC Concepts Applying ODC to the DP Process Summary

Defining Defect Prevention What does it mean to you? Defect prevention is a process whose purpose is to: identify the common causes of defects, and change the relevant process(es) to prevent that type of defect from recurring. (SEI) Take what we already know and apply it to what we think we know to produce quality software. Key points: It is a process in and of itself that focuses on the system of software development What we already know -> DP uses historical data – either ancient history or recent history, but what’s done is done and we learn from our mistakes Apply it: This feeds an intentional and specific action or set of actions to eliminate the conditions that allowed the defect. What we think we know: We think that the current process will be similar enough to predict future actions. Quality software: Quality depends on business needs and goals. Sufficiently good software, mission critical, etc.

A Bug’s Life Requirements Design Code Test This simple software lifecycle shows many stages at which point bugs may be injected and/or detected. Injected: Any activity whose purpose is to create software or analyze acceptable behavior (requirements, coding, debugging, testing). Detection: Any activity whose purpose is to find defects (testing, review, inspections, etc.) Bugs are injected into the process at different stages Click: We have strategies in place to detect and kill the bugs Click: Optimally, we detect bugs as close to the point of injection as possible – this is called defect containment Test

A Bird’s Eye View of DP Defect Prevention Historical Project Decrease Defects Injected Increase Detected Defect Prevention Historical Project Current Project Optimally, defect prevention uses defect data throughout the software lifecycle to improve both defect injection and defect detection processes for the next phase. This will ultimately result in higher quality software, and most likely cheaper.

Applying Defect Prevention A defect in the software is also a defect in the process (injection and/or detection) For DP to work, we need to turn software defects into actionable process defects Software defects Process DP In order for DP to work, you need to understand how the software defect manifested itself. How do you do that?

And then there was ODC… Orthogonal Defect Classification Developed at IBM in the 1990s by Ram Chillarege Methodology to characterize software defects and translate into process defects

Agenda About Defect Prevention ODC Concepts Applying ODC to the DP Process Summary

Simple ODC Classification Scheme Type Defect ODC creates defect profiles and other defect data that can be used for various metrics Trigger

Defect Types FUNCTION: Affects significant capability, end-user interfaces, product interfaces, interface with hardware architecture or global data structure(s). LOGIC: Affects the functionality of a code module and can be fixed by re-implementing an algorithm or local data structure without a need for requesting a high level design change. INTERFACE: Affects the interaction of components via macros, call statements and/or parameters. CHECKING: Affects program logic that would properly validate data and values before they are stored or used in computation. ASSIGNMENT: Requires change in a few lines of code, such as initialization of control blocks or data structures. TIMING/SERIALIZATION: Affects the proper management of shared and real-time resources. BUILD/PACKAGE/MERGE: Affects product version or configuration; requires a formal changes to reconfigure or rebuild the product.

Defect Triggers Fault Failure Trigger 1 Trigger 2 Trigger n A trigger is any event that allowed a fault to become a failure. Multiple triggers may uncover the same fault. The same trigger may be used at any point in the software lifecycle. There are 3 families of triggers: review/inspection, function test, and system test.

Review/Inspection Triggers DESIGN CONFORMANCE: Comparing the implemented design against a reference –design document, pattern, or guideline. LOGIC/FLOW: Checking for correctness or flaws using knowledge of the practice. BACKWARD COMPATIBILITY: Examining compatibility with prior version of the product. LATERAL COMPATIBILITY: Examining for compatibility with other products and platforms that need to work with this release. CONCURRENCY: Serialization, shared resources, multi-threaded tasks, timing, etc. INTERNAL DOCUMENT: Inconsistencies in prologs, and sections in the same work product. LANGUAGE DEPENDENCY: Programming standards, specific implementation considerations, environment restrictions, execution modes, etc. SIDE EFFECTS: Usage behavior beyond design, but relevant in context. Do A; B happens. RARE SITUATION: Unusual issues related to idiosyncrasy of environment, hardware, or software.

Function Test Triggers SIMPLE PATH: White box – Straightforward usage of code path and branches. COMPLEX PATH: White box – Exercising conditionals, and circuitous coverage. COVERAGE: Black box – Straightforward usage of function or single parameterized execution. VARIATION: Black box – Straightforward like coverage, but with a variety of parameters. SEQUENCING: Black box – Multiple functions, with a specific sequence (order is important). INTERACTION: Black box – When two or more bodies of code are involved (order is not important).

System Test Triggers WORKLOAD STRESS: Pushing the limits of performance, resources, users, queues, traffic, etc. RECOVERY: Invoke exception handling, recovery, termination, error percolation, etc. STARTUP/RESTART: Major events of turning on, off, or changing the degree of service availability. HARDWARE CONFIGURATION: Issues surfaced as a consequence of changes in hardware setup. SOFTWARE CONFIGURATION: Issues surfaced as a consequence of changes in software setup. BLOCKED TEST/NORMAL MODE: Test could not run during System Test, or customer found nonspecific trigger. Look for additional trigger.

ODC v5.11 Classification Scheme Trigger Activity Impact Opener Closer Defect Age Target IBM uses more fields, with their own field options This scheme can be customized to fit your own organization’s process and metrics needs Type Source Qualifier Source: ODC v5.11, IBM Center for Software Engineering, www.research.ibm.com/softeng

Agenda About Defect Prevention ODC Concepts Applying ODC to the DP Process Summary

When to Apply DP DP can be applied to one or more phases of the software lifecycle Depends on maturity, goals, etc. RQTS DSGN CODE TEST Defect Prevention Field Analysis Action

Defect Analysis Orthogonal Defect Classification What are the attributes of the defects? Defect Causal Analysis When are defects being injected? Defect Trigger Analysis How are defects being detected? Two types of analyses The more data you collect, the more you can analyze (but be careful to avoid collecting data just to collect data – use it or lose it) ODC provides a structure for the defect data

Defect Causal Analysis Purpose: Characterize process issues that lead to injection of defects. Step 1: ID a set of defects. Step 2: Identify Defect Types with team of experts. Step 3: Plot the distribution of Defect Types. Step 4: Map Defect Types back to development activity. Step 5: Develop action plan to address process deficiencies. Step 6: Monitor process to ensure changes were effective. If you are performing the typing in hindsight, best to get a team of engineers that were involved with the code If you add the field to the bug reports, then you just need to plot periodically

Defect Type Mapped to Phase H/L Level Design Interface Configuration Mgmt Build/Package/Merge Implementation Timing/Serialization Assignment LLD/Implementation Checking Low Level Design Logic Architecture/HLD Function

Sample Distribution Plot number or percentage, then nice to plot biggest to smallest; work on biggest first

Defect Trigger Analysis Purpose: Characterize process issues that allowed defects to escape to later phases. Step 1: ID a set of defects. Step 2: Identify Defect Trigger with team of experts. Step 3: Plot distributions: Defect Trigger by Family, Review Triggers, Function Test Triggers, System Test Triggers. Step 4: Map Defect Trigger family back to detection activity(ies). Step 5: Develop action plan to increase missed Defect Triggers. Step 6: Monitor process to ensure changes were effective. DTA can be used in two ways; one, for process improvement, and 2, for status

Defect Trigger Family Mapped to Phase Testing in real-world scenarios or extreme scenarios System Test Testing of specified scenarios Function Test Peer Reviews or Inspections Review/Inspection

Sample Distributions

Additional Use for DTA Purpose: Determine quality status. Step 1: Plot distribution of Defect Triggers after each detection activity. Step 2: Compare to historical defect profile to determine if profile is as expected. Step 3: Develop action plan, if necessary, to get back on track. If your project thinks it’s one phase, but the distribution is of an earlier phase, then you’re really in the earlier phase!

Agenda About Defect Prevention ODC Concepts Applying ODC to the DP Process Summary

The Big Picture: DP & ODC Mission: Improve software quality by using readily available data to decrease defects injected and increase defects detected. Defect Prevention Analysis Action RQTS DSGN CODE TEST Field ODC Classification of Software Defects

Important Points about ODC A defect in the software is a defect in the process. Implementing ODC is very cost-effective Enhances data already collected (software defects) Adding fields that are completed real-time make data collection virtually free! Tooled to quickly identify process defects (mapping) ODC can be implemented in stages Start with field defects, then move to in-process analysis Utilize defect profiling in-process to predict quality and project status Fields can be tailored to your own organization

References Handbook of Software Reliability Engineering (Michael R. Lyu, Editor; IEEE Computer Society Press/McGraw-Hill) Ram Chillarege (www.chillarege.com) IBM Research Center for Software Engineering (www.research.ibm.com/softeng) Soheil Khajenoori, SIAGroup (soheil@siagroup.us)