TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Generating a Test Strategy with Bayesian Networks, … and Common Sense Presentation by: Jean-Jacques Gras and Rishabh Gupta Software and Systems Engineering Research Lab Motorola Labs

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 2 Contents Introduction Bayesian Networks Defect Models Defect Profile Modelling Test Strategy Generation Sample Results Conclusion and Future Work

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 3 Introduction Problem: Software Testing is Difficult –Testing is coming late in lifecycle, means under pressure. –Information about Software weaknesses available too late to produce effective Test Strategy. Solution: Anticipate with Defect Prediction Models –Need advanced decision support tool to optimise V&V activities. –Use Predictions to drive V&V Activities –Use Predictions to generate intelligent Test Strategy by focussing on High Risk Areas. –Need to base models on real quality drivers: causal models –Use Bayesian Networks based Defect Models of each Activity to predict Quality of software artefacts along lifecycle.

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 4 Bayesian Nets l A graphical framework to reason about uncertainty l Based on Bayes theorem: P(B|A) = P(B) * P(A|B) / P(A) l Prediction: P(Effect | Cause) but also Inference: P(Cause | Effect) A: Training B: Knowledge D:Quality C: Requirements Arcs = Influence relations P(B|A) P(D|B,C) P(C) NPT = Conditional probability tables B = C = GoodPoorGoodPoor High Low HighLow D = Nodes = Uncertain variables P(A) A={True, False} or A={1, 2,…, n} or Continuous

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 5 End-To-End Defect Prediction Requirements Design Coding Coding FTR Design FTR Requirements FTR Unit Test Integration Test System Test B AT B AT B AT B AT B AT B AT B AT B AT B AT X - X - X - X X X X - X - X - Req Size Des SizeCod Size - X - - X X X X Latent Defects X X X X

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 6 BN Design Process Identify Causal Factors –Identify Factors that drive Software Quality or are indicators of it –Each factor is given a precise definition that can be understood by end-users Identify States and Ranges –Identify Node Type (Discrete or Continuous) –Identify States (for Discrete) or Ranges (for Continuous) Establish Cause-Effect Relations –Identify relationship between nodes –Use intermediate nodes where appropriate Quantify Cause-Effect Relations –Establish Conditional Probability Tables –Must define each relation

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 7 Additional Model Working BN Models may not be available –Models may not have been yet calibrated –Subjective data may not be available Schedule Pressure Process not instituted into organisation Need Predictions for Test Strategy. –Defect prediction for each Component/Sub-System Use alternative Model –to bootstrap the BN deployment –Multiple defect models provide reinforcements and credibility. Defect Profile Model (DPM)

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 8 DPM Method Use Defect Data from past Releases to understand Profile of Defect Discovery across Lifecycle Req = 65% Des = 86% Cod = 94% UT = 98% Actual: Req Faults Found ≈ 40 Estimate: 40/86% Total Req Faults Inserted ≈ 47 Prediction: Req Faults Unfound ≈ 7

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 9 DPM Process

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 10 DPM Process

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 11 DPM Key Points Data availability –Defect Data is usually collected by mature organisations No need for any other metrics –e.g. size of project, total effort Based on Component/Sub-System History –Generate a Prediction per Component/Sub-System –One DPM model for each Component/Sub-System –Calculate average ratio across multiple releases Historic Projects must be mature –Need Defect Data from the Field –More mature Projects mean better Prediction

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 12 System Test Strategy Generation Test Strategy Services# TCDefects Service 1601 Service Service Service Service Service Service Service 8601 Service BN Services ServiceBox ABox BBox C Service 110%55%90% Service 20%17%60% Service 330%20%70% Service 470%50%80% Service 520%50%30% Service 660%50%70% Service 710%20%10% Mapping components influence Ranked Services: Defects exposed + Suggested # Test Cases Service (FA) Models: Defect exposure calculation Component latent defects Component Models Latent defects -> Component Models Latent defects -> Component Models Latent defects -> Development Survey

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 13 Test Strategy – “Exposure” Matrix Method –Defect “Exposure” weight based on System Design information –Test Strategy = ranked services (per Test Area) Benefit –Identifies weakest services –Determines number of test cases to write for each Service SubSys 1SubSys 2SubSys 3 … 0% 20% 80% 60%20% 100%80%0% *** (Exposed Faults) = 47 Faults left Strategy = 63 ………… = 52 “Exposure” weights from experts Service 4.1 Service 4.2 Service 4.3

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 14 Suite 2 Development Applying BTA to Drive Testing System Test Pre-test Development System Test New Feature Test TCs BTA Outputs Suite 1 Component Defects Test Selection Preliminary Test Strategy Updated Test Strategy Updated data Preliminary data Create Test Cases

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 15 End-to-End Defect Prediction Results PilotTypeSample SizeAccuracy ADevelopment Faults16 components+/- 25% to actuals, 88% of the time BSystem Test Effectiveness15 areas+/- 25% to actuals, 80% of the time CTest Case Selection8 components 19 services 48% more faults detected than random selection = 5 yr. expert DThird-Party Faults5 vendors+/- 25% to actuals, 80% of the time EDevelopment Faults5 component+/- 16.4%, 100% of the time 7% effort reduction FTest Strategy1 system release 67 test categories 1/3 number of test cases 2X functional areas Success Criteria: Prediction accuracy of +/- 25% to actuals, 75% of the time

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 16 Defects Found –BTA Test Strategy would have found all the Defects Found by the original Test Strategy Test Cases –BTA Test Strategy recommends 1/3 the number of Test Cases as the original Test Strategy Test Strategy Impact Functional Areas –BTA Test Strategy recommends 2x the number of Functional Areas as the original Test Strategy Test CasesFunctional AreasDefects Found

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 17 Test Strategy Optimization Original unoptimised Test Strategy Execution Time to remove all Defects Optimised Test Strategy Execution Time to remove all Defects Reduced Savings

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 18 Future Work & Conclusion BN Defect Models –Better calibration techniques – machine learning –Simpler “Lightweight” model DPM –A “2D” version to allow earlier predictions with less information –Integration BN models with DPM approach leverage strength of each Test Strategy –More detailed exposure information Include type or category of Defect Exposure per Defect Type, not just Component –Automatic extraction of Exposure information, e.g. through UML / MSC Models

TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006 MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Slide 19 Q & A Thank You Contacts