Risk Based Testing Robert Sabourin President AmiBug.Com, Inc. Montreal, Canada rsabourin@amibug.com April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Robert Sabourin , Software Evangelist President AmiBug.Com Inc. Montreal, Quebec, Canada rsabourin@amibug.com April 27, 2019 © Robert Sabourin, 2007

Overview Introduction Some Philosophy Purpose Context Project Risk Product Risk Triage Be Informed Fundamental Question April 27, 2019 © Robert Sabourin, 2007

April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Some Philosophy April 27, 2019 © Robert Sabourin, 2007

Fundamental Question How do you know when you are finished? April 27, 2019 © Robert Sabourin, 2007

Edsger W. Dijkstra “Program testing can be used to show the presence of bugs, but never to show their absence” April 27, 2019 © Robert Sabourin, 2007

Boris Beizer “Why software has bugs – the fundamental problem – Programming is a bitch.” The Frozen Keyboard - 1986 April 27, 2019 © Robert Sabourin, 2007

Watts S. Humphrey “… the job of the software engineer is to deliver high-quality software products at agreed cost and schedule …” “… even the most experienced software engineer injects about one defect for ten lines of code …” April 27, 2019 © Robert Sabourin, 2007

C. Northcote Parkinson Parkinson’s Law: “…work expands so as to fill the time available for its completion…” April 27, 2019 © Robert Sabourin, 2007

Yoda "No! Try not, Do. Or do not. There is no try." April 27, 2019 © Robert Sabourin, 2007

“…begin with the end in mind … Steve Covey “…begin with the end in mind … “…first things first …" April 27, 2019 © Robert Sabourin, 2007

Pareto Principal Vilfredo Pareto, 1848 - 1923, Economist 80% of the wealth was in the hands of 20% of the population April 27, 2019 © Robert Sabourin, 2007

Pareto Principal Joseph Juran, 1903 - present, Quality Control Engineer 1950 Quality Control Handbook 20% of the study population accounts for 80% of the measure under consideration April 27, 2019 © Robert Sabourin, 2007

Testing Risk “Software testing begins and ends with risk!” Rick Craig April 27, 2019 © Robert Sabourin, 2007

April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Purpose April 27, 2019 © Robert Sabourin, 2007

Purpose of Testing Common definition: Broader definition: To find bugs before our customers do! Broader definition: The role of testing is to provide objective input to facilitate business decisions! Keeps stakeholders aware of all issues or concerns that relate to shipping a product! April 27, 2019 © Robert Sabourin, 2007

Bug Defined To make our job more fun, whenever we have a concern with software, we call it a “bug”. April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing It’s all about people! (and the occasional bug too) April 27, 2019 © Robert Sabourin, 2007

About Bugs Bugs are not Good or Bad April 27, 2019 © Robert Sabourin, 2007

About Bugs Some bugs are important and have a high priority! April 27, 2019 © Robert Sabourin, 2007

About Bugs Some bugs are dangerous and have a high severity! April 27, 2019 © Robert Sabourin, 2007

About Bugs Setting the priority and severity of a bug is a business decision Changing business conditions impact the priority and severity of a bug! Always review previous decisions in light of changing business context Ensure staff assigning priority and severity are aware of all relevant business drivers April 27, 2019 © Robert Sabourin, 2007

Bug Quadrants April 27, 2019 © Robert Sabourin, 2007

Quadrant Changing Same technical bug can be in a different quadrant depending on the business context Monitor business drivers! Focus find and fix high priority/high severity bugs April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Context April 27, 2019 © Robert Sabourin, 2007

April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Project Risks April 27, 2019 © Robert Sabourin, 2007

Project Risks People Places Things Environment Training Resources Dependencies Contingencies Mitigation How? April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Product Risks April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Testing based on Risk Analysis How? April 27, 2019 © Robert Sabourin, 2007

Insurance Testing is similar to insurance Testing is protection against risk of failure How? April 27, 2019 © Robert Sabourin, 2007

Math of Risk Traditional Risk Variables Probability of Failure P(i) Consequence of Failure C(i) Expected Utility U(i) The Math U(i) = P(i) x C(i) How? April 27, 2019 © Robert Sabourin, 2007

Technical Risk What can break? Probability of Failure P(i) Technical Risk Changes Code Developers Technical Context Tools Architecture Environment Clusters How? April 27, 2019 © Robert Sabourin, 2007

Technical Risk Elicitation of Technical Risk Interview How? Developers Architects DBAs Technical Analysts System Administrators Vendors Technical community How? April 27, 2019 © Robert Sabourin, 2007

Technical Risk Measuring Technical Risk Static Analysis How? LOCS Branches Paths Complexity Tools Change in Source Code Inspect Reviews How? April 27, 2019 © Robert Sabourin, 2007

Technical Risk Quantifying Technical Risk How? Probability Almost impossible Relative Scales of measure (3 – 5 levels) High Medium Low Inconsistency Methods Business Focus on change How? April 27, 2019 © Robert Sabourin, 2007

Business Risk Impact of Failure? Consequence of Failure C(i) Business Importance Impact of failure on business SLA Violations Support Maintenance Costs Lost business Lost reputation Lost productivity How? April 27, 2019 © Robert Sabourin, 2007

Business Risk Elicitation of Business Risk Product Manager Project Manager Program Manager Contracts Support and Help Desk Legal Stakeholders Sales How? April 27, 2019 © Robert Sabourin, 2007

Business Risk Quantifying of Business Risk Value Dollars Scale (3 – 5 levels) High Medium Low Project Office How? April 27, 2019 © Robert Sabourin, 2007

Expected Utility Expected Utility Traditional Computation U(i) = P(i) x C(i) Traditional Computation Higher utility more focus Lower utility less focus Effort distribution Proportional to utility How? April 27, 2019 © Robert Sabourin, 2007

Problem with formulas Units? Math? Exposure(i) = Risk * Consequence Allocation(i) = (Exposure(i)/Total Exposure) * MAX Units? Math? April 27, 2019 © Robert Sabourin, 2007

April 27, 2019 © Robert Sabourin, 2007

April 27, 2019 © Robert Sabourin, 2007

Quality Factors April 27, 2019 © Robert Sabourin, 2007

Triage – Risk Based Decisions Risk Based Testing Triage – Risk Based Decisions April 27, 2019 © Robert Sabourin, 2007

Which test? Impact estimation For each test idea guesstimate: benefit of implementation consequence of implementation benefit for not implementing consequence of not implementing How credible are values? April 27, 2019 © Robert Sabourin, 2007

Understanding Complex Technology Quantitatively By Tom Gilb How to Decide? Rank Credibility 0.0 Wild guess, no credibility 0.1 We know it has been done somewhere 0.2 We have one measurement somewhere 0.3 There are several measurements in the estimated range 0.4 The measurements are relevant to our case 0.5 The method of measurement is considered reliable 0.6 We have used the method in-house 0.7 We have reliable measurements in-house 0.8 Reliable in-house measurements correlate to independent external measurements 0.9 We have used the idea on this project and measured it 1.0 Perfect credibility, we have rock solid, contract- guaranteed, long-term, credible experience with this idea on this project and, the results are unlikely to disappear April 27, 2019 © Robert Sabourin, 2007

Which test? Test Idea Rejection – What If? If the cost/benefit does not make business sense then consider implementing: part of the test, could that lead to part of the benefit at a more reasonable cost? more than the stated test, would that generate more benefit? a different test than the stated idea, could that generate more benefit for less cost? April 27, 2019 © Robert Sabourin, 2007

Test Triage Test Triage JIT Projects High Frequency Daily Test Triage Session Experience dictates Early AM (Rob Preference) Late PM (several clients) April 27, 2019 © Robert Sabourin, 2007

Test Triage Test Triage Meeting Review Context Business Technical Information since last triage Test results Bug results New testing ideas April 27, 2019 © Robert Sabourin, 2007

Test Triage Allocate Testing Assignments to Testers Make sure testers know context Best thing to test Best person to test it Best people to explore it Best lead Assign subject matter experts is required Sessions may be scripted or exploratory April 27, 2019 © Robert Sabourin, 2007

Test Triage Requirement Triage Change Control Test Triage Bug Flow Combined Equivalent to CCB Few people Fluid April 27, 2019 © Robert Sabourin, 2007

Test Triage Life of a test idea Comes into existence Clarified Prioritized Test Now (before further testing) Test before shipping Nice to have May be of interest in some future release Not of interest in current form Will never be of interest Integrate into a testing objective April 27, 2019 © Robert Sabourin, 2007

Which test is next? Magic crystal ball Ask the question Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next? How much effort are we willing to spend continuing to test this project? Can we ship yet? April 27, 2019 © Robert Sabourin, 2007

Which test is next? Magic crystal ball If it existed then how would you use it? What question would you ask it? What question would it ask you? April 27, 2019 © Robert Sabourin, 2007

Which test is next? Magic crystal ball Discover Example questions What question to ask? What information to have at hand? Example questions Given these test objectives how many sessions should I dedicate to them? Given that this part of the application is very buggy what should I test otherwise? April 27, 2019 © Robert Sabourin, 2007

Deciding what not to test? Time pressure Should we skip a test? If test failed could system still be of value to some stakeholder? If test was skipped could important bugs have been otherwise found? April 27, 2019 © Robert Sabourin, 2007

Guidelines and Decisions To each stakeholder risk of failure consequence of failure value of success how much certainty do we have is it a wild guess or an absolute truth? April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Be Informed April 27, 2019 © Robert Sabourin, 2007

Sources of Information Version control system Monitor changes Track where work is Track where stability is Encourage finding defects earlier than system testing Inspections of code, design, requirements Unit Testing Informal code check in peer reviews April 27, 2019 © Robert Sabourin, 2007

Philosophy We have precious little time to run tests! We must always be prepared! April 27, 2019 © Robert Sabourin, 2007

Time April 27, 2019 © Robert Sabourin, 2007

Getting Things Done Concern Concern Being Prepared! Being Prepared! - Information Flow - Information Flow Corporate information Key business drivers Sales Market Finance Corporate information Key business drivers Sales Market Finance - Technology Flow Architecture Technology churn Tools Techniques Training - Requirement Flow Defined Understood Interrupt Poll Prioritize Turbulence Status Truffle - Bug Flow Defined Understood Business Technical Efficient Expedient Reassess - Test Objectives Quality Factors Technical Risk Failure Modes Importance - Test Strategy Plan Analytic Exploratory Checklists Parallel Chunking Scenarios Data - Test Organization Scheduling Staffing Outsourcing Contractors Students - Testing Lab Multi-tier Server Client Platforms Swap Pattern Synchronized - Test Status Bug charts Test Plan Elaboration Status Pass Fail Execution Status April 27, 2019 © Robert Sabourin, 2007

April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Fundemental Question April 27, 2019 © Robert Sabourin, 2007

Finished? How do you know you are finished? April 27, 2019 © Robert Sabourin, 2007

You know you are finished when … … the only bugs left are the ones are acceptable (based on your objective test team input) ... April 27, 2019 © Robert Sabourin, 2007

You know you are finished when … … the only bugs left are the ones are acceptable (based on your objective test team input) ... At least for now! April 27, 2019 © Robert Sabourin, 2007

Thank You Questions? April 27, 2019 © Robert Sabourin, 2007

April 27, 2019 © Robert Sabourin, 2007

Risk Based Testing Case Studies April 27, 2019 © Robert Sabourin, 2007

Case Study System Test Planning Identify potential test objectives Elicit business impact of failure Elicit likelihood of failure Estimate priority Adjust scope Spread budget over test objectives April 27, 2019 © Robert Sabourin, 2007

Case Study Focused Regression Testing What changed? How significant is each change? Which testable object is impacted by each change? Aggregate impact to all testable objects Focus regression April 27, 2019 © Robert Sabourin, 2007

Case Study High Impact Scenario Based Testing Identify usage scenarios Storyboard flows Normal Alternate Error Variables are points of decision or input Walking through from start to completion Correct defects which block scenarios April 27, 2019 © Robert Sabourin, 2007