1. Risk Based Testing Robert Sabourin President AmiBug.Com, Inc.
Montreal, Canada
April 27, 2019
© Robert Sabourin, 2007

2. Risk Based Testing Robert Sabourin , Software Evangelist President
AmiBug.Com Inc.
Montreal, Quebec, Canada
April 27, 2019
© Robert Sabourin, 2007

3. Overview Introduction Some Philosophy Purpose Context Project Risk
Product Risk
Triage
Be Informed
Fundamental Question
April 27, 2019
© Robert Sabourin, 2007

4. April 27, 2019
© Robert Sabourin, 2007

5. Risk Based Testing Some Philosophy April 27, 2019
© Robert Sabourin, 2007

6. Fundamental Question How do you know when you are finished?
April 27, 2019
© Robert Sabourin, 2007

7. Edsger W. Dijkstra
“Program testing can be used to show the presence of bugs, but never to show their absence”
April 27, 2019
© Robert Sabourin, 2007

8. Boris Beizer
“Why software has bugs – the fundamental problem – Programming is a bitch.”
The Frozen Keyboard
April 27, 2019
© Robert Sabourin, 2007

9. Watts S. Humphrey
“… the job of the software engineer is to deliver high-quality software products at agreed cost and schedule …”
“… even the most experienced software engineer injects about one defect for ten lines of code …”
April 27, 2019
© Robert Sabourin, 2007

10. C. Northcote Parkinson Parkinson’s Law:
“…work expands so as to fill the time available for its completion…”
April 27, 2019
© Robert Sabourin, 2007

11. Yoda "No! Try not, Do. Or do not. There is no try." April 27, 2019
© Robert Sabourin, 2007

12. “…begin with the end in mind …
Steve Covey
“…begin with the end in mind …
“…first things first …"
April 27, 2019
© Robert Sabourin, 2007

13. Pareto Principal Vilfredo Pareto, 1848 - 1923, Economist
80% of the wealth was in the hands of 20% of the population
April 27, 2019
© Robert Sabourin, 2007

14. Pareto Principal
Joseph Juran, present, Quality Control Engineer
1950 Quality Control Handbook
20% of the study population accounts for 80% of the measure under consideration
April 27, 2019
© Robert Sabourin, 2007

15. Testing Risk “Software testing begins and ends with risk!” Rick Craig
April 27, 2019
© Robert Sabourin, 2007

16. April 27, 2019
© Robert Sabourin, 2007

17. Risk Based Testing
Purpose
April 27, 2019
© Robert Sabourin, 2007

18. Purpose of Testing Common definition: Broader definition:
To find bugs before our customers do!
Broader definition:
The role of testing is to provide objective input to facilitate business decisions!
Keeps stakeholders aware of all issues or concerns that relate to shipping a product!
April 27, 2019
© Robert Sabourin, 2007

19. Bug Defined
To make our job more fun, whenever we have a concern with software, we call it a “bug”.
April 27, 2019
© Robert Sabourin, 2007

20. Risk Based Testing It’s all about people! (and the occasional bug too)
April 27, 2019
© Robert Sabourin, 2007

21. About Bugs Bugs are not Good or Bad April 27, 2019
© Robert Sabourin, 2007

22. About Bugs Some bugs are important and have a high priority!
April 27, 2019
© Robert Sabourin, 2007

23. About Bugs Some bugs are dangerous and have a high severity!
April 27, 2019
© Robert Sabourin, 2007

24. About Bugs
Setting the priority and severity of a bug is a business decision
Changing business conditions impact the priority and severity of a bug!
Always review previous decisions in light of changing business context
Ensure staff assigning priority and severity are aware of all relevant business drivers
April 27, 2019
© Robert Sabourin, 2007

25. Bug Quadrants
April 27, 2019
© Robert Sabourin, 2007

26. Quadrant Changing
Same technical bug can be in a different quadrant depending on the business context
Monitor business drivers!
Focus find and fix high priority/high severity bugs
April 27, 2019
© Robert Sabourin, 2007

27. Risk Based Testing
Context
April 27, 2019
© Robert Sabourin, 2007

28. April 27, 2019
© Robert Sabourin, 2007

29. Risk Based Testing Project Risks April 27, 2019
© Robert Sabourin, 2007

30. Project Risks People Places Things Environment Training Resources
Dependencies
Contingencies
Mitigation
How?
April 27, 2019
© Robert Sabourin, 2007

31. Risk Based Testing Product Risks April 27, 2019
© Robert Sabourin, 2007

32. Risk Based Testing Testing based on Risk Analysis How? April 27, 2019
© Robert Sabourin, 2007

33. Insurance Testing is similar to insurance
Testing is protection against risk of failure
How?
April 27, 2019
© Robert Sabourin, 2007

34. Math of Risk Traditional Risk Variables Probability of Failure
P(i)
Consequence of Failure
C(i)
Expected Utility
U(i)
The Math
U(i) = P(i) x C(i)
How?
April 27, 2019
© Robert Sabourin, 2007

35. Technical Risk What can break?
Probability of Failure
P(i)
Technical Risk
Changes
Code
Developers
Technical Context
Tools
Architecture
Environment
Clusters
How?
April 27, 2019
© Robert Sabourin, 2007

36. Technical Risk Elicitation of Technical Risk Interview How? Developers
Architects
DBAs
Technical Analysts
System Administrators
Vendors
Technical community
How?
April 27, 2019
© Robert Sabourin, 2007

37. Technical Risk Measuring Technical Risk Static Analysis How? LOCS
Branches
Paths
Complexity
Tools
Change in Source Code
Inspect
Reviews
How?
April 27, 2019
© Robert Sabourin, 2007

38. Technical Risk Quantifying Technical Risk How? Probability
Almost impossible
Relative
Scales of measure (3 – 5 levels)
High
Medium
Low
Inconsistency
Methods
Business
Focus on change
How?
April 27, 2019
© Robert Sabourin, 2007

39. Business Risk Impact of Failure?
Consequence of Failure
C(i)
Business Importance
Impact of failure on business
SLA Violations
Support Maintenance Costs
Lost business
Lost reputation
Lost productivity
How?
April 27, 2019
© Robert Sabourin, 2007

40. Business Risk Elicitation of Business Risk Product Manager
Project Manager
Program Manager
Contracts
Support and Help Desk
Legal Stakeholders
Sales
How?
April 27, 2019
© Robert Sabourin, 2007

41. Business Risk Quantifying of Business Risk Value Dollars
Scale (3 – 5 levels)
High
Medium
Low
Project Office
How?
April 27, 2019
© Robert Sabourin, 2007

42. Expected Utility Expected Utility Traditional Computation
U(i) = P(i) x C(i)
Traditional Computation
Higher utility
more focus
Lower utility
less focus
Effort distribution
Proportional to utility
How?
April 27, 2019
© Robert Sabourin, 2007

43. Problem with formulas Units? Math? Exposure(i) = Risk * Consequence
Allocation(i) = (Exposure(i)/Total Exposure) * MAX
Units?
Math?
April 27, 2019
© Robert Sabourin, 2007

44. April 27, 2019
© Robert Sabourin, 2007

45. April 27, 2019
© Robert Sabourin, 2007

46. Quality Factors
April 27, 2019
© Robert Sabourin, 2007

47. Triage – Risk Based Decisions
Risk Based Testing
Triage – Risk Based Decisions
April 27, 2019
© Robert Sabourin, 2007

48. Which test? Impact estimation For each test idea guesstimate:
benefit of implementation
consequence of implementation
benefit for not implementing
consequence of not implementing
How credible are values?
April 27, 2019
© Robert Sabourin, 2007

49. Understanding Complex Technology Quantitatively By Tom Gilb
How to Decide?
Rank
Credibility
0.0
Wild guess, no credibility
0.1
We know it has been done somewhere
0.2
We have one measurement somewhere
0.3
There are several measurements in the estimated range
0.4
The measurements are relevant to our case
0.5
The method of measurement is considered reliable
0.6
We have used the method in-house
0.7
We have reliable measurements in-house
0.8
Reliable in-house measurements correlate to independent external measurements
0.9
We have used the idea on this project and measured it
1.0
Perfect credibility, we have rock solid, contract- guaranteed, long-term, credible experience with this idea on this project and, the results are unlikely to disappear
April 27, 2019
© Robert Sabourin, 2007

50. Which test? Test Idea Rejection – What If?
If the cost/benefit does not make business sense then consider implementing:
part of the test, could that lead to part of the benefit at a more reasonable cost?
more than the stated test, would that generate more benefit?
a different test than the stated idea, could that generate more benefit for less cost?
April 27, 2019
© Robert Sabourin, 2007

51. Test Triage Test Triage JIT Projects High Frequency
Daily Test Triage Session
Experience dictates
Early AM (Rob Preference)
Late PM (several clients)
April 27, 2019
© Robert Sabourin, 2007

52. Test Triage Test Triage Meeting Review Context
Business
Technical
Information since last triage
Test results
Bug results
New testing ideas
April 27, 2019
© Robert Sabourin, 2007

53. Test Triage Allocate Testing Assignments to Testers
Make sure testers know context
Best thing to test
Best person to test it
Best people to explore it
Best lead
Assign subject matter experts is required
Sessions may be scripted or exploratory
April 27, 2019
© Robert Sabourin, 2007

54. Test Triage Requirement Triage Change Control Test Triage Bug Flow
Combined
Equivalent to CCB
Few people
Fluid
April 27, 2019
© Robert Sabourin, 2007

55. Test Triage Life of a test idea Comes into existence Clarified
Prioritized
Test Now (before further testing)
Test before shipping
Nice to have
May be of interest in some future release
Not of interest in current form
Will never be of interest
Integrate into a testing objective
April 27, 2019
© Robert Sabourin, 2007

56. Which test is next? Magic crystal ball Ask the question
Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?
How much effort are we willing to spend continuing to test this project?
Can we ship yet?
April 27, 2019
© Robert Sabourin, 2007

57. Which test is next? Magic crystal ball
If it existed then how would you use it?
What question would you ask it?
What question would it ask you?
April 27, 2019
© Robert Sabourin, 2007

58. Which test is next? Magic crystal ball Discover Example questions
What question to ask?
What information to have at hand?
Example questions
Given these test objectives how many sessions should I dedicate to them?
Given that this part of the application is very buggy what should I test otherwise?
April 27, 2019
© Robert Sabourin, 2007

59. Deciding what not to test?
Time pressure
Should we skip a test?
If test failed could system still be of value to some stakeholder?
If test was skipped could important bugs have been otherwise found?
April 27, 2019
© Robert Sabourin, 2007

60. Guidelines and Decisions
To each stakeholder
risk of failure
consequence of failure
value of success
how much certainty do we have
is it a wild guess or an absolute truth?
April 27, 2019
© Robert Sabourin, 2007

61. Risk Based Testing
Be Informed
April 27, 2019
© Robert Sabourin, 2007

62. Sources of Information
Version control system
Monitor changes
Track where work is
Track where stability is
Encourage finding defects earlier than system testing
Inspections of code, design, requirements
Unit Testing
Informal code check in peer reviews
April 27, 2019
© Robert Sabourin, 2007

63. Philosophy We have precious little time to run tests!
We must always be prepared!
April 27, 2019
© Robert Sabourin, 2007

64. Time
April 27, 2019
© Robert Sabourin, 2007

65. Getting Things Done Concern Concern Being Prepared! Being Prepared!
- Information Flow
- Information Flow
Corporate information Key business drivers Sales Market Finance
Corporate information Key business drivers Sales Market Finance
- Technology Flow
Architecture Technology churn Tools Techniques Training
- Requirement Flow
Defined Understood Interrupt Poll Prioritize Turbulence Status Truffle
- Bug Flow
Defined Understood Business Technical Efficient Expedient Reassess
- Test Objectives
Quality Factors Technical Risk Failure Modes Importance
- Test Strategy
Plan Analytic Exploratory Checklists Parallel Chunking Scenarios Data
- Test Organization
Scheduling Staffing Outsourcing Contractors Students
- Testing Lab
Multi-tier Server Client Platforms Swap Pattern Synchronized
- Test Status
Bug charts Test Plan Elaboration Status Pass Fail Execution Status
April 27, 2019
© Robert Sabourin, 2007

66. April 27, 2019
© Robert Sabourin, 2007

67. Risk Based Testing Fundemental Question April 27, 2019
© Robert Sabourin, 2007

68. Finished? How do you know you are finished? April 27, 2019
© Robert Sabourin, 2007

69. You know you are finished when …
… the only bugs left are the ones are acceptable (based on your objective test team input) ...
April 27, 2019
© Robert Sabourin, 2007

70. You know you are finished when …
… the only bugs left are the ones are acceptable (based on your objective test team input) ...
At least for now!
April 27, 2019
© Robert Sabourin, 2007

71. Thank You
Questions?
April 27, 2019
© Robert Sabourin, 2007

72. April 27, 2019
© Robert Sabourin, 2007

73. Risk Based Testing
Case Studies
April 27, 2019
© Robert Sabourin, 2007

74. Case Study System Test Planning Identify potential test objectives
Elicit business impact of failure
Elicit likelihood of failure
Estimate priority
Adjust scope
Spread budget over test objectives
April 27, 2019
© Robert Sabourin, 2007

75. Case Study Focused Regression Testing What changed?
How significant is each change?
Which testable object is impacted by each change?
Aggregate impact to all testable objects
Focus regression
April 27, 2019
© Robert Sabourin, 2007

76. Case Study High Impact Scenario Based Testing Identify usage scenarios
Storyboard flows
Normal
Alternate
Error
Variables are points of decision or input
Walking through from start to completion
Correct defects which block scenarios
April 27, 2019
© Robert Sabourin, 2007