1. Just-In-Time Testing Workshop
Robert Sabourin
AmiBug.Com, Inc.
Montreal, Canada
March 25, 2017
© Robert Sabourin, 2012

2. Overview Welcome Some Philosophy Be Pre[pared Testing Ideas
Test Triage
March 25, 2017
© Robert Sabourin, 2012

3. Just In Time Testing Robert Sabourin , Software Evangelist President
AmiBug.Com Inc.
Montreal, Quebec, Canada
March 25, 2017
© Robert Sabourin, 2012

4. Just-In-Time Testing
Welcome
March 25, 2017
© Robert Sabourin, 2012

5. Just-In-Time Testing Pain points? What hurts? How Much? March 25, 2017
© Robert Sabourin, 2012

6. Just-In-Time Testing Just In Time For ________? March 25, 2017
© Robert Sabourin, 2012

7. Just-In-Time Testing Some Philosophy March 25, 2017
© Robert Sabourin, 2012

8. Fundamental Question How do you know when you are finished?
March 25, 2017
© Robert Sabourin, 2012

9. Crosby on Quality “Quality is defined as conformance to requirements”
“Quality is not a measure of GOODNESS”
Phil B. Crosby, Quality is Free
March 25, 2017
© Robert Sabourin, 2012

10. Joseph Juran “Quality is fitness for use” Quality Control Handbook
March 25, 2017
© Robert Sabourin, 2012

11. Gerald M. Weinberg “Quality is value to some person”
Exploring Requirements Quality Before Design
Dorset House
March 25, 2017
© Robert Sabourin, 2012

12. Edsger W. Dijkstra
“Program testing can be used to show the presence of bugs, but never to show their absence”
March 25, 2017
© Robert Sabourin, 2012

13. Quiz Application screens are selected with three controls:
(a) has 5 options
(b) has 6 options
(c) has 2 options
How many screens can a user choose?
March 25, 2017
© Robert Sabourin, 2012

14. Quiz Total Combinations = 6 x 5 x 2 = 60
To exercise each combination once a total of 60 tests would be required.
March 25, 2017
© Robert Sabourin, 2012

15. Quiz
How many tests would be required to exercise all possible screens in every possible order?.
March 25, 2017
© Robert Sabourin, 2012

16. Pop Quiz To exercise all screens in every possible order would require
60! Test cases
60! = 60 x 59 x 58 x x 2 x 1
60! ≈ 8.32 x 10**81
March 25, 2017
© Robert Sabourin, 2012

17. Pop Quiz How many atoms are in the observable universe?
From 7.0 × 10**79
To × 10**82
March 25, 2017
© Robert Sabourin, 2012

18. Control Flow Testing Model flow Create control flow diagram
Find basis paths
Minimal set of transactions
Exercise at least once
Each step
Each decision
March 25, 2017
© Robert Sabourin, 2012

19. Control Flow Testing Minimal basis paths N=9 E=10 P=10-9+2 P=3
N – number of nodes
E – number of edges
P – number of basis paths
P = E – N + 2
McCabe Cyclomatic Complexity C D E F G H I
March 25, 2017
© Robert Sabourin, 2012

20. Control Flow Testing Finding basis paths: N=9 E=10 P=10-9+2 P=3
Start with a typical baseline
Flip first decision keep rest as similar as possible
Continue flipping decisions on baseline
After all decisions on baseline have been flipped continue on next path
Stop when all paths have been exhausted C D E F G H I
March 25, 2017
© Robert Sabourin, 2012

21. Control Flow Testing
March 25, 2017
© Robert Sabourin, 2012

22. March 25, 2017
© Robert Sabourin, 2012

23. ST Di AD AC X L K I A Y Z AE M O J B AF AA AG AH AJ AB N C AM AL P H DAI AN AK Q S E AO AP AQ R T G F U V W
March 25, 2017
© Robert Sabourin, 2012

24. March 25, 2017
© Robert Sabourin, 2012

25. Purpose of Testing Common definition: Broader definition:
To find bugs before our customers do!
Broader definition:
The role of testing is to provide objective input to facilitate business decisions!
Keeps stakeholders aware of all issues or concerns that relate to shipping a product!
March 25, 2017
© Robert Sabourin, 2012

26. Bug Defined
To make our job more fun, whenever we have a concern with software, we call it a “bug”.
March 25, 2017
© Robert Sabourin, 2012

27. Just-In-Time Testing
It’s all about people! (and the occasional bug too)
March 25, 2017
© Robert Sabourin, 2012

28. Just-In-Time Testing Context Drivers March 25, 2017
© Robert Sabourin, 2012

29. Context Drivers - BTO Business Technology Organization Value To whom?
Why?
Technology
Solutions
Organization
Corporate Structure
Team Structure
Roles and Responsibilities
March 25, 2017
© Robert Sabourin, 2012

30. March 25, 2017
© Robert Sabourin, 2012

31. Context Listeners Find Sources Monitor Drivers Anticipate Change React
March 25, 2017
© Robert Sabourin, 2012

32. Get Ready, Get Set, Cause here it comes
Just In Time Testing
Get Ready, Get Set,
Cause here it comes
March 25, 2017
© Robert Sabourin, 2012

33. Just-In-Time Testing
Turbulence
March 25, 2017
© Robert Sabourin, 2012

34. Just-In-Time Testing
Unprepared
March 25, 2017
© Robert Sabourin, 2012

35. Just-In-Time Testing Sharpen Testing Skills Thinker Detective Reporter
Diplomat
Negotiator
Cheer Leader
Pragmatist
March 25, 2017
© Robert Sabourin, 2012

36. Philosophy We have precious little time to run tests!
We must always be prepared!
March 25, 2017
© Robert Sabourin, 2012

37. Time
March 25, 2017
© Robert Sabourin, 2012

38. Just In Time Testing Test Triage March 25, 2017
© Robert Sabourin, 2012

39. Yoda Plan to support change "No! Try not, Do. Or do not.
There is no try."
March 25, 2017
© Robert Sabourin, 2012

40. Testing Ideas Collect all testing ideas you can find! List Sort
Organize
Shuffle
Plan to support change
March 25, 2017
© Robert Sabourin, 2012

41. Testing Ideas How to find them?
Does system do what it is suppose to do?
Does the system do things it is not supposed to?
How can the system break?
How does the system react to it’s environment?
What characteristics must the system have?
Why have similar systems failed?
How have previous projects failed?
Plan to support change
March 25, 2017
© Robert Sabourin, 2012

42. Testing Ideas Collect testing ideas
From testing ideas build a series of testing objectives
Each can be assigned as work to testers
Each can include all, part of, or multiple testing ideas
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

43. Testing Ideas I often use Index Cards Unique id
One testing idea per card
Colour indicates source
Shuffled and reviewed
Organized and reorganized
Sorted, grouped, prioritized and collected
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

44. Test Idea Sources Capabilities Failure Modes Quality Factors
Usage Scenarios
Creative Ideas
States
Data
Environments
White Box
Taxonomies
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

45. Testing Ideas Investigative approaches
We become truffle snorting pigs and try to find useful information in all evidence we discover
We can even get good ideas from out of date sources
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

46. Testing Ideas Capabilities Use cases Functional requirements
Documented requirements
Implicit requirements
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

47. Testing Ideas Failure Modes Capture testing ideas What can break?
Reaction to invalid input?
How does software behave in constrained environment?
Memory
Disk Space
Network Bandwidth
CPU capacity
Shared resources
Stress, Load, Volume
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

48. Quality Factors Importance Different Application Types
Capture testing ideas
Quality Factors Importance
Different Application Types
March 25, 2017
© Robert Sabourin, 2012

49. Testing Ideas Usage Scenarios Identify classes of users
Identify how users will use system
Describe scenarios
Use Story board or similar approaches
Identify variations
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

50. Testing Ideas Creative approaches Action verbs Mind Maps Soap Operas
Lateral Thinking
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

51. Testing Ideas State Models Capture testing ideas power up service
needed
reset button
idle
coin inserted
inserting
coins
coin return
cup removed
no cups
OR no coffee
OR sensor jam
right amount
entered
coin return
make
coffee
user
choose
Capture testing ideas
button pushed
State Models
March 25, 2017
© Robert Sabourin, 2012

52. Testing Ideas Data Flow Structure Create Update Change
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

53. Testing Ideas Environment Capture testing ideas Hardware Software
Operating systems
Locales
Browsers
Plug-ins
Co-dependent software
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

54. Testing Ideas White Box Capture testing ideas Design
Internal structure
Code
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

55. Testing Ideas Bug taxonomies Collections of possible bugs
Appendix A of Testing Computer Software, Kaner, Falk, Nguyen
Boris Biezer Taxonomy Otto Vinter manages
Shopping cart taxonomy Giri Vijayaraghavan
Capture testing ideas
March 25, 2017
© Robert Sabourin, 2012

56. March 25, 2017
© Robert Sabourin, 2012

57. Which test? Impact estimation For each test idea guesstimate:
benefit of implementation
consequence of implementation
benefit for not implementing
consequence of not implementing
How credible is the information?
Triage testing ideas
March 25, 2017
© Robert Sabourin, 2012

58. Which test? Test Idea Rejection – What If?
If the cost/benefit does not make business sense then consider implementing:
part of the test, could that lead to part of the benefit at a more reasonable cost?
more than the stated test, would that generate more benefit?
a different test than the stated idea, could that generate more benefit for less cost?
Triage testing ideas
March 25, 2017
© Robert Sabourin, 2012

59. Test Triage Test Triage Meeting Review Context New Information
Business
Technical
Organizational
New Information
Test results
Bug results
New testing ideas
Triage testing ideas
March 25, 2017
© Robert Sabourin, 2012

60. Test Triage Allocate Testing Assignments to Testers
Make sure testers know context
Best thing to test
Best person to test it
Best people to explore it
Best lead
Are subject matter experts required
Triage testing ideas
March 25, 2017
© Robert Sabourin, 2012

61. Test Triage Life of a test idea Triage testing ideas
Comes into existence
Clarified
Prioritized
Test Now (before further testing)
Test before shipping
Nice to have
May be of interest in some future release
Not of interest in current form
Will never be of interest
Integrate into a testing objective
Triage testing ideas
March 25, 2017
© Robert Sabourin, 2012

62. Which test is next? Questions
Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?
How much effort are we willing to spend continuing to test this project?
Can we ship yet?
Triage testing ideas
March 25, 2017
© Robert Sabourin, 2012

63. Deciding what not to test?
Time pressure
Should we skip a test?
If test failed could system still be of value to some stakeholder?
If test was skipped could important bugs have been otherwise found?
Triage testing ideas
March 25, 2017
© Robert Sabourin, 2012

64. Guidelines and Decisions
To each stakeholder
risk of failure
consequence of failure
value of success
how much certainty do we have
is it a wild guess or an absolute truth?
Get Started Right
March 25, 2017
© Robert Sabourin, 2012

65. Bottom Line
My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it!
Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now?
Get Started Right
March 25, 2017
© Robert Sabourin, 2012

66. Thank You
Questions?
March 25, 2017
© Robert Sabourin, 2012