1. Just-In-Time Testing Robert Sabourin President AmiBug.Com, Inc.
Montreal, Canada
December 3, 2018
© Robert Sabourin, 2008

2. Overview Some Philosophy Context Drivers Turbulence Skills
Testing Ideas
Test Triage
December 3, 2018
© Robert Sabourin, 2008

3. Just In Time Testing Robert Sabourin , Software Evangelist President
AmiBug.Com Inc.
Montreal, Quebec, Canada
December 3, 2018
© Robert Sabourin, 2008

4. Fundamental Question How do you know when you are finished?
Some Philosophy
December 3, 2018
© Robert Sabourin, 2008

5. Gerald M. Weinberg “Quality is value to some person”
Exploring Requirements Quality Before Design
Dorset House
Some Philosophy
December 3, 2018
© Robert Sabourin, 2008

6. Edsger W. Dijkstra
“Program testing can be used to show the presence of bugs, but never to show their absence”
Some Philosophy
December 3, 2018
© Robert Sabourin, 2008

7. C. Northcote Parkinson Parkinson’s Law:
“…work expands so as to fill the time available for its completion…”
Some Philosophy
December 3, 2018
© Robert Sabourin, 2008

8. “…begin with the end in mind …
Steve Covey
“…begin with the end in mind …
“…first things first …"
Some Philosophy
December 3, 2018
© Robert Sabourin, 2008

9. Purpose of Testing Common definition: Broader definition:
To find bugs before our customers do!
Broader definition:
The role of testing is to provide objective input to facilitate business decisions!
Keeps stakeholders aware of all issues or concerns that relate to shipping a product!
Some Philosophy
December 3, 2018
© Robert Sabourin, 2008

10. JIT Core Principals Core Values RRR – Keep it Real Recent & Relevant
Purposeful Testing – On Time On Quality On Budget are meaningless if you are not on Purpose
Fundamental Question of Software Engineering: How do you know you are finished?
First Things First: Begin with the end in mind.
BTO - Context Drivers: Business Technology Organization
Be Prepared – Be ready for anything
Testing is Intractable: Tests cannot show software is bug free
Collect new test Ideas from many sources always
Triage Ruthlessly
Decide what not to test on purpose
Bugs are concerns about the project
There is no try: Do or Do Not
It’s all about people – and the occasional bug!
Core Values
December 3, 2018
© Robert Sabourin, 2008

11. Context Drivers - BTO Business Technology Organization Value To whom?
Why?
Technology
Solutions
Organization
Corporate Structure
Team Structure
Roles and Responsibilities
Context
December 3, 2018
© Robert Sabourin, 2008

12. Context
December 3, 2018
© Robert Sabourin, 2008

13. Just-In-Time Testing Turbulence Turbulence December 3, 2018
© Robert Sabourin, 2008

14. Just-In-Time Testing Unprepared Test Skills December 3, 2018
© Robert Sabourin, 2008

15. Just-In-Time Testing Sharpen Testing Skills Thinker Detective Reporter
Diplomat
Negotiator
Cheer Leader
Pragmatist
Test Skills
December 3, 2018
© Robert Sabourin, 2008

16. Just-In-Time Testing So what exactly did they throw over the wall?
Test Focus
December 3, 2018
© Robert Sabourin, 2008

17. First Things First Begin with the end in mind Gain Consensus
Goals
How do we know we are finished?
Purpose
Why are we doing this project?
How will be react to change?
Meaning
What is a bug?
What is a test?
Test Focus
December 3, 2018
© Robert Sabourin, 2008

18. Be Prepared We have precious little time to run tests!
We must always be prepared!
Test Triage
December 3, 2018
© Robert Sabourin, 2008

19. Time
Test Triage
December 3, 2018
© Robert Sabourin, 2008

20. Yoda "No! Try not, Do. Or do not. There is no try." Test Triage
December 3, 2018
© Robert Sabourin, 2008

21. Testing Ideas Collect all testing ideas you can find! List Sort
Organize
Shuffle
Test Triage
December 3, 2018
© Robert Sabourin, 2008

22. Testing Ideas How to find them?
Does system do what it is suppose to do?
Does the system do things it is not supposed to?
How can the system break?
How does the system react to it’s environment?
What characteristics must the system have?
Why have similar systems failed?
How have previous projects failed?
Test Triage
December 3, 2018
© Robert Sabourin, 2008

23. Testing Ideas Collect testing ideas
From testing ideas build a series of testing objectives
Each can be assigned as work to testers
Each can include all, part of, or multiple testing ideas
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

24. Testing Ideas I often use Index Cards Unique id
One testing idea per card
Colour indicates source
Shuffled and reviewed
Organized and reorganized
Sorted, grouped, prioritized and collected
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

25. Test Idea Sources Capabilities Failure Modes Quality Factors
Usage Scenarios
Creative Ideas
States
Data
Environments
White Box
Taxonomies
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

26. Testing Ideas Investigative approaches
We become truffle snorting pigs and try to find useful information in all evidence we discover
We can even get good ideas from out of date sources
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

27. Testing Ideas Requirements Use cases Functional requirements
Quality factors
Constraints
Written requirements
Implicit requirements
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

28. Testing Ideas Failure Modes Capture testing ideas What can break?
Reaction to invalid input?
How does software behave in constrained environment?
Memory
Disk Space
Network Bandwidth
CPU capacity
Shared resources
Stress, Load, Volume
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

29. Quality Factors Importance Different Application Types
Capture testing ideas
Quality Factors Importance
Different Application Types
December 3, 2018
© Robert Sabourin, 2008

30. Testing Ideas Usage Scenarios Identify classes of users
Identify how users will use system
Describe scenarios
Use Story board or similar approaches
Identify variations
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

31. Testing Ideas Creative approaches Action verbs Mind Maps Soap Operas
Lateral Thinking
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

32. Testing Ideas State Models Capture testing ideas power up service
needed
reset button
idle
coin inserted
inserting
coins
coin return
cup removed
no cups
OR no coffee
OR sensor jam
right amount
entered
coin return
make
coffee
user
choose
Capture testing ideas
button pushed
State Models
December 3, 2018
© Robert Sabourin, 2008

33. Testing Ideas Data Flow Structure Create Update Change
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

34. Testing Ideas Environment Capture testing ideas Hardware Software
Operating systems
Locales
Browsers
Plug-ins
Co-dependent software
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

35. Testing Ideas White Box Capture testing ideas Design
Internal structure
Code
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

36. Testing Ideas Bug taxonomies Collections of possible bugs
Appendix A of Testing Computer Software, Kaner, Falk, Nguyen
Boris Biezer Taxonomy Otto Vinter manages
Shopping cart taxonomy Giri Vijayaraghavan
Capture testing ideas
December 3, 2018
© Robert Sabourin, 2008

37. Which test? Impact estimation For each test idea guesstimate:
benefit of implementation
consequence of implementation
benefit for not implementing
consequence of not implementing
How credible is the information?
Triage testing ideas
December 3, 2018
© Robert Sabourin, 2008

38. Which test? Test Idea Rejection – What If?
If the cost/benefit does not make business sense then consider implementing:
part of the test, could that lead to part of the benefit at a more reasonable cost?
more than the stated test, would that generate more benefit?
a different test than the stated idea, could that generate more benefit for less cost?
Triage testing ideas
December 3, 2018
© Robert Sabourin, 2008

39. Which test is next? Questions
Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?
How much effort are we willing to spend continuing to test this project?
Can we ship yet?
Triage testing ideas
December 3, 2018
© Robert Sabourin, 2008

40. Which test is next? Magic crystal ball Triage testing ideas
If it existed how would you use it?
What question would you ask?
What question would it ask?
Triage testing ideas
December 3, 2018
© Robert Sabourin, 2008

41. Deciding what not to test?
Time pressure
Should we skip a test?
If test failed could system still be of value to some stakeholder?
If test was skipped could important bugs have been otherwise found?
Triage testing ideas
December 3, 2018
© Robert Sabourin, 2008

42. Test Triage Test Triage Meeting Review Context
Business
Technical
Information since last triage
Test results
Bug results
New testing ideas
Triage testing ideas
December 3, 2018
© Robert Sabourin, 2008

43. Test Triage Allocate Testing Assignments to Testers
Make sure testers know context
Best thing to test
Best person to test it
Best people to explore it
Best lead
Are subject matter experts required?
Blend scripted & exploratory testing
Triage testing ideas
December 3, 2018
© Robert Sabourin, 2008

44. Bottom Line
My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it!
Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now?
Get Started Right
December 3, 2018
© Robert Sabourin, 2008