1. Just-In-Time Testing Robert Sabourin President AmiBug.Com, Inc.
Montreal, Canada
December 27, 2018
© Robert Sabourin, 2006

2. Just-In-Time Testing
Turbulent web development projects experience almost daily requirement changes coupled with frequent changes to user interfaces and the continual integration of new functions and features often using new technologies.
This workshop teaches experienced test managers effective techniques which can help keep testing efforts on track while reacting to fast-changing priorities, technologies and requirements.
There are no silver bullets – but several simple techniques can be applied to almost any development project enabling successful systematic testing efforts.
December 27, 2018
© Robert Sabourin, 2006

3. Just-In-Time Testing
After taking this workshop Test managers will be able to
Set up key decision making workflows for Test Prioritization and Bug Triage
Blend session based exploratory and scripted testing
Implement meaningful status tracking and reporting
Discover testing objectives, learning what is important to deliver, with or without written requirements
Identify sources of information from developers and project stakeholders
Adapt testing focus as priorities change, choosing what to test, what not to test, and in what order testing should be done.
Identify technical risks and respect business importance
Interact with key software engineering workflows
Avoid time wasting activities
Use systematic approaches to develop some Just-In-Time (JIT) Testing tools and templates and checklists
December 27, 2018
© Robert Sabourin, 2006

4. Overview Introduction Some Philosophy Testing Ideas Test Triage
Testing Just In Time
Platforms
Exploratory Testing
Scenarios
Status
December 27, 2018
© Robert Sabourin, 2006

5. Elevator Parable Robert Sabourin , Software Evangelist President
AmiBug.Com Inc.
Montreal, Quebec, Canada
December 27, 2018
© Robert Sabourin, 2006

6. Just-In-Time Testing Some Philosophy December 27, 2018
© Robert Sabourin, 2006

7. Fundamental Question How do you know when you are finished?
December 27, 2018
© Robert Sabourin, 2006

8. Edsger W. Dijkstra
“Program testing can be used to show the presence of bugs, but never to show their absence”
December 27, 2018
© Robert Sabourin, 2006

9. Boris Beizer
“Why software has bugs – the fundamental problem – Programming is a bitch.”
The Frozen Keyboard
December 27, 2018
© Robert Sabourin, 2006

10. Watts S. Humphrey
“… the job of the software engineer is to deliver high-quality software products at agreed cost and schedule …”
“… even the most experienced software engineer injects about one defect for ten lines of code …”
December 27, 2018
© Robert Sabourin, 2006

11. C. Northcote Parkinson Parkinson’s Law:
“…work expands so as to fill the time available for its completion…”
December 27, 2018
© Robert Sabourin, 2006

12. Yoda "No! Try not, Do. Or do not. There is no try." December 27, 2018
© Robert Sabourin, 2006

13. “…begin with the end in mind …
Steve Covey
“…begin with the end in mind …
“…first things first …"
December 27, 2018
© Robert Sabourin, 2006

14. Purpose of Testing Common definition: Broader definition:
To find bugs before our customers do!
Broader definition:
The role of testing is to provide objective input to facilitate business decisions!
Keeps stakeholders aware of all issues or concerns that relate to shipping a product!
December 27, 2018
© Robert Sabourin, 2006

15. Bug Defined
To make our job more fun, whenever we have a concern with software, we call it a “bug”.
December 27, 2018
© Robert Sabourin, 2006

16. Just-In-Time Testing
It’s all about people! (and the occasional bug too)
December 27, 2018
© Robert Sabourin, 2006

17. About Bugs Bugs are not Good or Bad December 27, 2018
© Robert Sabourin, 2006

18. About Bugs Some bugs are important and have a high priority!
December 27, 2018
© Robert Sabourin, 2006

19. About Bugs Some bugs are dangerous and have a high severity!
December 27, 2018
© Robert Sabourin, 2006

20. About Bugs
Setting the priority and severity of a bug is a business decision
Changing business conditions impact the priority and severity of a bug!
Always review previous decisions in light of changing business context
Ensure staff assigning priority and severity are aware of all relevant business drivers
December 27, 2018
© Robert Sabourin, 2006

21. Bug Quadrants
December 27, 2018
© Robert Sabourin, 2006

22. Quadrant Changing
Same technical bug can be in a different quadrant depending on the business context
Monitor business drivers!
Focus find and fix high priority/high severity bugs
December 27, 2018
© Robert Sabourin, 2006

23. Get Ready, Get Set, Cause here it comes
Just In Time Testing
Get Ready, Get Set,
Cause here it comes
December 27, 2018
© Robert Sabourin, 2006

24. Just-In-Time Testing Turbulence December 27, 2018
© Robert Sabourin, 2006

25. Just-In-Time Testing Unprepared December 27, 2018
© Robert Sabourin, 2006

26. Just-In-Time Testing So what exactly did they throw over the wall?
December 27, 2018
© Robert Sabourin, 2006

27. Getting Things Done Metaphors Themes RUTHLESS TRIAGE RING BELL
WELL OILED MACHINES
Themes
Parallelism wherever and always
Think and test 24/7, early and always
Chunking
December 27, 2018
© Robert Sabourin, 2006

28. First Things First Begin with the end in mind Gain Consensus Goals
How do we know we are finished?
Purpose
Why are we doing this project?
How will be react to change?
Meaning
What is a bug?
What is a test?
December 27, 2018
© Robert Sabourin, 2006

29. Just In Time Testing Testing Ideas December 27, 2018
© Robert Sabourin, 2006

30. Testing Ideas Collect all testing ideas you can find! List Sort
Organize
Shuffle
December 27, 2018
© Robert Sabourin, 2006

31. Testing Ideas How to find them?
Does system do what it is suppose to do?
Does the system do things it is not supposed to?
How can the system break?
How does the system react to it’s environment?
What characteristics must the system conform to?
Why have previous or similar projects failed?
How have previous or similar systems failed?
December 27, 2018
© Robert Sabourin, 2006

32. Testing Ideas Collect testing ideas
From testing ideas build a series of testing objectives (TO)
Each can be assigned as work to a tester
Each can include all, part of, or multiple testing ideas
December 27, 2018
© Robert Sabourin, 2006

33. Testing Ideas Index Cards Each card has a unique id
Can be shuffled and reviewed
Can be organized and reorganized
Have one testing idea per card
Can be sorted, grouped, prioritized and collected
Concept of a prioritized heap
December 27, 2018
© Robert Sabourin, 2006

34. Testing Ideas Creative approaches Action verbs Mind Maps Soap Operas
Lateral Thinking
December 27, 2018
© Robert Sabourin, 2006

35. Testing Ideas Investigative approaches
We become truffle snorting pigs and try to find useful information in all evidence we discover
We can even get good ideas from out of date sources
December 27, 2018
© Robert Sabourin, 2006

36. Testing Ideas Bug taxonomies Collections of possible bugs
Appendix A of Testing Computer Software, Kaner, Falk, Nguyen
Boris Biezer Taxonomy Otto Vinter manages
Shopping cart taxonomy Giri Vijayaraghavan
December 27, 2018
© Robert Sabourin, 2006

37. Testing Ideas Requirements Use cases Functional requirements
Quality factors
Constraints
Written requirements
Implicit requirements
December 27, 2018
© Robert Sabourin, 2006

38. Testing Ideas Usage Scenarios Identify classes of users
Identify how users will use system
Describe scenarios
Use Story board or similar approaches
Identify variations
December 27, 2018
© Robert Sabourin, 2006

39. Testing Ideas Functionality Analysis
Requirements, Design or Prototypes can give insights into
Domain Analysis
Equivalence classes
Boundary analysis
CRUD
December 27, 2018
© Robert Sabourin, 2006

40. Testing Ideas Failure Modes What can break? Reaction to invalid input?
How does software behave in constrained environment?
Memory
Disk Space
Network Bandwidth
CPU capacity
Shared resources
Stress, Load, Volume
December 27, 2018
© Robert Sabourin, 2006

41. Quality Factors Importance For Different Web Application Types
December 27, 2018
© Robert Sabourin, 2006

42. Brainstorming Testing Ideas
Min Max Missing
Group of people
Kick off meeting
Individual build list
Logging meeting
Synergy
Estimate size in sessions
Identify missing or redundant ideas
December 27, 2018
© Robert Sabourin, 2006

43. Testing Ideas Oracle Collection Strategies to assess correctness
Similar systems
Old systems
Subject matter experts
GURUs
Standards
December 27, 2018
© Robert Sabourin, 2006

44. Testing Tools Tools Status Dynamic analysis Static analysis
Test engines
Home brew with correct skills
Danny Faught Open Testware ideas
December 27, 2018
© Robert Sabourin, 2006

45. Just In Time Testing Test Triage December 27, 2018
© Robert Sabourin, 2006

46. Which test? Impact estimation For each test idea guesstimate:
benefit of implementation
consequence of implementation
benefit for not implementing
consequence of not implementing
How credible are values?
December 27, 2018
© Robert Sabourin, 2006

47. Understanding Complex Technology Quantitatively By Tom Gilb
How to Decide?
Rank
Credibility
0.0
Wild guess, no credibility
0.1
We know it has been done somewhere
0.2
We have one measurement somewhere
0.3
There are several measurements in the estimated range
0.4
The measurements are relevant to our case
0.5
The method of measurement is considered reliable
0.6
We have used the method in-house
0.7
We have reliable measurements in-house
0.8
Reliable in-house measurements correlate to independent external measurements
0.9
We have used the idea on this project and measured it
1.0
Perfect credibility, we have rock solid, contract- guaranteed, long-term, credible experience with this idea on this project and, the results are unlikely to disappear
December 27, 2018
© Robert Sabourin, 2006

48. Which test? Test Idea Rejection – What If?
If the cost/benefit does not make business sense then consider implementing:
part of the test, could that lead to part of the benefit at a more reasonable cost?
more than the stated test, would that generate more benefit?
a different test than the stated idea, could that generate more benefit for less cost?
December 27, 2018
© Robert Sabourin, 2006

49. Test Triage Test Triage JIT Projects High Frequency
Daily Test Triage Session
Experience dictates
Early AM (Rob Preference)
Late PM (several clients)
December 27, 2018
© Robert Sabourin, 2006

50. Test Triage Test Triage Meeting Review Context
Business
Technical
Information since last triage
Test results
Bug results
New testing ideas
December 27, 2018
© Robert Sabourin, 2006

51. Test Triage Allocate Testing Assignments to Testers
Make sure testers know context
Best thing to test
Best person to test it
Best people to explore it
Best lead
Assign subject matter experts is required
Sessions may be scripted or exploratory
December 27, 2018
© Robert Sabourin, 2006

52. Test Triage Requirement Triage Change Control Test Triage Bug Flow
Combined
Equivalent to CCB
Few people
Fluid
December 27, 2018
© Robert Sabourin, 2006

53. Test Triage Life of a test idea Comes into existence Clarified
Prioritized
Test Now (before further testing)
Test before shipping
Nice to have
May be of interest in some future release
Not of interest in current form
Will never be of interest
Integrate into a testing objective
December 27, 2018
© Robert Sabourin, 2006

54. Which test is next? Magic crystal ball Ask the question
Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?
How much effort are we willing to spend continuing to test this project?
Can we ship yet?
December 27, 2018
© Robert Sabourin, 2006

55. Which test is next? Magic crystal ball
If it existed then how would you use it?
What question would you ask it?
What question would it ask you?
December 27, 2018
© Robert Sabourin, 2006

56. Which test is next? Magic crystal ball Discover Example questions
What question to ask?
What information to have at hand?
Example questions
Given these test objectives how many sessions should I dedicate to them?
Given that this part of the application is very buggy what should I test otherwise?
December 27, 2018
© Robert Sabourin, 2006

57. Deciding what not to test?
Time pressure
Should we skip a test?
If test failed could system still be of value to some stakeholder?
If test was skipped could important bugs have been otherwise found?
December 27, 2018
© Robert Sabourin, 2006

58. Problem with formulas Units? Math? Exposure(i) = Risk * Consequence
Allocation(i) = (Exposure(i)/Total Exposure) * MAX
Units?
Math?
December 27, 2018
© Robert Sabourin, 2006

59. Guidelines and Decisions
To each stakeholder
risk of failure
consequence of failure
value of success
how much certainty do we have
is it a wild guess or an absolute truth?
December 27, 2018
© Robert Sabourin, 2006

60. Sources of Information
Version control system
Monitor changes
Track where work is
Track where stability is
Encourage finding defects earlier than system testing
Inspections of code, design, requirements
Unit Testing
Informal code check in peer reviews
December 27, 2018
© Robert Sabourin, 2006

61. Testing Just In Time
December 27, 2018
© Robert Sabourin, 2006

62. Philosophy We have precious little time to run tests!
We must always be prepared!
December 27, 2018
© Robert Sabourin, 2006

63. Time
December 27, 2018
© Robert Sabourin, 2006

64. Getting Things Done Development BUG REQ FLOW FLOW Release Cycle
- Who manages them?
- How are they prioritized?
- Where can I find them?
- Are the communicated?
- Do they get reprioritized?
- Are business drivers known?
- Are technical risks known?
Getting Things Done
Development
BUG
FLOW
REQ
FLOW
- Are builds delivered?
- Where do developers work?
- Configuration management?
- Source control? Baseline?
- Transition? Periodic?
- Smoke tests?
- Owners:Dev IT DBA SQA?
- Who manages them?
- What are they?
- Where can I find them?
- When are they updated?
- Why are they changing?
- How are they evolving?
- Do we observe turbulence?
Release
Cycle
December 27, 2018
© Robert Sabourin, 2006

65. Getting Things Done Concern Concern Being Prepared! Being Prepared!
- Information Flow
- Information Flow
Corporate information Key business drivers Sales Market Finance
Corporate information Key business drivers Sales Market Finance
- Technology Flow
Architecture Technology churn Tools Techniques Training
- Requirement Flow
Defined Understood Interrupt Poll Prioritize Turbulence Status Truffle
- Bug Flow
Defined Understood Business Technical Efficient Expedient Reassess
- Test Objectives
Quality Factors Technical Risk Failure Modes Importance
- Test Strategy
Plan Analytic Exploratory Checklists Parallel Chunking Scenarios Data
- Test Organization
Scheduling Staffing Outsourcing Contractors Students
- Testing Lab
Multi-tier Server Client Platforms Swap Pattern Synchronized
- Test Status
Bug charts Test Plan Elaboration Status Pass Fail Execution Status
December 27, 2018
© Robert Sabourin, 2006

66. Getting Things Done Where tests are run! Developers Desktop
During development task assignments
Unit Testing
Development Servers
During development and integration task assignments
Unit and Integration testing
Test Lab Servers
During Integration and system test assignments
Integration and System Testing
Staging Servers
Acceptance testing
As close as possible to live technologies
Live Servers
Operational system
Site monitoring
December 27, 2018
© Robert Sabourin, 2006

67. Never update servers when tests are running!
Getting Things Done
Where tests are run!
Developers Desktop
Development Servers
Test Lab Servers
Staging Servers
Live Servers
- Synchronize releasing
builds from the development
team to the testing team!
- Ensure a new builds passes
the Smoke Test before it
replaces the build currently
being tested!
Never update servers when tests are running!
December 27, 2018
© Robert Sabourin, 2006

68. Getting Things Done Test Scheduling Adapt to change
Revised risks?
New test objectives? New Objectives?
Triage Testing Objectives
Assign testing Objectives to testing team members.
Analysis and exploration to more senior team members
DAILY
Prioritize Bugs
Relative business importance of testing objective?
Any test objective more important than any other?
Track Progress
Total budget effort
Spread across testing objectives
Smoke Test
Should the new build be tested at all?
On failure continue with previous build in test.
FAST Test
Each area of functionality has a simple test.
Is functional area stable enough to test
BUILD
Regression Test
Does application still work as expected?
Did we accidentally break something?
Confirmation Test
Have bugs really been fixed?
Double check in test lab for each bug!
Stress Testing
How well does the application behave in harsh conditions?
Treated as an experiment.
December 27, 2018
© Robert Sabourin, 2006

69. Getting Things Done Testing Activities Elaboration Exploration
Define test procedure and test cases.
Methods, techniques, test cases. All must be repeatable.
Exploration
Primarily to identify areas of weakness or instability!
Use exploratory testing techniques.
Execution
Follow defined test procedures or execute automated test scripts.
While testing identify bugs! (use checklists!)
PLATFORM
Bug Isolation
Narrow down how to repeat bug. Be practical.
May need triage to determine if more isolation is needed.
Track Progress
Track defects Open open over time. (TREND CURVE)
Track test Objective status over time. (SPREED SHEET)
Stress Test Experiment
Execute in collaboration with development.
As required report bugs and always report stress test results!
December 27, 2018
© Robert Sabourin, 2006

70. Just In Time Testing Exploratory Testing December 27, 2018
© Robert Sabourin, 2006

71. Exploratory Testing
Approach formalized by James Bach (
Used in General Functionality and Stability Test Procedure for Windows 2000 Application Certification
December 27, 2018
© Robert Sabourin, 2006

72. Mandate to explore
William Clark
Meriwether Lewis
The object of your mission is to explore the Missouri river,
& such principal streams of it, as, by its course and communication
with the waters of the Pacific ocean...may offer the most direct &
practicable water communication across this continent for the
purposes of commerce.
- Thomas Jefferson's letter to Meriwether Lewis, June 1803
December 27, 2018
© Robert Sabourin, 2006

73. Make intelligent decisions
Take notes about your decisions
Map out where you have been
Others can use the result
December 27, 2018
© Robert Sabourin, 2006

74. Chart as you explore
Further exploration yields a good idea of the state of the world!
One bit at a time
December 27, 2018
© Robert Sabourin, 2006

75. Exploration Notes - Tabular - Chronological - Schematic - Point form
- Concise
December 27, 2018
© Robert Sabourin, 2006

76. Exploratory Testing Test cases
Not known in advance
Defined & executed “on the fly” while you learn about the product
Testers need to “hone up” their skills in making maps!
Consistent note taking style
Practice
We need to have a standard template
Uniform way to capture details
December 27, 2018
© Robert Sabourin, 2006

77. Exploratory Testing During test we must capture
Function, options or sub-functions being explored
Test cases attempted
Comments, notes, images or attachments
Hints, reminders and observations which may be useful to future testers
Date, Platform, Build or Configuration under test
Name of person running test
Oracles, “strategy to assess correctness”
Other relevant details
December 27, 2018
© Robert Sabourin, 2006

78. An Exploratory Test Process
Confirm Test Objective
Ensure context known
Ensure HW and SW OK
All tools available
Kick Off
Chunk of 90 to 120 min
Test, Plan, Discover
Prepare
Run
Wrap up
Collect all notes data
Complete
Review results with
Test Lead
Review
Follow Up
Reassess goals
Piece together map
December 27, 2018
© Robert Sabourin, 2006

79. Just In Time Testing Scenarios December 27, 2018
© Robert Sabourin, 2006

80. Scenario Based Testing
Scenarios
Your testing department should develop a series of typical, real, usage scenarios for each user type
Can be done in parallel with development effort
Could be based on Use Case Analysis
Could be based on Story Boards
Testing Objectives can be dedicated to scenario elaboration or execution
Elaboration (scripting) by testing professionals
Execution by contractors, students, support staff etc
Lots of parallelism!
Individual scenarios can cover many functions!
December 27, 2018
© Robert Sabourin, 2006

81. December 27, 2018
© Robert Sabourin, 2006

82. December 27, 2018
© Robert Sabourin, 2006

83. December 27, 2018
© Robert Sabourin, 2006

84. Scenario Based Testing
User Experience Test Case
Parameterize experience
Walk through scenario from start to end
Use pre selected input for each case
Always run every test as if it were a user experience
December 27, 2018
© Robert Sabourin, 2006

85. Just In Time Testing
Status
December 27, 2018
© Robert Sabourin, 2006

86. JIT Status Reporting Status Test Objectives Test Results Scoreboard
Bug Summary
What we know
What is left to do
December 27, 2018
© Robert Sabourin, 2006

87. JIT Status
December 27, 2018
© Robert Sabourin, 2006

88. JIT Status
December 27, 2018
© Robert Sabourin, 2006

89. JIT Status Test Lead Must Track Efforts Every Day!
Number of Tests to elaborate?
Number of Tests Passed?
Number of Tests Failed?
Number of High Priority Bugs to be fixed?
Number of Bug Fixes to confirm?
Count every day and plot trend graphs!
Publish results.
December 27, 2018
© Robert Sabourin, 2006

90. JIT Status Trend Chart Open Bugs December 27, 2018
© Robert Sabourin, 2006

91. JIT Status Trend Chart Open Bugs By Type December 27, 2018
© Robert Sabourin, 2006

92. Finished? How do you know you are finished? December 27, 2018
© Robert Sabourin, 2006

93. You know you are finished when …
… the only bugs left are the ones are acceptable (based on your objective test team input) ...
December 27, 2018
© Robert Sabourin, 2006

94. You know you are finished when …
… the only bugs left are the ones are acceptable (based on your objective test team input) ...
At least for now!
December 27, 2018
© Robert Sabourin, 2006

95. Thank You
Questions?
December 27, 2018
© Robert Sabourin, 2006