Robert Sabourin AmiBug.Com, Inc. Montreal, Canada rsabourin@amibug.com Just-In-Time Testing Robert Sabourin AmiBug.Com, Inc. Montreal, Canada rsabourin@amibug.com © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Overview Welcome Some Philosophy Context Drivers Decision Making Testing Ideas Test Triage Exploratory Testing © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just In Time Testing Robert Sabourin , Software Evangelist President AmiBug.Com Inc. Montreal, Quebec, Canada rsabourin@amibug.com www.amibugshare.com © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

© 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just-In-Time Testing Welcome © 2014 Robert Sabourin JIW V14.0

Just-In-Time Testing Pain points? What hurts? How Much? © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just-In-Time Testing Just In Time For ________? © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

© 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just-In-Time Testing Some Philosophy © 2014 Robert Sabourin JIW V14.0

Fundamental Question How do you know when you are finished? © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just-In-Time Testing What is Quality? © 2014 Robert Sabourin JIW V14.0

Phil Crosby “Quality is defined as conformance to requirements” “Quality is not a measure of GOODNESS” Phil B. Crosby, Quality is Free © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Joseph Juran “Quality is fitness for use” Quality Control Handbook © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Gerald M. Weinberg “Quality is value to some person” Exploring Requirements Quality Before Design Dorset House © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

© 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0 Conforming to requirements Suiting purpose Delivering value to stakeholders © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

© 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Circa 25 BC © 2014 Robert Sabourin JIW V14.0

Circa 1979 © 2014 Robert Sabourin JIW V14.0

Just-In-Time Testing Nature of testing © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Edsger W. Dijkstra “Program testing can be used to show the presence of bugs, but never to show their absence” © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Quiz Application screens are selected with three controls: (a) has 5 options (b) has 6 options (c) has 2 options How many screens can a user choose? © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Quiz Total Combinations = 6 x 5 x 2 = 60 To exercise each combination once a total of 60 tests would be required. © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Quiz How many tests would be required to exercise all possible screens in every possible order?. © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Pop Quiz To exercise all screens in every possible order would require 60! Test cases 60! = 60 x 59 x 58 x ... 3 x 2 x 1 60! ≈ 8.32 x 10**81 © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Pop Quiz How many atoms are in the observable universe? From 7.0 × 10**79 To 1.5 × 10**82 © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

© 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Boris Beizer “Why software has bugs – the fundamental problem – Programming is a bleep.” The Frozen Keyboard - 1986 © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Watts S. Humphrey “… even the most experienced software engineer injects about one defect for ten lines of code …” © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

“…begin with the end in mind … Steve Covey “…begin with the end in mind … “…first things first …" © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

C. Northcote Parkinson Parkinson’s Law: “…work expands so as to fill the time available for its completion…” © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

James Bach “The classical approach to test design is like playing ‘20 Questions’ by writing out all the questions in advance.” © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing versus Checking Michael Bolton Testing versus Checking Testing Exploring Learning Checking Confirming Verifying © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Purpose of Testing Common definition: Broader definition: To find bugs before our customers do! Broader definition: The role of testing is to provide objective input to facilitate business decisions! Keeps stakeholders aware of all issues or concerns that relate to shipping a product! © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Bug Defined To make our job more fun, whenever we have a concern with software, we call it a “bug”. © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just-In-Time Testing It’s all about people! (and the occasional bug too) © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

© 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just-In-Time Testing Context Drivers © 2014 Robert Sabourin JIW V14.0

Context Drivers - BTO Business Technology Organization Value To whom? Why? Technology Solutions Organization Corporate Structure Team Structure Roles and Responsibilities © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

© 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Context Listeners Find Sources Monitor Drivers Anticipate Change React © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just In Time Testing Decision Making © 2014 Robert Sabourin JIW V14.0

Decision Making © 2014 Robert Sabourin JIW V14.0 Workflows Requirements Tests Bugs © 2014 Robert Sabourin JIW V14.0

Decision Making © 2014 Robert Sabourin JIW V14.0 Requirement Workflow Priority Acceptance Criteria Change © 2014 Robert Sabourin JIW V14.0

Decision Making © 2014 Robert Sabourin JIW V14.0 Test Workflow Focus Scope Depth © 2014 Robert Sabourin JIW V14.0

Scope of Testing © 2014 Robert Sabourin JIW V14.0 Environment Hardware Software Co-resident Data Production Generated Sampled Characteristics Performance Security Usability Coverage Requirement Scenarios Structural © 2014 Robert Sabourin JIW V14.0

Depth of Testing © 2014 Robert Sabourin JIW V14.0 Light Touch Shallow Deep Harsh © 2014 Robert Sabourin JIW V14.0

Decision Making © 2014 Robert Sabourin JIW V14.0 Bug Workflow Priority Severity Good enough? © 2014 Robert Sabourin JIW V14.0

Defining a Workflow Seven Steps Identify key stakeholders Learn about decisions Define priority & severity List steps to follow Build state model Get stakeholders “buy in” Adapt as required © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

1 Defining a Workflow Identify key stakeholders © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

2 Defining a Workflow How is a decision made? © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

3 Defining a Workflow Define Priority and Severity Scheme © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

4 Defining a Workflow Define steps to follow Fits Software Development Lifecycle Model Sensitive to context Flexible Change is expected Understood by all team members 4 © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

5 Defining a Workflow Build State Model Entered Reviewed Prioritized REFUSE Entered Reviewed Prioritized Assigned CHECK TRIAGE DESIGNATE CORRECT MANDATE Unassigned Fixed Closed CONFIRM FAILURE Build State Model © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

6 Defining a Workflow Get stakeholder “buy-in” © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

© 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0 R2-D2 and Chewbacca are playing the holographic game aboard the Millennium Falcon Chewbacca: Aaaaaaaaaaaaaaaarrrgh C-3PO: He made a fair move. Screaming about it can't help you. Han Solo: Let him have it. It's not wise to upset a Wookiee. C-3PO: But sir, nobody worries about upsetting a droid. Han Solo: That's 'cause droids don't pull people's arms out of their sockets when they lose. Wookiees are known to do that. Chewbacca: Grrf. C-3PO: I see your point, sir. I suggest a new strategy, R2: let the Wookiee win. © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

7 Defining a Workflow Adapt as required © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just In Time Testing Test Triage © 2014 Robert Sabourin JIW V14.0

Yoda Plan to support change "No! Try not, Do. Or do not. There is no try." © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Collect all testing ideas you can find! List Sort Organize Shuffle Plan to support change © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas How to find them? Does system do what it is suppose to do? Does the system do things it is not supposed to? How can the system break? How does the system react to it’s environment? What characteristics must the system have? Why have similar systems failed? How have previous projects failed? Plan to support change © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Collect testing ideas From testing ideas build a series of testing objectives Each can be assigned as work to testers Each can include all, part of, or multiple testing ideas Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas I often use Index Cards Unique id One testing idea per card Colour indicates source Shuffled and reviewed Organized and reorganized Sorted, grouped, prioritized and collected Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Test Idea Sources Capture testing ideas Capabilities Failure Modes Quality Factors Usage Scenarios Creative Ideas States Data Environments White Box Taxonomies Across Story Relationships Software Breaking End to End Testing Sequencing Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Investigative approaches We become truffle snorting pigs and try to find useful information in all evidence we discover We can even get good ideas from out of date sources Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Capabilities Use cases Functional requirements Documented requirements Implicit requirements Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Failure Modes Capture testing ideas What can break? Reaction to invalid input? How does software behave in constrained environment? Memory Disk Space Network Bandwidth CPU capacity Shared resources Stress, Load, Volume Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Quality Factors Importance Different Application Types Capture testing ideas Quality Factors Importance Different Application Types © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Usage Scenarios Identify classes of users Identify how users will use system Describe scenarios Use Story board or similar approaches Identify variations Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Creative approaches Action verbs Mind Maps Soap Operas Lateral Thinking Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas State Models Capture testing ideas power up service needed reset button idle coin inserted inserting coins coin return cup removed no cups OR no coffee OR sensor jam right amount entered coin return make coffee user choose Capture testing ideas button pushed State Models © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Data Flow Structure Create Update Change Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Environment Capture testing ideas Hardware Software Operating systems Locales Browsers Plug-ins Co-dependent software Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas White Box Capture testing ideas Design Internal structure Code Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Bug taxonomies Collections of possible bugs Appendix A of Testing Computer Software, Kaner, Falk, Nguyen Boris Biezer Taxonomy Otto Vinter manages Shopping cart taxonomy Giri Vijayaraghavan Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Across Story Relationships Interference Resource sharing Inconsistent behaviour Out of order Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Software Breaking James Whitaker, How to Break Software Create a fault model Identify weakness Apply attack User interface attack System interface attack Data layer attack Security attack Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas End to End Testing Exercise entire process chain to complete a transaction Automatic steps Manual steps External systems Third-party systems Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Testing Ideas Sequences Explore paths Vary Operation order Valid Invalid Multiple Concurrent Capture testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Triage Criticality Resources Trade offs Credibility Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Which test? Impact estimation For each test idea guesstimate: benefit of implementation consequence of implementation benefit for not implementing consequence of not implementing How credible is the information? Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Understanding Complex Technology Quantitatively By Tom Gilb How to Decide? Rank Credibility 0.0 Wild guess, no credibility 0.1 We know it has been done somewhere 0.2 We have one measurement somewhere 0.3 There are several measurements in the estimated range 0.4 The measurements are relevant to our case 0.5 The method of measurement is considered reliable 0.6 We have used the method in-house 0.7 We have reliable measurements in-house 0.8 Reliable in-house measurements correlate to independent external measurements 0.9 We have used the idea on this project and measured it 1.0 Perfect credibility, we have rock solid, contract- guaranteed, long-term, credible experience with this idea on this project and, the results are unlikely to disappear Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Which test? Test Idea Rejection – What If? If the cost/benefit does not make business sense then consider implementing: part of the test, could that lead to part of the benefit at a more reasonable cost? more than the stated test, would that generate more benefit? a different test than the stated idea, could that generate more benefit for less cost? Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Test Triage Test Triage Turbulent Projects Agile Projects High Frequency Daily +++ Agile Projects On demand At stand up meeting Stable Products Periodically Same as “bug review” Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Test Triage Review New Context New Info Bugs New testing ideas Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Test Triage Review New Context Business Technical Organizational Cultural Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Test Triage Review New Info Test Findings Development Other Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Test Triage Review Bugs New Fixed Causes Patterns Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Test Triage Review New testing ideas Testers Developers Support Trainers Administrators Customers Users Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Test Triage Allocate Testing Assignments to Testers Make sure testers know context Best thing to test Best person to test it Best people to explore it Best lead Are subject matter experts required Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Test Triage Life of a test idea Triage testing ideas Comes into existence Clarified Prioritized Test Now (before further testing) Test before shipping Nice to have May be of interest in some future release Not of interest in current form Will never be of interest Integrate into a testing objective Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Which test is next? Questions Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next? How much effort are we willing to spend continuing to test this project? Can we ship yet? Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Which test is next? Magic crystal ball Triage testing ideas If it existed how would you use it? What question would you ask? What question would it ask? Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Deciding what not to test? Time pressure Should we skip a test? If test failed could system still be of value to some stakeholder? If test was skipped could important bugs have been otherwise found? Triage testing ideas © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Bottom Line My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it! Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now? Get Started Right © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Just In Time Testing Exploratory Testing © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Exploratory Testing James Bach (www.satisfice.com) General Functionality and Stability Test Procedure for Windows 2000 Application Certification © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Mandate to explore William Clark Meriwether Lewis The object of your mission is to explore the Missouri river, & such principle streams of it, as, by its course and communication with the waters of the Pacific ocean...may offer the most direct & practicable water communication across this continent for the purposes of commerce. - Thomas Jefferson's letter to Meriwether Lewis, June 1803 © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Make intelligent decisions Take notes about your decisions Map out where you have been Others can use the result © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Chart as you explore Further exploration yields a good idea of the state of the world! One bit at a time © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Exploration Notes - Tabular - Chronological - Schematic - Point form - Concise © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Exploratory Testing Test Cases Map Making Skills Not known in advance Defined & executed “on the fly” while you learn about the product Map Making Skills Consistent note taking style Practice © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

Exploratory Testing During test we must capture Function, options or sub-functions being explored Test cases attempted Comments, notes, images or attachments Hints, reminders and observations which may be useful to future testers Date, Platform, Build or Configuration under test Name of person running test Oracles, “strategy to assess correctness” Other relevant details © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

An Exploratory Test Process Confirm Charter Prepare HW, SW, DATA Tools, Environments Kick Off Testing Session Prepare Run Wrap up Collect all notes data Complete Debrief Review findings Review Follow Up Reassess goals Piece together map © 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0

© 2014 Robert Sabourin JIW V14.0 © 2013 Robert Sabourin JIW V13.0