AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 1 Toward an Exploratory Testing Culture Robert Sabourin President & Principal consultant AmiBug.Com, Inc. Montreal, Canada

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 2 Toward an Exploratory Testing Culture Traditional testing teams often agonize over exploratory testing. How can they plan and design tests without detailed up-front documentation? Stubborn testers may want to quit because they are being asked to move out of their comfort zone. Can a team’s testing culture be changed? Rob Sabourin describes how several teams have undergone dramatic shifts to embrace exploratory testing. Learn how to blend cognitive thinking skills, subject matter expertise, and “hard earned” experience to help refocus your team and improve your outcomes. Learn to separate bureaucracy from thinking and paperwork from value. Explore motivations for change and resistance to it in different project contexts. Leverage Parkinson’s Law—work expands to fill the time available—and Dijkstra’s Principle—testing can show the presence of bugs, but not their absence—to inspire and motivate you and your team to get comfortable in the world of exploratory testing.

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 3 –Defined –Philosophy Fundamental Question Dijkstra’s Principle Parkinson’s Law Covey –Purpose –People –Learning –Context –Diverse Testing Ideas –Case Study Toward an Exploratory Testing Culture

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 4 Exploratory Testing Defined

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 5 Concurrent –Learning –Design –Execution ET Defined Toward an Exploratory Testing Culture

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 6 Exploratory Testing An Example: Bug Confirmation Testing ET Defined

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 7 Bug Confirmation Confirm that the “fixed” bug is really fixed ET Defined

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 8 Bug Confirmation Was bug fixed? Did bug move? Did developers accidentally inject a new bug? ET Defined

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 9 Bug Confirmation Was bug really fixed? 1.Use repro steps from bug report 2.Confirm bug was in old build 3.Show bug is not in new build ET Defined

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 10 Bug Confirmation Explore: Did the bug move? 1.Exercise related capabilities 2.Vary steps to reproduce 3.Investigate features that share code ET Defined

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 11 Bug Confirmation Explore: Did developers inject a new bug? 1.Exercise application capabilities 2.Vary data 3.Walk through scenarios ET Defined

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 12 Bug Confirmation Jason’s Approach? 1.Speak with developer 2.Confirm bug was in old build (repro) 3.Confirm bug is not in new build (repro) 4.Identify a usage scenario which would have exposed the bug 5.Walk through the scenario many times with variation of steps and user data 6.Stop when perturbation is absurd ET Defined

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 13 Exploratory Testing Philosophy

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 14 Fundamental Question How do you know when you are finished? Philosophy

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 15 Edsger W. Dijkstra “Program testing can be used to show the presence of bugs, but never to show their absence” Philosophy

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 16 C. Northcote Parkinson Parkinson’s Law: “…work expands so as to fill the time available for its completion…” Philosophy

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 17 “…begin with the end in mind … “…first things first …" Steve Covey Philosophy

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 18 "No! Try not, Do. Or do not. There is no try." Yoda Philosophy

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 19 Exploratory Testing Purpose

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 20 Purpose of Testing Common definition: –To find bugs before our customers do! Broader definition: –The role of testing is to provide objective input to facilitate business decisions! –Keeps stakeholders aware of all issues or concerns that relate to shipping a product! Purpose

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 21 Bug Defined To make our job more fun, whenever we have a concern with software, we call it a “bug”. Purpose

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 22 Exploratory Testing People

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 23 It’s all about people! (and the occasional bug too) People Toward an Exploratory Testing Culture

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 24 Leadership Different Strokes for different folks at different times! –Adapt leadership style to the situation –Choose leadership styles deliberately! People

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 25 Leadership Basic Leadership Styles Directing Coaching Supporting Delegating People

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 26 Leadership Directing Leadership Tell people specifically what to do Provide constant feedback, praising and redirection Used when someone is new to a task and uncertain as to how to successfully achieve the task Used sometimes in an emergency situation People

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 27 Leadership Coaching Leadership Provide guidance and advice on how to achieve goals Does not need close direction but needs to learn how to achieve success Used when someone has a proven track record but is new to this specific task Team member is mature enough to ask for assistance People

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 28 Leadership Supporting Leadership Staff participate in decision making with leader Staff works with leader to establish goals and milestones Person can work quite autonomously but needs leaders help Team member is mature People

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 29 Leadership Delegating Leadership Staff is given broad goal and takes full ownership of task Constant feedback is not required Confirm big picture business drivers have not changed Team member is autonomous People

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 30 Exploratory Testing Learning

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 31 Unprepared Learning

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 32 Just-In-Time Testing Sharpen Testing Skills Thinker Detective Reporter Diplomat Negotiator Cheer Leader Pragmatist Learning

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 33 Exploratory Testing Context

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 34 Context Drivers - BTO Business –Value –To whom? –Why? Technology –Solutions Organization –Corporate Structure –Team Structure –Roles and Responsibilities Context

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 35 Context

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 36 Exploratory Testing Diverse Testing Ideas

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 37 Testing Ideas Collect testing ideas From testing ideas build a series of testing objectives –Each can be assigned as work to testers –Each can include all, part of, or multiple testing ideas Diverse testing ideas

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 38 Testing Ideas Investigative approaches –We become truffle snorting pigs and try to find useful information in all evidence we discover –We can even get good ideas from out of date sources Diverse testing ideas

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 39 Test Idea Sources Capabilities Failure Modes Quality Factors Usage Scenarios Creative Ideas States Data Environments White Box Taxonomies Diverse testing ideas

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 40 Test Triage Allocate Testing Assignments to Testers –Make sure testers know context –Best thing to test –Best person to test it –Best people to explore it –Best lead –Are subject matter experts required Diverse testing ideas

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 41 Thank You Questions?

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 42 Out of the Frying Pan into the Fire The case of the distracting documentation

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 43 The case of the distracting documentation Corporate goal –respond to market pressure without destroying ongoing development projects Case of the Distracting Documentation

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 44 The case of the distracting documentation Before –In Testing Heavily scripted Subject Matter Experts Independent team of system testers Case of the Distracting Documentation

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 45 The case of the distracting documentation Before –In Development Teams torn apart to react to emergencies Heroic Knights in shining armor Development uses several levels of documentation Tons of "integration testing” before release to test team Inconsistent unit testing Case of the Distracting Documentation

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 46 The case of the distracting documentation Before –In Product Management Many different roles managing work Many different requirement lists Dominant consensus management from highest levels Case of the Distracting Documentation

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 47 The case of the distracting documentation Setup –Urgent project selected as pilot –Storyboards replaced traditional requirements –Assigned team members by surprise –No training (wing it) Case of the Distracting Documentation

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 48 The case of the distracting documentation Problems –Testing team very uncomfortable –Discovered exploratory testing –Tons of overtime –Product management not active –Treated as a heroic effort Case of the Distracting Documentation

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 49 The case of the distracting documentation Remedy –Active product management –Consolidate backlog –Train team in method –Involve testers in planning sessions –Coaching on exploratory and Just in time testing –Consistently use unit test framework –Teach team how to use "stories" for requirements and testing –Team was committed during iteration Case of the Distracting Documentation