© 2012 WIPRO LTD | 1 Structured Exploratory Testing – Method in Madness Vivek Devarajan Senior Consultant, Wipro Technologies

© 2012 WIPRO LTD | 2 Scripted versus Non-Scripted Testing Some Exploratory Testing Ideas Session Based Test Management Benefits of Exploratory Testing Case Studies

© 2012 WIPRO LTD | 3 Scripted versus Non-Scripted Testing

© 2012 WIPRO LTD | 4 Scripted versus Non-Scripted Testing Freestyle Testing Scripted Testing Structured Exploratory Testing / Session Based Test Management Elaborate Documentation Static Test Suite Higher Testing Costs Limited Defect Detection Concise Documentation Dynamic Test Optimization Lower Testing Costs Better Defect Detection No Accountability No Predictability High Possibility of Defect Slippage Leveraging the benefits of both Testing Styles Methodical Madness

© 2012 WIPRO LTD | 5 Some Exploratory Testing Ideas

© 2012 WIPRO LTD | 6 Exploratory Testing using Question Patterns While testing a system, how often have you felt “I have tested something similar before?” The Domain Factor On the other hand, how often have you been bogged down by an application, just because it was from a different domain that you are not so familiar with? The Reuse Factor Is there a way to develop testing ideas once and reuse them across domains / applications? The Knowledge Factor Is there a way an experienced tester or domain expert can impart his/her experience to others? The Exploratory Testing Context Is there a way to explore the application systematically and find bugs?

© 2012 WIPRO LTD | 7 Question Patterns and Exploratory Testing Question Patterns: A Set of interrelated Questions grouped together, focused on an aspect of the system under test that Enable someone to understand the application better Can be used as a technique for requirements capture Patterns help the tester explore the application systematically and find bugs Example: Parent and Child Entity Pattern Scenario A).Consider an online shopping application where the user can add / modify / delete multiple items in his shopping cart order. In this case, the shopping cart is a business entity which has multiple child entities (items) within itself (a one- to-many relationship between the cart and the items) where the child entity (item) can be added/updated or deleted. Scenario B). Consider an application where the user can apply for an insurance policy online, to which he can add / modify / delete multiple beneficiaries. In this case, the policy is a business entity which has multiple child entities (beneficiaries) within itself (a one-to-many relationship between the policy and the beneficiaries) where the child entity (beneficiary) can be added/updated or deleted.

© 2012 WIPRO LTD | 8 Question Patterns and Exploratory Testing Questions that can be asked: Parent and Child Entity Pattern What are the business entities involved? What is the relationship between them? (One to Many, One-to-One, etc) What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search) What are the field level and business validations for each entity? Answers to such questions enable testers to find gaps in requirements (or even capture requirements). Answers to such questions also serve as a guideline and a checklist during exploratory testing.

© 2012 WIPRO LTD | 9 Question Patterns and Exploratory Testing Questions that can be asked: Parent and Child Entity Pattern What are the business entities involved? What is the relationship between them? (One to Many, One-to-One, etc) What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search) What are the field level and business validations for each entity? Answers to such questions enable testers to find gaps in requirements (or even capture requirements). Answers to such questions also serve as a guideline and a checklist during exploratory testing.

© 2012 WIPRO LTD | 10 Exploratory Testing by Looking for Specific Problems Purpose of Testing – Looking for Bugs/Problems that could impact the stakeholders Defect Seeking v/s Defect Finding Make a list of possible issues Design tests to find ways to stimulate the possible issues identified

© 2012 WIPRO LTD | 11 Exploratory Testing by Analyzing Data Flow Most Applications process / manage data Identify the input and output data Verify the output data for various combinations of input data (including green path values, invalid values, boundary conditions, large values, etc) Try modifying the data at various stages in the flow and see the impact Identify the right combinations / representatives of data

© 2012 WIPRO LTD | 12 Exploratory Testing by Analyzing Flows Identify end-to-end flows and branching Depict the branching graphically, if needed Try parallel flows Try varying the sequencing / timing of the flows Analyze the impact of input data values on the flows Check the main flows as well as alternate/exceptional flows Check for state transition flows

© 2012 WIPRO LTD | 13 Session Based Test Management – Accountable and Manageable Exploratory Testing

© 2012 WIPRO LTD | 14 Session Based Test Management - Approach  Test Execution is divided into time-boxed Test Sessions, where the tester does (uninterrupted) exploratory testing with a specific charter  Tester takes brief notes about the testing done and the issues found  Tester uses a checklist of testing ideas as a guide for testing  Further tests are dynamically optimized depending on the results of the current test  The Test Lead conducts a ‘Debrief’ session with the testers on the test findings and plans further executions based on the findings

© 2012 WIPRO LTD | 15 Session Based Test Management – Session Contents The contents of a typical test session are as follows: Charter: Contains the mission of the given session. Typically it mentions the application areas to be covered / explored, at a high level. Tester name, Session Time Start and End Task Breakdown: Time taken for Test execution v/s Bug Reporting, Charter (Original Session Charter) v/s Opportunity (Diversions taken) Test Notes: Notes of what the tester executed, in brief. Bugs/Issues found

© 2012 WIPRO LTD | 16 Session Based Test Management – Accountability Tester’s Activities: Tracked via Test Session reports Coverage and Testing Progress Tracking: High Level Coverage: Number of functional areas and quality criteria covered Granular Coverage: Number of checklists/items/test ideas covered for each functional area Traceability: Each test idea and item in the checklist can be mapped to requirements

© 2012 WIPRO LTD | 17 Benefits of Exploratory Testing

© 2012 WIPRO LTD | 18 Exploratory Testing - Benefits Lesser Preparation and lesser effort Quicker Bug Detection Leverages on Tester’s Abilities, experience and creativity Works even if requirements are not fully frozen

© 2012 WIPRO LTD | 19 Skilled Testing v/s Deskilling – The Classic Debate Deskilling via Detailed Test Cases leads to higher test design and maintenance costs and limited bug detection capabilities Testers with lesser experience can be trained in skilled exploratory testing Skilled Testing is more cost effective and results in better product quality

© 2012 WIPRO LTD | 20 Case Studies

© 2012 WIPRO LTD | 21 Case Study 1 – The Problem The Context: Previous customer concerns over quality in beta releases Large volume of functional tests to be executed in a short time span - Conventional Testing Estimations: 3000 test cases to be designed and executed in 2 months (including KAP phase), requiring 14 testers Inexperienced Testers with no domain knowledge Customer more focused on bugs/issues than in testing processes A Police IT Application for a Government Customer

© 2012 WIPRO LTD | 22 Case Study 1 – Testing Approach and Results Question Patterns used to arrive at checklist of testing ideas -Entity Relationship pattern - State Transition Pattern - Master Data – Transaction Data Pattern Checklists consisting of Business Conditions and Testing Ideas Initial Test Sessions focused on exploration of the business operations of entities Deep Dive Test Sessions focused on finding specific problems Test Sessions for End to End Testing focused on Application Flow and Data flow Results: 65% reduction in testing efforts and 60% better defect detection A Police IT Application for a Government Customer

© 2012 WIPRO LTD | 23 Case Study 2 – The Problem The Context: The application is intended to enable healthcare providers and pharmacies to place orders quickly Bugs not being found early enough High cost of testing Frequent patch releases with incomplete functionality Mix of experienced and inexperienced testers, with fair domain knowledge Parallel run of conventional testing and Session Based Test Management An Order Management Application for a Healthcare customer

© 2012 WIPRO LTD | 24 Case Study 2 – Testing Approach and Results Question Patterns used to arrive at checklist of testing ideas for regression testing Checklists consisting of Business Conditions and functional changes to be checked for Initial Test Sessions focused on the exploration of new functionalities Subsequent Test Sessions focused on user experience Results: 50% reduction in test design efforts and quicker defect detection An Order Management Application for a Healthcare customer

© 2012 WIPRO LTD | 25 Thank you Vivek Devarajan Senior Consultant, Wipro Technologies

© 2012 WIPRO LTD | 26 Appendix

© 2012 WIPRO LTD | 27 Artifacts Created Checklists Created for Session Based Test Management: Sample Test Session:

© 2012 WIPRO LTD | 28 Session Based Test Management Freestyle Testing Scripted Testing Session Based Test Management Exploratory Testing SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles S. No. FactorScripted TestingSession Based Test Management Freestyle Testing 1CostVery HighMedium to LowVery Low 2Defect Detection Limited Effectiveness Sometimes late in the cycle Highly Effective Early in the cycle Highly Effective Very unpredictable 3AccountabilityVery High Very Low

© 2012 WIPRO LTD | 29 SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles S. No. FactorScripted TestingSession Based Test Management Freestyle Testing 4PredictabilityVery HighMediumVery Low 5Possibility of Defect Slippage MediumLowHigh 6Documentatio n Level Very HighMediumVery Low 7Requirements Coverage Very High Very Low 8Requirements Traceability Very HighHighVery Low Session Based Test Management

© 2012 WIPRO LTD | 30 Comparison of Structured v/s Freestyle Testing Freestyle Testing Scripted Testing Benefits: High Predictability Deskilling (Test Execution) High Accountability Drawbacks: Very High Costs (Resources + Tools) for Test Creation as well as Maintenance Limited Defect Detection Capabilities (Lack of Proactive/Creative Approaches) Needs well documented requirements Benefits: Very Low Costs for Test Creation/Maintenance Highly effective in quick Defect Seeking, especially effective for tricky bugs Does not need well documented requirements Drawbacks: Low Accountability Relies on the creativity of the Tester

© 2012 WIPRO LTD | 31 Session Based Test Management Freestyle Testing Scripted Testing Automated Testing Session Based Test Management Exploratory Testing Elaborate Documentation Static Test Suite Higher Testing Costs Limited Defect Detection Concise Documentation Dynamic Test Optimization Lower Testing Costs Better Defect Detection No Accountability No Predictability High Possibility of Defect Slippage SBTM - Leveraging the benefits of both Testing Styles

© 2012 WIPRO LTD | 32 Evolution of Session Based Test Management Adhoc / Freestyle Testing Exploratory Testing Session Based Test Management Evolution

© 2012 WIPRO LTD | 33 Salient Features of Session Based Test Management Application Learning, Test Design and Test Execution happen in Parallel A Structured, measurable and accountable method of Exploratory Testing Crisp Checklists to guide the tester on what needs to be tested Execution happens in Test Sessions (Uninterrupted Test Execution Time) with specific charters, after which tester takes notes and makes a report Tester takes a different execution route, if required, depending on issues found in a Test Session Future test sessions are planned based on results from current session

© 2012 WIPRO LTD | 34 Metrics for Session Based Test Management Test Session MetricsCumulative Defect TrendDefect TrendExecution Metrics