Or how to find bugs faster… Exploratory Testing Or how to find bugs faster…

Summary What is Exploratory Testing? Why are we doing black box testing? Session Based Testing Where do we test? Types of testing How outlines can help How flow charts could help

What is Exploratory Testing? Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1983,[1] now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."[2] While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run. Exploratory testing is often thought of as a black box testing technique. Instead, those who have studied it consider it a test approach that can be applied to any test technique, at any stage in the development process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement of the tester, and the tester's responsibility for managing his or her time.[3] - http://en.wikipedia.org/wiki/Exploratory_testing

Exploratory Testing is not just adhoc and saying adhoc is a turn off to people who don't understand exploratory testing. Why? True adhoc is literally monkey pressing buttons. … is a learning process. The code will change as the project progresses. Doing the same thing over and over again has diminishing returns Not a replacement for doing/running test cases

Exploratory Testing Is not a replacement for automation testing It’s a supplement to reduce the risk of the unknown quantity of bugs Comment : I would implore various strategies of depth and breadth type testing in trying to reduce the risk of an unknown quality and severity of bugs to make them known.

Why do we do black box testing? Because human thoughts and behavior changes from individuals as well as over time. We don’t necessarily do the same thing over and over again like automation testing does Bottom line : people think. And people think differently.

Human Cognitive Behavior Be aware of yourself and why you may miss bugs: http://viscog.beckman.illinois.edu/media/mistakes.html http://viscog.beckman.illinois.edu/flashmovie/15.php

Testing Be a scientist! Poke around! Try to figure things out Go beyond just doing the test case… Talk to each other about it, learn from each other If you don’t understand the bug, then how do you expect other people to? It’s like chess, try to think in steps ahead Test cases aren’t the rule book; they are a guideline The more you test the more experience you will gain. You can learn from any source; I often learn from people I have mentored as much as I’ve learned from the people who have taught me!

Session Based Testing Session-based testing is a software test method that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. The method can also be used in conjunction with scenario testing. Session-based testing was developed in 2000 by Jonathan and James Bach. Highly recommend looking at this… this is what top blackbox testers do in finding bugs as well as other things discussed later on.

Where to test? New Code Change Trends in bugs Looking at code change Trends in bugs Talking to Dev; what might they be worried about? Talking to other QA; what seems fragile? Risk areas What’s dangerous if it fails for human safety Embarrassment (functionality) Complexity untested

Types of testing MVC = Model View Controller UI Testing Functional/Business Logic testing Data/Input Testing Does user flow make sense? (Usability) Culturally effective? Which culture? Does it look cluttered?

Types of testing Functional Testing (Positive Testing) Negative Testing Boundary Testing Race Conditions Environment conditions Device specific Localization Initialization Load/Endurance 508 Compliance

Examples Functional Testing (Positive Testing) Negative Testing Use Cases and specifications Negative Testing Error handling, like bad email and incorrect website address Boundary Testing Max characters, or with integers -1, 0, 1, max -1, max, max + 1 Race Conditions Running music app, video app, and getting a phone call at the same time

Examples Environment conditions Device specific Localization Rural area testing of telephony vs urban Device specific Battery running low, graphics issues Localization Date format, starting day, fitting in ui, punctuations Initialization Start up of device, FTE, start up of apps versus OTA and continual usage

Examples Load/Endurance 508 Compliance Loading tons of facebook, versus keeping music app running for days on end 508 Compliance Handicap ie : blind, deaf user interfaces

How outlines/test plans can help From 1.0 https://wiki.mozilla.org/B2G/QA/Gaia_Test_Plan/Browser https://wiki.mozilla.org/Releases/Firefox_25/Test_Plan https://wiki.mozilla.org/QA/Fennec/Milestones/2.0/TestPlan:Virtual_Keyboard/Android_VKB https://wiki.mozilla.org/QA/Fennec/24/Beta/8

B2G Specific Example https://docs.google.com/spreadsheet/ccc?key=0Arku3jleCA0UdF9ZZWtyYm9iV1NLOGRTWkQzWHFsS2c#gid=5

How flowcharts can help Mapping out user flow Demo

Resources QA conferences QA newsgroups/forums/websites They can be pricey and advertisey QA newsgroups/forums/websites QA magazines/books/articles http://en.wikipedia.org/wiki/Software_testing http://www.amazon.com/exec/obidos/ASIN/0471358460/softwarqaandtest

B2G Architecture https://wiki.mozilla.org/B2G/Architecture https://github.com/sotaroikeda/firefox-diagrams/wiki/Firefox-Diagrams Git Hub: https://github.com/mozilla-b2g/gaia/commits/master Compare to sources.xml file : https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-central-unagi/latest/sources.xml Tool for searching code https://mxr.mozilla.org/