More and Better Test Ideas Rikard Edgren TIBCO Spotfire EuroSTAR share one-liner test ideas

Acknowledgements This approach was invented and refined at my workplace, and at many other places as well A lot of good talking around test ideas: Kaner, Bach, Bolton Many tips & tricks at many places See paper for details and references

Agenda Test idea definition Two examples More test ideas Better test ideas Summary

Test Idea Definitions "a brief statement of something that should be tested. (Brian Marick) test condition: An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element.(ISTQB 1.1) Test-Ideas List: An enumerated list of ideas, often partially formed, that identify potentially useful tests to conduct (Rational Unified Process)

More Test Idea Definitions A test idea is the thought that guides our creation of a test (Cem Kaner) Test Idea: an idea for testing something (James Bach) maybe no definition is needed, you immediately understand what a test idea is.

Example #1 Spare time project from Canada A tool for exploratory testers Test specification from summer 2009 Used for Beta release 0.2

Example #1 A combination of generic and detailed test ideas 53 test ideas cover many important aspects Possible to see what is missing It is a good idea to combine several test ideas, and execute them together Efficient to make the list while trying the software

Scripted & Exploratory Testing As approaches to software testing: –ST focus on control, precision –ET focus on learning and the freedom of the tester As methods/activities/techniques: –ST designs and reviews tests in advance –ET design and execute simultaneously Test ideas enable early feedback, add visibility, save time, and can act as a check list for ET

Example #2 Product development in Gothenburg, Sweden Test specification from 2008 Used for a 1.0 release Good starting point for next project

Example #2 Test Ideas range from vague to very vague Test Ideas can be converted to detailed test scripts, together with necessary details Test Ideas can be executed as is, trusting the tester to design the tests and choose thoroughness

More Test Ideas Requirements arent enough For each part and its interactions –What should be right? –What can go wrong? –What quality attributes matter? –What is important? Be creative, and work hard –for each test idea, there are two more

Test Idea Triggers So many inputs, methods, heuristics... –requirements/specifications/prototypes/code –bugs/error catalogs/support/customers –technology/tools/systems/interactions/models –quality objectives/attributes –testing techniques –test idea lists –people your subjectivity, knowledge, experience...

My Favorites #1 Quality attributes –Capability –Reliability –Usability –Security –Scalability –Performance –Installability –Compatibility –Supportability –Testability –Maintainability –Portability –Localizability –Accessibility –Interoperability

My Favorites #2 Delete/Remove/Empty/Null Read the manual Look at all deliverables Use the same data/document a very long time Real usage of the product

Better Test Ideas Powerful Yield significant results Credible Likely Easy to evaluate Useful for troubleshooting Informative Appropriately complex Giving insightful information Easy to understand* Fast to execute* * not in Kaners What is a good test case

Better Test Ideas Test ideas get better through collaboration...with people that care, and understand whats important Try to get feedback from many sources Discard test ideas that seem weaker You will get even better test ideas as you learn more about the product

Test Ideas in Practice Many different types One-liners are fast to read Easy to update Vague to leave freedom Useful now-then, me-otherpeople Adjust granularity for better readability

Find Five Faults...but there are other things than these faults that are more important...

Limitations Test executors might need a lot of testing and product knowledge Might be difficult to evaluate results anyway Can be difficult to get good feedback Generic test ideas might obscure the new context

Summary Many different test ideas is a good thing More people will give better perspectives Test ideas are fast to read, write and update Test ideas could be the glue you need for using test scripts and an exploratory testing approach Documented test ideas is a simple idea, you need to use it to understand the value

Questions ???