1. More and Better Test Ideas
share one-liner test ideas
Rikard Edgren
TIBCO Spotfire
EuroSTAR
I have two messages:
* You should use more test ideas, and go far beyond the explicit requirements
* Writing test ideas in short sentences is a very efficient method
Version 1.0.2

2. Acknowledgements
This approach was “invented” and refined at my workplace, and at many other places as well
A lot of good talking around test ideas: Kaner, Bach, Bolton
Many tips & tricks at many places
See paper for details and references
Kaner, Bach Bolton has not so much about documenting or reviewing test ideas, but a lot about how to think.

3. Agenda Test idea definition Two examples More test ideas
Better test ideas
Summary

4. Test Idea Definitions
"a brief statement of something that should be tested.“ (Brian Marick)
“test condition: An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element.”(ISTQB 1.1)
Test-Ideas List: “An enumerated list of ideas, often partially formed, that identify potentially useful tests to conduct” (Rational Unified Process)
My definition: ”a brief notion about something that could be tested”
”condition: A logical expression that can be evaluated as True or False, e.g. A>B. See also test condition.” (ISTQB 1.3)
ISTQB is a bit too formal for my taste.
It shouldn’t be a surprise that this is included in RUP, everything is included there.

5. More Test Idea Definitions
”A test idea is the thought that guides our creation of a test” (Cem Kaner)
“Test Idea: an idea for testing something” (James Bach)
maybe no definition is needed, you immediately understand what a test idea is.
For me, test ideas are the most important part of software testing.
In this presentation a test idea is a documented idea about something to test.

6. Example #1 Spare time project from Canada
A tool for exploratory testers
Test specification from summer 2009
Used for Beta release 0.2
I’m gonna read 53 one-liner test ideas, and hopefully you will come up with new ideas for your own project.
It takes about 3 minutes to read all test ideas very fast.

7. This list is available at Session Tester Forum: http://clearspace

8. Example #1 A combination of generic and detailed test ideas
53 test ideas cover many important aspects
Possible to see what is missing
It is a good idea to combine several test ideas, and execute them together
Efficient to make the list while trying the software
Generic test ideas could also be called ongoing test ideas, and be spread to other test areas as well.
It should be noted that in 3 minutes you get an overview of what testing will be performed.

9. Scripted & Exploratory Testing
As approaches to software testing:
ST focus on control, precision
ET focus on learning and the freedom of the tester
As methods/activities/techniques:
ST designs and reviews tests in advance
ET design and execute simultaneously
Test ideas enable early feedback, add visibility, save time, and can act as a check list for ET
Exploratory Testing is weak on the review part
Scripted Testing is weak on the learning part
If scripts are used to control the testing, you are using a scripted testing approach
If scripts are used as a baseline for further testing, you are using an exploratory approach
Test ideas is a great tool for the exploratory tester, it is a way communicate early, to make sure that you test the most important things (including the explicit requirements)
One-liner test ideas can also be very good for scripted testing, better to choose between test ideas early, feedback can come from more people, and visibility for stakeholders et.al.
Test ideas don’t belong to either, but is extra powerful for Exploratory Testing
One or a group of test ideas might make very good topics for Exploratory Testing in Session-Based Test Management.

10. Potato testing:
The square symbolizes the features and bugs you will find with test cases stemming from requirements (that can’t and shouldn’t be complete)
The blue area are all bugs, including things that maybe no customers would consider bugs.
The brown area is all important bugs, those bugs you’d want to find and fix.
So how do you go from the requirements to ”all important bugs”?
You won’t have time to create test scripts for ”everything”.
So maybe you do exploratory testing (thin lines in many directions), and hope for the best.
Or maybe you test around one-liners (thicker vertical lines), that are more distinct, that are reviewed, and have a better chance of finding the important things.
Either option, some part of luck, and a large portion of hard work is needed.
But I think you have a much better chance if you are using one-liners, especially if it’s a larger project.
Lately I have realized that one-liners aren’t essential; that this problem has been solved many times at many places with many different approaches. What is common could be that testers learn a lot of things from many different sources, combine things, think critically and design tests (in advance or on-the-fly) that will cover the important areas. Maybe we need a name for this method; it could be Grounded Test Design

11. Example #2 Product development in Gothenburg, Sweden
Test specification from 2008
Used for a 1.0 release
Good starting point for next project
I asked my colleagues about a good example of a test specification, and one of them answered:
”Basically you could use anyone I have written.”
But then again, he is a member of ”the best and most humble test team in the world.”

12. I’ll let you read for yourself. It’s OK to ask questions.
7.1.1 is the base; make sure all requirements are met.”
The other test ideas are either things omitted in requirements, or details that will guide the testing.
”Sanity checks” can be such since there are unit tests covering the details.
The test ideas leave room for freedom, and also trusts that the tester can work out the details when encountering the software.
For some things it might be necessary with more details about what needs to be tested, or checked.

13. Example #2 Test Ideas range from vague to very vague
Test Ideas can be converted to detailed test scripts, together with necessary details
Test Ideas can be executed as is, trusting the tester to design the tests and choose thoroughness

14. More Test Ideas Requirements aren’t enough
For each part and its interactions
What should be right?
What can go wrong?
What quality attributes matter?
What is important?
Be creative, and work hard
for each test idea, there are two more
Requirements aren’t complete: implicit and incorrect requirements
It would be insane to document everything in a requirements document.
There are a lot of implicit requirements, things that a user expect from the software.
Basically, you learn as much as you can, then decide what is important.
The numbers of test ideas are about infinite; there are so many things that can go wrong.
But, the number of important ways the program can fail, is enumerable; the testers’ job is to address most of those.

15. Test Idea Triggers So many inputs, methods, heuristics...
requirements/specifications/prototypes/code
bugs/error catalogs/support/customers
technology/tools/systems/interactions/models
quality objectives/attributes
testing techniques
test idea lists
people
your subjectivity, knowledge, experience...
You can get far by just thinking about the functionality, but it is good to have many more things that help you come up with good test ideas.
I call these Test Idea Triggers.
Of course you have the requirements and other specifications.
If you can try out a prototype, that’s great.
You can also look at the code if that is possible and helps.
The bug system has a lot of information regarding quality, and mistakes that have been made before might very well happen again; maybe you have gathered the most important types of defects in an error catalog.
What the customers tell the support department is very important information, talk to the support people about what they see and hear.
But not all customers will tell you about their problems. Some won’t bother reporting things, especially non-blocking, but annoying things; and those that never bought your software because of something they didn’t like won’t tell you either.
The more you know about the technology, the more you know about what can be done, and what shouldn’t be done.
There are many tools out there, and some might give you new ideas; maybe a test idea can be to use a tool in a certain way.
By looking at systems as a whole, you can see the bigger picture, the picture that is the most important. Remember: ”your box is never the biggest box”.
Interactions with other software and within your product is the source perhaps of a majority of defects.
Models help you understand how the software works; and then you can also understand how it can fail.
The quality attributes can be used at many levels, and if your company has quality objectives they should be first on your list.
You might create a bunch of ”ongoing test ideas” that all testers keep in the back of their head all the time.
There are many testing techniques, and one like Equivalence Partitioning is every tester using all of the time, sub-consciously, or maybe unconsciously. But if you learn more about the theory, you might be able to draw even more juice from it. Boundary Value Analysis, or rather boundary testing is good and common. Classification Trees and Decision Tables can also be used, but mostly on small, important parts of a system.
Also note that the test techniques is just a small part of the equation. Categories go into each other, but focusing only on requirements and test techniques has a big risk of giving a too shallow test effort.
Can you read other test idea lists, maybe from different companies? Maybe you only got one idea while hearing the Session Tester Test Ideas, but it was a good one?
Last, but not least, we have people. Talking to people, learn from another and inspire to more and better test ideas.
And of course you should bring in your experiences from testing, and your knowledge from any area.
It is good if you can use the software in a ”real” way, because you will understand it better, and get a feeling for what is missing
It is good to be subjective, because you know, the users are subjective..
Using all of these can be seen as a test technique, that I call Grounded Test Design.

16. My Favorites #1 Quality attributes Capability Reliability Usability
Security
Scalability
Performance
Installability
Compatibility
Supportability
Testability
Maintainability
Portability
Localizability
Accessibility
Interoperability
CRUSSPIC STMPL – a mnemonic by Bach, Bolton.
+ AI (Accessibility is often a part of Usability, but I think it is important enough to have its own bullet)
The quality attributes are important for all products, not all of them, but many of them.
The bold ones are the ones that usually give me new ideas.
Capability is the same as Functionality and is covered in other ways.
STMPL are more for the code, than for the customers, but it might be fruitful to think about those aspects from the users’ perspective.
How will they maintain, support, test or port your software and its artifacts?

17. My Favorites #2 Delete/Remove/Empty/Null Read the manual
Look at all deliverables
Use the same data/document a very long ”time”
”Real” usage of the product
ST-12. Verify that the Help matches the functionality
Use the same data – ”to not start from scratch all the time”

18. Better Test Ideas Powerful Yield significant results Credible Likely
Easy to evaluate
Useful for troubleshooting
Informative
Appropriately complex
Giving insightful information
Easy to understand*
Fast to execute*
Kaner’s advice that “Test cases can be “good” in a variety of ways. No test case will be good in all of them.”
* not in Kaner’s What is a good test case

19. Better Test Ideas Test ideas get better through collaboration
...with people that care, and understand what’s important
Try to get feedback from many sources
Discard test ideas that seem weaker
You will get even better test ideas as you learn more about the product
Your sources can include any member of the development team, but also stakeholders, customers you trust etc.
The more, the better!
When test ideas get better over time, it might be appropriate to update them, or it might not be important at all.
It should also be noted that test ideas can get better depending on how you use them.
Some might be perfect for a unit test, others appropriate for manual exploratory testing, while some might be a small part of a larger scenario.

20. Test Ideas in Practice Many different types
One-liners are fast to read
Easy to update
Vague to leave freedom
Useful now-then, me-otherpeople
Adjust granularity for better readability
Developers can get insights by reading test ideas
Managers can understand more about the actual testing
Stakeholders can have a say
All readers can see what testing is done, and make sure that nothing important is missing.
My experience says that it starts to get difficult if you have more than a 100 test ideas in a specification.
You can either write test ideas at a higher level, or split in several pieces.

21. Find Five Faults
Missing: Skalman, tail, flower, ape, Lille Skutts ear.
More important:
Are the pictures in a Bamse comic?
What happens on the strips before and after?
Is the picture a good part of the story?
Is the picture consistent with the style of the story?
Are the right people on the picture?
The right environment?
...but there are other things than these faults that are more important...

22. Limitations
Test executors might need a lot of testing and product knowledge
Might be difficult to evaluate results anyway
Can be difficult to get good feedback
Generic test ideas might obscure the new context
Sharing between companies/industries might not be allowed?
But I can assure you that it is easier to get good feedback (from most people) on test ideas than on test cases

23. Summary Many different test ideas is a good thing
More people will give better perspectives
Test ideas are fast to read, write and update
Test ideas could be the glue you need for using test scripts and an exploratory testing approach
Documented test ideas is a simple idea, you need to use it to understand the value
Test ideas enable early feedback, add visibility, save time, and is an effective check list (ET only)
”the glue” – maybe you can use test scripts for the requirements, and test ideas for everything else?

24. Questions ??? www.thetesteye.com redgren@tibco.com
Why isn't this more common?
I think the reason for a scripted environment is that the test designer is almost divine, so no good feedback is impossible.
And for an Exploratory environment, the tester is almost divine and will be restrained by documented test ideas.
I'm not kidding here, humility is very difficult for excellent testers, but we need feedback as well.
Do you have a template to share?
A template is not a solution, it is how you and your colleagues think that matters.
Could test ideas be used as titles for Session-Based testing?
Yes
Why do you bother writing test scripts?
We have requirements that tests need to be documented in a more thorough fashion, i.e. Test cases.