1. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 1 Black Box Software Testing Fall 2004 PART 7 -- TEST DESIGN by Cem Kaner, J.D., Ph.D. Professor of Software Engineering Florida Institute of Technology and James Bach Principal, Satisfice Inc. Copyright (c) Cem Kaner & James Bach, 2000-2004 This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. These notes are partially based on research that was supported by NSF Grant EIA-0113539 ITR/SY+PE: "Improving the Education of Software Testers." Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

2. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 2 Test Design: Some Readings Kaner, Bach & Pettichord, “Testing Techniques” in Lessons Learned in Software Testing. Kaner, C. (2003) What is a good test case, Conference on Software Testing, Analysis & Review (STAR East), Orlando, FL: http://www.testingeducation.org/articles/what_is_a_good_test_case_star_2003_paper. pdf http://www.testingeducation.org/articles/what_is_a_good_test_case_star_2003_paper. pdf Whittaker, What is testing? And why is it so hard? http://www.computer.org/software/so2000/pdf/s1070.pdf http://www.computer.org/software/so2000/pdf/s1070.pdf Whittaker & Atkin, Software Engineering is not Enough, http://www.sisecure.com/pdf/jwsasofteng.pdf http://www.sisecure.com/pdf/jwsasofteng.pdf

3. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 3 Test design: Let's take stock We've studied Domain testing –Reduce the number of test cases –Focus on variables (input / output / intermediate / control) –Pick high-power tests Risk-based testing –Identify potential failures –Develop powerful tests to expose errors Scenario testing –Credible, motivating stories –Complex combinations that can model experienced use

4. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 4 Test design Test design is a multi-dimensional challenge

5. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 5 Test design What is the difference between –User testing? –Usability testing? –User interface testing?

6. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 6 Test design Testing combines techniques that focus on: –Testers: who does the testing. –Coverage: what gets tested. –Potential problems: why you're testing (what risk you're testing for). –Activities: how you test. –Evaluation: how to tell whether the test passed or failed. All testing involves all five dimensions.

7. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 7 Test design A technique focuses your attention on one or a few dimensions, leaving the others open to your judgment. You can combine a technique focused on one dimension with techniques focused on the other dimensions to achieve the result you want. For example: –Domain testing Coverage (test every variable) Potential problems (identify risks for each variable) –Risk-based testing Potential problems –Scenario testing Activity (how you test) -- See Lessons Learned, ch. 3

8. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 8 What’s a test case? Focus on procedure? –“A set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.” (IEEE) Focus on the test idea? –“A test idea is a brief statement of something that should be tested. For example, if you're testing a square root function, one idea for a test would be ‘test a number less than zero’. The idea is to check if the code handles an error case.” (Marick)

9. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 9 Test cases In my view, The point of running the test is to gain information, for example whether the program will pass or fail the test. Implications of this approach: –The test must be CAPABLE of revealing valuable information –The SCOPE of a test changes over time, because the information value of tests changes as the program matures –The METRICS that count test cases are essentially meaningless because test cases merge or are abandoned as their information value diminishes. A test case is a question you ask of the program.

10. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 10 Test Cases Information Objectives Test Attributes Techniques Factors involved in test case quality

11. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 11 Test design Information Objectives? Let's try an exercise …

12. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 12 Information Objectives: Which Group is Better? Two groups test the same program. The functions are equally important The bugs are equally significant This is artificial, but it sets up a simple context for a discussion of tradeoffs. Testing Group 1 Testing Group 2 From Marick, Classic Testing Mistakes

13. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 13 Which group is better?

14. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 14 Information objectives Find defects Maximize bug count Block premature product releases Help managers make ship / no-ship decisions Minimize technical support costs Assess conformance to specification Conform to regulations Minimize safety-related lawsuit risk Find safe scenarios for use of the product Assess quality Verify correctness of the product Assure quality

15. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 15 Test Cases Information Objectives Test Attributes Techniques Factors involved in test case quality

16. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 16 Test attributes To different degrees, good tests have these attributes: Power. When a problem exists, the test will reveal it. Valid. When the test reveals a problem, it is a genuine problem. Value. It reveals things your clients want to know about the product or project. Credible. Your client will believe that people will do the things that are done in this test. Representative of events most likely to be encountered by the user. (xref. Musa's Software Reliability Engineering). Motivating. Your client will want to fix the problem exposed by this test. Performable. It can be performed as designed. Maintainable. Easy to revise in the face of product changes. Repeatable. It is easy and inexpensive to reuse the test. Pop. (short for Karl Popper) It reveal things about our basic or critical assumptions. Coverage. It exercises the product in a way that isn't already taken care of by other tests. Easy to evaluate. Supports troubleshooting. Provides useful information for the debugging programmer. Appropriately complex. As the program gets more stable, you can hit it with more complex tests and more closely simulate use by experienced users. Accountable. You can explain, justify, and prove you ran it. Cost. This includes time and effort, as well as direct costs. Opportunity Cost. Developing and performing this test may prevent you from doing other tests (or other work). »See Kaner, What IS a Good Test Case?

17. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 17 Test Techniques A test technique is a recipe for performing these tasks in order to reveal something worth reporting –Analyze the situation. –Model the test space. –Select what to cover. –Determine test oracles. –Configure the test system. –Operate the test system. –Observe the test system. –Evaluate the test results.

18. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 18 Eleven dominating techniques This list reflects our observations in the field. It is not exhaustive. We put a technique on the list if we’ve seen credible testers drive their thinking about black box testing in the way we describe. A paradigm for one person might merely be a technique for another. –Domain testing –Risk-based testing –Scenario testing –Function testing –Specification-based testing –Regression testing –Stress testing –User testing –State-model based testing –High volume automated testing –Exploratory testing

19. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 19 Paradigms A paradigm provides a structure of thinking about an area. –Typically, the description of a paradigm includes one or a few paradigmatic cases -- key examples. Much of the reasoning within the paradigm is based on analogies to these key cases. –The paradigm creates a mainstream of thinking--it provides insights and a direction for further research or work. But it implicitly also defines limits on what is relevant, interesting, or possible. Things outside of the paradigm are uninteresting. People who solve the types of puzzles that are characteristic of the paradigm are respected, whereas people who solve other types of puzzles instead are outsiders, and not well respected. A testing paradigm would define the types of tests that are (to the person operating under this paradigm) relevant and interesting.

20. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 20 Paradigms (Kuhn) See Thomas Kuhn, The Structure of Scientific Revolutions. He describes paradigms as follows: –A paradigm is a model based on a shared experiment that includes law, theory, application and instrumentation together and from which springs particular coherent traditions of scientific research. –A paradigm includes a concrete scientific achievement as a locus of professional commitment, prior to the various concepts, laws, theories and points abstracted from it. –The pre-paradigm period... is regularly marked by frequent and deep debate over legitimate methods, problems, and standards of solution, though these serve rather to define schools than to produce agreement.

21. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 21 Paradigms We’ll study a sample of these techniques. There is nothing magical about this list. They overlap. They don’t collectively cover everything that would be good to do. They are interesting because so many testers structure their thinking around one of these techniques. Imagine that you are one of the people who has adopted one of these techniques as your primary approach, your paradigm: –What makes for an excellent test? –What is your approach best for? –What are some weaknesses in your approach?

22. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 22 Black box testing paradigms Should you master all of these techniques? Perhaps, but be cautious about introducing too much change into your company (or your own practices) at one time. It might be most effective to master a few, adding more over time. Choose approaches that complement each other

23. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 23 Paradigm exercise Do any of the paradigms listed reflect a dominant approach in your company? Which one(s)? Looking at the paradigms as styles of testing, which styles are in use in your company? (List them from most common to least.) Of the ones that are not common or not in use in your company, is there one that looks useful, that you think you could add to your company’s repertoire?

24. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 24 Test Design Exercise Draw a table. –The columns are the 11 dominating test techniques (add a 12th or 13th if you have other favorites) –The rows are the 5 dimensions, testers, coverage, potential problems, activities, and evaluation. –In each cells, put an X if you think that this (column's) technique is primarily focused on that (cell's) dimension. Below the table (or on another page): –For each technique Start a new paragraph Write no more than 100 words Briefly define or describe the technique Explain why you think the technique is primarily focused on the dimensions you selected

25. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 25 Test Design Exercise Draw a table. –The columns are the 11 dominating test techniques (add a 12th or 13th if you have other favorites) –The rows are the 9 attributes: power, significance, credibility, representativeness, easy to evaluate, useful for troubleshooting, informative, appropriate complexity, triggers insight –In each cell, put an X if you think that this (column's) technique is primarily focused on that (cell's) dimension. Below the table (or on another page): –For each technique Start a new paragraph Write no more than 100 words Briefly define or describe the technique Explain why you think the technique is primarily focused on the dimensions you selected

26. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 26 Black Box Software Testing Function Testing

27. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 27 Function Testing Tag lines –“Black box unit testing.” –"Test what it can do" Fundamental question or goal –Test each function thoroughly, one at a time. Paradigmatic case(s) –Spreadsheet, test each item in isolation. –Database, test each report in isolation Strengths –Best for assessing capability rather than reliability –If used as a dominant technique, thorough analysis of each item tested Blind spots –Misses interactions, misses exploration of the benefits offered by the program.

28. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 28 Some Function Testing Tasks A function is something the product can do –Functions may be called features / commands or they may be identified only by what they do (usually, some of each) Identify each function and sub-function –From specifications or the draft user manual –From walking through the user interface –From trying commands at the command line –From searching the program or resource files for command names Determine how you would know if the function worked. Identify variables used by the functions and test their boundaries. Identify environmental variables that may constrain the function under test. Check that each function does what it's supposed to do and does not do what it's not supposed to do.