1. Black Box Testing mariamkayani@mcs.edu.pk Software Quality Engineering

2. Black Box Methods In a black box method, we choose our test cases based solely on the requirements, specification or (sometimes) design documents Normally, black box testing is based on the functional specification (requirements) for the software system Black box testing requires no knowledge of the internal paths, structure, or implementation of the software under test (SUT).

3. Applicability 3 Black box testing can be applied at all levels of system development—unit, integration, system, and acceptance.

4. Applicability 4 As we move up in size from module to subsystem to system the box gets larger, with more complex inputs and more complex outputs, but the approach remains the same. Also, as we move up in size, we are forced to the black box approach; there are simply too many paths through the SUT to perform white box testing.

5. Disadvantage-1 5  When using black box testing, the tester can never be sure of how much of the SUT has been tested. No matter how clever or diligent the tester, some execution paths may never be exercised.  For example, what is the probability a tester would select a test case to discover this "feature"? if (name=="Lee" && employeeNumber=="1234" && employmentStatus=="RecentlyTerminatedForCause") { send Lee a check for $100,000; }

6. Disadvantage-2 6 To find every defect using black box testing, the tester would have to create every possible combination of input data, both valid and invalid. This exhaustive input testing is almost always impossible. We can only choose a subset (often a very small subset) of the input combinations.

7. Advantages 7 Even though we can't test everything, formal black box testing directs the tester to choose subsets of tests that are both efficient and effective in finding defects. As such, these subsets will find more defects than a randomly created equivalent number of tests. Black box testing helps maximize the return on our testing investment.

8. Use of Black Box Testing 8  Can’t look at particular implementation  Sub-factors testable  Output correctness, Documentation, Availability  Reliability  Stress/load  Security  Training, Operational  Maintainability  Testability  Portability  Interoperability: software, equipment

9. Results of Black Box Testing (Outputs) 9  Types of errors  Missing/incorrect capability  Inadequate performance, deadlock  Incorrect output  Abnormal termination  Causes of errors  Elicitation errors  Specification errors  Programming errors  via incorrect results  Configuration/integration errors

10. Techniques of Black Box Testing 10  Equivalence Class Testing  Boundary Value Testing  Decision Table Testing  Pair-wise Testing  State-Transition Testing  Domain Analysis Testing  Use Case Testing

11. Equivalence Class Testing Group input and outputs of a component into classes which can be treated similarly – Assumed to be treated equally by component – A value in a partition is taken to be representative all the entire partition If the test passes for the representative value it is assumed that all other values in the group will also pass 11

12. Example Partitions  Values between boundaries  Numeric values 0  Dates 2000; leap/non-leap years  Months of various lengths  0–16 Don't hire  16–18 Can hire on a part-time basis only  18–55 Can hire as a full-time employee  55–99 Don't hire  Objects with different state  Empty/non-empty strings/lists/containers  Files with different permissions  Objects/references which exist/don’t exist  Valid vs invalid input and output values 12

13. Design of Test Cases Test cases must exercise all partitions Each test case lists – Inputs – Partition(s) to be exercised – Expected outcome of the test case 13

14. Equivalence Class Testing:Key Point A group of tests forms an equivalence class if: – They all test the same thing. – If one test catches a bug, the others probably will too. – If one test doesn't catch a bug, the others probably won't either. 14

15. Equivalence Partitioning[1][2] Test selection using equivalence partitioning allows a tester to subdivide the input domain into a relatively small number of sub-domains, say N>1 In strict mathematical terms, the sub-domains by definition are disjoint. The four subsets shown in (a) constitute a partition of the input domain while the subsets in (b) are not. Each subset is known as an equivalence class.

16. Program behavior and equivalence classes The equivalence classes are created assuming that the program under test exhibits the same behavior on all elements, i.e. tests, within a class. This assumption allow the tester to select exactly one test from each equivalence class resulting in a test suite of exactly N tests.

17. Faults targeted The entire set of inputs to any application can be divided into at least two subsets: one containing all the expected, or legal, inputs (E) and the other containing all unexpected, or illegal, inputs (U). Each of the two subsets, can be further subdivided into subsets on which the application is required to behave differently (e.g. E1, E2, E3, and U1, U2).

18. Example 1 Consider an application A that takes an integer denoted by age as input. Let us suppose that the only legal values of age are in the range [1..120]. The set of input values is now divided into a set E containing all integers in the range [1..120] and a set U containing the remaining integers. All integers [1..120] Other integers

19. Example 1 (contd.) Further, assume that the application is required to process all values in the range [1..61] in accordance with requirement R1 and those in the range [62..120] according to requirement R2. Thus E is further subdivided into two regions depending on the expected behavior. Similarly, it is expected that all invalid inputs less than or equal to 1 are to be treated in one way while all greater than 120 are to be treated differently. This leads to a subdivision of U into two categories.

20. Example 1 (contd.) All integers [62-120] [1..61] <1 >120

21. Example 1 (contd.) It is expected that any single test selected from the range [1..61] will reveal any fault with respect to R1. Similarly, any test selected from the region [62..120] will reveal any fault with respect to R2. A similar expectation applies to the two regions containing the unexpected inputs. Tests selected using the equivalence partitioning technique aim at targeting faults in the application under test with respect to inputs in any of the four regions, i.e. two regions containing expected inputs and two regions containing the unexpected inputs.

22. Equivalence Class Partitioning Example 2 Consider that wordCount method takes a word w and a filename f as input and returns the number of occurrences of w in the text contained in the file named f. An exception is raised if there is no file with name f.

23. Example 2 (contd.) begin String w, f Input w, f if (not exists(f) {raise exception; return(0);} if(length(w)==0)return(0); if(empty(f))return(0); return(getCount(w,f)); end Using the partitioning method described in the examples above, we obtain the following equivalence classes.

24. Example 2 (contd.) Equivalence classwf E1non-nullexists, not empty E2non-nulldoes not exist E3non-nullexists, empty E4nullexists, not empty E5nulldoes not exist E6nullexists, empty

25. Equivalence classes based on program output In some cases the equivalence classes are based on the output generated by the program. For example, suppose that a program outputs an integer. It is worth asking: ``Does the program ever generate a 0? What are the maximum and minimum possible values of the output?" These two questions lead to two the following equivalence classes based on outputs:

26. Equivalence classes based on program output (contd.) E1: Output value v is 0. E2: Output value v is the maximum possible. E3: Output value v is the minimum possible. E4: All other output values. Based on the output equivalence classes one may now derive equivalence classes for the inputs. Thus each of the four classes given above might lead to one equivalence class consisting of inputs.

27. Valid and Invalid Classes: Ranges (1) If a specification includes input conditions, these can be used to derive equivalence classes: 1.If an input condition specifies a range of values, this defines three classes: – within range: a valid input equivalence class – too large: an invalid input equivalence class – too small: a invalid input equivalence class

28. Valid and Invalid Classes : Ranges (2) 2.If an input condition specifies the number of values (for example, “one through six owners can be listed for the automobile”), identify one valid equivalence class and two invalid equivalence classes (no owners and more than six owners).

29. Valid and Invalid Classes: Enumerations 3.If an input condition specifies an enumerated set of values (e.g. “car”, “truck”, etc.): One valid equivalence class for each value in the enumeration. One invalid equivalence class: all values not in the enumerated set (i.e. everything else).

30. Valid and Invalid Classes: Presence / absence 4.If an input condition specifies a “must be”, situation (e.g. “first character of the identifier must be a letter”), this leads to: One valid equivalence class (e.g. the first character is a letter). One invalid equivalence class (e.g. the first character is not a letter).

31. Equivalence classes for variables: range Eq. ClassesExample One class with values inside the range and two with values outside the range. speed  [60..90] {50}, {75}, {92} area: float area  0.0 {{-1.0}, {15.52}} age: int{{-1}, {56}, {200}} letter: bool{{J}, {3}} Constraints Classes

32. Equivalence classes for variables: strings Eq. ClassesExample At least one containing all legal strings and one all illegal strings based on any constraints. firstname: string {{  }, {Sue}, {Loooong Name}} Constraints Classes

33. Equivalence classes for variables: arrays Eq. ClassesExample One class containing all legal arrays, one containing the empty array, and one containing a larger than expected array. int [ ] aName: new int[3]; {[ ]}, {[-10, 20]}, {[-9, 0, 12, 15]} Constraints Classes

34. Equivalence classes for variables: compound data type Structures in C++ are compound types. Such input types may arise while testing components of an application such as a function or an object. While generating equivalence classes for such inputs, one must consider legal and illegal values for each component of the structure. The next example illustrates the derivation of equivalence classes for an input variable that has a compound type.

35. Equivalence classes for variables: compound data type: Example struct transcript { string fName; // First name. string lName; // Last name. string cTitle [200]; // Course titles. char grades [200]; // Letter grades corresponding to course titles. }

36. Unidimensional partitioning One way to partition the input domain is to consider one input variable at a time. Thus each input variable leads to a partition of the input domain. We refer to this style of partitioning as unidimensional equivalence partitioning or simply unidimensional partitioning. This type of partitioning is commonly used.

37. Multidimensional partitioning Another way is to consider the input domain I as the set product of the input variables and define a relation on I. This procedure creates one partition consisting of several equivalence classes. We refer to this method as multidimensional equivalence partitioning or simply multidimensional partitioning. Multidimensional partitioning leads to a large number of equivalence classes that are difficult to manage manually. Many classes so created might be infeasible. Nevertheless, equivalence classes so created offer an increased variety of tests as is illustrated in the next section.

38. Partitioning Example Consider an application that requires two integer inputs x and y. Each of these inputs is expected to lie in the following ranges: 3  x  7 and 5  y  9. For unidimensional partitioning we apply the partitioning guidelines to x and y individually. This leads to the following six equivalence classes.

39. Partitioning Example (contd.) E1: x<3 E2: 3  x  7 E3: x>7 y ignored. E4: y<5 E5: 5  y  9 E6: y>9 x ignored. For multidimensional partitioning we consider the input domain to be the set product X x Y. This leads to 9 equivalence classes.

40. Partitioning Example (contd.) E1: x<3, y<5 E2: x<3, 5  y  9 E3: x 9 E4: 3  x  7, y<5 E5: 3  x  7, 5  y  9 E6: 3  x  7, y>9 E7: >7, y<5 E8: x>7, 5  y  9 E9: x>7, y>9

41. Systematic procedure for equivalence partitioning 1. Identify the input domain: Read the requirements carefully and identify all input and output variables, their types, and any conditions associated with their use.

42. Systematic procedure for equivalence partitioning (contd.) 2. Equivalence classing: Partition the set of values of each variable into disjoint subsets. Each subset is an equivalence class. Together, the equivalence classes based on an input variable partition the input domain. partitioning the input domain using values of one variable, is done based on the the expected behavior of the program. Values for which the program is expected to behave in the ``same way" are grouped together. Note that ``same way" needs to be defined by the tester.

43. Systematic procedure for equivalence partitioning (contd.) The equivalence classes are combined using the multidimensional partitioning approach described earlier. 3. Combine equivalence classes: This step is usually omitted and the equivalence classes defined for each variable are directly used to select test cases. However, by not combining the equivalence classes, one misses the opportunity to generate useful tests.

44. Equivalence Class Table External conditionValid equivalence classes Invalid equivalence classes integer value between 1 and 10 one of X, Y, or Z V1: [1,10] V2: [X] V3: [Y] V4: [Z] I1: [-∞,0] I2: [11,+∞] I3: [non-integer] I4: [not X, Y, or Z]

45. Test Case Strategy Once the set of equivalence classes has been identified, here is how to derive test cases: 1.Assign a unique identifier to each equivalence class. 2.Until all valid equivalence classes have been covered by at least one test case, write a new test case covering as many of the valid equivalence classes as possible. 3.Until all invalid equivalence classes have been covered, write a test case that covers one, and only one, of the uncovered invalid equivalence classes. – For each test case, annotate it with the equivalence class identifiers that it covers.

46. Boundary Value Analysis

47. BVA[1] [2] When you’re presented with a software test problem that involves identifying boundaries, look for the following types: – Numeric Speed – Character Location – Position Size – Quantity And, think about the following characteristics of those types: – First/Last Min/Max – Start/Finish Over/Under – Empty/Full Shortest/Longest – Slowest/Fastest Soonest/Latest – Largest/Smallest Highest/Lowest

48. Look at a few examples so you can start thinking about all the boundary possibilities: – If a text entry field allows 1 to 255 characters, try entering 1 character and 255 characters as the valid partition.Enter 0 and 256 characters as the invalid partitions. – If a program reads and writes to a floppy disk, try saving a file that’s very small, maybe with one entry. Save a file that’s very large—just at the limit for what a floppy holds. Also try saving an empty file and a file that’s too large to fit on the disk.

49. – If a program allows you to print multiple pages onto a single page, try printing just one (the standard case) and try printing the most pages that it allows. If you can, try printing zero pages and one more than it allows. – Maybe the software has a data-entry field for a 9-digit ZIP code. Try 00000-0000, the simplest and smallest. Try entering 99999-9999 as the largest. Try entering one more or one less digit than what’s allowed.

50. 50 Errors at the boundaries Experience indicates that programmers make mistakes in processing values at and near the boundaries of equivalence classes. For example, suppose that method M is required to compute a function f1 when x  0 is true and function f2 otherwise. However, M has an error due to which it computes f1 for x<0 and f2 otherwise. Obviously, this fault is revealed, though not necessarily, when M is tested against x=0 but not if the input test set is, for example, {-4, 7} derived using equivalence partitioning. In this example, the value x=0, lies at the boundary of the equivalence classes x  0 and x>0.

51. 51 Boundary value analysis (BVA) Boundary value analysis is a test selection technique that targets faults in applications at the boundaries of equivalence classes. While equivalence partitioning selects tests from within equivalence classes, boundary value analysis focuses on tests at and near the boundaries of equivalence classes. Certainly, tests derived using either of the two techniques may overlap.

52. 52 BVA: Procedure 1 Partition the input domain using unidimensional partitioning. This leads to as many partitions as there are input variables. Alternately, a single partition of an input domain can be created using multidimensional partitioning. We will generate several sub-domains in this step. 2 Identify the boundaries for each partition. Boundaries may also be identified using special relationships amongst the inputs. 3 Select test data such that each boundary value occurs in at least one test input.

53. 53 BVA: Example: 1. Create equivalence classes Assuming that an item code must be in the range 99..999 and quantity in the range 1..100, Equivalence classes for code: E1: Values less than 99. E2: Values in the range. E3: Values greater than 999. Equivalence classes for qty: E4: Values less than 1. E5: Values in the range. E6: Values greater than 100.

54. 54 BVA: Example: 2. Identify boundaries Equivalence classes and boundaries for findPrice. Boundaries are indicated with an x. Points near the boundary are marked *. E1 E2 E3 981009981000 99 999 x x ** * * E4 E5 E6 0299101 1 100 x x *** *

55. 55 BVA: Example: 3. Construct test set Test selection based on the boundary value analysis technique requires that tests must include, for each variable, values at and around the boundary. Consider the following test set: T={t1: (code=98, qty=0), t2: (code=99, qty=1), t3: (code=100, qty=2), t4: (code=998, qty=99), t5: (code=999, qty=100), t6: (code=1000, qty=101) } Illegal values of code and qty included.

56. Cause effect graphing[3] Nodes: Cause distinct instance of input condition or EC Effect observable result of change in system state Intermediate node combination of causes representing an expression on input conditions 56

57. Step 1: Break the specification down into workable pieces. First, the functional requirements are decomposed and analyzed. To do this, the functional requirements are partitioned into logical groupings, – for example, commands, actions, and menu options. Each logical grouping is then further analyzed and decomposed into a list of detailed functions, sub-functions, and so forth

58. Step two: Identify the causes and effects. a) Identify the causes (the distinct or equivalence classes of input conditions) and assign each one a unique number. – A cause can also be referred to as an input, as a distinct input condition, or as an equivalence class of input conditions. – Examining the specification, or other similar artifact, word-by-word and underlining words or phrases that describe inputs helps to identify the causes. – An input (cause) is an event that is generated outside an application that the application must react to in some fashion. – Examples of inputs include hardware events (e.g. keystrokes, pushed buttons, mouse clicks, sensor activations), API calls, return codes, and so forth.

59. Step two: Identify the causes and effects. b) Identify the effects or system transformation and assign each one a unique number. – An effect can also be referred to as an output action, as a distinct output condition, as an equivalence class of output conditions or as an output such as a confirmation message or error message. – An output (effect) is an event that an application generates that is sent outside the application. – Examples of output include a message printed on the screen, a string sent to a database a command sent to the hardware, a request to the operating system and so forth. – System transformations such as file or database record updates are considered effects as well. – As with causes, examining the specification, or other similar artifact, word-by- word and underlining words or phrases that describe outputs or system transformations helps to identify the effects.

60. Example Requirements for Calculating Car Insurance Premiums: – R00101 For females less than 65 years of age, the premium is $500 – R00102 For males less than 25 years of age, the premium is $3000 – R00103 For males between 25 and 64 years of age, the premium is $1000 – R00104 For anyone 65 years of age or more, the premium is $1500

61. When examining these requirements, we see that there are two variables that will determine what the premium will be: gender and age. Therefore there are 5 distinct causes or input conditions: – Gender is either male or female, and age is either less than 25, between 25 and 64 inclusive, or 65 or more. There are also 4 distinct effects or ouput conditions: – the premium is either $500, $1000, $1500, or $3000. Each cause and each effect is assigned an arbitrary unique number as part of this process step.

62. Causes and Effects Causes (input conditions) Effects (output conditions) 1. Gender is Male 2. Gender is Female 3. Age is <25 4. Age is >=25 and < 65 5. Age is >= 65 100. Premium is $1000 101. Premium is $3000 102. Premium is $1500 103. Premium is $500

63. Step 3: Creating Cause- Effect Graph The semantic content of the specification is analyzed and transformed into a Boolean graph linking the causes and effects. This is the cause-effect graph. – Semantics, in this step’s instructions, reflect the meaning of the programs or functions. – This meaning is discerned from the specification and transformed into a boolean graph that maps the causes to the resulting effects. – It is easier to derive the boolean function for each effect from their separate CEGs, then these can be overlaid to produce a single decision table for all effects

64. CEGInterpretation Causes: 1. Gender is Male and ( ^ ) 4. Age is >=25 and < 65 Effect: 100: Premium is $1000 Causes: 1. Gender is Male and (^ ) 3. Age is <25 Effect: 101: Premium is $3000 Causes: 1. Gender is Male and ( ^) 5. Age is >= 65 OR 2. Gender is Female and ( ^ ) 5. Age is >= 65 Effect: 102: Premium is $1500 Causes: 2. Gender is Female and ( ) 3. Age is <25 OR 2. Gender is Female and ( ) 4. Age is >=25 and < 65 Effect: 103: Premium is $500

65. Step 4: Add constraints Annotate the graph with constraints describing combinations of causes and/or effects that are impossible because of syntactic or environmental constraints or considerations. – In most software programs, certain combinations of causes are impossible because of syntactic or environmental considerations. – For example, for the purpose of calculating insurance premium in the above example, a person cannot be both a “Male” and a “Female” simultaneously. – To show this, the CEG is annotated, as appropriate, with the constraint symbols

66. Example: Adding Constraint

67. Step 5: Creating Decision Table Methodically trace state conditions in the graphs, converting them into a limited-entry decision table. Each column in the table represents a test case. – The ones (1) in the limited entry decision table column indicate that the cause (or effect) is true in the CEG and zeros (0) indicate that it is false. – Table 4 below illustrates the limited-entry decision table created by converting the CEG from the Calculating Car Insurance Premiums example. For example, the CEG #1, from Table 2 in step 3, converts into test case column 1 in the table below. From CEG #1, causes 1 and 3 being true result in effect 101 being true.

68. Decision Table

69. Test Cases

70. Reference [1]Foundations of Software Testing by Aditya Mathur [2]Software Testing by Ron Patton [3]Cause-Effect Graphing By Theresa Hunt