Testing and Test Case Development A “primitive” method of testing, with NO test preparation, may include the following steps : – Initiate the system – Provide the requested input(s) {unplanned value} – Observe the outputs/results – Record and analyze the results Note that such testing (ad hoc or random) is very difficult to analyze: – Is the test coverage complete? – How much duplication exist?

Checklist Testing A more orderly testing than ad hoc testing is to question the “coverage” in some systematic way. A Check-list of coverage is developed: 1.The test cases designed & developed are based on the checklist items which may cover different items such as: Major Functionalities (black-box) Modules in the system (white-box) All Error Messages or All UI Screens (white-box) 2.The test execution may be terminated & analyzed via: the coverage - the % of the checklist test cases executed the coverage - the % of checklist items executed successfully

Checklist Example An application “functional” checklist (perhaps derived from Requirements doc.) of an on-line- order software may include the following: – Entering the order items – Validating the order items – Checking availability/price of order items – Summarizing the order items and the total price into an invoice – Display and validate the invoice You may “check-off” the items as you tested each of the above functionalities.

More on Checklist A checklist may be further expanded to form a “tree” of checklists to provide some depth. Several checklists may be combined to form a “matrix” of checklists. Some logical intersection of checklists may be developed into a “new” checklist. Some drawbacks of checklist: – Too high level to cover some of the details One high level item may require extensive analysis (complex scenario and thus require several related test cases to test a scenario.) – There may exist “overlaps” at high level – Complex interactions may also exist besides overlapping – Too difficult to combine multiple views such as black-box and white- box testing even with combined checklists. Can we improve on checklist ? ----

Partition We care about partition because we want to divide our test cases up so that they (a) uniquely (with non or minimal duplication) cover (b) all the situations of interest. The formal mathematical definition of a partition is: – Consider a set S – The set S is partitioned into subsets C1, C2, ----, Cn if: For any x, y, x≠y then Cx ∩ Cy = Ø (there is no overlap between Ci’s) ∑ Ci = S (union of all Ci’s is S)

Partition of set S of marbles “by size” C1 C4 C2 C3 C5 Note that: 1) For any x, y, x≠y then Cx ∩ Cy = Ø (there is no overlap between Ci’s) 2) ∑ Ci = S (union of all Ci’s is S)

Some More “Math” on Partition Each partition, Cx, of set S is considered an equivalence class. (this is because every member inside Cx is the “same” with respect to the property, size.) A relation “= size” used to partition the set S. – Another example: the set S may be a set of dogs, and we partitioned S by the color of the dogs {black, white, brown, buff, gray, multi-color}. Then every dog in the partition “black” is the “same” with respect to that color. The relation for members in an equivalence class satisfy the following rules. Let R represent the “equivalence” relation then: – R(a,b) => R(b,a) [symmetric] {e.g.} Samecolor(a,b) => Samecolor(b,a) – R(a,b) Λ R(b,c) => R(a,c) [transitive] {e.g.} { Samecolor(a.b) Λ a Samecolor (b,c) } => Samecolor(a,c) – R(a,a) [reflexive] {e.g.} Samecolor (a,a)

Example of Testing Program Q Consider a program Q that i) reads inputs a, b, c and ii) determines the roots, r1 and r2, for the quadratic equation ax 2 +bx +c = 0. From high school math, we remember the 2 roots for this equation is given by: r1 = [ -b + √(b 2 -4ac) ]/2a and r2 = [ -b - √(b 2 -4ac) ]/2a To test this program, we may consider all kinds of approaches How should we decide on the test cases for our testing of this program?

1st Example of Test cases for the Program Q Consider the inputs to the program Q: a, b, c – We could divide the three inputs into valid (numerical) and invalid (non- numerical) to see if program Q handles input validation properly – Then we would have 2 cases for each input variable or 2 x 2 x 2 = 8 test cases to cover all the possible “valid-invalid” input combinations: (a = valid, b= valid, c= valid) (a = valid, b= valid, c= invalid) (a = valid, b= invalid, c= valid) (a= valid, b= invalid, c= invalid) (a= invalid, b= valid, c= valid) (a= invalid, b= valid, c= invalid) (a= invalid, b= invalid, c= valid) (a= invalid, b= invalid, c= invalid) – Note: that this is a checklist and is also a partitioning of the input based on our coming up with the relation of “valid and invalid’ inputs

Partitioning Input, (a,b,c), into 8 Equivalence Categories (a = valid, b= valid, c= valid) (a = valid, b= valid, c= invalid) (a = valid, b= invalid, c= valid) (a = valid, b= invalid, c= invalid) (a = invalid, b= valid, c= invalid) (a = invalid, b= valid, c= valid) (a = invalid, b= invalid, c= valid) (a = invalid, b= invalid, c= invalid) -Note that within each equivalence category there may be infinite choices! -Note that Choosing 1 from each category covers a) all 8 categories and b) each category is unique ---- partition

2nd Example of Test cases for the Program Q Again, consider the inputs to the program Q: a, b, c – We consider the situation of =0, >0 and <0 values for each input to test the implementation of algorithm for root – Then we will have 3 x 3 x 3 = 27 test cases to cover all the possible combinations: (a=0, b=0, c =0) (a>0, b=0, c =0) (a<0, b=0, c =0) (a=0, b=0, c >0) (a>0, b=0, c >0) (a 0) (a=0, b=0, c 0, b=0, c <0) (a<0, b=0, c <0) (a=0, b>0, c =0) (a>0, b>0, c =0) (a 0, c =0) (a=0, b>0, c >0) (a>0, b>0, c >0) (a 0, c >0) (a=0, b>0, c 0, b>0, c 0, c <0) (a=0, b 0, b<0, c =0) (a<0, b<0, c =0) (a=0, b 0) (a>0, b 0) (a 0) (a=0, b 0, b<0, c <0) (a<0, b<0, c <0) – Note that this is also a checklist and is also a partitioning of the input based on our coming up with the relation of =0, >0, and <0 values for the 3 inputs

3rd Example of Test cases for the Program Q This time consider the outputs to the program Q: r1, r2 We test the two major categories of possible outputs: a) r1 = r2 and b) r1 ≠ r2 If (b 2 -4ac) = 0, then r1 = r2 because r1 = (-b + 0)/2a and r2 = (-b – 0)/2a If (b 2 -4ac) >0, then r1 ≠ r2 We have partitioned our “outputs” into 2 partitions : r1=r2 and r1 ≠ r2 ; this in turn partitioned our inputs of a,b,c into: – set of (a,b,c)’s such that b 2 = 4ac – set of (a,b,c)’s such that b 2 > 4ac 1)Note that this is a checklist and also a partition of 2 sets of input (a,b,c)’s based on covering the 2 partitions of outputs where r1 = r2 and r1 ≠ r2. 2)Note that your text includes the case of output being imaginary number.

Partitions Test Case partitioning technique may be applied over different types of properties of interest: – Inputs – Outputs – Functionalities – Usages – Logical paths – etc. We can combine the different partitioning (Examples 1, 2, and 3) to cover different aspect of Program Q: input validation, algorithm correctness, output coverage)

Usage Based Testing Usage Based testing may be viewed as a type of partitioning. – We formulate a set of user-operational profiles (partitioning the users by some characteristics: e.g. user role & access of different functionalities by role) We may attach probabilities to the user profile (e.g. different probabilities of access to different functionalities based on the role.) – We may formulate a resource usage profile (partitioning the resources by some characteristics: e.g. number of times accessed by resources) We may attach frequency of usage count to each resource (e.g. develop a statistical usage frequency distribution of the resources) The idea is to test more on the more heavily “used” categories because more heavily used category is more “important”

User-Operational Profile Example (naïve user versus experienced user) Start (80 users) naïve users (20/80) experienced users (60/80) default f1 Choice of f1,f2,f3 f4 f2 f1 f3 f4 f6 f7 f4 f5 term (10/60) (40/60) (10/60) (20/20) (5/10) (40/40) (5/10) (3/10) (7/10) prob =.25 √ prob =.063 prob =.50 √√ prob =.037 prob =.087 (5/5) (40/40) (3/3) (7/7) Note that 2 usage cases take up 75% of operational profile !

Some Interesting Observations/Usage of the User Operational Profile From a testing priority and resource allocation perspective, we may conduct our testing in a sequence equal to the highest probability scenario (choice of f1,f2,f3 - f2 – f7 ) to the lowest probability scenario (choice of f1,f2,f3 – f3 – f4). If we want to claim some form of “coverage,” we may use the user-profile and test all the scenarios thoroughly except (choice of f1,f2,f3 – f3 – f4) and claim 96% coverage reliability.

Resource-Usage Profile Example (DB-table Usage Frequency distribution) employeepayrollbenefitstaxdependenttax 3ktimes 6k times 9k times Usage/Month DB - Tables employee table is “used” the most; thus more test cases should be created to for accessing that table