Testing Testing Techniques to Design Tests

Testing:Example Problem: Find a mode and its frequency given an ordered list (array) of with one or more integer elements. void getModeFreq (ArrayType a, int size, int & mode, int & freq); n Test case categories  Empty array ?  Normal cases?  More than one mode  All elements unique

Test to specification n Tests can have more than one answer and still be consistent with specification.  Find THE mode is a constraint on the values in the array.  Find A mode  several modes can exist. n Specification and test design developed concurrently

Black Box Testing n Happy Path n Interface Testing n Equivalence Partitioning n Boundary value analysis n Functional testing n Cause-effect graphing testing n Comparison testing n Error testing n Random inputs

Black Box Testing n Happy Path  Test cases and input values chosen from those known or expected to work. n Interface Testing  Unit interfaces and I/O interfaces (Automated test tools) n Equivalence Partitioning  Input class: generate 1 test point for each input class  Output class – generate 1 test point that gives result in each output class  Subdivide into subclasses (middle of range and 2 extremes)  Error Handling Routines/Exception output conditions (input beyond accepted range)

Boundary Value Analysis Derive test cases from boundaries of equivalence partitions. 1. Add equivalence partitions for output to input classes. 2. For each input and output class: a.Write valid test cases for representative value and high and low values. b.Write invalid test cases for values just beyond boundary.

Black Box Testing n Functional testing  Functions within certain math classes can be distinguished by their values on a small number of points F(x) = y, if x > 0; y-1, if x <= 0  Functions of more than one variable? z=x+y+1  Limitation: most programming problems not simple math problems n Random inputs n Cause-effect graphing testing n Comparison testing n Error testing

Error Guessing Test designer uses skill and experience to devise test cases to uncover errors. n null input. n long input. n random input. n almost correct input.  spaces in strings.  quoted strings.  all CAPS. n negative numbers.

White Box Testing n Based on program structure n Coverage metrics for thoroughness  Statement coverage : all statements are executed at least once. This is harder than it sounds: defensive programming makes some code hard to get to. if (f = fopen(path)) { perror("fopen"); }  Insufficient for “if P then S” (only need true condition to test statement)

White Box Testing  Branch coverage all branches are taken all conditions on branches are evaluated. Includes statement coverage Only this will catch bug above. Consider If (x < 0) x--; else x++; if (y < 0) z = sqrt (-x); else z = x+y; Can branch cover with then- then and else-else without detecting the problem of calling sqrt with a negative value

White Box Testing  Path coverage (every combination of T/Fs) Impractical – too many paths Eliminate infeasible paths (no systematic way) If (y < 0) x = y –1; else x = y+1; if (x > 0) … then-then infeasible Missing paths (“special cases”) – can involve 1 single input data point (if y = 0)

More…Path coverage  Coincidental correctness read x; y = x + 2; vs. y = x * 2; write y;  Can’t distinguish if input is 2  Implies one point per path is insufficient

Paths Loop <= possible paths At 1 task/msec = 3170 years

More White Box Testing n Data Flow Coverage  All definitions and all uses of variables n Mutation Analysis  Create mutant programs  See if mutant programs give identical output for each test; If yes, mutant is live If no, mutant is killed  Display set of live mutants, if all killed, confidence in Test  Examples: Replace 1 constant or variable with another Replace one arithmetic operator with another Similarly, relational and logical operators Delete a statement Increment change