WARNING 40 min These slides are not optimized for printing or exam preparation. These are for lecture delivery only. These slides are made for PowerPoint 2010. They may not show up well on other PowerPoint versions. You can download PowerPoint 2010 viewer from here. These slides contain a lot of animations. For optimal results, watch in slideshow mode.

CS2103/T, Lecture 8, Part 3, [March, 2017] Pre-determined or on-the-fly? How to choose? Heuristics for Better Test Case Design CS2103/T, Lecture 8, Part 3, [March, 2017] How much info revealed?

SUT How to choose? Scripted+Exploratory Black/Glass/Gray-box Test case

How to choose? Why choose? Some heuristics Test case

[extra] BSoD This is a simulation  [CS2103]

How to choose? Why choose? Test case Cost in term of time + money

How to choose? Why choose? [Effectiveness & Efficiency]

Both tested the same system How to choose? Why choose? Dilbert Wally Both tested the same system [Effectiveness & Efficiency] Found 20 bugs Found 10 bugs

Both tested the same system How to choose? Why choose? Dilbert Wally Both tested the same system [Effectiveness & Efficiency] Found 20 bugs Found 20 bugs Used 100 test cases Used 40 test cases

How to choose? Why choose? [Effectiveness & Efficiency] A test case should have a reasonable chance of finding a new bug not found by the other test cases.

How to choose? E&E of testing Some heuristics Test case

How to choose? E&E of testing Some heuristics Test case

Equivalence partitioning How to choose? E&E of testing Some heuristics Test case Equivalence partitioning

Equivalence partitioning How to choose? E&E of testing Some heuristics Test case Equivalence partitioning

Equivalence partitioning

77577 OR tinyurl.com/answerpost Equivalence partitioning isValidMonth (int m): boolean Description: checks if m is in the range [1..12]. returns true if m is in the range, false otherwise. Which values would you use to test this isValidMonth method? cases {value1 value2 …} e.g. cases 1 4 9 77577 OR tinyurl.com/answerpost

Pass/fail? Hmmm… interesting. Let me try 275 Should he? Why not? Equivalence partitioning isValidMonth (int m): boolean Description: checks if m is in the range [1..12]. returns true if m is in the range, false otherwise. Pass/fail? Hmmm… interesting. Let me try 275 Test case 23: Input: 273 Expected: false Test Case 12 Test Case 13 Test case 23: Input: 274 Expected: false Should he? Why not? Actual: true Actual: true

275 Must be broken too! Broken! Broken! isValidMonth (int m): boolean Equivalence partitioning isValidMonth (int m): boolean 275 273 274 Must be broken too! Broken! Broken!

Equivalence partitioning isValidMonth (int m): boolean 275 273 274

275 isValidMonth (int m): boolean 273 274 Equivalence partitioning MIN_INT 1 12 13 MAX_INT 275 273 274

isValidMonth (int m): boolean Equivalence partitioning isValidMonth (int m): boolean MIN_INT 1 12 13 MAX_INT

Equivalence partitioning Input space Output space SUT

SUT Equivalence partitioning Output space Input space Input partition 1 Input partition 2 Input partition 3 Output 1 Output 2 Output 3 Input space Output space Input partition 1a SUT false true

SUT Equivalence partitioning Input partition 1 Output 1 Input partition 1a Input partition 1b SUT false true

Equivalence partitioning

How many equivalence partition? Equivalence partitioning How many equivalence partition? int square (int i) Description: returns the the square value of i

Description: returns the the square value of i Equivalence partitioning int square (int i) Description: returns the the square value of i +√MAX -√MAX -1 MIN MAX

Description: returns the the square value of i Equivalence partitioning int square (int i) Description: returns the the square value of i +√MAX -√MAX -1 MIN MAX

It helps to improve E&E by … How to choose? E&E of testing Some heuristics Test case Equivalence partitioning An Eq. partition is a … It helps to improve E&E by …

Equivalence partitioning How to choose? E&E of testing Some heuristics Test case Equivalence partitioning

Equivalence partitioning Boundary value analysis How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis

Equivalence partitioning Boundary value analysis How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis

Boundary value analysis

isValidMonth (int m): boolean Boundary value analysis isValidMonth (int m): boolean MIN_INT 1 12 13 MAX_INT

isValidMonth (int m): boolean Boundary value analysis isValidMonth (int m): boolean MIN_INT 1 12 13 MAX_INT

isValidMonth (int m): boolean Boundary value analysis isValidMonth (int m): boolean MIN_INT 1 12 13 MAX_INT if (m>=1) && (m<12) … if (m>=1) && (m<=12) …

isValidMonth (int m): boolean Boundary value analysis isValidMonth (int m): boolean MIN_INT 1 12 13 MAX_INT

What are the boundary values for i? Boundary value analysis What are the boundary values for i? isPrimeNumber (int i):boolean Description: Checks if i is a prime number. Returns true if i is a prime number, false otherwise. Not prime numbers Prime numbers

What are the boundary values for i? Boundary value analysis What are the boundary values for i? isPrimeNumber (int i):boolean Description: Checks if i is a prime number. Returns true if i is a prime number, false otherwise.

BVA helps to improve E&E by … How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis BVA helps to improve E&E by …

Equivalence partitioning Boundary value analysis How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis Combining multiple inputs

Equivalence partitioning Boundary value analysis How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis Combining multiple inputs

Combining multiple inputs :MSLogic markCellAt(x,y)

Combining multiple inputs markCellAt(x,y) markCellAt(x,y)

Combining multiple inputs markCellAt(x,y) MSLogic x y Cell GAME_STATE

[MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] Combining multiple inputs markCellAt(x,y) x y Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST [MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] [H..MAX_INT] W -> Width H -> Height

[MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] Combining multiple inputs markCellAt(x,y) x y Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST [MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] [H..MAX_INT] MIN_INT, -2, -1, 0, 2, (W-1), W, W+1, MAX_INT

[MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] Combining multiple inputs markCellAt(x,y) x y Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST [MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] [H..MAX_INT] MIN_INT, -2, -1, 0, 2, (W-1), W, W+1, MAX_INT MIN_INT, -2, -1, 0, 2, (H-1), H, H+1, MAX_INT

HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY Combining multiple inputs markCellAt(x,y) x y Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST MIN_INT, -2, -1, 0, 2, (W-1), W, W+1, MAX_INT MIN_INT, -2, -1, 0, 2, (H-1), H, H+1, MAX_INT

HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY Combining multiple inputs markCellAt(x,y) Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST y MIN_INT, -2, -1, 0, 2, (H-1), H, H+1, MAX_INT x MIN_INT, -2, -1, 0, 2, (W-1), W, W+1, MAX_INT

HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY Combining multiple inputs markCellAt(x,y) Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST y MIN_INT, -2, -1, 0, 2, (H-1), H, H+1, MAX_INT x MIN_INT, -2, -1, 0, 2, (W-1), W, W+1, MAX_INT

2025 HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY Combining multiple inputs markCellAt(x,y) Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST y MIN_INT, -2, -1, 0, 2, (H-1), H, H+1, MAX_INT x MIN_INT, -2, -1, 0, 2, (W-1), W, W+1, MAX_INT 9x9x5x5= 2025

Combining multiple inputs

77577 OR tinyurl.com/answerpost Combining multiple inputs gender size speed female  large  fast  male  medium  slow  blank  small  3x3x2= 18 Payment for testing: $100 Cost per test case: $10 Penalty for uncaught bug: $200 Work out a set of test cases that is both E&E. How many do you have? count {answer} e.g. count 9 77577 OR tinyurl.com/answerpost

18 gender size speed female large fast male medium slow blank small Combining multiple inputs gender size speed female large fast male medium slow blank small 3x3x2= 18 Case gender size speed Output 1 female large fast behavior 1 2 male medium slow error 1 3 blank small -any - error 2

18 gender size speed female large fast male medium slow blank small Combining multiple inputs gender size speed female large fast male medium slow blank small 3x3x2= 18 Case gender size speed Output 1 female large fast behavior 1 2 male medium slow error 1 3 blank small -any - error 2

18 gender size speed female large fast male medium slow blank small Combining multiple inputs gender size speed female large fast male medium slow blank small 3x3x2= 18 Case gender size speed Output 1 female large fast behavior 1 2 male medium slow error 1 3 blank small -any - error 2

Case gender size speed Output 1 female large fast behavior 1 2 male medium slow error 1 3 blank small -any - error 2 Heuristic : Each valid input should appear at least once without invalid inputs

Case gender size speed Output 1 female large fast behavior 1 2 male medium slow error 1 3 blank small -any - error 2 Heuristic : Each valid input should appear at least once without invalid inputs

Case gender size speed Output 1 female large fast behavior 1 2 male medium slow error 1 3 blank small -any - error 2 Case gender size speed Output 1 female large fast behavior 1 2 male -any valid - slow behavior 2 2a medium error 1 3 blank small -any - error 2 Heuristic : Each valid input should appear at least once without invalid inputs

Case gender size speed Output 1 female large fast behavior 1 2 male medium slow error 1 3 blank small -any - error 2 Case gender size speed Output 1 female large fast behavior 1 2 male -any valid - slow behavior 2 2a medium error 1 3 blank small -any - error 2 Heuristic : Each valid input should appear at least once without invalid inputs

Heuristic: Only one invalid input per test case gender size speed Output 1 female large fast behavior 1 2 male medium slow error 1 3 blank small -any - error 2 Case gender size speed Output 1 female large fast behavior 1 2 male -any valid - slow behavior 2 2a medium error 1 3 blank small -any - error 2 Heuristic: Only one invalid input per test case

Heuristic: Only one invalid input per test case gender size speed Output 1 female large fast behavior 1 2 male medium slow error 1 3 blank small -any - error 2 Case gender size speed Output 1 female large fast behavior 1 2 male -any valid - slow behavior 2 2a medium error 1 3 blank small -any - error 2 Heuristic: Only one invalid input per test case

Heuristic: Only one invalid input per test case gender size speed Output 1 female large fast behavior 1 2 male -any valid - slow behavior 2 2a medium error 1 3 blank small error 2 5 3x3x2= 18 Case gender size speed Output 1 female large fast behavior 1 2 male -any valid - slow behavior 2 2a medium error 1 3 blank error 2 4 small error 3 Heuristic: Only one invalid input per test case

Combining multiple inputs Equivalence partitioning How to choose? E&E of testing Some heuristics Combining multiple inputs Test case Equivalence partitioning Boundary value analysis

Apply in *some* places. e.g. //5 is a boundary value for size assertEquals(5, getSize());

Test case design heuristics Other QA techniques Test case design heuristics