WARNING These slides are not optimized for printing or exam preparation. These are for lecture delivery only. These slides are made for PowerPoint 2010. They may not show up well on other PowerPoint versions. You can download PowerPoint 2010 viewer from here. These slides contain a lot of animations. For optimal results, watch in slideshow mode.

SUT Test case

SUT Test case Test case

SUT Test case Test case To do this, we have to design these test cases

When do we define test cases? Are they pre-determined or on-the-fly? SUT Test case Test case

SUT Pre-determined or on-the-fly? Test case Test case How much internal info about the SUT should be revealed to the test case designer?

SUT Pre-determined or on-the-fly? Test case How to choose which test cases to use and which to discard? Test case How much info revealed?

Heuristics for Better Test Case Design Pre-determined or on-the-fly? How to choose? Heuristics for Better Test Case Design How much info revealed?

Pre-determined or on-the-fly? SUT Test case

Pre-determined or on-the-fly? SUT Test case

SUT scripted testing  which one? exploratory testing  which one? Pre-determined or on-the-fly? SUT Test case scripted testing  which one? exploratory testing  which one?

SUT scripted testing  pre-determined exploratory testing  on-the-fly Pre-determined or on-the-fly? SUT Test case scripted testing  pre-determined exploratory testing  on-the-fly Most of the jokes done by comedians are scripted. They sometimes throw in a few on-the-spot jokes, but even they are not totally random. how to choose?

Scripted+Exploratory SUT Test case

Scripted+Exploratory SUT Test case Test case How much info revealed?

Scripted+Exploratory SUT Test case Test case How much info revealed?

SUT Developer Tester Test case How much info revealed?

SUT Developer Tester Test case How much info revealed?

Glass-box / a.k.a white box Gray-box Test case Black-box How much info revealed?

SUT How to choose? Scripted+Exploratory Black/Glass/Gray-box Test case

How to choose? Why choose? Some heuristics Test case

Do not panic! This is a simulation  [extra] BSoD Do not panic! This is a simulation  [CS2103]

Windows without bugs? That’s crazy talk! [extra] BSoD Windows without bugs? That’s crazy talk! [CS2103]

How to choose? Why choose? Test case

How to choose? Why choose? [Effectiveness & Efficiency]

Both tested the same system How to choose? Why choose? Dilbert Wally Both tested the same system [Effectiveness & Efficiency] Found 20 bugs Found 10 bugs

Both tested the same system How to choose? Why choose? Dilbert Wally Both tested the same system [Effectiveness & Efficiency] Found 20 bugs Found 20 bugs Used 100 test cases Used 40 test cases

How to choose? Why choose? [Effectiveness & Efficiency] A test case should have a reasonable chance of finding a new bug not found by the other test cases.

How to choose? E&E of testing Some heuristics Test case

Equivalence partitioning How to choose? E&E of testing Some heuristics Test case Equivalence partitioning

Equivalence partitioning How to choose? E&E of testing Some heuristics Test case Equivalence partitioning

Equivalence partitioning

77577 OR tinyurl.com/answerpost Equivalence partitioning isValidMonth (int m): boolean Description: checks if m is in the range [1..12]. returns true if m is in the range, false otherwise. Which values would you use to test this isValidMonth method? cases {value1 value2 …} e.g. cases 1 4 9 77577 OR tinyurl.com/answerpost

Pass/fail? Hmmm… interesting. Let me try 275 Should he? Actual: true Equivalence partitioning isValidMonth (int m): boolean Description: checks if m is in the range [1..12]. returns true if m is in the range, false otherwise. Pass/fail? Hmmm… interesting. Let me try 275 Test case 23: Input: 273 Expected: false Test Case 12 Test Case 13 Test case 23: Input: 274 Expected: false Should he? Actual: true Actual: true

275 Must be broken too! Broken! Broken! isValidMonth (int m): boolean Equivalence partitioning isValidMonth (int m): boolean 275 273 274 Must be broken too! Broken! Broken!

Equivalence partitioning isValidMonth (int m): boolean 275 273 274

275 isValidMonth (int m): boolean 273 274 Equivalence partitioning MIN_INT 1 12 13 MAX_INT 275 273 274

isValidMonth (int m): boolean Equivalence partitioning isValidMonth (int m): boolean MIN_INT 1 12 13 MAX_INT

Equivalence partitioning How to partition?

Equivalence partitioning Input space Output space SUT

SUT Equivalence partitioning Output space Input space Input partition 1 Input partition 2 Input partition 3 Output 1 Output 2 Output 3 Input space Output space Input partition 1a SUT false true

SUT Equivalence partitioning Input partition 1 Output 1 Input partition 1a Input partition 1b SUT false true

Equivalence partitioning

Why does i need more than one eq. partition? Equivalence partitioning Why does i need more than one eq. partition? int square (int i) Description: returns the the square value of i

Description: returns the the square value of i Equivalence partitioning int square (int i) Description: returns the the square value of i +√MAX -√MAX -1 MIN MAX

Description: returns the the square value of i Equivalence partitioning int square (int i) Description: returns the the square value of i +√MAX -√MAX -1 MIN MAX

Equivalence partitioning is … It helps to improve E&E by … How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Equivalence partitioning is … It helps to improve E&E by …

Equivalence partitioning Boundary value analysis How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis

Equivalence partitioning Boundary value analysis How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis

isValidMonth (int m): boolean Boundary value analysis isValidMonth (int m): boolean MIN_INT 1 12 13 MAX_INT

isValidMonth (int m): boolean Boundary value analysis isValidMonth (int m): boolean MIN_INT 1 12 13 MAX_INT if(m>=1)||(m<12) …

isValidMonth (int m): boolean Boundary value analysis isValidMonth (int m): boolean MIN_INT 1 12 13 MAX_INT

What are the boundary values for i? Boundary value analysis What are the boundary values for i? isPrimeNumber (int i):boolean Description: Checks if i is a prime number. Returns true if i is a prime number, false otherwise. Not prime numbers Prime numbers

What are the boundary values for i? Boundary value analysis What are the boundary values for i? isPrimeNumber (int i):boolean Description: Checks if i is a prime number. Returns true if i is a prime number, false otherwise.

BVA helps to improve E&E by … How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis BVA helps to improve E&E by …

Equivalence partitioning Boundary value analysis How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis Combining multiple inputs

Equivalence partitioning Boundary value analysis How to choose? E&E of testing Some heuristics Test case Equivalence partitioning Boundary value analysis Combining multiple inputs

Combining multiple inputs :MSLogic markCellAt(x,y)

Combining multiple inputs markCellAt(x,y) markCellAt(x,y)

Combining multiple inputs markCellAt(x,y) MSLogic x y Cell GAME_STATE

[MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] Combining multiple inputs markCellAt(x,y) x y Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST [MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] [H..MAX_INT] W -> Width H -> Height

[MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] Combining multiple inputs markCellAt(x,y) x y Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST [MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] [H..MAX_INT] MIN_INT, -2, -1,0, 2, (W-1), W, W+1, MAX_INT

[MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] Combining multiple inputs markCellAt(x,y) x y Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST [MIN_INT..-1] [0..(W-1)] [W..MAX_INT] [MIN_INT..-1] [0..(H-1)] [H..MAX_INT] MIN_INT, -2, -1,0, 2, (W-1), W, W+1, MAX_INT MIN_INT, -2, -1,0, 2, (H-1), H, H+1, MAX_INT

HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY Combining multiple inputs markCellAt(x,y) x y Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST MIN_INT, -2, -1,0, 2, (W-1), W, W+1, MAX_INT MIN_INT, -2, -1,0, 2, (H-1), H, H+1, MAX_INT

HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY Combining multiple inputs markCellAt(x,y) Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST y MIN_INT, -2, -1,0, 2, (H-1), H, H+1, MAX_INT x MIN_INT, -2, -1,0, 2, (W-1), W, W+1, MAX_INT

HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY Combining multiple inputs markCellAt(x,y) Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST y MIN_INT, -2, -1,0, 2, (H-1), H, H+1, MAX_INT x MIN_INT, -2, -1,0, 2, (W-1), W, W+1, MAX_INT

2025 HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY Combining multiple inputs markCellAt(x,y) Cell GAME_STATE HIDDEN MARKED CLEARED INC_MARKED INC_CLEARED PRE_GAME READY IN_PLAY WON LOST y MIN_INT, -2, -1,0, 2, (H-1), H, H+1, MAX_INT x MIN_INT, -2, -1,0, 2, (W-1), W, W+1, MAX_INT 9x9x5x5= 2025

Combining multiple inputs

77577 OR tinyurl.com/answerpost Combining multiple inputs a b c valid1 valid3 valid4 valid2 invalid2 valid5 invalid1 invalid3 3x3x2= 18 Payment for testing: $100 Cost per test case: $10 Penalty for uncaught bug: $200 Work out a set of test cases that is both E&E. How many do you have? count {answer} e.g. count 9 77577 OR tinyurl.com/answerpost

18 a b c valid1 valid3 valid4 valid2 invalid2 valid5 invalid1 invalid3 Combining multiple inputs a b c valid1 valid3 valid4 valid2 invalid2 valid5 invalid1 invalid3 3x3x2= 18 Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 invalid2 valid5 error 1 3 invalid1 invalid3 -any - error 2

18 a b c valid1 valid3 valid4 valid2 invalid2 valid5 invalid1 invalid3 Combining multiple inputs a b c valid1 valid3 valid4 valid2 invalid2 valid5 invalid1 invalid3 3x3x2= 18 Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 invalid2 valid5 error 1 3 invalid1 invalid3 -any - error 2

18 a b c valid1 valid3 valid4 valid2 invalid2 valid5 invalid1 invalid3 Combining multiple inputs a b c valid1 valid3 valid4 valid2 invalid2 valid5 invalid1 invalid3 3x3x2= 18 Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 invalid2 valid5 error 1 3 invalid1 invalid3 -any - error 2

Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 invalid2 valid5 error 1 3 invalid1 invalid3 -any - error 2 Heuristic : Each valid input should appear at least once without invalid inputs

Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 invalid2 valid5 error 1 3 invalid1 invalid3 -any - error 2 Heuristic : Each valid input should appear at least once without invalid inputs

Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 invalid2 valid5 error 1 3 invalid1 invalid3 -any - error 2 Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 -any valid - valid5 behavior 2 2a invalid2 error 1 3 invalid1 invalid3 -any - error 2 Heuristic : Each valid input should appear at least once without invalid inputs

Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 invalid2 valid5 error 1 3 invalid1 invalid3 -any - error 2 Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 -any valid - valid5 behavior 2 2a invalid2 error 1 3 invalid1 invalid3 -any - error 2 Heuristic : Each valid input should appear at least once without invalid inputs

Heuristic: Only one invalid input per test case b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 invalid2 valid5 error 1 3 invalid1 invalid3 -any - error 2 Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 -any valid - valid5 behavior 2 2a invalid2 error 1 3 invalid1 invalid3 -any - error 2 Heuristic: Only one invalid input per test case

Heuristic: Only one invalid input per test case b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 invalid2 valid5 error 1 3 invalid1 invalid3 -any - error 2 Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 -any valid - valid5 behavior 2 2a invalid2 error 1 3 invalid1 invalid3 -any - error 2 Heuristic: Only one invalid input per test case

Heuristic: Only one invalid input per test case b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 -any valid - valid5 behavior 2 2a invalid2 error 1 3 invalid1 invalid3 -any - error 2 5 3x3x2= 18 Case a b c Output 1 valid1 valid3 valid4 behavior 1 2 valid2 -any valid - valid5 behavior 2 2a invalid2 error 1 3 invalid1 error 2 4 invalid3 error 3 Heuristic: Only one invalid input per test case

Combining multiple inputs Equivalence partitioning How to choose? E&E of testing Some heuristics Combining multiple inputs Test case Equivalence partitioning Boundary value analysis

Apply in *some* places. e.g. //5 is a boundary value for size assertEquals(5, getSize());

Test case design heuristics Other QA techniques Test case design heuristics