Testing in the Small (aka Unit Testing, Class Testing) 1209

Unit Testing Focus on the smallest units of code: Functions Methods Subroutines Frequently done (at least partially) during code development by code developers Often the target of testing frameworks such as JUnit

Each component is Tested in isolation from the rest of the system Tested in a controlled environment Uses appropriately chosen input data Uses component-level design description as guide

During Unit Tests Data transformations across the module are tested Data structures are tested to ensure data integrity

Unit Test Procedures Driver Module to be tested Results Test cases Stub Stub

Two fundamental approaches: Black box Based on specification Inner structure of test object is not considered White box Inner structure of test object is the basis of test case selection

Usefulness Effectiveness of black box is similar to white box, but the mistakes found are different. (Hetzel 1976, Myers 1978) Examples of errors not detected by white box testing: Lock up PC if keyboard used during disk IO (neglected to turn off interrupts in pre ’84 versions of DOS) Monitor destroyed if both color and monochrome monitors used to play PC games

Black Box Unit Testing 1209

What is black-box testing? Unit (code, module) seen as a black box No access to the internal or logical structure Determine if given input produces expected output. Input Output

Black Box Testing Test set is derived from requirements Goal is to cover the input space Lots of approaches to describing input space: Equivalence Classes Boundary Value Analysis Decision Tables State Transitions Use Cases . . .

Advantage and Disadvantage Advantage: it does not require access to the internal logic of a component Disadvantage: in most real-world applications, impossible to test all possible inputs Need to define an efficient strategy to limit number of test cases

General Black Box Process Analyze requirements Select valid and invalid inputs Determine expected outputs Construct tests Run tests Compare actual outputs to expected outputs

Equivalence Relations: Reflexive Symmetric Transitive

Equivalence Relations: Reflexive Symmetric Transitive What are these? G

Equivalence Relations: Reflexive x R(x,x) Symmetric x,y R(x,y)  R(y,x) Transitive x,y,z R(x,y)  R(y,z)  R(x,z)

Equivalence classes: Basic Strategy Partition the input into equivalence classes This is the tricky part It’s an equivalence class if: Every test using one element of the class tests the same thing that any other element of the class tests If an error is found with one element, it should be found with any element If an error is not found with some element, it is not found by any element Test a subset from each class

Basic strategy: Example: fact(n): if n<0, or n>=200 error if 0<=n<=20, exact value if 20<n<200, approximate value within .1% What classes can you see? G

Basic strategy: Example: fact(n): if n<0, or n>=200 error if 0<=n<=20, exact value if 20<n<200, approximate value within .1% Obvious classes are n<0, 0<=n<=20, 20<n<200, and 200<=n. Might be some subclasses here, also, but this is a good start.

Simple Example: Suppose you are building an airline reservation system. A traveler can be a child, an adult, or a senior. The price depends on the type of traveler. The seat reservation does not depend on the type of traveler. How many test cases can you identify for the reservation component and the billing component? G

Finding equivalence classes: Identify restrictions for inputs and outputs in the specification

Finding equivalence classes: Identify restrictions for inputs and outputs in the specification If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN)

Finding equivalence classes: Identify restrictions for inputs and outputs in the specification If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN) If a number of values is required, create one valid and two invalid classes (one invalid, two invalid)

Finding equivalence classes: Identify restrictions for inputs and outputs in the specification If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN) If a number of values is required, create one valid and two invalid classes If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set

Finding equivalence classes: Identify restrictions for inputs and outputs in the specification If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN) If a number of values is required, create one valid and two invalid classes If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set If there is a condition, create two classes, one satisfying and one not satisfying the condition

Boundary Values: Programs that fail at interior elements of a class usually fail at the boundaries too. Test the boundaries. if it should work for 1-99, test 0, 1, 99, 100. if it works for A-Z, try @, A, Z, [, a, and z The hard part is identifying boundaries

Hints If a domain is a restricted set, check the boundaries. e.g., D=[1,10], test 0, 1, 10, 11 It may be possible to test the boundaries of outputs, also. For ordered sets, check the first and last elements For complex data structures, the empty list, full lists, the zero array, and the null pointer should be tested Extremely large data sets should be tested Check for off-by-one errors

More Hints Some boundaries are not obvious and may depend on the implementation (use gray box testing if needed) Numeric limits (e.g., test 255 and 256 for 8-bit values) Implementation limits (e.g., max array size)

Boundary Value Exercise Determine the boundary values for US Postal Service ZIP codes Determine the boundary values for a 15- character last name entry. G

Decision Tables Construct a table (to help organize the testing) Identify each rule or condition in the system that depends on some input For each input to one of these rules, list the combinations of inputs and the expected results

Decision Table Example Theater ticket prices are discounted for senior citizens and students. Test Case C1 Student C2 Senior Result Discount? 1 T 2 F 3 4

Pairwise Testing

Problem 1 From Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech House Publishers, 2004. A web site must operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; and FireFox 2, 3, and 4. It must work using RealPlayer, MediaPlayer, or no plugin. It needs to run under Windows ME, NT, 2000, XP, and Vista, and 7.0 It needs to accept pages from IIS, Apache and WebLogic running on Windows NT, 2000, and Linux servers How many different configurations are there? G

Problem 1 A web site must operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; and FireFox 2, 3, and 4. It must work using RealPlayer, MediaPlayer, or no plugin. It needs to run under Windows ME, NT, 2000, XP, and Vista, and 7.0 It needs to accept pages from IIS, Apache and WebLogic running on Windows NT, 2000, and Linux servers How many different configurations are there? 8 browsers x 3 plugins x 6 OS x 3 web servers x 3 server OSs = 1296 combinations

Problem 2 A bank is ready to test a data processing system From Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech House Publishers, 2004. A bank is ready to test a data processing system Customer types Gold (i.e., normal) Platinum (i.e., important) Business Non profits Account types Checking Savings Mortgages Consumer loans Commercial loans States (with different rules): CA, NV, UT, ID, AZ, NM How many different configurations are there? G

When given a large number of combinations: (options improve …) Give up and don’t test

When given a large number of combinations: (options improve …) Give up and don’t test Test all combinations … miss targets, delay product launch, and go out of business

When given a large number of combinations: (options improve …) Give up and don’t test Test all combinations … miss targets, delay product launch, and go out of business Choose one or two cases

When given a large number of combinations: (options improve …) Give up and don’t test Test all combinations … miss targets, delay product launch, and go out of business Choose one or two cases Choose a few that the programmers already ran

When given a large number of combinations: (options improve …) Give up and don’t test Test all combinations … miss targets, delay product launch, and go out of business Choose one or two cases Choose a few that the programmers already ran Choose the tests that are easy to create

When given a large number of combinations: (options improve …) Give up and don’t test Test all combinations … miss targets, delay product launch, and go out of business Choose one or two cases Choose a few that the programmers already ran Choose the tests that are easy to create List all combinations and choose first few

When given a large number of combinations: (options improve …) Give up and don’t test Test all combinations … miss targets, delay product launch, and go out of business Choose one or two cases Choose a few that the programmers already ran Choose the tests that are easy to create List all combinations and choose first few List all combinations and randomly choose some

When given a large number of combinations: (options improve …) Give up and don’t test Test all combinations … miss targets, delay product launch, and go out of business Choose one or two cases Choose a few that the programmers already ran Choose the tests that are easy to create List all combinations and choose first few List all combinations and randomly choose some “Magically” choose a small subset with high probability of revealing defects

Orthogonal Arrays A 2-D array with the property All pairwise combinations occur in every pair of columns Example: consider 3 variables (columns) with {a,b}, {1,2}, and (α,β) 1 2 3 a α b β 4

Orthogonal Array Example 1 2 3 a α b β 4 Look at each pair of columns (1 and 2),( 1 and 3), and (2 and 3) Does each of the 4 pairings appear in each? (Yes, of course!)

Pairwise Testing Algorithm Identify variables Determine choices for each variable Locate an orthogonal array Map test cases to the orthogonal array Construct tests

Example using Bank Identify variables Customers (4) Account types (5) States (6) Determine choices for each variable Customer types (Gold, Platinum, Business, Non Profit) Account types (Checking, Savings, Mortgages, Consumer loans, Commercial loans States (CA, NV, UT, ID, AZ, NM) Locate an orthogonal array Map test cases to the orthogonal array Construct tests

Locate an orthogonal array (look up on web) 1 2 3 4 5 1 2 3 4 6

https://controls. engin. umich. edu/wiki/index https://controls.engin.umich.edu/wiki/index.php/Design_of_experiments_via_taguchi_methods:_orthogonal_arrays

Map Test Cases (30) CA Check Gold Save Plat Mort Busi Cons NonP NV UT ID AZ Comm Gold Check Plat Save Busi Mort NonP NM CA NV UT ID Cons

Pairwise Summary When the combination is large, test all pairs, not all combinations Studies indicate that most defects are single or double-mode effects Can be found in pairing Few defects require more than two modes Orthogonal arrays have the pairwise property and can be found or generated

State Transition Testing

State Transition Testing Build STD of system or component Cover the STD Visit each state Trigger each event Exercises each transition Exercise each path* * Might not be possible

Groups: How many tests to visit each state? Example Payment made Ticket Printed Paid Res Made Ticketed Cancel Cancel/refund Cust Order/start timer Time Expires Cancelled Cust Cancelled Non Pay Ticket Delivered Cancel/refund Used Customer makes reservation and has limited time to pay. May cancel at any time. Groups: How many tests to visit each state?

Example: Each State Paid Res Made Ticketed Cancelled Cancelled Cust Payment made Ticket Printed Paid Res Made Ticketed Cancel Cancel/refund Cust Order/start timer Time Expires Cancelled Cust Cancelled Non Pay Ticket Delivered Cancel/refund Used 3 tests needed: From start to final From start to cancelled non-pay From start to Res Made to Cancelled Cust

Example: Each Event Paid Res Made Ticketed Cancelled Cancelled Cust Payment made Ticket Printed Paid Res Made Ticketed Cancel Cancel/refund Cust Order/start timer Time Expires Cancelled Cust Cancelled Non Pay Ticket Delivered Cancel/refund Used 3 tests needed: From start to final From start to cancelled non-pay From start to Res Made to Cancelled Cust

Example: Each Transition Payment made Ticket Printed Paid Res Made Ticketed Cancel Cancel/refund Cust Order/start timer Time Expires Cancelled Cust Cancelled Non Pay Ticket Delivered Cancel/refund Used 5 tests needed

Use Case Testing

Use Case Testing Use the use cases to specify test cases Use case specifies both normal, alternate, and exceptional operations Use cases may not have sufficient detail Beizer estimates that 30-40% of effort in testing transactions is generating the test data Budget for this!

White-Box Testing Pfleeger, S. Software Engineering Theory and Practice 2nd Edition. Prentice Hall, 2001. Ghezzi, C. et al., Fundamentals of Software Engineering. Prentice Hall, 2002. Pressman, R., Software Engineering A Practitioner’s Approach, Mc Graw Hill, 2005. Hutchinson, Marnie, Software Testing Fundamentals, Wiley, 2003. 0310

White Box Testing Also known as: Glass box testing Structural testing Test set is derived from structure of code Code structure represented as a Control Flow Graph Goal is to cover the CFG

Control Flow Graphs Programs are made of three kinds of statements: Sequence (i.e., series of statements) Condition (i.e., if statement) Iteration (i.e., while, for, repeat statements)

Control Flow Graphs Programs are made of three kinds of statements: Sequence (i.e., series of statements) Condition (i.e., if statement) Iteration (i.e., while, for, repeat statements) CFG: visual representation of flow of control. Node represents a sequence of statements with single entry and single exit Edge represents transfer of control from one node to another

Control Flow Graph (CFG) Join Join Sequence If-then-else If-then Iterative

Control Flow Graph (CFG) Join Join Sequence If-then-else If-then Iterative When drawing CFG, ensure that there is one exit: include the join node if needed

Example 1: CFG 1. read (result); 2. read (x,k) 3. while result < 0 then { 4. ptr  false 5. if x > k then 6. ptr  true 7. x  x + 1 8. result  result + 1 } 9. print result Draw CFG

Example 1: CFG 1 2 3 4 9 5 6 8 1. read (result); 2. read (x,k) 3. while result < 0 then { 4. ptr  false 5. if x > k then 6. ptr  true 7. x  x + 1 8. result  result + 1 } 9. print result 3 4 9 5 6 7 8

Example 1: CFG 1 1,2 2 3 3 9 4 9 4,5 5 6 6 7,8 8 1. read (result); 2. read (x,k) 3. while result < 0 then { 4. ptr  false 5. if x > k then 6. ptr  true 7. x  x + 1 8. result  result + 1 } 9. print result 5 6 6 Join 7 7,8 8

In Class: Write CFGs Example 2 if (a < b) then while(a < n) 3. a  a + 1; 4. else 5. while(b < n) 6. b  b + 1; Example 3 1. read (a,b); 2. if (a  0 && b  0) then { 3. c  a + b; if c > 10 then c  max else c  min } 7. else print ‘Done’ Example 4 1. read (a,b); 2. if (a  0 || b  0) then { 3. c  a + b 4. while( c < 100) 5. c  a + b; } 6. c  a * b

Write a CFG 1 2 3 5 6 Join Example 2 1. if (a < b) then 2. while (a < n) 3. a  a + 1; 4. else 5. while (b < n) 6. b  b + 1;

Answers 1,2 3,4 5 6 7 Join Example 3 1. read (a,b); 2. if (a  0 && b  0) then { 3. c  a + b; 4. if c > 10 then 5. c  max else c  min } 7. else print ‘Done’

Answers 1, 2 3 4 Join 5 6 Example 4 1. read (a,b); 2. if (a  0 || b  0) then { 3. c  a + b 4. while( c < 100) 5. c  a + b; } 6. c  a * b 1, 2 3 Join 4 5 6

Coverage Statement Branch Condition Path Def-use Others

Statement Coverage Every statement gets executed at least once Every node in the CFG gets visited at least once

Example (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 5

Examples: number of paths needed for Statement Coverage Examples: number of paths needed for Statement Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 5

Examples: number of paths needed for Statement Coverage Examples: number of paths needed for Statement Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 4 1 5 5

Branch coverage Every decision is made true and false Every edge in a CFG of the program gets traversed at least once Also known as Decision coverage All edge coverage Basis path coverage

Examples: number of paths needed for Branch Coverage Examples: number of paths needed for Branch Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 5

Examples: number of paths needed for Branch Coverage Examples: number of paths needed for Branch Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 1 5 5

Condition coverage Every complex condition is made true and false by every possible combination E.G., (x and y) x = true, y = true x=false, y=true x = true, y= false x =false, y = false There are lots of different types of condition coverage: Condition, multiple condition, condition/decision, modified condition/decision (MCDC), …I’m only covering the simplest. Under most circumstances, achieving Condition achieves Branch.

Condition Coverage: CFGs One way to determine the number of paths is to break the compound conditional into atomic conditionals Suppose you were writing the CFG for the assembly language implementation of the control construct If (A AND B) then C Endif (short circuit eval) (no short circuit eval) LD A LD A ; in general, lots of BZ :endif LAND B ; code for A and B LD B BZ :endif BZ :endif JSR C JSR C :endif nop :endif nop

AND Condition 1 2A 2B 4 3 Join 1. read (a,b); 2. if (a == 0 && b == 0) then { 3. c  a + b } 4. else c  a * b paths: 1, 2A,2B,3, J 1, 2A, 2B, 4, J 1, 2A, 4, J 2A 2B 4 3 Join

OR Condition 1 2A 3 2B 4 5 Join 1. read (a,b); 2. if (a == 0 || b == 0) then } 3. c  a + b; 4. while( c < 100) 5. c  a + b;} Paths: 1, 2A, 3, 4, 5, 4 … J 1, 2A, 3, 4, J 1, 2A, 2B, 3, 4, 5, 4, … J 1,2A, 2B, J 2A 3 2B 4 5 Join

Path A path is a sequence of statements A path is sequence of branches A path is a sequence of edges

Examples: number of paths needed for Path Coverage Examples: number of paths needed for Path Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 5

Examples: number of paths needed for Path Coverage Examples: number of paths needed for Path Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 16 5 5

Path Coverage-1 Every distinct path through code is executed at least once Example 1. read (x) 2. read (z) 3. if x  0 then begin 4. y  x * z; 5. x  z end 6. else print ‘Invalid’ 7. if y > 1 then 8. print y 9. else print ‘Invalid’ Test Paths: 1, 2, 3, 4, 5, J1, 7, 8, J2 1, 2, 3, 4, 5, J1, 7, 9, J2 1, 2, 3, 6, J1, 7, 8, J2, 1, 2, 3, 6, J1, 7, 9, J2 1,2,3 4,5 6 Join1 P353 Pfleeger 2nd ed 7 8 9 Join2

Cyclomatic Complexity Software metric for the logical complexity of a program. Defines the number of independent paths in the basis set of a program Provides the upper bound for the number of tests that must be conducted to ensure that all statements been have executed at least once For Edges (E) and Nodes (N) V(G) = E – N + 2

Examples: Complexity of CFGs 3,4 3,4 1 1 5 5 6 2 Join Join 3 N=3 E=2 E-N+2= 2-3+2= 1=V(G) N=4 E=4 E-N+2= 4-4+2= 2=V(G) N=3 E=3 E-N+2= 3-3+2= 2=V(G) N=1 E=0 E-N+2= 0-1+2= 1=V(G)

Example 1 V(G) = 11 – 9 + 2 = 4 Independent paths: 1-11 1-2-3-4-5-10-1-11 1-2-3-6-8-9-10-1…-11 1-2-3-6-7-9-10-1…-11 2,3 6 4,5 8 7 9 10 11

In Class: Compute the cyclomatic complexity 1,2 Join 3,4 5 6 7 1, 2 3 Join 4 5 6 1,2 3 9 4,5 1 2 3 Join 5 6 6 Join 7,8

Example 1: CFG 1,2 N=7 E=8 E-N+2= 8-7+2= 3=V(G) 3 9 4,5 6 Join 7,8

Example 2 1 2 3 Join 5 6 N=6 E=8 E-N+2= 8-6+2= 4=V(G)

Answers 1,2 Join 3,4 5 6 7 N=7 E=8 E-N+2= 8-7+2= 3=V(G)

Answers N=6 E=7 E-N+2= 7-6+2= 3=V(G) 1, 2 3 Join 4 5 6

Independent Path Coverage Basis set does not yield minimal test set Example 1. read (x) 2. read (z) 3. if x  0 then begin 4. y  x * z; 5. x  z end 6. else print ‘Invalid’ 7. if y > 1 then 8. print y 9. else print ‘Invalid’ Cyclomatic complexity: 3 Test Paths: 1, 2, 3, 4, 5, J1, 7, 8, J2 1, 2, 3, 4, 5, J1, 7, 9, J2 1, 2, 3, 6, J1, 7, 8, J2, 1,2,3 4,5 6 Join1 P353 Pfleeger 2nd ed 7 8 9 Join2 96

Def-Use Coverage 1,2,3 Example 4,5 6 Join 7 8 9 Join Def-use coverage: every path from every definition of every variable to every use of that definition is exercised in some test. Example 1. read (x) 2. read (z) 3. if x  0 then begin 4. y  x * z; 5. x  z end 6. else print ‘Invalid’ 7. if y > 1 then 8. print y 9. else print ‘Invalid’ 1,2,3 Def: x, z Use: x Def: y, x Use: x, z 4,5 6 Use: none Join 7 Use: y 8 9 Use: none Use: y Test Path: 1, 2, 3, 4, 5, 7, 8, J Join

Strength of Coverage Statement Arrows point from weaker to stronger coverage. Stronger coverage requires more test cases. Branch Def-Use Condition Path

What paths don’t tell you Timing errors Unanticipated error conditions User interface inconsistency (or anything else) Configuration errors Capacity errors