1. CIS 4932 Software Testing White-Box Testing
2/2000
CIS 4932 Software Testing White-Box Testing
1) Welcome to session #1 Unit Test CONCEPTS
2) A few administrative matters
1. Sign in -- the sheet is circulating
2. Attend each session
3. You will be asked to complete a course evaluation at the end of Session #3.
3) How many of you are
Test: [ ] Lead [ ] Analyst [ ] Coordinator
Development
A&D
Now, open your notebook.
1. Read the welcome page
2. Turn to Tab1
This session takes around 2.5 hours.
Handouts -- Unit Test Concepts

2. 2/2000
WHITE-BOX TESTING
Software
under
Test
Stimuli
Response(s)
Testing to ensure that software does not do what is not supposed to do. Test ALL of it!
White-Box Testing is an UNDER-THE-HOOD testing approach.
The goal is to ensure that the software does not contain a TICKING BOMB waiting to go off unexpectedly.
No-Surprise Software!!
Tick -- Tick -- Tick
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

3. 2/2000
WHITE-BOX TESTING
Software
under
Test
Stimuli
Response(s)
Focus is thorough execution of program elements during the testing process.
Warning: Tests only what is built, not what was intended!
The one way to ensure that there is no ticking time bomb is to execute all the code under all the possible circumstances -- to see how the code responds.
WARNING: Success in white-box testing is measured in terms of how thoroughly the code was executed -- EVEN IF IT'S THE WRONG CODE!
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

4. 2/2000
WHITE-BOX TESTING
Concept of coverage. Numeric measure of thoroughness of testing, relative to
Statements
Branches
Conditions
Paths
1) White-box testing can be MEASURED.
2) The granularity of the measurement is defined in terms of code elements
a) Statements (or blocks)
b) Branches caused by decision statements (if, loops, switches)
c) Conditions used to make decisions
d) Execution paths through the code
3) COVERAGE is the numeric measure -- a percentage of the countable code elements
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

5. CONTROL FLOW GRAPH Defines the flow of control through unit. 1 2 4 3 5
2/2000
CONTROL FLOW GRAPH
Defines the flow of control through unit. 1 2 4 3
The execution of a unit can be described by a graph that depicts the FLOW OF CONTROL through elements of the code.
Drawing this graph IS NOT REQUIRED (whew!), but the graph explains the important relationship between code complexity and testing difficulty. 5
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

6. CONTROL FLOW GRAPH TERMINOLOGY
2/2000
CONTROL FLOW GRAPH TERMINOLOGY
5 NODES
sequential blocks of code terminated by a branch
3 PATHS:
[1,2,3,5], [1,2,5], [1,4,5]
2 BRANCHES
1 and 2 are decision nodes 1 2 4 5 3
A unit can be described by this graph. Certain features of the graph are relevant to white-box testing.
1) NODE -- block of statements in the code
2) BRANCH -- code statement at which sequential flow is broken based on the outcome of a decision
3) PATH -- end-to-end sequence of nodes visited during one execution of the unit.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

7. CONTROL FLOW GRAPH COVERAGE
2/2000
CONTROL FLOW GRAPH COVERAGE
One test case forces execution of one path (red)
Paths are determined by branches (decision nodes)
A thorough test set forces execution of all paths (red, green, blue). 1 2 4 5 3
The measure of thoroughness is called COVERAGE.
The notion of THOROUGHNESS can be made VISUAL by coloring elements of the graph that have been executed during testing.
There are tools that produce such pictures.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

8. 2/2000
COVERAGE LEVELS (%)
Statement -- a statement has been executed at least once during testing
Branch -- each outcome of a branch has been performed at least once during testing
Path -- a path through the code has been executed at least once during during testing
Condition -- a condition has evaluated to true and to false at least once during testing
These coverage levels are arranged in order of RELATIVE STRENGTH and INCREASING DIFFICULTY.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

9. CONTROL FLOW GRAPH COVERAGE MEASUREMENT
2/2000
CONTROL FLOW GRAPH COVERAGE MEASUREMENT
For 2 test cases (red, green)
Node (statement) cov = 4/5
Branch cov = 1/2 [2]
Path cov = 2/3
Acceptable coverage levels
Statement cov = 90%
Branch cov = 80%
Path cov = 70% 1 2 4 3 5
You may be surprised that acceptable coverage levels are not all 100%.
You will see in the next few slides some of the things take make it hard to achieve 100% coverage.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

10. BRANCH vs CONDITION COVERAGE
2/2000
BRANCH vs CONDITION COVERAGE
Code example 1
if (x<1 && y>1) x = x + y;
else y = y - x; 2 4
100% Branch coverage [1]
(x=0,y=2), (x=1,y=2) [TT,FT]
But not 100% Condition coverage
Need case TF (x=0,y=1)
It is common to write code that uses COMPLEX CONDITIONS of logical expressions involving AND and OR operators.
From experience, it is HARD to get programs with complex logic to work properly. If they are hard to get to work, they are also HARD TO TEST!
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

11. THE PROBLEM WITH COMPOUND CONDITIONS
2/2000
THE PROBLEM WITH COMPOUND CONDITIONS
Makes complex logic appear simpler than it really is
Test cases may be omitted
Logic results in 3 paths, not 2!! 1 2 3 1 2 5 3 4
if (x<1)
{if (y>1) x=x+y;
else y=y-x; }
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

12. THE PROBLEM WITH PATH COVERAGE
2/2000
THE PROBLEM WITH PATH COVERAGE
Not all paths are feasible
No test case can force path [1,2,3,4,5]. Consecutive decisions mutually exclusive. 1 2 4 3 5
if (x<1)
y=2;
if (x >= 1)
y=3;
z=y;
1) There may be paths in code that CAN NEVER BE EXECUTED. These are called INFEASIBLE PATHS.
2) For the example:
a) Test case x=0, path is [1,2,3,5]
b) Test case x=2, path is [1,3,4.5]
c) NO TEST CASE for path [1,2,3,4,5]
3) One can TEST FOREVER and NEVER REACH 100% path coverage.
4) HOWEVER -- These test cases achieved 100% BRANCH and CONDITION COVERAGE!!
5) Designing test cases for path coverage results in high levels of branch/condition coverage.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

13. Measuring Path Testing Difficulty
2/2000
Measuring Path Testing Difficulty
McCabe metric -- logical code complexity
Formula: 1 + #decisions in control flow graph
Test Significance: #basis paths through code
Design use: complexity of code
Test use: min #test cases for 100% path coverage
McCabe measures test (development) difficulty
1) There is a surprisingly simple way to determine the minimum number of test cases required to achieve 100% PATH COVERAGE (assuming feasibility)
2) The metric, McCabe's Cyclomatic Complexity
= 1 + #elementary_decisions
3) Most widely used metric for software design
a) SPLIT unit when McCabe exceeds threshold
b) High McCabe means ERROR-PRONE
4) McCabe
= MIN# TEST CASES for path coverage
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

14. How To Design White Box Tests
2/2000
How To Design White Box Tests
Test cases must execute different paths
Decision tables
Rows -- elementary conditions in the code
Columns -- combinations of conditions in the code
Column based test case forces flow through different logic paths in the code
Decision table built from code reflects what was built versus intended (from spec)
Decision analysis for white-box testing.
There are two options for designing white-box test cases:
1. Reverse Engineer Decision Table from Code
a) Each column represents a path
b) Create test case for each column
NOTE: Decisions based on computed values may complicate test case generation.
2. Use Black-Box Test Cases
a) Adequate scheme when implementation matches the specification -- or is very close
b) Misleading otherwise
NOTE: Use of black-box test cases during code inspection may keep implementation close to spec.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

15. WHITE-BOX TESTING TOOLS
2/2000
WHITE-BOX TESTING TOOLS
Tools instrument source code and gathers coverage data when tests
Compiled languages
Script languages -- coming to market
Some provide test case design assistance
List of paths not covered
Data conditions to force branch/path
Graphic depiction of graph coverage
1) Fortunately, there are tools for white-box testing. Basic features
a) Instrument source code by inserting probes to gather coverage data
b) Report generation following test execution
c) Graphical display of coverage
d) Assistance with test case design
2) Unfortunately, these tools are not in widespread or consistent use in SC.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

16. Problem P1 Code (v1) & Decision Table
2/2000
Problem P1 Code (v1) & Decision Table
Age> | Y N N N N N
Weight>300 | - Y N N N N
Age<= | Y N N N
Age> | Y N N
Weight<120 | Y N
Pills = | C
Note: C: 2+(Weight/120)/50
if (Age>80 ||
Weight>300)
return 0;
if (Age <= 12)
return 1;
if (Age > 65)
return 2;
if (Weight < 120)
return 2
else
return 2+(Weight-120)/50;
1) On the next two slides you will be shown how to build a decision table from code.
2) Afterwards, you will be given "Your Turn" at doing the same.
3) NOTE: This is an implementation of the Compute Dosage. It has McCabe of 6, so six test cases are needed to achieve 100% path coverage.
4) The reverse engineered decision table has 6 columns, as expected. So deriving a test case for each column should force execution of the 6 different paths.
McCabe = 6
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

17. Problem P1 Code (v2) & Decision Table
2/2000
Problem P1 Code (v2) & Decision Table
Age> | Y N N N N N N
Weight>300 | - Y N N N N N
Age<= | Y N N N
Age> | Y N N
Weight<120 | Y N
Pills = | C
if (Age>80 ||
Weight>300)
return 0;
if (Age <= 12)
return 1;
if (Age > 65 ||
(Age<=65 && Weight<120))
return 2;
return 2+(Weight-120)/50;
1) This is a different implementation of Compute Dosage. It is slightly more complex -- largely due to the use of complex conditions.
NOTE: The decision table is different from your notes -- the condition rows have been reordered to match the order the conditions APPEAR IN THE CODE.
2) WARNING: REMEMBER that these test cases force the execution of the paths of the AS BUILT CODE. Passing these tests does not mean the code matches the specification.
McCabe = 7
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

18. Your Turn -- White-Box Testing (1) Construct Decision Table
2/2000
Your Turn -- White-Box Testing (1) Construct Decision Table
____________ |
Pills = |
Note: C: 2+(Weight/120)/50
pills=0;
if (Age < 80 && Weight <300) {
pills=1;
if (Age >= 65)
pills=2;
else if (Age > 12)
pills=2+(Weight-120)/50; }
return pills;
YOUR TURN:
(1) Reverse engineer the decision table from the code. In determining expected result, GO BY THE CODE, not the original spec.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

19. Your Turn -- P1 (2) Derive White-Box Test Cases
2/2000
Your Turn -- P1 (2) Derive White-Box Test Cases
Case
Age ___ ___ ___ ___ ___ ___ ___ ___ ___
Weight ___ ___ ___ ___ ___ ___ ___ ___ ___
Pills ___ ___ ___ ___ ___ ___ ___ ___ ___
YOUR TURN
(1) Create test cases for path coverage, one per column of the decision table.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

20. OBSERVATIONS -- WHITE-BOX TEST CASES
2/2000
OBSERVATIONS -- WHITE-BOX TEST CASES
Code may not be complete with respect to input combinations from the specification
Decision table constructed from code is simpler -- subset of black-box table
Claim: black-box test cases force coverage of logic
Unless the code implements the wrong (a different) function
In the ideal world, the CODE implements the SPEC, so that EACH black-box test case triggers a distinct behavior from the code .. and EACH DISTINCT BEHAVIOR has a UNIQUE PATH through the code.
USE BLACK-BOX TEST CASES FOR WHITE-BOX TESTING
a) Measure actual coverage (using tool)
b) Create only those test cases needed to achieve coverage goal
WITHOUT TOOLS, USE SOUND ENGINEERING JUDGMENT.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts

21. MAIN POINTS WHITE-BOX TESTING
2/2000
MAIN POINTS WHITE-BOX TESTING
White-box = logic testing
Limitation: can't tell what's missing
Don't forget exceptions -- throwing, catching, propagating (debugger)
Perform decision analysis of code
Coverage tools help.
Use black-box test cases.
1) The real goal of white-box testing is to TEST IT ALL!
2) COVERAGE answers HOW MUCH SO FAR?
3) TEST CASE DESIGN uses REVERSE ENGINEERING of code to decision table
4) CODE INSPECTIONS are the best way to address whether black-box test cases are adequate for achieving white-box coverage.
5) WITHOUT TOOLS you are limited in what you can BE SURE OF.
2/2000
Unit Test Concepts
Handouts -- Unit Test Concepts