Chapter 10 – Testing and Debugging

Goals Learn techniques to test your code Learn techniques to test your code Learn to carry out unit tests Learn to carry out unit tests Understand principles of test case selection and evaluation Understand principles of test case selection and evaluation Learn to use logging Learn to use logging Become familiar with a debugger Become familiar with a debugger

Unit Testing The single most important testing tool The single most important testing tool Checks a single method or a set of cooperating methods Checks a single method or a set of cooperating methods You don't test the complete program that you are developing; you test the classes in isolation You don't test the complete program that you are developing; you test the classes in isolation

Unit Testing Avoids confusion of interactions Avoids confusion of interactions More startup cost, but less work in the end More startup cost, but less work in the end Analogy: Testing car Analogy: Testing car

Test Harness For each test, you provide a simple class called a test harness For each test, you provide a simple class called a test harness Test harness feeds parameters to the methods being tested Test harness feeds parameters to the methods being tested

Example To compute the square root of a use a common algorithm: To compute the square root of a use a common algorithm: Guess a value x that might be somewhat close to the desired square root (x = a is ok) Guess a value x that might be somewhat close to the desired square root (x = a is ok) Actual square root lies between x and a/x Actual square root lies between x and a/x Take midpoint (x + a/x) / 2 as a better guess Take midpoint (x + a/x) / 2 as a better guess Repeat the procedure. Stop when two successive approximations are very close to each other Repeat the procedure. Stop when two successive approximations are very close to each other

Example Method converges rapidly. Square root of 100: Guess #1: 50.5 Guess #2: Guess #3: Guess #4: Guess #5: Guess #6: Guess #7: Guess #8: 10.0 Method converges rapidly. Square root of 100: Guess #1: 50.5 Guess #2: Guess #3: Guess #4: Guess #5: Guess #6: Guess #7: Guess #8: 10.0

Testing RootApproximator class RootApproximator class getRoot() getRoot() nextGuess() nextGuess() Numeric class Numeric class approxEqual(double, double) approxEqual(double, double)

06:public class RootApproximatorTester 07:{ 08: public static void main(String[] args) 09: { 10: System.out.print("Enter a number: "); 11: Scanner in = new Scanner(System.in); 12: double x = in.nextDouble(); 13: RootApproximator r = new RootApproximator(x); 14: final int MAX_TRIES = 10; 15: for (int tries = 1; tries <= MAX_TRIES; tries++) 16: { 17: double y = r.nextGuess(); 18: System.out.println("Guess #"+tries+ ": " + y); 19:} 20: System.out.println("Square root: " + r.getRoot()); 21: } 22:}

Unit Testing What is needed to make the testing more robust? What is needed to make the testing more robust? More tests More tests Limitation: Hard to replicate tests when bugs are fixed Limitation: Hard to replicate tests when bugs are fixed Solution: Use test harness to repeat values Solution: Use test harness to repeat values

Providing Test Input Solution #1: Hardwire series of tests Solution #1: Hardwire series of tests No need to memorize series of tests, already taken care of. Just run tester class. No need to memorize series of tests, already taken care of. Just run tester class.

public class RootApproximatorHarness1 { public static void main(String[] args) { double[] testInputs = { 100, 4, 2, 1, 0.25, 0.01 }; for (int x = 0; x < testInputs.length; x++) { RootApproximator r = new RootApproximator(testInputs[x]); double y = r.getRoot(); System.out.println("square root of " + testInputs[x] + " = " + y); }}}

Generate Test Cases Automatically Instead of hardcoding array of values Instead of hardcoding array of values 1. Loop through a sample range of values for (int x = MIN; x <= MAX; x += INCREMENT){ RootApproximator r = new RootApproximator(x); double y = r.getRoot(); System.out.println("square root of " + x + " = " + y); }

Generate Test Cases Automatically Instead of hardcoding array of values Instead of hardcoding array of values 2. Use random number generator final int SAMPLES = 100; Random generator = new Random(); for (int i = 0; i < SAMPLES; i++){ double x = 1000 * generator.nextDouble(); RootApproximator r = new RootApproximator(x); double y = r.getRoot(); System.out.println("square root of " + x + " = " + y); }

What to Test for? Good testing requires testing good cases Good testing requires testing good cases Reduces debugging time and ensures better product Reduces debugging time and ensures better product Test all of the features of the method Test all of the features of the method Positive tests – normal behavior of method Positive tests – normal behavior of method Boundary test cases (e.g. x = 0), make sure end conditions are correct Boundary test cases (e.g. x = 0), make sure end conditions are correct Negative cases – program should reject Negative cases – program should reject

Test Case Evaluation How do you know whether the output is correct? How do you know whether the output is correct? Calculate correct values by hand E.g., for a payroll program, compute taxes manually Calculate correct values by hand E.g., for a payroll program, compute taxes manually Supply test inputs for which you know the answer E.g., square root of 4 is 2 and square root of 100 is 10 Supply test inputs for which you know the answer E.g., square root of 4 is 2 and square root of 100 is 10 Verify that the output values fulfill certain properties E.g., square root squared = original value Verify that the output values fulfill certain properties E.g., square root squared = original value

for (int i = 1; i <= SAMPLES; i++) { double x = 1000 * generator.nextDouble(); RootApproximator r = new RootApproximator(x); double y = r.getRoot(); if (Numeric.approxEqual(y * y, x)){ System.out.print("Test passed: "); passcount++; } else { System.out.print("Test failed: "); failcount++;} System.out.println("x = " + x 37: + ", root squared = " + y * y); }

Test Case Evaluation Use an Oracle: a slow but reliable method to compute a result for testing purposes E.g., use Math.pow to slower calculate x 1/2 (equivalent to the square root of x) Use an Oracle: a slow but reliable method to compute a result for testing purposes E.g., use Math.pow to slower calculate x 1/2 (equivalent to the square root of x)

for (int i = 1; i <= SAMPLES; i++) { double x = 1000 * generator.nextDouble(); RootApproximator r = new RootApproximator(x); double y = r.getRoot(); double oracleValue = Math.pow(x, 0.5); if (Numeric.approxEqual(y,oracleValue)){ System.out.print("Test passed: "); passcount++; } else { System.out.print("Test failed: "); failcount++;}}

Regression Testing and Test Coverage Save test cases Save test cases Particularly tests that reveal bugs Particularly tests that reveal bugs Use saved test cases in subsequent versions Use saved test cases in subsequent versions A test suite is a set of tests for repeated testing A test suite is a set of tests for repeated testing

Why keep a test case? Very common for bugs to show up later Very common for bugs to show up later We often think we fixed a bug, but just covered it up We often think we fixed a bug, but just covered it up Cycling = bug that is fixed but reappears in later versions Cycling = bug that is fixed but reappears in later versions Regression testing: repeating previous tests to ensure that known failures of prior versions do not appear in new versions Regression testing: repeating previous tests to ensure that known failures of prior versions do not appear in new versions

Test Coverage Test coverage: measure of how many parts of a program have been tested Test coverage: measure of how many parts of a program have been tested Good testing tests all parts of the program Good testing tests all parts of the program E.g. every line of code is executed in at least one test E.g. every line of code is executed in at least one test Example: Test all possible branches inside of an if statement Example: Test all possible branches inside of an if statement In Tax code program, 6 tests (3 $ brackets * 2 types of status) In Tax code program, 6 tests (3 $ brackets * 2 types of status)

Test Coverage Black-box testing: test functionality without consideration of internal structure of implementation Black-box testing: test functionality without consideration of internal structure of implementation Useful since this is what the user interacts with Useful since this is what the user interacts with White-box testing: take internal structure into account when designing tests White-box testing: take internal structure into account when designing tests Unit testing Unit testing Why? Cannot prove absence of bugs, only presence Why? Cannot prove absence of bugs, only presence

Testing Levels The part of the program that is being tested. The part of the program that is being tested. System System Unit Unit

System Level Testing Test interactions of the various parts of the application. Test interactions of the various parts of the application. Completed after the unit tests pass. Completed after the unit tests pass.

Unit Level Testing Test individual units (classes) in isolation. Test individual units (classes) in isolation. Test features of the unit. Test features of the unit. Multiple test cases for each feature. Multiple test cases for each feature.

Black Box Testing a.k.a. behavioral, functional, closed a.k.a. behavioral, functional, closed against the specification against the specification does not need the program source does not need the program source internal implementation is unknown internal implementation is unknown

Black Box Testing Advantages: Advantages: designed from the specifications designed from the specifications user's point of view user's point of view less biased less biased can discover if part of the specification has not been fulfilled. can discover if part of the specification has not been fulfilled.

Black Box Testing Disadvantages: Disadvantages: may be redundant may be redundant can be difficult to design. can be difficult to design. testing every possible input stream is unrealistic testing every possible input stream is unrealistic

White Box Testing a.k.a. structural, clear box, open a.k.a. structural, clear box, open tests against the implementation tests against the implementation every line of source code is executed at least once. every line of source code is executed at least once. tests exception handling code. tests exception handling code.

White Box Testing Advantages Advantages can discover if any part is faulty can discover if any part is faulty test coverage (test all paths) test coverage (test all paths) Disadvantages Disadvantages will not discover if part is missing will not discover if part is missing only if the programmer knows what the program is supposed to do only if the programmer knows what the program is supposed to do code must also be visible to the tester. code must also be visible to the tester.

"Fully" Tested To fully test a software product, the following are required: To fully test a software product, the following are required: unit testing and system level testing unit testing and system level testing black and white box testing black and white box testing Does this ensure there are no bugs? Does this ensure there are no bugs?

Testing Tips Develop cases to take into account end conditions Develop cases to take into account end conditions Tip: write first test cases before program is written completely → gives insight into what program should do Tip: write first test cases before program is written completely → gives insight into what program should do Build up as program goes along and bugs are found Build up as program goes along and bugs are found

Logging To figure out where your program “goes”, use program trace statements To figure out where your program “goes”, use program trace statements Help determine the “flow” of execution Help determine the “flow” of execution if (status == SINGLE) { System.out.println("status is SINGLE");... }

Problem When done debugging, have to remove all System.out.println trace messages When done debugging, have to remove all System.out.println trace messages What if we find a bug again? What if we find a bug again? Solution: Create a separate class for the purpose of logging Solution: Create a separate class for the purpose of logging

Logging Logging messages can be deactivated when testing is complete Logging messages can be deactivated when testing is complete Use global object Logger.global in place of System.out Use global object Logger.global in place of System.out Log a message Log a message Logger.global.info("status is SINGLE");

Turn off/on How does this class help? How does this class help? Can easily switch logging on and off Can easily switch logging on and off Default: On Default: On To turn off, insert at top of main(): To turn off, insert at top of main():Logger.global.setLevel(Level.OFF);

What to print out Common to need to know enter/exit conditions of methods Common to need to know enter/exit conditions of methods public TaxReturn(double anIncome, int aStatus){ Logger.global.info("Parameters: anIncome = " + anIncome + " aStatus = " + aStatus);... }//Exit public double getTax(){... Logger.global.info("Return value = " + tax); return tax; }

Logging Obviously other output is needed Obviously other output is needed Bug is that certain condition isn’t true when it is supposed to be – track values as you go along Bug is that certain condition isn’t true when it is supposed to be – track values as you go along Logging class has many other options (see API) Logging class has many other options (see API) Disadvantages: Disadvantages: Time consuming Time consuming Too much/little output Too much/little output

Debuggers To avoid logging problems, most professionals use a debugger To avoid logging problems, most professionals use a debugger Programs rarely (never) run perfectly the first time Programs rarely (never) run perfectly the first time Think of perfect first draft to paper Think of perfect first draft to paper The larger your programs, the harder to debug them simply by logging The larger your programs, the harder to debug them simply by logging

Debuggers Most current development environments contain a debugger Most current development environments contain a debugger Debugger - program to run your program and analyze its run-time behavior Debugger - program to run your program and analyze its run-time behavior A debugger lets you stop and restart your program, see contents of variables, and step through it A debugger lets you stop and restart your program, see contents of variables, and step through it

Debuggers Debuggers can be part of your IDE (Eclipse, BlueJ) or separate programs (JSwat) Debuggers can be part of your IDE (Eclipse, BlueJ) or separate programs (JSwat) Three key concepts: Three key concepts: Breakpoints Breakpoints Single-stepping Single-stepping Inspecting variables Inspecting variables

Breakpoint Breakpoint – the debugger doesn’t stop running the program until it hits a defined breakpoint Breakpoint – the debugger doesn’t stop running the program until it hits a defined breakpoint Setup by programmer Setup by programmer Once stopped, you can see the state of all the variables Once stopped, you can see the state of all the variables

Step-through Once a breakpoint is hit, two options Once a breakpoint is hit, two options Step through the next few statements carefully inspecting Step through the next few statements carefully inspecting Slow, but useful in heavy calculations Slow, but useful in heavy calculations Single-step – line by line execution Single-step – line by line execution Step into – doesn’t just skip line to line, but goes into method calls within each line Step into – doesn’t just skip line to line, but goes into method calls within each line Step over – Execute the enter statement and stop Step over – Execute the enter statement and stop Run at full speed until the next breakpoint Run at full speed until the next breakpoint

Example Current line: Current line: String input = in.next(); Word w = new Word(input); int syllables = w.countSyllables(); System.out.println("Syllables in " + input + ": " + syllables);

Step Over Next step: Next step: String input = in.next(); Word w = new Word(input); int syllables = w.countSyllables(); System.out.println("Syllables in " + input + ": " + syllables);

Step Into public int countSyllables() { int count = 0; int end = text.length() - 1;... }

Which to choose If the method is suspect (it may be the cause of a problem) – Step into If the method is suspect (it may be the cause of a problem) – Step into If you are sure the method works correctly – step over If you are sure the method works correctly – step over

What if a test fails? It's time to debug! It's time to debug! We will want (need) a strategy. We will want (need) a strategy. Debugging Strategy Debugging Strategy A systematic way to find and fix bugs. A systematic way to find and fix bugs.

Debug Strategy 1: Trial & Error 1. Write code 2. Run program 3. Fix bugs Advantages Advantages no planning required no planning required Disadvantages Disadvantages Spend more time trying to find and fix bugs than you did writing the program. Spend more time trying to find and fix bugs than you did writing the program.

Debug Strategy 2: Trial & Error with a Debugger 1. Write code 2. Run program 3. Use Debugger program to fix bugs Advantages Advantages no planning required no planning required Disadvantages Disadvantages Spend lots of time in the debugger program trying to find and fix the bugs Spend lots of time in the debugger program trying to find and fix the bugs

Debug Strategy 3: Incremental Development with Testing 1. Write small amount of code with testing and debugging in mind. 2. Document and write tests for each new piece of code. 3. Run tests on new code 4. Fix any bugs before adding new code. Disadvantage: Sometimes you have to debug code that you didn't write. Disadvantage: Sometimes you have to debug code that you didn't write.

Debug Strategy 3: Incremental Development with Testing Disadvantages Disadvantages Takes more time (harder) to design the solution Takes more time (harder) to design the solution Must write more code (tests) Must write more code (tests) Only possible if you're the one who wrote the program Only possible if you're the one who wrote the program Advantages Advantages A small amount of code to review and fix. A small amount of code to review and fix. Bug is fixed before it get hidden by some other bug. Bug is fixed before it get hidden by some other bug. If small parts work, the bigger parts will usually work too. If not, it's only the public interfaces that have to be debugged. If small parts work, the bigger parts will usually work too. If not, it's only the public interfaces that have to be debugged.

Debugging Strategy 4: Test and use a Debugger 1. Identify the bug. Must be repeatable and testable. 2. Write test(s) that fails for the bug. 3. Locate the source of the bug. HOW? 4. Understand the problem before you fix it. 5. Try your fix and run all tests. 6. Repeat until all tests pass.

How do we find the source of the bug? Pick a point to start looking Pick a point to start looking Beginning (main method) Beginning (main method) Divide and conquer Divide and conquer Trace the code. ( Look for variables with unexpected values) Trace the code. ( Look for variables with unexpected values) Manual trace Manual trace Trace messages Trace messages System.out.println(…) System.out.println(…) Java 1.5: Use java.util.logging.Logger class Java 1.5: Use java.util.logging.Logger class Debugger program Debugger program