Chapter 22 Developer testing Peter J. Lane

Testing can be difficult for developers to follow  Testing’s goal runs counter to the goals of the other development activities.  The goal in testing a software is to break it.  It can never completely prove the absence of errors.  Testing alone does not improve the software’s quality.  Fix it after you test it.  You are required to assume that you’ll find errors in your code.  If you assume you won’t, then that is more than likely so. Even when there are errors.

Testing During Construction  “White/glass – box” testing  Can look into the class while testing  “Black – Box” testing  See no further than the interface while testing  Easier to debug if you test a routine or class one at a time vs. several at once.

Recommended approach, cover this ground:  Test each relevant requirement to make sure that the requirements have been implemented.  Test each relevant design concern to make sure that the design has been implemented.  “Basis testing” adds detailed test cases to those that the test requirements and the design.  Make a checklist for past errors.

Test First or Last?  Writing test cases before code does not take any more effort than after.  If first, you can detect defects earlier and you can correct them more easily.  Writing cases first forces you to think a little bit about the requirements and design.  It exposes requirement problems sooner before code is written.

Limitations  Developer tests tend to be “clean” tests.  Tries to how it works rather than how it breaks.  Tends to have an optimistic view of test coverage.  Most programmers assume they’ve covered more than they actually did,  Tends to skip more sophisticated kinds of test coverage.

Incomplete Testing  Pick test cases that are most likely to find errors.  Concentrate on picking a few that tell you different things instead of a set that says the same thing again and again.

Structured Basis Covering  Start with 1 test case for the routine itself.  Add 1 test case for every logic statement (if, while, for, and and or)

Data-Flow Testing  Defined – The data has been initialized, but not used yet.  Used – Data that’s been used for computation.  Killed – Once defined data that has now been undefined.  Entered – Control flow enters routine immediately before the variable is acted upon.  Exited – The control flow leaves the routine immediately after the variable acted upon.

Equivalence Partitioning  If two test cases bring out exactly the same errors, you need only one of them.  Helps when you are not looking at it from source code.

Error Guessing  Heuristic approach  Sophisticated guessing.

Boundary analysis & Compound Boundaries  Test cases that exercise the maximum and minimum of allowable values.

Classes of Bad Data  Too little data (or no data)  Too much data  Invalid data  Wrong size of data  Uninitialized data

Classes of Good Data  Nominal cases – middle-of-the-road, expected values  Min and Max normal configurations  Compatibility with old data

Test Cases that make Hand checks convenient  Using numbers in values that are easy to calculate by hand for testing.

Which classes contain most errors  80%of the errors are found in 20% of a project’s classes or routines.  50% of the errors are found in 5% of a project’s classes.

Errors by Classification  Scope of most errors is fairly limited  Many errors are outside the domain of construction  Thin application domain knowledge, conflicting requirements, bad communication  Most construction errors are the programmer’s fault  95% programmer, 2% system software, 2% other software, 1% hardware  Typos are surprisingly common source problems  Misunderstanding the design is a recurring theme in studies of programmers errors  Most are easy to fix  It’s a good idea to measure your own organization’s experience with errors.

Proportion from Faulty Construction  On small projects, construction defects make up the vast bulk of all errors.  Construction defects account for at least 35% of all projects regardless of size.  Although cheaper to fix than requirements and design errors, construction errors are still expensive.

Errors in Testing Itself  Check your work  Plan test cases as you develop your software  Keep your test cases  Plug unit tests into a test framework

Chapter 23 Debugging

The scientific Method of Debugging 1.Stabilize the error  Unpredictable errors are often initialization errors  Requires narrowing a test case that produces the error. 2.Locate the source of error  Gather data that produces defect  Analyze the data that has been gathered, and form a hypothesis about the defect  Determine how to prove/disprove the hypothesis, by testing or examining the code and do so. 3.Fix defect 4.Test fix 5.Look for similar errors

Tips for finding defects  Use all the data available to make hypothesis  Refine the test case that produce the error  Exercise the code in your unit test suite  Test code in isolation  Use available tools  Reproduce the error several different ways

Tips (part 2)  Generate more data to generate more hypotheses  Use results of negative tests  Brainstorm for possible hypotheses  Make list of things to try  Narrow the suspicious region of the code

Tips (part 3)  Be suspicious of classes and routines that have had defects before  Check code that’s changed recently  Expand the suspicious region of the code  Integrate incrementally  Check for common defects  Talk to someone else about the problem  Take a break from the problem

Syntax Errors  Don’t trust line numbers in compiler messages  Compiler may place error on the line before or after the actual error.  Don’t trust compiler messages  May need to read between the lines to understand  Don’t trust compiler’s second message  Errors following the first may no mean anything  Divide and conquer  Temporarily remove part of code and compile again.  Find misplaced comments and quotation marks

Fixing defects  Understand the problem before you fix it  Understand the program, not just the problem  Confirm the defect diagnosis  Relax  Save original source code  Fix the problem, not the symptom  Change code only for good reasons; be certain that it will work  One change at a time

….. Okay, I’m done.