Software Testing

Testing Levels (McConnel) Unit Testing Single programmer involved in writing tested code Component Testing Multiple programmers involved in writing test code Integration Testing Testing two or more independently developed, interacting modules together Regression Testing Use a set of tests that can be rerun when changes are made to the system

Testing Transparency White-box Testing Black-box Testing Tester has access to the code Black-box Testing Tester has no access to the code Relies on functional descriptions

Test Plan Test plan consists of a set of tests and the expected results

Code coverage What parts of the code are actually executed by a test? Statement coverage What combinations of sections of code are executed by a test? Conditional Coverage (all conditions are evaluated as true at least once and as false at least once)

Boundary Values Identify equivalence classes Array declaration example: Good test cases usually test the boundary values of inputs Empty strings Max number of array elements

Finding Equivalence Classes: C++ Array Declaration Input Condition Valid Equivalence classes Invalid Equivalence classes Size of array name <255 chars (1) >255 truncates to 255 (3) <0 char (2) Array name Has letters (4), has digits(5), has underscore(6) Has something else (7) Number of dimensions Has 1 dimension (8) Has a lot of dimensions(9) Has 0 dimensions Spaces Spaces between name and bracket (10), Spaces between brackets(11), No spaces between name and bracket (12), No spaces between brackets(13) Spaces within name (14)

Boundary-Value Analysis Successful test cases frequently include values that are “directly on, above or beneath the edges of input equivalence classes and output equivalence classes.” (Myers) Each edge of an equivalence class is subject to test Result space is considered as well as input space

Boundary-Value Analysis (Myers) Write test cases for the ends of input ranges Input range is [-1.0, 1.0] Write cases for -1.0, 1.0, -1.0001, 1.0001 Number of possible inputs is [1,255] Write cases for 0, 1, 255, 256 Write test cases for the ends of output ranges Calculated SS Tax (FICA) is [$0,$6621.60] Write cases that yield results: $0, $.01, $6621.60 , and $6621.61 Number of possible output is [1,255] Write cases for 0,1 255, 256 outputs Use “your ingenuity” to search for other boundary conditions. Boundaries are not always so obvious

When to write tests? “All in all, I think test-first programming is one of the most beneficial software practices to emerge during the past decade…” Steve McConnell If you have to write tests eventually anyway, why not write them first so they can be useful longer. Detect bugs earlier Think more about requirements Think more about how others will use your code

Research Results on Tests Two studies 80% of errors are found in 20% of the classes 50% of errors are found in 5% of classes Most of the cost of a system might be represented by a very small amount of code. Error prone routines should be redesigned and/or re- implemented

Characterizing Errors 85% corrected within a single routine Three most common causes of errors Thin-domain knowledge Fluctuating/Conflicting Requirements Communication/Coordination Breakdown Almost always the programmer’s fault Typos/Spelling errors are very common Three most expensive errors involved the change of a single character ($1.6 Billion, $900 Million, $245 Million) Misinterpretation of Design 85% of errors can be fixed in a few hours It pays to know where you make errors (personal process) and where your project makes errors

Number of Errors to Expect Industry average is estimated at 1-25 errors per 1000 lines of code MS Applications Division gets 10-20/1000 lines in in-house tests: .5/1000 in released code. Space shuttle claims 0 defeats in 500,000 lines of code

Unit Tests Unit tests test an individual class or module We use unit tests in TDD Unit tests need to work independently from other parts of the system Unit tests need to be fast Unit tests typically test one thing The software under unit test may rely on other components in the systems. These components might be unavailable/undesirable during testing. They might be under development They might be unreliable They might be too slow

Definitions (Art of Unit Testing, by Roy Osherove) An external dependency is an object in your system that your code under test interacts with, and over which you have no control. A stub is a controllable replacement for an existing dependency (or collaborator) in the system. By using a stub, you can test your code without dealing with the dependency directly. Stubs are one instance of a class of tools called fakes. Fakes can provide a wide range of aids in testing. s.

Definitions (Art of Unit Testing, by Roy Osherove) A mock object is a fake object in the system that decides whether the unit test has passed on failed. It does so by verifying whether the object under test interacted as expected with the fake object. There’s usually no more than one mock per test. Numerous software frameworks are available to support the use of fakes Moq, Rhino Mocks, Microsoft Fakes