Higher-Level Testing and Integration Testing

Context Higher-level testing begins with the integration of (already unit-tested) modules to form higher-level program entities (e.g., components). The primary objective of integration testing is to discover interface and design errors associated with interactions between elements being integrated. Once the elements have been integrated (i.e., once they are able to function together), the higher-level element can be tested thoroughly (via component, product, or system testing).

Unit Testing (Reminder) Unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual function or procedure. In object-oriented programming a unit is a method or class. Unit tests are created by programmers or occasionally by white box testers.

Integration Testing Integration testing is carried out when we combine “lower-level” system elements (individual units or groups of them). The strategy employed can significantly affect the time and effort required to yield a working, higher-level element.

Why do we do Integration Testing? Unit tests only test the unit in isolation Many failures result from faults in the interaction of units or subsystems Often many off-the-shelf components are used that cannot be unit tested Failures not discovered until system testing are more expensive to fix. Failures that are not discovered in integration testing may not be discovered until after the system is deployed and can be very expensive.

Integration Testing (cont’d) Note that ‘‘integration testing’’ is sometimes defined as the level of testing between unit and system. We use a more general model of the testing process, in which integration testing is used when we combine: Units or modules to form a component Components to form a sub-system Sub-systems to form a system

Levels of Testing A More General View A Popular View System Test Integration Test System Test Subsystem Test Integration Test Integration Test Unit Test Component Test Integration Test Unit Test

Arch. Diagram Example: Compiler tokens AST Lexical Analyzer source code Parser High Level Optimizer AST optimized FRONT END optimized object code Code Generator Low Level Optimizer object code BACK END

Integration Testing Strategies The first issue to address is the choice between instantaneous and incremental integration testing. The former is sometimes referred to as the big bang approach.

Disadvantages of Big Bang Defects present at the interfaces of components are identified at a very late stage. It is very difficult to isolate the defects found, as it is very difficult to tell whether a defect is in component or interface. There is a high probability of missing some critical defects which might surfaced in production. It is very difficult to make sure that all the cases for integration testing are covered.

Incremental Integration Testing Strategies Incremental integration testing results in some additional overhead, but can significantly reduce error localization and correction time. The optimum incremental approach is inherently dependent on the individual project and the pros and cons of the various alternatives.

Modeling the Integration Process Graphically driver A A common system model used when considering integration testing is a call graph. B C C D D E F G G H H I I In a call graph, nodes represent system modules, and an edge from N1 to N2 indicates that N1 calls N2 J K L

Modeling the Integration Process Graphically driver A More generally, though, the relationship we’re interested in depicting by edges is one of “dependence” of one module on another. For example, M1 might not call M2, but it might require its services in order to proceed – for example, if M2 makes some data available that M1 requires, via say a database or other data transmission method This is more like the “Uses” relationship. B C C D D E F G G H H I I J K L

Incremental Strategies Suppose we are integrating a group of modules to form a component, the control structure of which forms the following uses graph. An incremental integration strategy imposes a specific, well-thought-out order to the way in which nodes in the graph (modules) are considered. A B C D E F G H I J K L

Incremental Strategies (cont’d) In what order should the modules be integrated? From the top (“root”) module toward the bottom? From bottom (“leaf”) modules toward the top? By function? Critical or high-risk modules first? By availability?

Incremental Strategies (cont’d) How many should be combined at a time? What scaffolding (i.e., drivers and stubs to exercise the modules and oracles to interpret or inspect test results) will be required? Is scaffolding ever required outside the context of integration testing?

Stubs and Drivers Driver: A component, that calls the Tested Unit Controls the test cases Stub: A component the Tested Unit depends on Partial implementation Returns fake values. Driver Tested Unit Stub

Stubs int main (int argc, char **argv) { if (argc<4) usage(); double x = f(argv[1],argv[2]); … } void usage() { return; double f(char *arg1, char *arg2) { return 3.0

Drive execution of sqrt function with inputs from 1 to 10 Drivers int main( int argc, char **argv) { int i; for (i=0; i<10;i++) { printf(“SQRT of %i is %i”, i, sqrt(i)); } Drive execution of sqrt function with inputs from 1 to 10

Top-Down Strategy Start with the ‘‘root’’ and one or more used modules. Test this group using stubs to take the place of missing used modules, and one driver (if necessary) to call the root module. Add one or more other called modules, replacing and providing new stubs as necessary. Continue the process until all elements have been integrated and tested.

Top-Down Strategy (cont’d) B C C D D E F G G H H I I J K L

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B stub stub stub stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C stub stub stub stub stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C D D stub stub stub stub stub stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C D D E stub stub stub stub stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C D D E F stub stub stub stub stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C D D E F G G stub stub stub stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C D D E F G G H H stub stub stub stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C D D E F G G H H I I stub stub stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C D D E F G G H H I I J stub stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C D D E F G G H H I I J K stub

Top-Down Strategy (cont’d) driver A B C D E F G H I J K L A B C C D D E F G G H H I I J K L

Top-Down Strategy (cont’d) Advantages: Allows early verification of high-level behavior. One driver (at most) is required. Modules can be added one at a time with each step if desired. Supports both ‘‘breadth first’’ and ‘‘depth first’’ approaches.

Top-Down Strategy (cont’d) Disadvantages: Delays verification of low-level behavior. Stubs are required for missing elements. When stubs replace lower-level modules it is more difficult to formulate tests in which significant data flows “upward”

Bottom-Up Strategy Start at the bottom of the hierarchy with two or more sibling leaf modules, or an only-child leaf module with its parent. Test this group using a driver. (No stubs are required.) Add one or more additional siblings, replacing drivers with modules only when all modules they use have been integrated. (cont’d)

Bottom-Up Strategy (cont’d) Continue the process until all elements of the subtree have been integrated and tested. Repeat the steps above for the remaining subtrees in the hierarchy (or handle in parallel). Incrementally combine the sub-trees and then add the root module.

Bottom-Up Strategy (cont’d) F G G H H I I J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver F J

Bottom-Up Strategy (cont’d) F G H I J K L driver E F J

Bottom-Up Strategy (cont’d) F G H I J K L driver B E F J

Bottom-Up Strategy (cont’d) F G H I J K L driver B E F driver J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver B driver E F H H J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver B driver E F H H I I J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver driver B D D E F H H I I J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver driver B D D driver E F G G H H I I J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver driver driver B C C D D E F G G H H I I J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver driver B C C D D E F G G H H I I J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver driver B C C D D E F G G H H I I J K K L

Bottom-Up Strategy (cont’d) F G H I J K L driver B C C D D E F G G H H I I J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver B C C D D E F G G H H I I J K L

Bottom-Up Strategy (cont’d) F G H I J K L driver A B C C D D E F G G H H I I J K L

Bottom-Up Strategy (cont’d) Advantages: Allows early verification of low-level behavior. No stubs are required. Easier to formulate input data for some subtrees. Easier to interpret output data for others.

Bottom-Up Strategy (cont’d) Disadvantages: Delays verification of high-level behavior. Drivers are required for missing elements. As subtrees are combined, a large number of elements may be integrated at one time.

Sandwich Integration Combination of bottom-up and top-down integrations System is viewed as composed of layers Approach 1: Top-down approach is used for the top layer Bottom-up approach is used for the bottom layer Allows integration to begin early in the testing phase

Sandwich Integration Approach 2: Start with a layer in the middle Use drivers to and stubs to check Work out from middle Allows integration to begin early in the testing phase

Hybrid Incremental Integration Approaches Risk Driven Start by integrating the most critical or complex modules together with modules they call or are called by. Schedule Driven To the extent possible, integrate modules as they become available.

Hybrid Incremental Integration Approaches (cont’d) Function or Thread Driven Integrate the modules associated with a key function (thread); continue the process by selecting another function, etc. A B C D E F G H I J K L