IS301 – Software Engineering mailto:mkabay@norwich.edu V: 802.479.7937 Software Testing IS301 – Software Engineering Lecture #31 – 2004-11-12 M. E. Kabay, PhD, CISSP Assoc. Prof. Information Assurance Division of Business & Management, Norwich University mailto:mkabay@norwich.edu V: 802.479.7937 M. E. Kabay, PhD, CISSP Copyright © 2003 M. E. Kabay. All rights reserved.

Topics Defect Testing Integration Testing Object-Oriented Testing Testing Workbenches

Copyright © 2003 M. E. Kabay. All rights reserved. Defect Testing Testing programs to establish presence of system defects Copyright © 2003 M. E. Kabay. All rights reserved.

Testing Process Component testing Testing of individual program components Usually responsibility of component developer (except sometimes for critical systems) Tests derived from developer’s experience Integration testing Testing of groups of components integrated to create system or sub-system responsibility of independent testing team Tests based on system specification

Testing Phases

Defect Testing Goal of defect testing to discover defects in programs Successful defect test test which causes program to behave in anomalous way Tests show presence not absence of defects

Copyright © 2003 M. E. Kabay. All rights reserved. Testing priorities Only exhaustive testing can show program free from defects. However, exhaustive testing impossible Tests should exercise system's capabilities rather than its components Testing old capabilities more important than testing new capabilities Testing typical situations more important than boundary value cases Copyright © 2003 M. E. Kabay. All rights reserved.

Test Data and Test Cases Inputs which have been devised to test system Test cases Inputs to test system plus Predicted outputs from these inputs If system operates according to its specification Copyright © 2003 M. E. Kabay. All rights reserved.

Defect Testing Process

Black-Box Testing Program considered as ‘black-box’ Test cases based on system specification Test planning can begin early in software process

Copyright © 2003 M. E. Kabay. All rights reserved. Black-Box Testing Copyright © 2003 M. E. Kabay. All rights reserved.

Equivalence Partitioning Input data and output results often fall into different classes where all members of class related Each of these classes = equivalence partition where program behaves in equivalent way for each class member Test cases should be chosen from each partition

Equivalence Partitioning (1) Copyright © 2003 M. E. Kabay. All rights reserved.

Equivalence partitioning (2) Partition system inputs and outputs into ‘equivalence sets’; e.g., If input 5-digit integer between 10,000 and 99,999, then equivalence partitions are <10,000, 10,000-99,999 and >99,999 Choose test cases at boundaries of these sets: 00000, 09999, 10000, 10001, 99998, 99999, & 100000 Copyright © 2003 M. E. Kabay. All rights reserved.

Equivalence Partitioning (3)

Search Routine Specification procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ; Pre-condition -- array has at least one element T’FIRST <= T’LAST Post-condition -- element found and referenced by L ( Found and T (L) = Key) or -- element not in array ( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key )) Copyright © 2003 M. E. Kabay. All rights reserved.

Search Routine - Input Partitions Inputs which conform to pre-conditions Inputs where pre-condition does not hold Inputs where key element is member of array Inputs where key element not member of array Copyright © 2003 M. E. Kabay. All rights reserved.

Testing Guidelines (Sequences) Test software with sequences which have only single value Use sequences of different sizes in different tests Derive tests so that first, middle and last elements of sequence accessed Test with sequences of zero length

Search Routine - Input Partitions

Copyright © 2003 M. E. Kabay. All rights reserved. Structural Testing Sometime called white-box testing Derivation of test cases according to program structure. Knowledge of program used to identify additional test cases Objective to exercise all program statements not all path combinations Copyright © 2003 M. E. Kabay. All rights reserved.

Copyright © 2003 M. E. Kabay. All rights reserved. White-Box Testing Copyright © 2003 M. E. Kabay. All rights reserved.

Binary Search – Equiv. Partitions Pre-conditions satisfied, key element in array Pre-conditions satisfied, key element not in array Pre-conditions unsatisfied, key element in array Pre-conditions unsatisfied, key element not in array Input array has single value Input array has even number of values Input array has odd number of values Copyright © 2003 M. E. Kabay. All rights reserved.

Binary Search – Equiv. Partitions Copyright © 2003 M. E. Kabay. All rights reserved.

Binary Search – Test Cases Copyright © 2003 M. E. Kabay. All rights reserved.

Path Testing Objective: ensure each path through program executed at least once Starting point: flow graph Shows nodes representing program decisions Arcs representing flow of control Statements with conditions therefore = nodes in flow graph

Copyright © 2003 M. E. Kabay. All rights reserved. Program Flow Graphs Each branch shown as separate path Loops shown by arrows looping back to loop condition node Basis for computing cyclomatic complexity Cyclomatic complexity = Number of edges - Number of nodes + 2 Copyright © 2003 M. E. Kabay. All rights reserved.

Cyclomatic Complexity Number of tests needed to test all control statements equals cyclomatic complexity Cyclomatic complexity equals number of conditions in program Useful if used with care. Does not imply adequacy of testing. Although all paths executed, all combinations of paths not necessarily executed Copyright © 2003 M. E. Kabay. All rights reserved.

Binary Search Flow Graph

Copyright © 2003 M. E. Kabay. All rights reserved. Independent Paths 1, 2, 3, 8, 9 1, 2, 3, 4, 6, 7, 2 1, 2, 3, 4, 5, 7, 2 1, 2, 3, 4, 6, 7, 2, 8, 9 Test cases should be derived so that all of these paths executed Dynamic program analyzer may be used to check that paths have been executed Copyright © 2003 M. E. Kabay. All rights reserved.

Integration Testing Tests complete systems or subsystems composed of integrated components Integration testing should be black-box testing with tests derived from specification Main difficulty is localizing errors Incremental integration testing reduces this problem

Incremental Integration Testing

Approaches to Integration Testing Top-down testing Start with high-level system and integrate from top-down replacing individual components by stubs where appropriate Bottom-up testing Integrate individual components in levels until complete system created In practice, most integration involves combination of these strategies

Top-Down Testing

Bottom-Up Testing

Testing Approaches Architectural validation Top-down integration testing better at discovering errors in system architecture System demonstration Top-down integration testing allows limited demonstration at early stage in development Test implementation Often easier with bottom-up integration testing Test observation – what’s happening during test? Problems with both approaches Extra code may be required to observe tests Instrumenting the code

Copyright © 2003 M. E. Kabay. All rights reserved. Interface Testing Takes place when modules or sub-systems integrated to create larger systems Objective: Detect faults due to Interface errors or Invalid assumptions about interfaces Particularly important for object-oriented development Objects defined by their interfaces Copyright © 2003 M. E. Kabay. All rights reserved.

Interface Testing

Interface Types Parameter interfaces Data passed from one procedure to another Shared-memory interfaces Block of memory shared between / among procedures Procedural interfaces Sub-system encapsulates set of procedures to be called by other sub-systems Message-passing interfaces Sub-systems request services from other sub-systems

Interface Errors Interface misuse Calling component makes error use of interface; e.g. parameters in wrong order Interface misunderstanding Calling component embeds incorrect assumptions about behavior of called component Timing errors Called and calling component operate at different speeds Out-of-date information accessed “Race conditions”

Interface Testing Guidelines Parameters to called procedure at extreme ends of their ranges Always test pointer parameters with null pointers Design tests which cause component to fail Use stress testing in message passing systems (see next slide) In shared-memory systems, vary order in which components activated Why?

Stress Testing Exercises system beyond its maximum design load Stressing system often causes defects to come to light Stressing system tests failure behavior Should not fail catastrophically Check for unacceptable loss of service or data Distributed systems Non-linear performance degradation

Approaches to Cluster Testing Use-case or scenario testing Testing based on user interactions with system Tests system features as experienced by users Thread testing Tests systems response to events as processing threads through system Object interaction testing Tests sequences of object interactions Stop when object operation does not call on services from another object

Scenario-Based Testing Identify scenarios from use-cases Supplement with interaction diagrams Show objects involved in scenario Consider scenario in weather station system where report generated (next slide)

Testing Workbenches Testing expensive Workbenches Range of tools Reduce Time required and Total testing costs Most testing workbenches are open systems Testing needs are organization-specific Difficult to integrate with closed design and analysis workbenches

Testing Workbench

Testing Workbench Adaptation Scripts User interface simulators and Patterns for test data generators Test outputs May have to be prepared manually for comparison Special-purpose file comparators may be developed

Required Homework Read-recite-review Chapter 23 of Sommerville’s text Survey-Question Chapter 24 for Monday Quiz on WEDNESDAY 17th Nov: Chapters 17-21 Required For Fri 19 Nov 2004 for 35 points 23.1 & 23.3 (@5) 23.4 & 23.5 (@10) 23.7 (@5)

Optional Homework By Mon 29 Nov 2004, for up to 14 extra points, complete any or all of 23.2, 23.6, 23.8 (@2) 23.9 (@5) 23.10 (@3)

Copyright © 2003 M. E. Kabay. All rights reserved. DISCUSSION Copyright © 2003 M. E. Kabay. All rights reserved.