Software Testing Techniques Chapter 14 Software Testing Techniques Highlights of Software Testing Techniques

Review - What is Testing? - Testing is the process of exercising/examining a program with the intent of finding errors prior to delivery to the end user. - Errors may belong to a variety of classes (functional, behaviour, performance). - Among others, the objectives of testing are to: > find uncovered errors, > demonstrate conformance to requirements, and > get indication of quality and reliability. - A testing strategy is a plan that outlines detailed testing activities (steps, test case design, test execution, effort, time, resources), and it results in Test Specification document. Note: 30% - 40% of development effort is spent on testing!

Testing Principles Among others, some basic testing principles include: - Test cases should be traceable to requirements. - Design of test cases should starts as early as completing software requirements and before testing starts. - Testing should begin “in the small” and progress toward testing “in the large”. - Exhaustive testing is not possible. Test selective execution paths. - Testing should be conducted by independent team. Pareto Principle: 80% of all uncovered errors during testing is likely be traceable to 20% of all program components. See section 5.5.2, page 114.

Software Testability As testing become integral part of development, software is developed with “testability” in mind! (how easy to test the program) Testability results from the design (good design leads to better testability) Testability attributes (next slides) are mapped back to the quality of design and implementation. When testing is performed by an independent tester, the tester must learn the system, try to “break” it, and keep the testing quality-driven.

Testability Attributes (1) Characteristics that lead to a testable software: Operability: (the program operates cleanly) - no or few known bugs - no serious bugs to prevent execution of test cases - ability to conduct simultaneous testing on components Observability: (the results of each test case are readily observed) - distinct output for each input - ability to see system state and variables - ability to easily identify incorrect outputs - ability of detect internal errors and report them - access to source code

Testability Attributes (2) Controllability: (the degree to which testing can be automated and optimized) - ability to generate all possible outputs (via inputs) - all code is executable using combination of inputs - ability to control SW/HW states and variables - consistent input/output format - tests may be automated and re-produced Decomposeability: (component targeted testing) - modular design of the software - ability to test modules independently

Testability Attributes (3) Simplicity: (reduce complex architecture and logic to simplify testing) - functional simplicity (specific functions of requirements) - structural simplicity (modularized architecture) - code simplicity (following coding and documentation standards) Stability: (few changes are required during testing) - changes are infrequent, controlled, don’t invalidate test cases - the program can recover from failures

Testability Attributes (4) Understandability: (good knowledge of the design) - understanding component design - understanding of component hierarchy - understanding of design changes - accessibility to technical documents > readily available > well organized > detailed and accurate content

Test Case Attributes What is a “good” test? - It has high probability of finding errors. This requires good understanding of the software. - It is not redundant. Each test case must have a well-defined and unique purpose and objective. - It is neither too simple nor too complex to avoid error masking in case of combining test case together. - It should be the “best of breed” (select the best from among similar test cases)

Test Case Design OBJECTIVE: CRITERIA: CONSTRAINT: to uncover errors in a complete manner with a minimum of effort and time "Bugs lurk in corners and congregate at boundaries ...“ Boris Beizer

Exhaustive Testing is not Possible! - Two nested loops - Each loop runs 20 iterations - Four if-then-else constructs are defined inside the inner loop There are about 1014 possible executable paths! - If we execute one test per millisecond, it would take 3,170 years to test this program!! See page 392, Info Box

Testing Selected Paths Due to large number of possible paths, select limited number of “important” logical paths for testing! (e.g., Basic Path Testing) Selected path

Testing Methods Methods Strategies white-box methods black-box

White-Box Testing (1) The goal is to ensure that all statements and conditions have been executed at least once

White-Box Testing (2) With white-box testing, test - all independent paths within the module/component - all decision structures for their true and false values - all loop structures for their boundaries and operations - all data structures (arrays, sets, maps, stacks, etc…)

Why White-Box Testing? - Logical errors and incorrect assumptions are inversely proportional to a path's execution probability. - It is often believed that a logical path is not likely to be executed when it may be executed on regular basis! - It is likely that untested paths will contain some typographical errors. - Others?

Basic Path Testing (1) Basic path method is white-box techniques that enables the designer to derive a logical complexity measure (number of independent paths) that can be used for defining basic set of execution paths. Steps: - Construct flow graph (nodes and edges) - Determine the cyclomatic complexity value - Identify the basic set of execution paths - Design test cases for basic set paths

Flow (Program) Graph 1 2 3 4 5 6 9 10 8 7

Cyclomatic Complexity 6 2 4 10 3 9 8 7 5 1 R4 R2 R3 R1 Cyclomatic complexity V(G) is a metric that provides quantitative measure of the logical complexity of code (that is, number of independent paths). Computed as: 1. V(G) = E - N + 2 2. V(G) = P + 1 3. V(G) = # of regions on the graph where E is # of edges N is # of nodes P is # of predicate nodes (red nodes) (for combined conditions, see page 395 figure)

Basic Set of Paths 6 2 4 10 3 9 8 7 5 1 Independent path: Is a path that includes a new edge(s) that have not need visited by other paths. Since V(G) = 4, there are 4 independent paths (called Basic Set): Path 1: 1,2,3,6,7,8,9,10 Path 2: 1,2,3,5,7,8,9,10 Path 3: 1,2,4,8,9,10 Path 4: 1,2,4,8,9,2, ... ,9,10

Design Test Cases Each test case must force the execution of the intended path. Note: A path my not be tested stand-alone. It must be tested as part of another path(s) (e.g., loop paths) See complete example on page 397, figure 14.4, Procedure average.

Basic Paths - Notes You don't need a flow chart, 1 2 3 4 5 6 9 10 8 7 You don't need a flow chart, but the picture will help when you trace program paths. Count each simple logical test, compound tests count as 2 or more nodes. Basis path testing should be applied to critical modules.

Graph Matrices A graph matrix is a square matrix whose size (i.e., number of rows and columns) is equal to the number of nodes on a flow graph (see page 399, figure 14.6) Each row and column corresponds to an identified node, and matrix entries correspond to connections (an edge) between nodes. Adding a link weight to each matrix entry, the graph matrix can be used to calculate useful info such as Probability of execution Required memory Processing time Other resources

Cyclomatic Complexity - Note A number of industry studies have indicated that the higher the cyclomatic complexity V(G) of a module, the higher the probability of errors in that module. modules V(G) modules in this range are more error prone

Other Testing Techniques (concepts) Other White-Box based testing techniques include: 1. Condition Testing: Focus on testing logical conditions. 2. Data Flow Testing: Focus on testing variables. 3. Loop Testing: Focus on testing loops. Note: these techniques are used to test “suspect” modules and not necessarily the entire software.

Condition Testing Concept Conditions are potential places for errors. Condition testing is designing test cases to detect errors in conditional structures. That is, - Arithmetic expressions - Relational operators - Boolean operator (with compound conditions) - Boolean variables (with simple conditions) - Formation of parenthesis A condition may require several test cases for complete coverage.

Data Flow Testing Concept Data flow testing is designing test cases based on the definition and use of a variable. The selected path includes the module that defines the targeted variable and all modules that make use of that variable. Data flow testing may be used with paths that include nested loops and if-structures. The difficulty with this approach is measuring test coverage and selecting test paths.

Loop Testing Concept Loop testing is designing test cases to validate a loop structure and it boundaries. A loop structure can be - simple - nested - concatenated - unstructured Each structure requires different test cases.

Loop Structures Unstructured Loops Nested Concatenated Simple Loop

Testing Simple Loops Typical strategy to test a simple loop that iterates n times: 1. Skip the loop entirely 2. Test only one pass through the loop 3. Test two passes through the loop 4. Test m passes through the loop m < n 5. Test iterations n-1, n, and n+1

Testing Nested Loops Typical strategy to test nested loop: 1. Set all outer loops to their minimum iteration parameter values. 2. Test the the innermost for min+1, typical, max-1, and max values, while holding the outer loops at their minimum values. 3. Move out one loop and test it as in step 2, holding all other inner loops at typical values and outer loops at minimum values. Continue this step until the outermost loop has been tested.

Testing Concatenated Loops If the loops are independent of one another then test them as simple loop, otherwise, test them as nested loops. Two loops are dependent is when the loop counter value of first loop is used to initialize the second loop. Avoid unstructured loop! They can be re-designed using other loop structures.

Black-Box (behavior) Testing (1) requirements events input output

Black-Box (behavior) Testing (2) Black-Box testing focuses on testing functional requirements of program components. Test cases should be designed to fully exercise the outlined functions of the component. It complements White-Box testing and uncovers: - Incorrect/missing functions - Interface errors - Module initialization and termination errors - External data errors - Performance errors

Black-Box (behavior) Testing (3) Black-Box testing is applied in later stages of the testing process. Test cases are designed to answer these questions: How is functional validity tested? How are system behavior and performance tested? What classes of input will make good test cases? Is the system particularly sensitive to certain input values? How are the boundaries of a data class isolated? What data rates and data volume can the system tolerate? What effect will specific combinations of data have on system operation?

Black-Box Testing Techniques (1) Black-Box based testing techniques include: 1. Graph-Based methods: Focus on testing relationship between modules. Different graphs my be utilized (Transition Flow, Finite State, Data Flow, Execution Time) 2. Equivalence Partitioning: Focus on dividing input data into equivalence classes from which test cases are derived to test different classes of errors. (classes may include user queries, mouse click, prompts, output format, inputs, etc…) (valid and invalid values (or ranges of values) for each class are defined)

Black-Box Testing Techniques (2) 3. Boundary Value Analysis: Extends Equivalence partitioning and focus on testing values at the edges of each partitioning (ranges of valid and invalid values) 4. Comparison Testing: Focus on testing software systems with redundancy. (multiple instances of the program) 5. Orthogonal Testing: Focus on minimizing number of test cases by designing test cases with maximum coverage. (“testing one item at a time” vs. “multiple items at a time”) (see figure 14.10) See textbook for details.

OO Testing By nature, OO design results in layered subsystems of collaborating classes OO testing focuses on uncovering errors in class collaborations. OO testing starts “in the small” (testing classes and their collaborations) toward testing “in the large” (testing interfacing among subsystems) OO testing is derived by “class state” changes (sequence of operations to test class states) Encapsulation can be an obstacle for testing! How? Inheritance can be an obstacle for testing! How? White-box techniques can be applied to class operations! or shouldn't be?

OO Testing Methods (1) 1. Fault-based testing: - The tester looks for plausible faults (part of the implementation that may result in defects). To determine whether these faults exist, test cases are designed to exercise the design or code. - Test cases focus on class methods. Class testing and the class hierarchy: - Inheritance complicates testing since testing derived classes does not preclude the need to test derived classes. - Test cases for a base class may be used to test a derived class, but expect different results.

OO Testing Methods (2) 3. Scenario-based testing: - It focuses on user (actor) interaction (what the user does not what the system does) - It uncovers interaction errors among subsystems - Its test cases are more complex to design (than other methods) 4. Surface and deep structure testing: - Surface structure testing is black-box testing (focus on what the user see in (and can do with) the system) - Deep structure testing is white-box testing (focus on the technical (analysis and design) details of the system, such as interactions, behavior, and dependencies)

Class-Level Testing Methods (1) 1. Random testing: - identify operations applicable to a class. - define constraints on their use. - identify a minimum test sequence (sequence of operation) that defines the minimum life history of the class (object). - generate a variety of random (but valid) test sequences to exercise other (more complex) class instance life histories. See class “Account” testing sequences page 415.

Class-Level Testing Methods (2) 2. Partition testing: - It is similar to equivalence partitioning for conventional software. - It reduces the number of test cases required to test a class. - State-based partitioning groups and test operations based on their ability to change the state of the class. - Attribute-based partitioning groups and tests operations based on the attributes that they use in the class. - Category-based partitioning groups and tests operations based on the generic function each operation performs. See class “Account” testing sequences page 416.

Inter-Class Testing Methods (1) 1. Inter-Class testing: (multi-class integration testing) - For each class, use the list of class operations to generate a series of random test sequences. - Use the operations to send messages to other classes. - For each generated message, determine the collaborator class and the corresponding operator in that class. - For each operation in the called class, determine the messages that it transmits. - For each of the messages, determine the next level of operations that are invoked and incorporate these into the test sequence.

Inter-Class Testing Methods (2) 2. Behavior-based integration testing - Test sequences are derived based on the class behavior and the classes it collaborates with. - The tests should achieve all state coverage (for examples, the operation sequences should cause the “Account” class to make transition through all allowable states) (see figure 14.12 for Account class states) - In some cases, multiple state diagrams may be need to derive test sequences.

Specialized Testing Black and white box testing may be specialized for different environments, architectures and applications. See textbook for testing guideline for (page 420): 1. GUI development environment 2. Client/Server architecture 3. Documentation and help application 4. Real-Time system

Testing Patterns Test patterns may be developed to define generic templates for designing test cases, similar to analysis and design patterns. Many testing pattern have nee proposed in the literature. See textbook, page 424, for three example patterns: Pair testing: It describes a technique in which two testers work together to design and execute a series of tests for unit, integration, or validation testing activities. Separate test interface: It describes how to create a test interface that can be used to describe specific tests for classes local to a component. Scenario testing: It describes a technique for exercising the software from the user’s point of view. A failure at this level indicates that the software has failed to meet a user visible requirement.

Suggested Problems Consider working the following problems from chapter 14, page 427: 3, 9, 10, 11, and 12. No submission is required for practice assignments. Work it for yourself!

Last Slide End of chapter 14