Software Testing Mistake in coding is called error , Error found by tester is called defect, Defect accepted by development team is called bug , Product does not meet the requirements then it Is failure.”

The Importance of Test There are hundreds of stories about failures of computer systems that have been attributed to errors in software There are many reasons why systems fail but the issue that stands out the most is the lack of adequate testing. Example Pepsi - $42 Billion Error Philippines -1992 Due to a software error, 800,000 bottle caps were produced with number 349 instead of one. It was equivalent to $32 billion in prize money instead of $40,000

Testing Levels

Testing Levels Unit Module Each of testing activities is performed during the process of building an application Unit ….. Unit Build Application on final platform Application on test bed ….. ….. Module Interface testing ….. system testing installation testing usability testing Unit Build acceptance testing Unit Module integration testing ….. Unit Regression testing throughout

Testing Goals Based on Test Process Testing Levels (Bezier) Level 0 There is no difference between testing and debugging Level 1 The purpose of testing is to show the correctness (software works). Level 2 The purpose of testing is to show that the software doesn’t work.(the purpose is to show failures) Level 3 Testing shows the presence , not absence or failure. If we use software, we expose some risk. The risk may be small and unimportant, or the risk may be great and the consequences catastrophic, but risk is always there (The purpose of testing is to reduce the risk of software ; both testers and developers work together to reduce risk. Level 4 The purpose of testing is to improve the ability of the developers to produce high quality software. (Level 4 testing shows that testers and developers are on the same team)

Debugging and Testing Debugging is done in the development phase by the developer In debugging phase identified bugs are fixed Testing is done in the tester phase by the tester. In testing, locating and identifying of the bugs are included.

Software Bug A software bug is an error, flaw, mistake, failure, or fault in a computer program or system Bug is terminology of Tester Software bug produces an incorrect or unexpected result, or causes it to behave in unintended ways

Software Fault A static (physical)defect*, imperfection, flaw in the software. short between wires break in transistor infinite program loop A fault means that there is a problem within the code of a program which causes it to behave incorrectly. *defect: Mismatch between the requirements

Software Error Error is an incorrect internal state that is the demonstration of some fault. In other words; Error is a deviation from correctness or accuracy. Example Suppose a line is physically shortened to 0. (there is a fault). As long as the value on line is supposed to be 0, there is no error. Errors are usually associated with incorrect values in the system state.

Software Failure A failure means that the program has not performed as expected, and the actual performance has not met the minimum standards for success. External, incorrect behavior with respect to the requirements or other description of the expected behavior Example Suppose a circuit controls a lamp (0 = turn off, 1 = turn on) and the output is physically shortened to 0 (there is a fault). As long as the user wants the lamp off, there is no failure.

Example: Failure & Fault & Error Consider a medical doctor making a diagnosis for a patient. The patient enters the doctor’s office with a list of failures (that is, symptoms). The doctor then must discover the fault, or root cause of the symptom. To aid in the diagnosis, a doctor may order tests that look for anomalous internal conditions. In our terminology, these anomalous internal conditions correspond to errors.

Cause-and-Effect Relationship Faults can result in errors. Errors can lead to system failures Errors are the effect of faults. Failures are the effect of errors Bug in a program is a fault. Possible incorrect values caused by this bug is an error. Possible crush of the operating system is a failure

Program Example class numZero { public static int numZero (int[] x) { // if x == null throw NullPointerException // else return the number of occurrences of 0 in x int count = 0; for (int i = 1; i < x.length; i++) { if (x[i] == 0) count++; } } return count; } }

The Fault ,Error and Failure in the Example The fault is that the program starts looking for zeroes at index 1 instead of index 0 For example, numZero ([2, 7, 0]) correctly results with 1, while numZero ([0, 7, 2]) incorrectly results with 0. In both of these cases the fault is executed. Both of the cases result in an error (because of next slide) But the first case results with correct value. Only the second case results in failure.

Error States in the Example To understand the error states, we need to identify the state for the program. The state for numZero consists of values for the variables x, count , i , program counter (PC). For the first case: The state at the if statement on the first iteration ( x = [2, 7, 0], count = 0, i = 1, PC = if) This state is in error because the value of i should be zero on the first iteration. But the value of count is correct, the error state does not propagate to the output, and the software does not fail. In other words, a state is in error if it is not the expected state, even if all of the values in the state are acceptable.

Error States in the Example In the second case the corresponding (error) state is (x = [0, 7, 2], count = 0, i = 1, PC = if). In this case, the error propagates to the variable count and is present in the return value of the method. Hence a failure results.

Distinguish Testing from Debugging The definitions of fault and failure allow us to distinguish testing from debugging. The difference is that debugging is conducted by a programmer and the programmer fix the errors during debugging phase. Tester never fixes the errors, but rather find them and return to programmer.

Testing versus Debugging Testing activity is carried down by a team of testers, in order to find the defect* in the software. Testers run their tests on the piece of software and if they encounter any defect,they report it to the development team. Testers also have to report at what point the defect occurred and what happened due the occurrence of that defect. All this information is used by development team to DEBUG the defect. After finding out the bug, he tries to modify that portion of code and then he rechecks if the defect has been removed After fixing the bug, developers send the software back to testers. *actual results don't match expected results

What is Software Quality ? According to the IEEE Software Quality is: The degree to which a system, component, or process meets specified requirements. The degree to which a system, component, or process meets customer or user needs or expectations.

Software Quality Factors

Software Quality Assurance Verification – are we building the product right ? – performed at the end of a phase to ensure that requirements established during previous phase have been met Validation – are we building the right product ? – performed at the end of the development process to ensure compliance with product requirements

Verification & Validation Verification: The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase. Validation: The process of evaluating software at the end of software development to ensure compliance with intended usage.

Difference Between Verification &Validation- I Verification is preventing mechanism to detect possible failures before the testing begin. It involves reviews, meetings, evaluating documents, plans, code, inspections, specifications etc. Validation occurs after verification and it's the actual testing to find defects against the functionality or the specifications

The Difference between Verification &Validation- II Verification is usually a more technical activity that uses knowledge about the individual software artifacts, requirements, and specifications. Validation usually depends on domain knowledge; that is, knowledge of the application for which the software is written. For example, validation of software for an airplane requires knowledge from aerospace engineers and pilots.

Software Testing Types Black box testing : You don't need to know the internal design in detail or have a good knowledge about the code for this test. It's mainly based on functionality and specifications, requirements. White box testing : This test is based on detailed knowledge of the internal design and code. Tests are performed for specific code statements and coding styles.

Traditional Testing Levels System testing : Test the overall functionality of the system main Class P Acceptance testing : Is the software acceptable to the user? Module testing (developer testing) : Test each class, file, module or component Integration testing : Test how modules interact with each other Class A method mA1() method mA2() Class B method mB1() method mB2() Unit testing (developer testing) : Test each unit (method) individually

Object-Oriented Testing Levels Inter-class testing : Test multiple classes together Class A Class B Intra-class testing : Test an entire class as sequences of calls method mA1() method mB1() method mA2() method mB2() Inter-method testing : Test pairs of methods in the same class Intra-method testing : Test each method individually

Coverage Criteria Even small programs have too many inputs to fully test them all private static computeAverage (int A, int B, int C) On a 32-bit machine, each variable has over 4 billion possible values Over 80 octillion possible tests!! Input space might as well be infinite Testers search a huge input space Trying to find the fewest inputs that will find the most problems Coverage criteria give structured, practical ways to search the input space Search the input space thoroughly Not much overlap in the tests

Test Requirements and Criteria Test Criterion : A collection of rules and a process that define test requirements Cover every statement Cover every functional requirement Test Requirements : Specific things that must be satisfied or covered during testing Each statement is a test requirement Each functional requirement is a test requirement Testing researchers have defined dozens of criteria, but they are all really just a few criteria on four types of structures … Graphs Logic expressions Input domains Syntax descriptions

Coverage Criterion A coverage criterion is simply a recipe for generating test requirements in a systematic way: Coverage Criterion: A coverage criterion is a rule or collection of rules that impose test requirements on a test set.

Structural Coverage Criteria -I Node Coverage (Formal Definition): For each node n ∈ reachG(N0), TR contains the predicate “visit n.” Node Coverage (NC): TR contains each reachable node in G. Node Coverage (NC) (Standard Definition) Test set T satisfies node coverage on graph G if and only if for every syntactically reachable node n in N, there is some path p in path(T) such that p visits n. TR = { 0, 1, 2 } path(t1) = [0, 1 , 2] T1={t1} T1 satisfies node coverage on the graph 1 2

Structural Coverage Criteria - II Edge coverage is slightly stronger than node coverage Edge Coverage (EC) : TR contains each reachable path of length up to 1, inclusive, in G. The “length up to 1” allows for graphs with one node and no edges Edge Coverage : TR = { (0,1), (0, 2), (1, 2) } path (t1 ) = [ 0,1,2] path (t2 ) = [ 0, 2 ] T2={t1, t2} Test Paths = [ 0, 1, 2 ] [ 0, 2 ] T2 satisfies edge coverage on the graph 1 2

Edge-Pair Coverage TR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6], [2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] } path (t1 )= [ 0, 1, 2, 3, 6 ] path (t2 )= [ 0, 1, 2, 4, 6 ] path (t3 )=[ 0, 2, 3, 6 ] path (t4 )= [ 0, 2, 4, 5, 4, 5, 4, 6 ] T={t1 , t2 , t3 , t4 } 1 2 Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 5, 4, 6 ] 3 4 5 6

Graph Coverage for Source Code Graph : Usually the control flow graph (CFG) Node coverage : Execute every statement Edge coverage : Execute every branch Loops : Looping structures such as for loops, while loops, etc. Data flow coverage : Augment the (CFG) control flow graph Defs are statements that assign values to variables Uses are statements that use variables

Control Flow Graphs : The if Statement if (x < y) { y = 0; x = x + 1; } else x = y; 4 1 2 3 x >= y x < y x = y y = 0 x = x + 1 3 1 2 x >= y x < y y = 0 x = x + 1 if (x < y) { y = 0; x = x + 1; }

Control Flow Graphs : while and for Loops 1 x = 0 x = 0; while (x < y) { y = f (x, y); x = x + 1; } 2 for (x = 0; x < y; x++) { y = f (x, y); } x >= y x < y x >= y x < y 4 3 y =f(x,y) x = x + 1 1 2 3 5 x >= y x < y y = f (x, y) 4 x = x + 1

Control Flow Graphs : do Loop, break and continue 1 x = 0 x = 0; do { y = f (x, y); x = x + 1; } while (x < y); println (y) x = 0; while (x < y) { y = f (x, y); if (y == 0) break; } else if y < 0) y = y*2; continue; } x = x + 1; print (y); 2 3 y =f(x,y) y == 0 4 break 1 x = 0 5 y < 0 2 y = f (x, y) x = x+1 6 y = y*2 continue x >= y x < y 7 x = x + 1 3 8

The Application of Graph Criteria: Source Code It is usually defined with the control flow graph (CFG) Node coverage is used to execute every statement Edge coverage is used to execute every branch Loops are used to execute looping structures such as for loops, while loops, etc. Data flow coverage is the extended form of control flow graph (CFG) with defs and uses defs are statements that assign values to variables uses are statements that use variables

System.out.println ("length: " + length); public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) sum += numbers [ i ]; } med = numbers [ length / 2]; mean = sum / (double) length; varsum = 0; varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println ("length: " + length); System.out.println ("mean: " + mean); System.out.println ("median: " + med); System.out.println ("variance: " + var); System.out.println ("standard deviation: " + sd); } 1 2 i = 0 3 4 5 6 i = 0 7 8 i >= length i < length i++

Post-Unit Testing Interface testing system testing Module Each of post-unit testing activities is performed during The process of building an application Unit ….. Unit Build Application on final platform Application on test bed ….. ….. Module Interface testing ….. system testing installation testing usability testing Unit Build acceptance testing Unit Module integration testing ….. Unit Regression testing throughout

How to select input values? Scenario I Random values For each input parameter we randomly select the values Tester Experience For each input we use our experience to select relevant values to test Domain knowledge We use requirements information or domain knowledge information to identify relevant values for inputs

How to select input values? Scenario II Equivalence classes (for black-box testing) We subdivide the input domain into a small number of sub-domains The equivalence classes are created assuming that they exhibits the same behavior on all elements Few values for each classes can be used for our testing Boundary values (for black-box testing) Is a test selection technique that targets faults in applications at the “boundaries” of equivalence classes Experience indicates that programmers make mistakes in processing values at and near the boundaries of equivalence classes

Black-Box Testing Example: Person a rents « the Matrix» on May5 Person b rents «Gone with the Wind» on May 6 Person a returns «the Matrix» on May 10 It does not take the manner in which the application was designed or implemented This is analogous to building an automobile and then testing it by driving it under various conditions. BUT If the car is built with a new design for its automatic transmission , we would be wise to use this knowledge White box testing is required White box tests are based on design and implementation

Parameter Space of computeFine() Each element represents a pair: (Days late, Movie name) new releases old releases one-of-a-kind The penalty also depends on the whether the movie is a «new» or «old» release or «one of a kind.» -5 5 10 15 Days late e.g., (6 days late, “Casablanca”)