Introduction to Software Testing Slides adapted from various sources including Software Engineering: A Practitioner’s Approach, 6/e by R.S. Pressman and Associates Inc. CSC 550

What is software testing? “Testing is the process of exercising a program with the intent of finding Errors.” “Testing is the process of exercising a program with the intent of proving its correctness.”

What is software testing? “Observing the execution of a software system to validate whether it behaves as intended and identify potential malfunctions.” Antonia Bertolino

What does testing show? errors requirements conformance performance an indication of quality

Why is testing important? Because the cost of failures is significant!

Expectation vs. Reality Do we expect software to work correctly? A carefully made program 5 faults/KLOC Windows XP has 45M LOC How many faults? 5 * 45M/1000 = 225,000! Why not remove all the faults?

Why is testing so difficult? 1 3 2 5 4 loop < =20 X a b c d e How many paths are there through this loop? How long would it take to execute all paths @ 1 test per millisecond (10-3)?

A Testing Scenario There are about 1014 possible paths 3 2 5 4 loop < =20 X a b c d e There are about 1014 possible paths How long would it take to execute all paths?

A Testing Scenario 1 3 2 5 4 loop < =20 X a b c d e How long would it take to execute 1014 possible paths @ 1 paths per 10-3 seconds? 1 year =31,536,000 seconds ~= 32,000,000,000 milliseconds Total time = 1014 / 32 * 109 ~= 105 /32 years About 3,170 years @ one test/millisecond About 3 years @ one test/microsecond About 11 days @ one test/nanosecond

Why is testing so difficult? 1 3 2 5 4 loop <= 20 X a b c d e Because exhaustive testing is impossible Exhaustive testing means testing a program with all combinations of input values and preconditions for each element of the software under test.

Why is testing so difficult? Looping : for (i=0; i< n; i++) etc. Loop header Conditional statement statement How long would it take to execute all paths @ 1 test per millisecond (10-3)?

Terms! Terms! Terms! Anomaly BUG Failure Error Defect Fault Exception

Testing Terminology Fault/defect Error Failure An incorrect step, process, condition, or data definition in a computer program. Software problem found after release (Pressman) Error An incorrect internal state resulting from a fault. A discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition (ISO). Software problem found before release (Pressman) Failure The inability of a system or component to perform its required functions within specified performance requirements (IEEE). Failures are caused by faults, but not all faults cause failures.

Testing Terminology: Fault vs. Error vs. Failure public static int numZaro(int[] x) { //Effects: if x == null throw NullPointerException // else return the number of occurrences of “0” (zero) in x. int count = 0; for (int i = 1; i < x.length; i++) { if (x[i] == 0) { count++; } return count; Fault = search starts at index 1 instead of 0. Error: numZero([2, 7, 0])  error but no failure Failure: numZero([0, 7, 2])  error and failure State at if given [2, 7, 0] = (x=[2,7,0], count=0, i=1, PC=if). i Should be zero S0, if required sequence of states is s0, s1, s2 and actual sequence is s0, s2, s3, then s2 is in error in second sequence.

Testing Terminology Anomaly Bug Exception Anything observed in the documentation or operation of software that deviates from expectations based on previously verified software products or reference documents (IEEE). Bug Things the software does that it is not supposed to do Something the software doesn't do that it is supposed to. Exception An event that causes suspension of normal program execution; types include addressing exception, data exception, operation exception, overflow exception, protection exception, underflow exception (IEEE).

Testing Principles Testing shows the presence of defects. Exhaustive testing is impossible. Early testing. Testing should start as early as possible in the SE lifecycle. Defect clustering. A small number of modules contains most defects detected. Pesticide paradox. If the same tests are over and over again, then eventually these tests will not find any bugs (review and create new test cases). Testing is context dependent. Different types of software must be tested differently. Absence of errors – fallacy.

Testing Terminology: Testing vs. Debugging Evaluating software to determine conformance to some objective. Debugging Finding a fault, given a failure.

Testing Terminology: The RIP Fault/Failure Model Three conditions are necessary for failures to be observable. Reachability: The location of the fault must be reached. Infection: After executing the location, the state of the program must be incorrect. Propagation: The infected state must propagate to cause incorrect program output.

Testing Terminology Validation Verification Does software meet its goals? User-centric! Verification Are we building the product right? E.g., does the code reflect the design? Developer-centric! Name for the artifact being tested Implementation under test (IUT) Method under test (MUT), object under test (OUT) Class/component under test (CUT), etc.

Testing Terminology: Test Plan A test plan specifies how we will demonstrate that the software is free of faults and behaves according to the requirements specification A test plan breaks the testing process into specific tests, addressing specific data items and values Each test has a test specification that documents the purpose of the test If a test is to be accomplished by a series of smaller tests, the test specification describes the relationship between the smaller and the larger tests The test specification must describe the conditions that indicate when the test is complete and a means for evaluating the results

Testing Terminology: Test Oracle A test oracle is the set of predicted results for a set of tests, and is used to determine the success of testing Test oracles are extremely difficult to create and are ideally created from the requirements specification

Testing Terminology: Test Case A test case is a specification of The pretest state of the IUT Prefix values: values needed to put software into state required before test execution. a set of inputs to the system Postfix values: values that need to be sent to the software after test execution. The expected results

Testing Terminology Test Point Test suite Domain A specific value for test case input and state variables. Domain Set of values that input or state variables of IUT may take. Heuristics for test point selection Equivalence classes – one values in set represents all values in set. Partition testing techniques Boundary value analysis Special values testing Test suite A collection of test cases.

Testing Terminology Fault model Test strategy Test model Test design A description about where faults are likely to occur in a program. Test strategy An algorithm or heuristic for creating test cases from a representation, an implementation or a test model Test model A description of relationships between elements of a representation or implementation. Test design The process of creating a test suite using a test strategy. Test effectiveness: relative ability of testing strategy to find bugs. Test efficiency: relative cost of finding bugs.

Testing Terminology: Testing Strategies Responsibility-based test design Behavioural, functional, black-box, etc. testing Implementation-based test design Relies on source code,, e.g., white-box Fault-based testing

Testing Strategy We begin by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’ For conventional software The module (component) is our initial focus Integration of modules follows For OO software “testing in the small”  OO class Has attributes and operations and implies communication and collaboration

Testing Strategy unit test integration test Black Box Testing White Box Testing system test validation test

A Classification of Testing: The V Model Requirements Analysis Acceptance Test Architectural Design System Test Subsystem Design Integration Test Detailed Design Module Test Implementation Unit Test

Unit Testing module to be tested software engineer results module to be tested test cases interface local data structures boundary conditions independent paths error handling paths The module being tested should be reviewed in context of the requirements specification The units comprising a system are individually tested The code is examined for faults in algorithms, data and syntax A set of test cases is formulated and input and the results are evaluated

Unit Test Environment test cases RESULTS driver Module stub stub interface local data structures Module boundary conditions independent paths error handling paths stub stub test cases RESULTS

Integration Testing The goal is to ensure that groups of components work together as specified in the requirements document Four kinds of integration tests exist Structure tests Functional tests Stress tests Performance tests

Integration Testing Strategies Options: • the “big bang” approach • an incremental construction strategy

Top Down Integration A top module is tested with stubs B F G stubs are replaced one at a time, "depth first" C as new modules are integrated, some subset of tests is re-run D E

Bottom-Up Integration F G drivers are replaced one at a time, "depth first" C worker modules are grouped into builds and integrated D E cluster

Sandwich Testing cluster A Top modules are tested with stubs B F G C Worker modules are grouped into builds and integrated D E cluster

Important Terms Validation & Verification (IEEE) Validation : The process of evaluating software at the end of software development to ensure compliance with intended usage Verification : The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase IV&V stands for “independent verification and validation” Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Who does testing: Test Engineer & Test Managers Test Engineer : An IT professional who is in charge of one or more technical test activities designing test inputs producing test values running test scripts analyzing results reporting results to developers and managers Test Manager : In charge of one or more test engineers sets test policies and processes interacts with other managers on the project otherwise helps the engineers do their work Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Test Engineer Activities Manager Test Designs Executable Tests design instantiate Test Engineer Test Engineer P Computer execute Evaluate Output Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Static and Dynamic Testing Static Testing : Testing without executing the program This include software inspections and some forms of analyses Very effective at finding certain kinds of problems – especially “potential” faults, that is, problems that could lead to faults when the program is modified Dynamic Testing : Testing by executing the program with real inputs Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

System Testing The goal is to ensure that all elements that interact to solve the problem do so correctly. Software, databases, hardware, people, procedures. Overall functionality and performance must be achieved.

Validation Testing The goal is to ensure that the system actually does what the customer expects it to do – fulfils requirements Testing is carried out by customers mimicking real world activities Customers should also intentionally enter erroneous values to determine the system behavior in those instances

Black Box Testing The tester knows nothing about the internal structure of the code Test cases are formulated based on expected output of methods Tester generates test cases to represent all possible situations in order to ensure that the observed and expected behavior is the same

White Box Testing The tester uses knowledge of the programming constructs to determine the test cases to use If one or more loops exist in a method, the tester would wish to test the execution of this loop for 0, 1, max, and max + 1, where max represents a possible maximum number of iterations Similarly, conditions would be tested for true and false

High Order Testing Validation testing System testing Focus is on software requirements System testing Focus is on system integration Alpha/Beta testing Focus is on customer usage Recovery testing forces the software to fail in a variety of ways and verifies that recovery is properly performed Security testing verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume Performance Testing test the run-time performance of software within the context of an integrated system

Test-Case Exercise …From Myers classic The Art of Software Testing (1978): Devise a test plan for a program that: reads three integer values from a card. The three values are interpreted as representing the length of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.

The Polygon Class abstract class Polygon { abstract void draw(int r, int g, int b); /* color closed area*/ abstract void erase(); /* set to background rgb */ abstract float area(); /* return area */ abstract float perimeter(); /* return sum of sides */ abstract Point center(); /* return centroid pixel */ }

The Triangle Class class Triangle extends Polygon { public Triangle(LineSegment a, LineSegment b, LineSegment c){…} public void setA(LineSegment a) {…} public void setB(LineSegment b) {…} public void setC(LineSegment c) {…} public LineSegment getA() {…} public LineSegment getB() {…} public LineSegment getC() {…} public boolean is_isoceles(){…} public boolean is_scalene(){…} public boolean is_equilateral(){…} public void draw(int r, int g, int b){…} public void erase(){…} public float area(){…} public float perimeter(){…} public Point center(){…} }

The LineSegment Class class LineSegment extends Figure { public LineSegment(Point x1, Point y1, Point x2, Point y2){…} public void setx1(Point x1) {…} public void sety1(Point y1) {…} public void setx2(Point x2) {…} public void sety2(Point y2) {…} public Point getx1() {…} public Point gety1() {…} public Point getx2() {…} public Point gety2() {…} }

Test-Case Exercise Specify as many test cases as you can for the Triangle class defined above. The class is used to determine if a triangle is: isosceles, equilateral or scalene. A triangle is isosceles if two of its sides are equal. A triangle is equilateral if all its sides are equal. A triangle is scalene if its sides are of different sizes.

Test-Case Exercise A valid triangle must meet two conditions: No side may have a length of zero. Each side must be smaller than the sum of all sides divided by two. i.e. Given s = (a + b + c)/2, then a < s, b < s, c < s must hold. Stated differently, a < s && b < s && c < s  a < b + c b < a + c and c < a + b.

Test-Case Exercise Specify as many test cases as you can for the Triangle class defined above. Test Case Description Test Case Input a b c Expected Output 1 Valid isosceles triangle 3 4 Isosceles 2 28 29 30 31 32 33 34 35

Solution To Test Case Exercise

Solution To Test Case Exercise

Solution To Test Case Exercise

Solution To Test Case Exercise

White Box Testing

Control-flow graph A control flow graph of a program is a graph that shows the paths that may be traversed when the program is executed.

Control-flow Graph int n = 0; do{ if(n++ % 2 == 1) dBase[i]++; else 3 4 5 6 7 8 int n = 0; do{ if(n++ % 2 == 1) dBase[i]++; else if(dBase[i] > 100) dBase[i] -= n; dBase[i] += n; }while(n < 20); 1 2 4 3 5 6 7

Cyclomatic Complexity – V(G) If G is the control flow graph of program P and G has e edges and n nodes, then the cyclomatic complexity of P, V(G) is given by: V(G) = e – n + 2 Or V(G) = the number of decision nodes + 1

Cyclomatic Complexity – V(G) 1 2 3 4 5 6 7 8 cyclomatic complexity V(G) = e – n + 2 = 11 – 7 = 4 Or V(G) = the number of decision nodes + 1 = 3 + 1 = 4

Cyclomatic Complexity A number of industry studies have indicated that the higher V(G), the higher the probability or errors. modules V(G) modules in this range are more error prone

Cyclomatic Complexity – V(G) 1 2 3 4 5 6 7 8 The cyclomatic complexity is a measure of the number of linearly independent paths in the graph. An independent path must move along at least one edge not previously traversed What are the independent paths for this graph? 1-2-3-5-7-8 1-2-3-6-7-8 1-2-4-7-8 1-2-4-7-2…8

Logic/White-box testing: Code coverage White box testing methods emphasize code coverage Statement coverage Every statement is executed at least once Branch coverage Every branch is exercised at least once i.e. every path from a node is traversed at least once Once true and once false evaluation Multiple condition coverage All combinations of simple conditions are exercised Basis-path model Exercise V(G) independent paths Data flow coverage Evaluation of state of instance variables.

Statement Coverage 1 2 3 4 5 6 7 8 Write enough test cases to execute each statement al least once, 1-2-3-5-7-8 1-2-3-6-7-8 1-2-4-7-8 May miss bugs for complex predicates and loops

Branch/Predicate Coverage 1 2 3 4 5 6 7 8 Determine number of branches Compute path for each branch 1-2-3-5-7-2 1-2-3-6-7-8 1-2-4-7-8 Write test cases to exercise each path

Combinatorial/Multiple Condition Coverage 1 2 3 4 5 6 7 8 Determine true/false combinations for conditions Write test cases to exercise each combination

Combinatorial/Multiple Condition Coverage How many test cases do we need to test this if statement? if(num > 5 || isValid(num) && (sum-num<findMax(num) || sum = 100) && isDivisible(num, 7)

Combinatorial/Multiple Condition Coverage if(num > 5 || isValid(num) && (sum-num<findMax(num) || sum = 100) && isDivisible(num, 7) Test Case # num > 5 isValid(num) (sum-num) < findMax(num) sum=100 isDiv (num, 7) 1 F 2 T 3 4 5 6 7 8 9 10 11

Basis-path Testing Compute the cyclomatic complexity 1 2 3 4 5 6 7 8 Compute the cyclomatic complexity Derive independent paths 1-2-3-5-7-8 1-2-3-6-7-8 1-2-4-7-8 1-2-4-7-2…7-8 Write test cases to exercise each path

Example 1 V(G) = 2 + 1 = 3 Statement coverage requires 2 test cases Branch coverage requires 2 test cases Basis-path coverage requires 3 test cases There are four distinct entry-exit paths 1-2-4-6-7 1-2-4-5-7 1-3-4-5-7 1-3-4-6-7 Which should be chosen? 1 2 4 3 5 6 7

Example 2 A D What is V(G)? How many test cases are required for statement coverage? How many test cases are required for branch coverage? How many test cases are required for basis-path coverage? How many unique entry-exit paths exist? F E K L

Introduction to Mutation Testing & Program Perturbation

Mutation Testing Faults are introduced into a program by creating many versions of the program Each version is called a mutant. Each mutant contains a single fault. Test cases are applied to the original program and to the mutant program. The goal is to cause the mutant program to fail, thus demonstrating the effectiveness of the test case.

Example of a Program Mutation 1 int max(int x, int y) 2 { 3 int mx = x; 4 if (x > y) 5 mx = x; 6 else 7 mx = y; 8 return mx; 9 } 1 int max(int x, int y) 2 { 3 int mx = x; 4 if (x < y) 5 mx = x; 6 else 7 mx = y; 8 return mx; 9 }

Mutation Testing Process Process (cont.) Figure 2. Mutation Testing Flowchart, Source: Mutation 2000, pg 3

Terminology Mutation testing Mutation The creation of a set of mutant programs of the program being tested. Mutation a single syntactic change that is made to a program statement. Each mutant differs from the original program by one mutation.

Terminology Mutant Mutant Killed Mutation Score Mutation Operator A faulty program; the original with one fault introduced Mutant Killed Test suite detects the mutant Mutation Score #of killed mutants/ # of non-equivalent mutants Mutation Operator The function that changes the source code to create a mutant

Mutation Operators Replace each operand with every other syntactically correct operand Modify expressions by replacing operators and inserting new operators Delete entire statements

Categories of Mutation Operators Operand Replacement Operators: Replace a single operand with another operand or constant. E.g., if(x > y) if (5 > y) Replacing x by constant 5. if (x > 5) Replacing y by constant 5. if (y > x) Replacing x and y with each other. E.g., if all operators are {+,-,*,**,/} then the following expression a = b * (c - d) will generate 8 mutants: 4 by replacing * 4 by replacing -.

Categories of Mutation Operators Expression Modification Operators: Replace an operator or insert new operators. E.g., if (x == y) if (x >= y) Replacing == by >=. if (x == ++y) Inserting ++.

Categories of Mutation Operators Statement Modification Operators: E.g., Delete the else part of the if-else statement. Delete the entire if-else statement. Replace line 3 by a return statement. 1 int max(int x, int y) 2 { 3 int mx = x; 4 if (x > y) 5 mx = x; 6 else 7 mx = y; 8 return mx; 9 } return x;

Mutation System Example The Mothra mutation system for FORTRAN77 supports 22 mutation operators. E.g., absolute value insertion constant for array reference replacement GOTO label replacement RETURN statement replacement statement deletion unary operator insertion logical connector replacement comparable array name replacement

Why Does Mutation Testing Work? The operators are limited to simple single syntactic changes on the basis of the competent programmer hypothesis.

The Competent Programmer Hypothesis Programmers are generally very competent and do not create “random” programs. For a given problem, a programmer, if mistaken, will create a program that is very close to a correct program. An incorrect program can be created from a correct program by making some minor change to the correct program.

Mutation Testing Algorithm Generate program test cases. Run each test case against the original program. If the output is incorrect, the program must be modified and re-tested. If the output is correct go to the next step ... Construct mutants E.g. using a tool like Mothra.

Mutation Testing Algorithm (Cont’d) Execute each test case against each alive mutant. If the output of the mutant differs from the output of the original program, the mutant is considered incorrect and is killed. Two kinds of mutants survive: Functionally equivalent to the original program: Cannot be killed. Killable: Test cases are insufficient to kill the mutant. New test cases must be created.

Mutation Score The mutation score for a set of test cases is the percentage of non-equivalent mutants killed by the test data. Mutation Score = 100 * D / (N - E) D = Dead mutants N = Number of mutants E = Number of equivalent mutants A set of test cases is mutation adequate if its mutation score is 100%.

Program Perturbation Program Perturbation is a technique to test a program’s robustness. It is based on unexpectedly changing the values of program data during run-time.

The Perturbation Function The perturbation function is a mathematical function that: takes a data state as its input; changes the data state according to some specified criteria; produces a modified data state as output.

Program Perturbation Example Assume the following perturbation function: 1. int perturbation (int x) 2. { 3. int newX; 4. newX = x + 20; 5. return newX; 6. }

Example of a Fault Injection 1. main() 2. { 3. int x; 4. x = ReadInt(); 4.1 x = perturbation(x); 5. if (x > 0) 6. printf(“X positive”); 7. else 8. printf(“X negative”); 9. } 1. main() 2. { 3. int x; 4. x = ReadInt(); 5. if (x > 0) 6. printf(“X positive”); 7. else 8. printf(“X negative”); 9. }

Model-Driven Test Design refined requirements / test specs model / structure test requirements DESIGN ABSTRACTION LEVEL IMPLEMENTATION ABSTRACTION LEVEL software artifact input values pass / fail test results test scripts test cases © Ammann & Offutt

Model-Driven Test Design – Steps criterion refine refined requirements / test specs model / structure test requirements generate analysis DESIGN ABSTRACTION LEVEL IMPLEMENTATION ABSTRACTION LEVEL software artifact input values prefix postfix expected evaluate execute automate pass / fail test results test scripts test cases © Ammann & Offutt

Model-Driven Test Design – Activities refined requirements / test specs Test Design model / structure test requirements DESIGN ABSTRACTION LEVEL Raising our abstraction level makes test design MUCH easier IMPLEMENTATION ABSTRACTION LEVEL software artifact input values Test Automation Test Evaluation pass / fail test results Test Execution test scripts test cases © Ammann & Offutt

Types of Test Activities Testing can be broken up into four general types of activities Test Design Test Automation Test Execution Test Evaluation Each type of activity requires different skills, background knowledge, education and training No reasonable software development organization uses the same people for requirements, design, implementation, integration and configuration control 1.a) Criteria-based 1.b) Human-based Why do test organizations still use the same people for all four test activities?? This is clearly a waste of resources © Ammann & Offutt

1. Test Design – (a) Criteria-Based Design test values to satisfy coverage criteria or other engineering goal This is the most technical job in software testing Requires knowledge of : Discrete math Programming Testing Requires much of a traditional CS degree This is intellectually stimulating, rewarding, and challenging Test design is analogous to software architecture on the development side Using people who are not qualified to design tests is a sure way to get ineffective tests © Ammann & Offutt

1. Test Design – (b) Human-Based Design test values based on domain knowledge of the program and human knowledge of testing This is much harder than it may seem to developers Criteria-based approaches can be blind to special situations Requires knowledge of : Domain, testing, and user interfaces Requires almost no traditional CS A background in the domain of the software is essential An empirical background is very helpful (biology, psychology, …) A logic background is very helpful (law, philosophy, math, …) This is intellectually stimulating, rewarding, and challenging But not to typical CS majors – they want to solve problems and build things © Ammann & Offutt

Embed test values into executable scripts 2. Test Automation Embed test values into executable scripts This is slightly less technical Requires knowledge of programming Fairly straightforward programming – small pieces and simple algorithms Requires very little theory May be boring for test designers Programming is out of reach for many domain experts Who is responsible for determining and embedding the expected outputs ? Test designers may not always know the expected outputs Test evaluators need to get involved early to help with this © Ammann & Offutt

Run tests on the software and record the results 3. Test Execution Run tests on the software and record the results This is easy – and trivial if the tests are well automated Requires basic computer skills Interns Employees with no technical background Asking qualified test designers to execute tests is a sure way to convince them to look for a development job If, for example, GUI tests are not well automated, this requires a lot of manual labor Test executors have to be very careful and meticulous with bookkeeping © Ammann & Offutt

Evaluate results of testing, report to developers 4. Test Evaluation Evaluate results of testing, report to developers This is much harder than it may seem Requires knowledge of : Domain Testing User interfaces and psychology Usually requires almost no traditional CS A background in the domain of the software is essential An empirical background is very helpful (biology, psychology, …) A logic background is very helpful (law, philosophy, math, …) This is intellectually stimulating, rewarding, and challenging But not to typical CS majors – they want to solve problems and build things © Ammann & Offutt

Caveat: Impact of New Tools and Techniques Naw - I already don’t plow as good as I know how... They’re teaching a new way of plowing over at the Grange tonight - you going? “Knowing is not enough, we must apply. Willing is not enough, we must do.” Goethe Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Testing in the 21st Century We are going through a time of change Software Defines Behavior network routers financial networks telephone switching networks other infrastructure Embedded Control Applications airplanes, air traffic control spaceships watches ovens remote controllers Safety critical, real-time software Testing ideas have matured enough to be used in practice PDAs memory seats DVD players garage door openers cell phones © Ammann & Offutt

Junit Overview Test Case Design From: http://www.google.com/url?sa=t&source=web&cd=1&sqi=2&ved=0CBcQFjAA&url=http%3A%2F%2Fwww2.informatik.hu-berlin.de%2F~hs%2FLehre%2F2004-WS_SWQS%2F20041029_Ex_JUnit.ppt&rct=j&q=ppt%20junit%20testing&ei=Y7yUTf7EEe2C0QHO7vn0Cw&usg=AFQjCNHiSKk_Md3YoqEUBez54lS0MJ1Evw&cad=rja and other sources. Test Case Design Junit Overview

JUnit 3

JUnit JUnit is a framework for writing unit tests From: http://www.google.com/url?sa=t&source=web&cd=1&sqi=2&ved=0CBcQFjAA&url=http%3A%2F%2Fwww2.informatik.hu-berlin.de%2F~hs%2FLehre%2F2004-WS_SWQS%2F20041029_Ex_JUnit.ppt&rct=j&q=ppt%20junit%20testing&ei=Y7yUTf7EEe2C0QHO7vn0Cw&usg=AFQjCNHiSKk_Md3YoqEUBez54lS0MJ1Evw&cad=rja JUnit JUnit is a framework for writing unit tests A unit test is a test of a single class A test case is a single test of a single method A test suite is a collection of test cases Unit testing is particularly important when software requirements change frequently Code often has to be refactored to incorporate the changes Unit testing helps ensure that the refactored code continues to work Satish Mishra Unit testing with JUnit

JUnit.. JUnit helps the programmer: Define and execute tests and test suites Formalize requirements and clarify architecture Write and debug code Integrate code and always be ready to release a working version Satish Mishra Unit testing with JUnit

What JUnit does JUnit runs a suite of tests and reports results For each test in the test suite: JUnit calls setUp() This method should create any objects you may need for testing Satish Mishra Unit testing with JUnit

What JUnit does… JUnit calls one test method The test method may comprise multiple test cases; that is, it may make multiple calls to the method you are testing In fact, since it’s your code, the test method can do anything you want The setUp() method ensures you entered the test method with a virgin set of objects; what you do with them is up to you JUnit calls tearDown() This method should remove any objects you created Satish Mishra Unit testing with JUnit

Creating a test class in JUnit Define a subclass of TestCase Override the setUp() method to initialize object(s) under test. Override the tearDown() method to release object(s) under test. Define one or more public testXXX() methods that exercise the object(s) under test and assert expected results. Define a static suite() factory method that creates a TestSuite containing all the testXXX() methods of the TestCase. Optionally define a main() method that runs the TestCase in batch mode. Satish Mishra Unit testing with JUnit

protected void setUp() { …} Fixtures A fixture is just a some code you want run before every test You get a fixture by overriding the method protected void setUp() { …} The general rule for running a test is: protected void runTest() { setUp(); <run the test> tearDown(); } so we can override setUp and/or tearDown, and that code will be run prior to or after every test case Satish Mishra Unit testing with JUnit

Implementing setUp() method Override setUp() to initialize the variables, and objects Since setUp() is your code, you can modify it any way you like (such as creating new objects in it) Reduces the duplication of code Satish Mishra Unit testing with JUnit

Implementing the tearDown() method In most cases, the tearDown() method doesn’t need to do anything The next time you run setUp(), your objects will be replaced, and the old objects will be available for garbage collection Like the finally clause in a try-catch-finally statement, tearDown() is where you would release system resources (such as streams) Satish Mishra Unit testing with JUnit

The structure of a test method A test method doesn’t return a result If the tests run correctly, a test method does nothing If a test fails, it throws an AssertionFailedError The JUnit framework catches the error and deals with it; you don’t have to do anything Satish Mishra Unit testing with JUnit

Test suites In practice, you want to run a group of related tests (e.g. all the tests for a class) To do so, group your test methods in a class which extends TestCase Running suites we will see in examples Satish Mishra Unit testing with JUnit

assertX methods static void assertTrue(boolean test) static void assertFalse(boolean test) assertEquals(expected, actual) assertSame(Object expected, Object actual) assertNotSame(Object expected, Object actual) assertNull(Object object) Satish Mishra Unit testing with JUnit

assertX methods assertNotNull(Object object) fail() All the above may take an optional String message as the first argument, for example, static void assertTrue(String message, boolean test) Satish Mishra Unit testing with JUnit

Organize The Tests Create test cases in the same package as the code under test For each Java package in your application, define a TestSuite class that contains all the tests for validating the code in the package Define similar TestSuite classes that create higher-level and lower-level test suites in the other packages (and sub-packages) of the application Make sure your build process includes the compilation of all tests Satish Mishra Unit testing with JUnit

JUnit framework Satish Mishra Unit testing with JUnit

Example code 1 import junit.framework.TestCase; public class JunitExamples extends TestCase { SortingComparison obj;//instance variable in the fixture public JunitExamples(String name) { super(name); } public void setUp() { obj = new SortingComparison(); public void tearDown () { //do something public void testBubble(){ //SortingComparison obj = new SortingComparison(); int[] qArray = {1,9,8,7,600000,5,4,3,2,10,0}; int[] rArray = {0,1,2,3,4,5,7,8,9,10,600000}; obj.bubbleSort(qArray); obj.printArray(qArray); obj.printArray(rArray); assertTrue(java.util.Arrays.equals(qArray, rArray)); }//end of method }//end of class

Example: Counter class For the sake of example, we will create and test a trivial “counter” class The constructor will create a counter and set it to zero The increment method will add one to the counter and return the new value The decrement method will subtract one from the counter and return the new value Satish Mishra Unit testing with JUnit

The Counter class itself public class Counter { int count = 0; public int increment() { return ++count; } public int decrement() { return --count; } public int getCount() { return count; } } Satish Mishra Unit testing with JUnit

JUnit tests for Counter public class CounterTest extends junit.framework.TestCase { Counter counter1; public CounterTest() { } // default constructor protected void setUp() { // creates a (simple) test fixture counter1 = new Counter(); } protected void tearDown() { } // no resources to release Satish Mishra Unit testing with JUnit

JUnit tests for Counter… public void testIncrement() { assertTrue(counter1.increment() == 1); assertTrue(counter1.increment() == 2); } public void testDecrement() { assertTrue(counter1.decrement() == -1); } } // End from last slide Satish Mishra Unit testing with JUnit

JUnit tests for Counter public class CounterTest extends junit.framework.TestCase { Counter counter1; public CounterTest() { } // default constructor protected void setUp() { // creates a (simple) test fixture counter1 = new Counter(); } public void testIncrement() { assertTrue(counter1.increment() == 1); assertTrue(counter1.increment() == 2); } public void testDecrement() { assertTrue(counter1.decrement() == -1); } } Note that each test begins with a brand new counter This means you don’t have to worry about the order in which the tests are run

Why JUnit Allow you to write code faster while increasing quality Elegantly simple Check their own results and provide immediate feedback Tests is inexpensive Increase the stability of software Developer tests Written in Java Free Gives proper uniderstanding of unit testing Satish Mishra Unit testing with JUnit

A Case Study class Money { private int fAmount; private String fCurrency;    public Money(int amount, String currency) {   fAmount= amount;         fCurrency= currency;     } public int amount() {         return fAmount;     }     public String currency() {         return fCurrency;     } public Money add(Money m) {     return new Money(amount()+m.amount(), currency()); } }

How to Write A TestCase (2) Exercises the objects in the fixture. public class MoneyTest extends TestCase {     //…     public void testSimpleAdd() {         Money m12CHF= new Money(12, "CHF");  // (1)         Money m14CHF= new Money(14, "CHF");                  Money expected= new Money(26, "CHF");         Money result= m12CHF.add(m14CHF);    // (2) Assert.assertTrue(expected.equals(result));     // (3)     } (1) Creates the objects we will interact with during the test. This testing context is commonly referred to as a test's fixture. All we need for the testSimpleAdd test are some Money objects. (2) Exercises the objects in the fixture. (3) Verifies the result

Assert assertEquals(expected, actual) assertEquals(message, expected, actual) assertEquals(expected, actual, delta) assertEquals(message, expected, actual, delta) assertFalse(condition) assertFalse(message, condition) AssertNotNull(object) AssertNotNull(message, object) AssertNotSame(expected, actual) AssertNotSame(message, expected, actual) assertTrue(condition) assertTrue(message, condition)

Structure setUp() Storing the fixture's objects in instance variables of your TestCase subclass and initialize them by overriding the setUp method tearDown() Releasing the fixture’s run() Defining how to run an individual test case. Defining how to run a test suite. testCase()

Structure of Writing A Test public class MoneyTest extends TestCase {     private Money f12CHF;     private Money f14CHF;         protected void setUp() {         f12CHF= new Money(12, "CHF");         f14CHF= new Money(14, "CHF");     } public void testSimpleAdd() {     Money expected= new Money(26, "CHF");     Money result= f12CHF.add(f14CHF);     Assert.assertTrue(expected.equals(result)); } TestCase test= new MoneyTest("simple add") {     public void runTest() {         testSimpleAdd();     } }

JUnit 4 with Annotations

Start to Use it 2. Installation unzip the junit.zip file 1. Download the latest version of JUnit from http://download.sourceforge.net/junit/ 2. Installation unzip the junit.zip file add junit.jar to the CLASSPATH. For example: set classpath=%classpath%;INSTALL_DIR\junit3\junit.jar 3. Testing Test the installation by using either the batch or the graphical TestRunner tool to run the tests that come with this release. All the tests should pass OK. for the batch TestRunner type:     java junit.textui.TestRunner junit.samples.AllTests for the graphical TestRunner type:     java junit.awtui.TestRunner junit.samples.AllTests for the Swing based graphical TestRunner type:     java junit.swingui.TestRunner junit.samples.AllTests Notice: The tests are not contained in the junit.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path Important: Don't install the junit.jar into the extension directory of your JDK installation. If you do so the test class on the files system will not be found. JUnit plug-in for Eclipse

Running JUnit java junit.textui.TestRunner JavaClassfileName java junit.swingui.TestRunner JavaClassfileName java junit.awtui.TestRunner JavaClassfileName Use eclipse

Eclipse plug-in http://dev.eclipse.org/viewcvs/index.cgi/jdt-ui-home/plugins/org.eclipse.jdt.junit/org.eclipse.pde.junit_3.0.0.zip

This will be filled in automatically JUnit in Eclipse To create a test class, select File New Other...  Java, JUnit, TestCase and enter the name of the class you will test Fill this in This will be filled in automatically

Running JUnit Second, use this pulldown menu First, select a Test class Third, Run As  JUnit Test

Results Your results are here

JUnit 4

Comparing JUnit 3 to JUnit 4 All the old assertXXX methods are the same Most things are about equally easy JUnit 4 makes it easier to test that exceptions are thrown when they should be JUnit 4 can still run JUnit 3 tests JUnit 4 provides protection against infinite loops JUnit 4 has some additional features

Migrating from JUnit 3 JUnit 4 requires Java 5 or newer Don’t extend junit.framework.TestCase; just use an ordinary class Import org.junit.* and org.junit.Assert.* Use a static import for org.junit.Assert.* Static imports replace inheritance from junit.framework.TestCase Use annotations instead of special method names: Instead of a setUp method, put @Before before some method Instead of a tearDown method, put @After before some method Instead of beginning test method names with ‘test’, put @Test before each test method

Writing a JUnit test class, I Start by importing the JUnit 4 classes you need import org.junit.*; import static org.junit.Assert.*; Declare your class in the usual way public class MyProgramTest { Declare any variables you are going to use frequently, typically including an instance of the class being tested MyProgram program; int [] array; int solution;

Writing a JUnit test class, II If you wish, you can declare one method to be executed just once, when the class is first loaded This is for expensive setup, such as connecting to a database @BeforeClass public static void setUpClass() throws Exception { // one-time initialization code } If you wish, you can declare one method to be executed just once, to do cleanup after all the tests have been completed @AfterClass public static void tearDownClass() throws Exception { // one-time cleanup code }

Writing a JUnit test class, III You can define one or more methods to be executed before each test; typically such methods initialize values, so that each test starts with a fresh set @Before public void setUp() { program = new MyProgram(); array = new int[] { 1, 2, 3, 4, 5 }; } You can define one or more methods to be executed after each test; typically such methods release resources, such as files @After public void tearDown() { }

@Before and @After methods You can have as many @Before and @After methods as you want Be warned: You don’t know in what order they will execute You can inherit @Before and @After methods from a superclass; execution is as follows: Execute the @Before methods in the superclass Execute the @Before methods in this class Execute a @Test method in this class Execute the @After methods in this class Execute the @After methods in the superclass

Writing a JUnit test class, IV A test method is annotated with @Test, takes no parameters, and returns no result All the usual assertXXX methods can be used @Test public void sum() { assertEquals(15, program.sum(array)); assertTrue(program.min(array) > 0); }

Special features of @Test You can limit how long a method is allowed to take This is good protection against infinite loops The time limit is specified in milliseconds The test fails if the method takes too long @Test (timeout=10) public void greatBig() { assertTrue(program.ackerman(5, 5) > 10e12); } Some method calls should throw an exception You can specify that a particular exception is expected The test will pass if the expected exception is thrown, and fail otherwise @Test (expected=IllegalArgumentException.class) public void factorial() { program.factorial(-5); }

Parameterized tests Using @RunWith(value=Parameterized.class) and a @Parameters method, you can run the same tests with multiple datasets @RunWith(value=Parameterized.class) public class FactorialTest { private long expected; private int value; @Parameters public static Collection data() { return Arrays.asList( new Object[ ][ ] { { 1, 0 }, { 1, 1 }, { 2, 2 }, { 120, 5 } }); } public FactorialTest(long expected, int value) { // constructor this.expected = expected; this.value = value; } @Test public void factorial() { assertEquals(expected, new Calculator().factorial(value)); } } Source: http://today.java.net/pub/a/today/2006/12/07/junit-reloaded.html

Ignoring a test The @Ignore annotation says to not run a test @Ignore("I don’t want Dave to know this doesn’t work") @Test public void add() { assertEquals(4, program.sum(2, 2)); } You shouldn’t use @Ignore without a very good reason!

Test suites As before, you can define a suite of tests @RunWith(value=Suite.class) @SuiteClasses(value={MyProgramTest.class, AnotherTest.class}) public class AllTests { … }

Other stuff Failed tests now throw an AssertionError, rather than JUnit 3’s AssertionFailedError There is now a version of assertEquals for arrays of objects: assertEquals(Object[] expected, Object[] actual) Unfortunately, there is still no assertEquals for arrays of primitives JUnit 3 had an assertEquals(p, p) method for each kind of primitive p, but JUnit 4 only has an assertEquals(object, object) and depends on autoboxing

A gotcha The following method: with the following test: gives: long sum(long x, long y) { return x + y; } with the following test: @Test public void sum() { assertEquals(4, s.sum(2, 2)); } gives: expected: <4> but was: <4> This is due to your friend, autoboxing assertEquals no longer exists for primitives, only for objects Hence, the 4 is autoboxed to an Integer, while sum returns a long The error message means: expected int 4, but got long 4 To make this work, change the 4 to a 4L

JUnit 4 in Eclipse and NetBeans As usual, the easiest way to create a test class is just to let your IDE do it for you Here is the recommended test-driven approach Create a class containing all the “stub” methods you initially think you will need Have the IDE create the test class, with all the test methods Repeat: Write a test Make sure the test fails Write the method being tested Make sure the test now succeeds Note: When you create the test class, NetBeans in particular puts a lot of garbage lines into each test method; you can just delete these and put in your own code

Summary? JUnit Exercise

More Information http://www.junit.org Download of JUnit Lots of information on using JUnit http://sourceforge.net/projects/cppunit C++ port of Junit http://www.thecoadletter.com Information on Test-Driven Development

CSC550, Devon M. Simmonds, Computer Science Department, University of North Carolina Wilmington Q u e s t i o n ? The End

Why is testing so difficult? 1 3 2 5 4 loop < =20 X a b c d e There are about 1014 possible paths How long would it take to execute all paths? About 3,170 years @ one test/millisecond

Example 2 Similar for right side What is V(G)? - 7 How many test cases are required for statement coverage? - 4 How many test cases are required for branch coverage? - 4 How many test cases are required for basis-path coverage? - 7 How many unique entry-exit paths exist? – 16 D F E K Similar for right side L ABDEGKMQS ABDEGKNQS ABDEHKMQS ABDEHKNQS ACDEGKMQS ACDEGKNQS ACDEHKMQS ACDEHKNQS

White Box Testing

White-box and Black-box Testing Black-box testing : Deriving tests from external descriptions of the software, including specifications, requirements, and design White-box testing : Deriving tests from the source code internals of the software, specifically including branches, individual conditions, and statements This view is really out of date. The more general question is: from what level of abstraction do we derive tests? Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Changing Notions of Testing Old view of testing is of testing at specific software development phases Unit, module, integration, system … New view is in terms of structures and criteria Graphs, logical expressions, syntax, input space Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Old : Testing at Different Levels System testing: Test the overall functionality of the system Acceptance testing: Is the software acceptable to the user? main Class P Module testing: Test each class, file, module or component Integration testing: Test how modules interact with each other Class A method mA1() method mA2() Class B method mB1() method mB2() Unit testing: Test each unit (method) individually Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Old : Find a Graph and Cover It Tailored to: a particular software artifact code, design, specifications a particular phase of the lifecycle requirements, specification, design, implementation This viewpoint obscures underlying similarities Graphs do not characterize all testing techniques well Four abstract models suffice … Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

New : Test Coverage Criteria A tester’s job is simple : Define a model of the software, then find ways to cover it Test Requirements : Specific things that must be satisfied or covered during testing Test Criterion : A collection of rules and a process that define test requirements Testing researchers have defined dozens of criteria, but they are all really just a few criteria on four types of structures … Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

New : Criteria Based on Structures Structures : Four ways to model software Graphs Logical Expressions Input Domain Characterization Syntactic Structures (not X or not Y) and A and B A: {0, 1, >1} B: {600, 700, 800} C: {swe, cs, isa, infs} if (x > y) z = x - y; else z = 2 * x; Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

1. Graph Coverage – Structural 6 5 3 2 1 7 4 Node (Statement) Cover every node 12567 1343567 Edge (Branch) Cover every edge 12567 1343567 1357 Path Cover every path 12567 1257 13567 1357 1343567 134357 … This graph may represent statements & branches methods & calls components & signals states and transitions Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

1. Graph Coverage – Data Flow This graph contains: defs: nodes & edges where variables get values uses: nodes & edges where values are accessed def = {x, y} def = {a , m} def = {a} def = {m} 6 5 3 2 1 7 4 use = {x} use = {a} use = {y} use = {m} Defs & Uses Pairs (x, 1, (1,2)), (x, 1, (1,3)) (y, 1, 4), (y, 1, 6) (a, 2, (5,6)), (a, 2, (5,7)), (a, 3, (5,6)), (a, 3, (5,7)), (m, 2, 7), (m, 4, 7), (m, 6, 7) All Defs Every def used once 1, 2, 5, 6, 7 1, 2, 5, 7 1, 3, 4, 3, 5, 7 All Uses Every def “reaches” every use 1, 2, 5, 6, 7 1, 2, 5, 7 1, 3, 5, 6, 7 1, 3, 5, 7 1, 3, 4, 3, 5,7 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

1. Graph - FSM Example Memory Seats in a Lexus ES 300 Guard (safety constraint) Trigger (input) Driver 1 Configuration Driver 2 [Ignition = off] | Button2 [Ignition = off] | Button1 sideMirrors () [Ignition = on] | Modified Configuration lumbar () [Ignition = on] | (to Modified) seatBottom () [Ignition = on] | seatBack () [Ignition = on] | Ignition = off New Configuration Driver 1 Driver 2 [Ignition = on] | Reset AND Button1 [Ignition = on] | Reset AND Button2 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

( (a > b) or G ) and (x < y) 2. Logical Expressions ( (a > b) or G ) and (x < y) Transitions Logical Expressions Program Decision Statements Software Specifications Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

( (a > b) or G ) and (x < y) 2. Logical Expressions ( (a > b) or G ) and (x < y) Predicate Coverage : Each predicate must be true and false ( (a>b) or G ) and (x < y) = True, False Clause Coverage : Each clause must be true and false (a > b) = True, False G = True, False (x < y) = True, False Combinatorial Coverage : Various combinations of clauses Active Clause Coverage: Each clause must determine the predicate’s result Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

2. Logic – Active Clause Coverage ( (a > b) or G ) and (x < y) 1 T F T 2 F F T With these values for G and (x<y), (a>b) determines the value of the predicate duplicate 3 F T T 4 F F T 5 T T T 6 T T F Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

3. Input Domain Characterization Describe the input domain of the software Identify inputs, parameters, or other categorization Partition each input into finite sets of representative values Choose combinations of values System level Number of students { 0, 1, >1 } Level of course { 600, 700, 800 } Major { swe, cs, isa, infs } Unit level Parameters F (int X, int Y) Possible values X: { <0, 0, 1, 2, >2 }, Y : { 10, 20, 30 } Tests F (-5, 10), F (0, 20), F (1, 30), F (2, 10), F (5, 20) Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

4. Syntactic Structures Based on a grammar, or other syntactic definition Primary example is mutation testing Induce small changes to the program: mutants Find tests that cause the mutant programs to fail: killing mutants Failure is defined as different output from the original program Check the output of useful tests on the original program Example program and mutants if (x > y) if (x >= y) z = x - y;  z = x + y;  z = x – m; else z = 2 * x; if (x > y) z = x - y; else z = 2 * x; Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Source of Structures These structures can be extracted from lots of software artifacts Graphs can be extracted from UML use cases, finite state machines, source code, … Logical expressions can be extracted from decisions in program source, guards on transitions, conditionals in use cases, … Model-based testing derives tests from a model that describes some aspects of the system under test The model usually describes part of the behavior The source is usually not considered a model Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Four Structures for Modeling Software Coverage Overview Four Structures for Modeling Software FSM = finite state machine DNF = Disjunctive normal form Graphs Logic Input Space Syntax Use cases Specs Design Source Applied to DNF Specs FSMs Source Applied to Input Models Integ Source Applied to Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Coverage Given a set of test requirements TR for coverage criterion C, a test set T satisfies C coverage if and only if for every test requirement tr in TR, there is at least one test t in T such that t satisfies tr Infeasible test requirements : test requirements that cannot be satisfied No test case values exist that meet the test requirements Dead code Detection of infeasible test requirements is formally undecidable for most test criteria Thus, 100% coverage is impossible in practice Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Two Ways to Use Test Criteria Directly generate test values to satisfy the criterion often assumed by the research community most obvious way to use criteria very hard without automated tools Generate test values externally and measure against the criterion usually favored by industry sometimes misleading if tests do not reach 100% coverage, what does that mean? Test criteria are sometimes called metrics Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Generators and Recognizers Generator : A procedure that automatically generates values to satisfy a criterion Recognizer : A procedure that decides whether a given set of test values satisfies a criterion Both problems are provably undecidable for most criteria It is possible to recognize whether test cases satisfy a criterion far more often than it is possible to generate tests that satisfy the criterion Coverage analysis tools are quite plentiful Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Comparing Criteria with Subsumption Criteria Subsumption : A test criterion C1 subsumes C2 if and only if every set of test cases that satisfies criterion C1 also satisfies C2 Must be true for every set of test cases Example : If a test set has covered every branch in a program (satisfied the branch criterion), then the test set is guaranteed to also have covered every statement Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Test Coverage Criteria Traditional software testing is expensive and labor-intensive Formal coverage criteria are used to decide which test inputs to use More likely that the tester will find problems Greater assurance that the software is of high quality and reliability A goal or stopping rule for testing Criteria makes testing more efficient and effective But how do we start to apply these ideas in practice? Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Testing Levels Based on Test Process Maturity Level 0 : There’s no difference between testing and debugging Level 1 : The purpose of testing is to show correctness Level 2 : The purpose of testing is to show that the software doesn’t work Level 3 : The purpose of testing is not to prove anything specific, but to reduce the risk of using the software Level 4 : Testing is a mental discipline that helps all IT professionals develop higher quality software Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

This is what we teach undergraduate CS majors Level 0 Thinking Testing is the same as debugging Does not distinguish between incorrect behavior and mistakes in the program Does not help develop software that is reliable or safe This is what we teach undergraduate CS majors Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

This is what hardware engineers often expect Level 1 Thinking Purpose is to show correctness Correctness is impossible to achieve What do we know if no failures? Good software or bad tests? Test engineers have no: Strict goal Real stopping rule Formal test technique Test managers are powerless This is what hardware engineers often expect Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Level 2 Thinking This describes most software companies. Purpose is to show failures Looking for failures is a negative activity Puts testers and developers into an adversarial relationship What if there are no failures? This describes most software companies. How can we move to a team approach ?? Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

This describes a few “enlightened” software companies Level 3 Thinking Testing can only show the presence of failures Whenever we use software, we incur some risk Risk may be small and consequences unimportant Risk may be great and the consequences catastrophic Testers and developers work together to reduce risk This describes a few “enlightened” software companies Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Level 4 Thinking A mental discipline that increases quality Testing is only one way to increase quality Test engineers can become technical leaders of the project Primary responsibility to measure and improve software quality Their expertise should help the developers This is the way “traditional” engineering works Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Summary More testing saves money Testing is no longer an “art form” Planning for testing saves lots of money Testing is no longer an “art form” Engineers have a tool box of test criteria When testers become engineers, the product gets better The developers get better Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Open Questions Which criteria work best on embedded, highly reliable software? Which software structure to use? How can we best automate this testing with robust tools? Deriving the software structure Constructing the test requirements Creating values from test requirements Creating full test scripts Solution to the “mapping problem” Empirical validation Technology transition Application to new domains Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt

Summary of Today’s New Ideas Why do we test – to reduce the risk of using the software Four types of test activities – test design, automation, execution and evaluation Software terms – faults, failures, the RIP model, observability and controllability Four structures – test requirements and criteria Test process maturity levels – level 4 is a mental discipline that improves the quality of the software Earlier and better testing can empower the test manager Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt