Introduction to Software Testing

Introduction to Software Testing
Slides adapted from various sources including Software Engineering: A Practitioner’s Approach, 6/e by R.S. Pressman and Associates Inc. CSC 550

What is software testing?
“Testing is the process of exercising a program with the intent of finding Errors.” “Testing is the process of exercising a program with the intent of proving its correctness.”

What is software testing?
“Observing the execution of a software system to validate whether it behaves as intended and identify potential malfunctions.” Antonia Bertolino

What does testing show? errors requirements conformance performance
an indication of quality

Why is testing important? Because the cost of failures is significant!

Expectation vs. Reality
Do we expect software to work correctly? A carefully made program 5 faults/KLOC Windows XP has 45M LOC How many faults? 5 * 45M/1000 = 225,000! Why not remove all the faults?

Why is testing so difficult?
1 3 2 5 4 loop < =20 X a b c d e How many paths are there through this loop? How long would it take to execute all 1 test per millisecond (10-3)?

A Testing Scenario There are about 1014 possible paths
3 2 5 4 loop < =20 X a b c d e There are about 1014 possible paths How long would it take to execute all paths?

A Testing Scenario 1 3 2 5 4 loop < =20 X a b c d e How long would it take to execute 1014 possible 1 paths per 10-3 seconds? 1 year =31,536,000 seconds ~= 32,000,000,000 milliseconds Total time = 1014 / 32 * 109 ~= 105 /32 years About 3,170 one test/millisecond About 3 one test/microsecond About 11 one test/nanosecond

1 3 2 5 4 loop <= 20 X a b c d e Because exhaustive testing is impossible Exhaustive testing means testing a program with all combinations of input values and preconditions for each element of the software under test.

Looping : for (i=0; i< n; i++) etc. Loop header Conditional statement statement How long would it take to execute all 1 test per millisecond (10-3)?

Terms! Terms! Terms! Anomaly BUG Failure Error Defect Fault Exception

Testing Terminology Fault/defect Error Failure
An incorrect step, process, condition, or data definition in a computer program. Software problem found after release (Pressman) Error An incorrect internal state resulting from a fault. A discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition (ISO). Software problem found before release (Pressman) Failure The inability of a system or component to perform its required functions within specified performance requirements (IEEE). Failures are caused by faults, but not all faults cause failures.

Testing Terminology: Fault vs. Error vs. Failure
public static int numZaro(int[] x) { //Effects: if x == null throw NullPointerException // else return the number of occurrences of “0” (zero) in x. int count = 0; for (int i = 1; i < x.length; i++) { if (x[i] == 0) { count++; } return count; Fault = search starts at index 1 instead of 0. Error: numZero([2, 7, 0])  error but no failure Failure: numZero([0, 7, 2])  error and failure State at if given [2, 7, 0] = (x=[2,7,0], count=0, i=1, PC=if). i Should be zero S0, if required sequence of states is s0, s1, s2 and actual sequence is s0, s2, s3, then s2 is in error in second sequence.

Testing Terminology Anomaly Bug Exception
Anything observed in the documentation or operation of software that deviates from expectations based on previously verified software products or reference documents (IEEE). Bug Things the software does that it is not supposed to do Something the software doesn't do that it is supposed to. Exception An event that causes suspension of normal program execution; types include addressing exception, data exception, operation exception, overflow exception, protection exception, underflow exception (IEEE).

Testing Terminology: Testing vs. Debugging
Evaluating software to determine conformance to some objective. Debugging Finding a fault, given a failure.

Testing Terminology: The RIP Fault/Failure Model
Three conditions are necessary for failures to be observable. Reachability: The location of the fault must be reached. Infection: After executing the location, the state of the program must be incorrect. Propagation: The infected state must propagate to cause incorrect program output.

Testing Terminology Validation Verification
Does software meet its goals? User-centric! Verification Are we building the product right? E.g., does the code reflect the design? Developer-centric! Name for the artifact being tested Implementation under test (IUT) Method under test (MUT), object under test (OUT) Class/component under test (CUT), etc.

Testing Terminology: Test Plan
A test plan specifies how we will demonstrate that the software is free of faults and behaves according to the requirements specification A test plan breaks the testing process into specific tests, addressing specific data items and values Each test has a test specification that documents the purpose of the test If a test is to be accomplished by a series of smaller tests, the test specification describes the relationship between the smaller and the larger tests The test specification must describe the conditions that indicate when the test is complete and a means for evaluating the results

Testing Terminology: Test Oracle
A test oracle is the set of predicted results for a set of tests, and is used to determine the success of testing Test oracles are extremely difficult to create and are ideally created from the requirements specification

Testing Terminology: Test Case
A test case is a specification of The pretest state of the IUT Prefix values: values needed to put software into state required before test execution. a set of inputs to the system Postfix values: values that need to be sent to the software after test execution. The expected results

Testing Terminology Test Point Test suite Domain
A specific value for test case input and state variables. Domain Set of values that input or state variables of IUT may take. Heuristics for test point selection Equivalence classes – one values in set represents all values in set. Partition testing techniques Boundary value analysis Special values testing Test suite A collection of test cases.

Testing Terminology Fault model Test strategy Test model Test design
A description about where faults are likely to occur in a program. Test strategy An algorithm or heuristic for creating test cases from a representation, an implementation or a test model Test model A description of relationships between elements of a representation or implementation. Test design The process of creating a test suite using a test strategy. Test effectiveness: relative ability of testing strategy to find bugs. Test efficiency: relative cost of finding bugs.

Testing Terminology: Testing Strategies
Responsibility-based test design Behavioural, functional, black-box, etc. testing Implementation-based test design Relies on source code,, e.g., white-box Fault-based testing

Testing Strategy We begin by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’ For conventional software The module (component) is our initial focus Integration of modules follows For OO software “testing in the small”  OO class Has attributes and operations and implies communication and collaboration

Testing Strategy unit test integration test Black Box Testing
White Box Testing system test validation test

Unit Testing module to be tested
software engineer results module to be tested test cases interface local data structures boundary conditions independent paths error handling paths The module being tested should be reviewed in context of the requirements specification The units comprising a system are individually tested The code is examined for faults in algorithms, data and syntax A set of test cases is formulated and input and the results are evaluated

Unit Test Environment test cases RESULTS driver Module stub stub
interface local data structures Module boundary conditions independent paths error handling paths stub stub test cases RESULTS

Integration Testing The goal is to ensure that groups of components work together as specified in the requirements document Four kinds of integration tests exist Structure tests Functional tests Stress tests Performance tests

Integration Testing Strategies
Options: • the “big bang” approach • an incremental construction strategy

Top Down Integration A top module is tested with stubs B F G
stubs are replaced one at a time, "depth first" C as new modules are integrated, some subset of tests is re-run D E

Bottom-Up Integration
F G drivers are replaced one at a time, "depth first" C worker modules are grouped into builds and integrated D E cluster

Sandwich Testing cluster A Top modules are tested with stubs B F G C
Worker modules are grouped into builds and integrated D E cluster

System Testing The goal is to ensure that all elements that interact to solve the problem do so correctly. Software, databases, hardware, people, procedures. Overall functionality and performance must be achieved.

Validation Testing The goal is to ensure that the system actually does what the customer expects it to do – fulfils requirements Testing is carried out by customers mimicking real world activities Customers should also intentionally enter erroneous values to determine the system behavior in those instances

Black Box Testing The tester knows nothing about the internal structure of the code Test cases are formulated based on expected output of methods Tester generates test cases to represent all possible situations in order to ensure that the observed and expected behavior is the same

White Box Testing The tester uses knowledge of the programming constructs to determine the test cases to use If one or more loops exist in a method, the tester would wish to test the execution of this loop for 0, 1, max, and max + 1, where max represents a possible maximum number of iterations Similarly, conditions would be tested for true and false

High Order Testing Validation testing System testing
Focus is on software requirements System testing Focus is on system integration Alpha/Beta testing Focus is on customer usage Recovery testing forces the software to fail in a variety of ways and verifies that recovery is properly performed Security testing verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume Performance Testing test the run-time performance of software within the context of an integrated system

A Classification of Testing: The V Model
Requirements Analysis Acceptance Test Architectural Design System Test Subsystem Design Integration Test Detailed Design Module Test Implementation Unit Test

A Classification of Testing
Graphs Logical expressions Input domain characterizations Syntactic descriptions

Test-Case Exercise UML class model for Figure Hierarchy

The Polygon Class abstract class Polygon {
abstract void draw(int r, int g, int b); /* color closed area*/ abstract void erase(); /* set to background rgb */ abstract float area(); /* return area */ abstract float perimeter(); /* return sum of sides */ abstract Point center(); /* return centroid pixel */ }

The Triangle Class class Triangle extends Polygon {
public Triangle(LineSegment a, LineSegment b, LineSegment c){…} public void setA(LineSegment a) {…} public void setB(LineSegment b) {…} public void setC(LineSegment c) {…} public LineSegment getA() {…} public LineSegment getB() {…} public LineSegment getC() {…} public boolean is_isoceles(){…} public boolean is_scalene(){…} public boolean is_equilateral(){…} public void draw(int r, int g, int b){…} public void erase(){…} public float area(){…} public float perimeter(){…} public Point center(){…} }

The LineSegment Class class LineSegment extends Figure {
public LineSegment(Point x1, Point y1, Point x2, Point y2){…} public void setx1(Point x1) {…} public void sety1(Point y1) {…} public void setx2(Point x2) {…} public void sety2(Point y2) {…} public Point getx1() {…} public Point gety1() {…} public Point getx2() {…} public Point gety2() {…} }

Test-Case Exercise Specify as many test cases as you can for the Triangle class defined above. The class is used to determine if a triangle is: isosceles, equilateral or scalene. A triangle is isosceles if two of its sides are equal. A triangle is equilateral if all its sides are equal. A triangle is scalene if its sides are of different sizes.

Test-Case Exercise A valid triangle must meet two conditions:
No side may have a length of zero. Each side must be smaller than the sum of all sides divided by two. i.e. Given s = (a + b + c)/2, then a < s, b < s, c < s must hold. Stated differently, a < s && b < s && c < s  a < b + c, b < a + c and c < a + b.

Test-Case Exercise Specify as many test cases as you can for the Triangle class defined above. Test Case Description Test Case Input a b c Expected Output 1 Valid isosceles triangle 3 4 Isosceles 2 28 29 30 31 32 33 34 35

Solution To Test Case Exercise

Caveat: Impact of New Tools and Techniques
Naw - I already don’t plow as good as I know how... They’re teaching a new way of plowing over at the Grange tonight - you going? “Knowing is not enough, we must apply. Willing is not enough, we must do.” Goethe Introduction to Software Testing (Ch 1), © Ammann & Offutt

Testing in the 21st Century
We are going through a time of change Software Defines Behavior network routers financial networks telephone switching networks other infrastructure Embedded Control Applications airplanes, air traffic control spaceships watches ovens remote controllers Safety critical, real-time software Testing ideas have matured enough to be used in practice PDAs memory seats DVD players garage door openers cell phones © Ammann & Offutt

Types of Test Activities
Testing can be broken up into four general types of activities Test Design Test Automation Test Execution Test Evaluation Each type of activity requires different skills, background knowledge, education and training No reasonable software development organization uses the same people for requirements, design, implementation, integration and configuration control 1.a) Criteria-based 1.b) Human-based Why do test organizations still use the same people for all four test activities?? This is clearly a waste of resources © Ammann & Offutt

1. Test Design – (a) Criteria-Based
Design test values to satisfy coverage criteria or other engineering goal This is the most technical job in software testing Requires knowledge of : Discrete math Programming Testing Requires much of a traditional CS degree This is intellectually stimulating, rewarding, and challenging Test design is analogous to software architecture on the development side Using people who are not qualified to design tests is a sure way to get ineffective tests © Ammann & Offutt

1. Test Design – (b) Human-Based
Design test values based on domain knowledge of the program and human knowledge of testing This is much harder than it may seem to developers Criteria-based approaches can be blind to special situations Requires knowledge of : Domain, testing, and user interfaces Requires almost no traditional CS A background in the domain of the software is essential An empirical background is very helpful (biology, psychology, …) A logic background is very helpful (law, philosophy, math, …) This is intellectually stimulating, rewarding, and challenging But not to typical CS majors – they want to solve problems and build things © Ammann & Offutt

Embed test values into executable scripts
2. Test Automation Embed test values into executable scripts This is slightly less technical Requires knowledge of programming Fairly straightforward programming – small pieces and simple algorithms Requires very little theory May be boring for test designers Programming is out of reach for many domain experts Who is responsible for determining and embedding the expected outputs ? Test designers may not always know the expected outputs Test evaluators need to get involved early to help with this © Ammann & Offutt

Run tests on the software and record the results
3. Test Execution Run tests on the software and record the results This is easy – and trivial if the tests are well automated Requires basic computer skills Interns Employees with no technical background Asking qualified test designers to execute tests is a sure way to convince them to look for a development job If, for example, GUI tests are not well automated, this requires a lot of manual labor Test executors have to be very careful and meticulous with bookkeeping © Ammann & Offutt

Evaluate results of testing, report to developers
4. Test Evaluation Evaluate results of testing, report to developers This is much harder than it may seem Requires knowledge of : Domain Testing User interfaces and psychology Usually requires almost no traditional CS A background in the domain of the software is essential An empirical background is very helpful (biology, psychology, …) A logic background is very helpful (law, philosophy, math, …) This is intellectually stimulating, rewarding, and challenging But not to typical CS majors – they want to solve problems and build things © Ammann & Offutt

White-box testing: control-flow graph
A control flow graph of a program is a graph that shows the paths that may be traversed when the program is executed.

Control-flow Graph int n = 0; do{ if(n++ % 2 == 1) dBase[i]++; else
3 4 5 6 7 8 int n = 0; do{ if(n++ % 2 == 1) dBase[i]++; else if(dBase[i] > 100) dBase[i] -= n; dBase[i] += n; }while(n < 20); 1 2 4 3 5 6 7

Cyclomatic Complexity – V(G)
If G is the control flow graph of program P and G has e edges and n nodes, then the cyclomatic complexity of P, V(G) is given by: V(G) = e – n + 2 Or V(G) = the number of decision nodes + 1

1 2 3 4 5 6 7 8 cyclomatic complexity V(G) = e – n + 2 = 11 – 7 = 4 Or V(G) = the number of decision nodes + 1 = = 4

Cyclomatic Complexity
A number of industry studies have indicated that the higher V(G), the higher the probability or errors. modules V(G) modules in this range are more error prone

1 2 3 4 5 6 7 8 The cyclomatic complexity is a measure of the number of linearly independent paths in the graph. An independent path must move along at least one edge not previously traversed What are the independent paths for this graph? …8

White-box testing: Code coverage
White box testing methods emphasize code coverage Statement coverage Every statement is executed at least once Branch coverage Every branch is exercised at least once i.e. every path from a node is traversed at least once Once true and once false evaluation Multiple condition coverage All combinations of simple conditions are exercised Basis-path model Exercise V(G) independent paths Data flow coverage Evaluation of state of instance variables.

Statement Coverage 1 2 3 4 5 6 7 8 Write enough test cases to execute each statement al least once, May miss bugs for complex predicates and loops

Branch Coverage Determine number of branches
1 2 3 4 5 6 7 8 Determine number of branches Compute path for each branch Write test cases to exercise each path

Multiple Condition Coverage
1 2 3 4 5 6 7 8 Determine true/false combinations for simple conditions Write test cases to exercise each combination

Basis-path Testing Compute the cyclomatic complexity
1 2 3 4 5 6 7 8 Compute the cyclomatic complexity Derive independent paths …7-8 Write test cases to exercise each path

Example 1 V(G) = 2 + 1 = 3 Statement coverage requires 2 test cases
Branch coverage requires 2 test cases Basis-path coverage requires 3 test cases There are four distinct entry-exit paths Which should be chosen? 1 2 4 3 5 6 7

Example 2 A D What is V(G)? How many test cases are required for statement coverage? How many test cases are required for branch coverage? How many test cases are required for basis-path coverage? How many unique entry-exit paths exist? F E K L

Introduction to Mutation Testing & Program Perturbation

Mutation Testing Faults are introduced into a program by creating many versions of the program Each version is called a mutant. Each mutant contains a single fault. Test cases are applied to the original program and to the mutant program. The goal is to cause the mutant program to fail, thus demonstrating the effectiveness of the test case.

Example of a Program Mutation
1 int max(int x, int y) 2 { 3 int mx = x; 4 if (x > y) 5 mx = x; 6 else 7 mx = y; 8 return mx; 9 } 1 int max(int x, int y) 2 { 3 int mx = x; 4 if (x < y) 5 mx = x; 6 else 7 mx = y; 8 return mx; 9 }

Mutation Testing Process
Process (cont.) Figure 2. Mutation Testing Flowchart, Source: Mutation 2000, pg 3

Terminology Mutation testing Mutation
The creation of a set of mutant programs of the program being tested. Mutation a single syntactic change that is made to a program statement. Each mutant differs from the original program by one mutation.

Terminology Mutant Mutant Killed Mutation Score Mutation Operator
A faulty program; the original with one fault introduced Mutant Killed Test suite detects the mutant Mutation Score #of killed mutants/ # of non-equivalent mutants Mutation Operator The function that changes the source code to create a mutant

Mutation Operators Replace each operand with every other syntactically correct operand Modify expressions by replacing operators and inserting new operators Delete entire statements

Categories of Mutation Operators
Operand Replacement Operators: Replace a single operand with another operand or constant. E.g., if(x > y) if (5 > y) Replacing x by constant 5. if (x > 5) Replacing y by constant 5. if (y > x) Replacing x and y with each other. E.g., if all operators are {+,-,*,**,/} then the following expression a = b * (c - d) will generate 8 mutants: 4 by replacing * 4 by replacing -.

Expression Modification Operators: Replace an operator or insert new operators. E.g., if (x == y) if (x >= y) Replacing == by >=. if (x == ++y) Inserting ++.

Statement Modification Operators: E.g., Delete the else part of the if-else statement. Delete the entire if-else statement. Replace line 3 by a return statement. 1 int max(int x, int y) 2 { 3 int mx = x; 4 if (x > y) 5 mx = x; 6 else 7 mx = y; 8 return mx; 9 } return x;

Mutation System Example
The Mothra mutation system for FORTRAN77 supports 22 mutation operators. E.g., absolute value insertion constant for array reference replacement GOTO label replacement RETURN statement replacement statement deletion unary operator insertion logical connector replacement comparable array name replacement

Why Does Mutation Testing Work?
The operators are limited to simple single syntactic changes on the basis of the competent programmer hypothesis.

The Competent Programmer Hypothesis
Programmers are generally very competent and do not create “random” programs. For a given problem, a programmer, if mistaken, will create a program that is very close to a correct program. An incorrect program can be created from a correct program by making some minor change to the correct program.

Mutation Testing Algorithm
Generate program test cases. Run each test case against the original program. If the output is incorrect, the program must be modified and re-tested. If the output is correct go to the next step ... Construct mutants E.g. using a tool like Mothra.

Mutation Testing Algorithm (Cont’d)
Execute each test case against each alive mutant. If the output of the mutant differs from the output of the original program, the mutant is considered incorrect and is killed. Two kinds of mutants survive: Functionally equivalent to the original program: Cannot be killed. Killable: Test cases are insufficient to kill the mutant. New test cases must be created.

Mutation Score The mutation score for a set of test cases is the percentage of non-equivalent mutants killed by the test data. Mutation Score = 100 * D / (N - E) D = Dead mutants N = Number of mutants E = Number of equivalent mutants A set of test cases is mutation adequate if its mutation score is 100%.

Program Perturbation Program Perturbation is a technique to test a program’s robustness. It is based on unexpectedly changing the values of program data during run-time.

The Perturbation Function
The perturbation function is a mathematical function that: takes a data state as its input; changes the data state according to some specified criteria; produces a modified data state as output.

Program Perturbation Example
Assume the following perturbation function: 1. int perturbation (int x) 2. { 3. int newX; 4. newX = x + 20; 5. return newX; 6. }

Example of a Fault Injection
1. main() 2. { 3. int x; 4. x = ReadInt(); 4.1 x = perturbation(x); 5. if (x > 0) printf(“X positive”); 7. else printf(“X negative”); 9. } 1. main() 2. { 3. int x; 4. x = ReadInt(); 5. if (x > 0) 6. printf(“X positive”); 7. else 8. printf(“X negative”); 9. }

Model-Driven Test Design
refined requirements / test specs model / structure test requirements DESIGN ABSTRACTION LEVEL IMPLEMENTATION ABSTRACTION LEVEL software artifact input values pass / fail test results test scripts test cases © Ammann & Offutt

Model-Driven Test Design – Steps
criterion refine refined requirements / test specs model / structure test requirements generate analysis DESIGN ABSTRACTION LEVEL IMPLEMENTATION ABSTRACTION LEVEL software artifact input values prefix postfix expected evaluate execute automate pass / fail test results test scripts test cases © Ammann & Offutt

Model-Driven Test Design – Activities
refined requirements / test specs Test Design model / structure test requirements DESIGN ABSTRACTION LEVEL Raising our abstraction level makes test design MUCH easier IMPLEMENTATION ABSTRACTION LEVEL software artifact input values Test Automation Test Evaluation pass / fail test results Test Execution test scripts test cases © Ammann & Offutt

From: http://www. google. com/url
From: and other sources. Junit Overview

JUnit 3

JUnit JUnit is a framework for writing unit tests
From: JUnit JUnit is a framework for writing unit tests A unit test is a test of a single class A test case is a single test of a single method A test suite is a collection of test cases Unit testing is particularly important when software requirements change frequently Code often has to be refactored to incorporate the changes Unit testing helps ensure that the refactored code continues to work Satish Mishra Unit testing with JUnit

JUnit.. JUnit helps the programmer:
Define and execute tests and test suites Formalize requirements and clarify architecture Write and debug code Integrate code and always be ready to release a working version Satish Mishra Unit testing with JUnit

What JUnit does JUnit runs a suite of tests and reports results
For each test in the test suite: JUnit calls setUp() This method should create any objects you may need for testing Satish Mishra Unit testing with JUnit

What JUnit does… JUnit calls one test method The test method may comprise multiple test cases; that is, it may make multiple calls to the method you are testing In fact, since it’s your code, the test method can do anything you want The setUp() method ensures you entered the test method with a virgin set of objects; what you do with them is up to you JUnit calls tearDown() This method should remove any objects you created Satish Mishra Unit testing with JUnit

Creating a test class in JUnit
Define a subclass of TestCase Override the setUp() method to initialize object(s) under test. Override the tearDown() method to release object(s) under test. Define one or more public testXXX() methods that exercise the object(s) under test and assert expected results. Define a static suite() factory method that creates a TestSuite containing all the testXXX() methods of the TestCase. Optionally define a main() method that runs the TestCase in batch mode. Satish Mishra Unit testing with JUnit

protected void setUp() { …}
Fixtures A fixture is just a some code you want run before every test You get a fixture by overriding the method protected void setUp() { …} The general rule for running a test is: protected void runTest() { setUp(); <run the test> tearDown(); } so we can override setUp and/or tearDown, and that code will be run prior to or after every test case Satish Mishra Unit testing with JUnit

Implementing setUp() method
Override setUp() to initialize the variables, and objects Since setUp() is your code, you can modify it any way you like (such as creating new objects in it) Reduces the duplication of code Satish Mishra Unit testing with JUnit

Implementing the tearDown() method
In most cases, the tearDown() method doesn’t need to do anything The next time you run setUp(), your objects will be replaced, and the old objects will be available for garbage collection Like the finally clause in a try-catch-finally statement, tearDown() is where you would release system resources (such as streams) Satish Mishra Unit testing with JUnit

The structure of a test method
A test method doesn’t return a result If the tests run correctly, a test method does nothing If a test fails, it throws an AssertionFailedError The JUnit framework catches the error and deals with it; you don’t have to do anything Satish Mishra Unit testing with JUnit

Test suites In practice, you want to run a group of related tests (e.g. all the tests for a class) To do so, group your test methods in a class which extends TestCase Running suites we will see in examples Satish Mishra Unit testing with JUnit

assertX methods static void assertTrue(boolean test)
static void assertFalse(boolean test) assertEquals(expected, actual) assertSame(Object expected, Object actual) assertNotSame(Object expected, Object actual) assertNull(Object object) Satish Mishra Unit testing with JUnit

assertX methods assertNotNull(Object object) fail()
All the above may take an optional String message as the first argument, for example, static void assertTrue(String message, boolean test) Satish Mishra Unit testing with JUnit

Organize The Tests Create test cases in the same package as the code under test For each Java package in your application, define a TestSuite class that contains all the tests for validating the code in the package Define similar TestSuite classes that create higher-level and lower-level test suites in the other packages (and sub-packages) of the application Make sure your build process includes the compilation of all tests Satish Mishra Unit testing with JUnit

JUnit framework Satish Mishra Unit testing with JUnit

Example: Counter class
For the sake of example, we will create and test a trivial “counter” class The constructor will create a counter and set it to zero The increment method will add one to the counter and return the new value The decrement method will subtract one from the counter and return the new value Satish Mishra Unit testing with JUnit

The Counter class itself
public class Counter { int count = 0; public int increment() { return ++count; } public int decrement() { return --count; } public int getCount() { return count; } } Satish Mishra Unit testing with JUnit

JUnit tests for Counter
public class CounterTest extends junit.framework.TestCase { Counter counter1; public CounterTest() { } // default constructor protected void setUp() { // creates a (simple) test fixture counter1 = new Counter(); } protected void tearDown() { } // no resources to release Satish Mishra Unit testing with JUnit

JUnit tests for Counter…
public void testIncrement() { assertTrue(counter1.increment() == 1); assertTrue(counter1.increment() == 2); } public void testDecrement() { assertTrue(counter1.decrement() == -1); } } // End from last slide Satish Mishra Unit testing with JUnit

JUnit tests for Counter
public class CounterTest extends junit.framework.TestCase { Counter counter1; public CounterTest() { } // default constructor protected void setUp() { // creates a (simple) test fixture counter1 = new Counter(); } public void testIncrement() { assertTrue(counter1.increment() == 1); assertTrue(counter1.increment() == 2); } public void testDecrement() { assertTrue(counter1.decrement() == -1); } } Note that each test begins with a brand new counter This means you don’t have to worry about the order in which the tests are run

Why JUnit Allow you to write code faster while increasing quality
Elegantly simple Check their own results and provide immediate feedback Tests is inexpensive Increase the stability of software Developer tests Written in Java Free Gives proper uniderstanding of unit testing Satish Mishra Unit testing with JUnit

A Case Study class Money { private int fAmount;
private String fCurrency; public Money(int amount, String currency) { fAmount= amount; fCurrency= currency; } public int amount() { return fAmount; } public String currency() { return fCurrency; } public Money add(Money m) { return new Money(amount()+m.amount(), currency()); } }

How to Write A TestCase (2) Exercises the objects in the fixture.
public class MoneyTest extends TestCase { //… public void testSimpleAdd() { Money m12CHF= new Money(12, "CHF"); // (1) Money m14CHF= new Money(14, "CHF"); Money expected= new Money(26, "CHF"); Money result= m12CHF.add(m14CHF); // (2) Assert.assertTrue(expected.equals(result)); // (3) } (1) Creates the objects we will interact with during the test. This testing context is commonly referred to as a test's fixture. All we need for the testSimpleAdd test are some Money objects. (2) Exercises the objects in the fixture. (3) Verifies the result

Assert assertEquals(expected, actual)
assertEquals(message, expected, actual) assertEquals(expected, actual, delta) assertEquals(message, expected, actual, delta) assertFalse(condition) assertFalse(message, condition) AssertNotNull(object) AssertNotNull(message, object) AssertNotSame(expected, actual) AssertNotSame(message, expected, actual) assertTrue(condition) assertTrue(message, condition)

Structure setUp() Storing the fixture's objects in instance variables of your TestCase subclass and initialize them by overriding the setUp method tearDown() Releasing the fixture’s run() Defining how to run an individual test case. Defining how to run a test suite. testCase()

Structure of Writing A Test
public class MoneyTest extends TestCase { private Money f12CHF; private Money f14CHF; protected void setUp() { f12CHF= new Money(12, "CHF"); f14CHF= new Money(14, "CHF"); } public void testSimpleAdd() { Money expected= new Money(26, "CHF"); Money result= f12CHF.add(f14CHF); Assert.assertTrue(expected.equals(result)); } TestCase test= new MoneyTest("simple add") { public void runTest() { testSimpleAdd(); } }

JUnit 4 with Annotations

Start to Use it 2. Installation unzip the junit.zip file
1. Download the latest version of JUnit from 2. Installation unzip the junit.zip file add junit.jar to the CLASSPATH. For example: set classpath=%classpath%;INSTALL_DIR\junit3\junit.jar 3. Testing Test the installation by using either the batch or the graphical TestRunner tool to run the tests that come with this release. All the tests should pass OK. for the batch TestRunner type: java junit.textui.TestRunner junit.samples.AllTests for the graphical TestRunner type: java junit.awtui.TestRunner junit.samples.AllTests for the Swing based graphical TestRunner type: java junit.swingui.TestRunner junit.samples.AllTests Notice: The tests are not contained in the junit.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path Important: Don't install the junit.jar into the extension directory of your JDK installation. If you do so the test class on the files system will not be found. JUnit plug-in for Eclipse

Running JUnit java junit.textui.TestRunner JavaClassfileName
java junit.swingui.TestRunner JavaClassfileName java junit.awtui.TestRunner JavaClassfileName Use eclipse

Eclipse plug-in

This will be filled in automatically
JUnit in Eclipse To create a test class, select File New Other...  Java, JUnit, TestCase and enter the name of the class you will test Fill this in This will be filled in automatically

Running JUnit Second, use this pulldown menu
First, select a Test class Third, Run As  JUnit Test

Results Your results are here

JUnit 4

Comparing JUnit 3 to JUnit 4
All the old assertXXX methods are the same Most things are about equally easy JUnit 4 makes it easier to test that exceptions are thrown when they should be JUnit 4 can still run JUnit 3 tests JUnit 4 provides protection against infinite loops JUnit 4 has some additional features

Migrating from JUnit 3 JUnit 4 requires Java 5 or newer
Don’t extend junit.framework.TestCase; just use an ordinary class Import org.junit.* and org.junit.Assert.* Use a static import for org.junit.Assert.* Static imports replace inheritance from junit.framework.TestCase Use annotations instead of special method names: Instead of a setUp method, before some method Instead of a tearDown method, before some method Instead of beginning test method names with ‘test’, before each test method

Writing a JUnit test class, I
Start by importing the JUnit 4 classes you need import org.junit.*; import static org.junit.Assert.*; Declare your class in the usual way public class MyProgramTest { Declare any variables you are going to use frequently, typically including an instance of the class being tested MyProgram program; int [] array; int solution;

Writing a JUnit test class, II
If you wish, you can declare one method to be executed just once, when the class is first loaded This is for expensive setup, such as connecting to a database @BeforeClass public static void setUpClass() throws Exception { // one-time initialization code } If you wish, you can declare one method to be executed just once, to do cleanup after all the tests have been completed @AfterClass public static void tearDownClass() throws Exception { // one-time cleanup code }

Writing a JUnit test class, III
You can define one or more methods to be executed before each test; typically such methods initialize values, so that each test starts with a fresh set @Before public void setUp() { program = new MyProgram(); array = new int[] { 1, 2, 3, 4, 5 }; } You can define one or more methods to be executed after each test; typically such methods release resources, such as files @After public void tearDown() { }

@Before and @After methods
You can have as methods as you want Be warned: You don’t know in what order they will execute You can methods from a superclass; execution is as follows: Execute methods in the superclass Execute methods in this class Execute method in this class Execute methods in this class Execute methods in the superclass

Writing a JUnit test class, IV
A test method is annotated takes no parameters, and returns no result All the usual assertXXX methods can be used @Test public void sum() { assertEquals(15, program.sum(array)); assertTrue(program.min(array) > 0); }

Special features of @Test
You can limit how long a method is allowed to take This is good protection against infinite loops The time limit is specified in milliseconds The test fails if the method takes too long @Test (timeout=10) public void greatBig() { assertTrue(program.ackerman(5, 5) > 10e12); } Some method calls should throw an exception You can specify that a particular exception is expected The test will pass if the expected exception is thrown, and fail otherwise @Test (expected=IllegalArgumentException.class) public void factorial() { program.factorial(-5); }

Parameterized tests and method, you can run the same tests with multiple datasets @RunWith(value=Parameterized.class) public class FactorialTest { private long expected; private int value; @Parameters public static Collection data() { return Arrays.asList( new Object[ ][ ] { { 1, 0 }, { 1, 1 }, { 2, 2 }, { 120, 5 } }); } public FactorialTest(long expected, int value) { // constructor this.expected = expected; this.value = value; } @Test public void factorial() { assertEquals(expected, new Calculator().factorial(value)); } } Source:

Ignoring a test annotation says to not run a test @Ignore("I don’t want Dave to know this doesn’t public void add() { assertEquals(4, program.sum(2, 2)); } You shouldn’t without a very good reason!

Test suites As before, you can define a suite of tests AnotherTest.class}) public class AllTests { … }

Other stuff Failed tests now throw an AssertionError, rather than JUnit 3’s AssertionFailedError There is now a version of assertEquals for arrays of objects: assertEquals(Object[] expected, Object[] actual) Unfortunately, there is still no assertEquals for arrays of primitives JUnit 3 had an assertEquals(p, p) method for each kind of primitive p, but JUnit 4 only has an assertEquals(object, object) and depends on autoboxing

A gotcha The following method: with the following test: gives:
long sum(long x, long y) { return x + y; } with the following test: @Test public void sum() { assertEquals(4, s.sum(2, 2)); } gives: expected: <4> but was: <4> This is due to your friend, autoboxing assertEquals no longer exists for primitives, only for objects Hence, the 4 is autoboxed to an Integer, while sum returns a long The error message means: expected int 4, but got long 4 To make this work, change the 4 to a 4L

JUnit 4 in Eclipse and NetBeans
As usual, the easiest way to create a test class is just to let your IDE do it for you Here is the recommended test-driven approach Create a class containing all the “stub” methods you initially think you will need Have the IDE create the test class, with all the test methods Repeat: Write a test Make sure the test fails Write the method being tested Make sure the test now succeeds Note: When you create the test class, NetBeans in particular puts a lot of garbage lines into each test method; you can just delete these and put in your own code

Summary? JUnit Exercise

More Information http://www.junit.org
Download of JUnit Lots of information on using JUnit C++ port of Junit Information on Test-Driven Development

CSC550, Devon M. Simmonds, Computer Science Department, University of North Carolina Wilmington
Q u e s t i o n ? The End

1 3 2 5 4 loop < =20 X a b c d e There are about 1014 possible paths How long would it take to execute all paths? About 3,170 one test/millisecond

Example 2 Similar for right side What is V(G)? - 7
How many test cases are required for statement coverage? - 4 How many test cases are required for branch coverage? - 4 How many test cases are required for basis-path coverage? - 7 How many unique entry-exit paths exist? – 16 D F E K Similar for right side L ABDEGKMQS ABDEGKNQS ABDEHKMQS ABDEHKNQS ACDEGKMQS ACDEGKNQS ACDEHKMQS ACDEHKNQS

Introduction to Software Testing

Similar presentations

Presentation on theme: "Introduction to Software Testing"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Introduction to Software Testing

Similar presentations

Presentation on theme: "Introduction to Software Testing"— Presentation transcript:

Similar presentations

About project

Feedback