CS 3773 Software Engineering Basic Concepts of Software Testing Unit Testing

2 Why testing? Examples  Mars Climate Orbiter ($165M, 1998)  Sent to Mars to relay signal from the Mars Lander  Smashed to the planet: failing to convert between English measure to metric values  Shooting down of Airbus A300 (290 death, 1988)  US CG-49 shoot down a Airbus A300  Misleading output of the tracking software  THERAC-25 Radiation Therapy (1985)  2 cancer patients at East Texas Cancer center received fatal overdoses  Miss-handling of race condition of the software in the equipment

3 Why testing? Numbers  On average, 1-5 bugs per KLOC (thousand lines of code)  In mature software  More than 10 bugs in prototypes  Windows2000  35MLOC  63K known bugs at the time of release  2 bugs per KLOC  $59.5B loss due to bugs in US 2002 (estimation by NIST)  It is not feasible to remove all bugs  But try to reduce critical bugs

4 Answer is testing, why?  “50% of my employees are testers, and the rest spends 50% of their time testing” ---- Bill Gates, 1995  More reliable than inspection, relatively cheap  Actually in the old days, when testing is expensive, inspection was the major answer  You get what you pay (linear rewards)  Compared to other 3 approaches  Inspection, static checking, formal proof

5 Testing: Concepts  Test case  Test oracle  Test suite  Test script  Test driver  Test result  Test coverage

6 Testing: Concepts  Test case  An execution of the software with a given list of input values  Include:  Input values, sometimes fed in different steps  Expected outputs  Test oracle  The expected outputs of software by feeding in a list of input values  A part of test cases  Hardest problem in auto-testing: test oracle problem

7 Testing: Concepts  Test suite  A collection of test cases  Usually these test cases share similar pre-requisites and configuration  Usually can be run together in sequence  Different test suites for different purposes  Smoke test, Certain platforms, Certain feature, performance, …  Test Script  A script to run a sequence of test cases or a test suite automatically

8 Testing: Concepts  Test Driver  A software framework that can load a collection of test cases or a test suite  It can usually handle the configuration and comparison between expected outputs and actual outputs  Test Coverage  A measurement to evaluate how well the testing is done  The measure can be based on multiple elements  Code  Input combinations  Specifications

9 Granularity of Testing: V-model

10 Granularity of testing  Unit Testing  Test of a single module  Integration Testing  Test the interaction between modules  System Testing  Test the system as a whole, by developers on test cases  Acceptance Testing  Validate the system against user requirements, by customers with no formal test cases  Regression Testing  Test a new version with old test cases

11 Unit testing  Testing of an basic module of the software  A function, a class, a component  Typical problems revealed  Local data structures  Algorithms  Boundary conditions  Error handling

12 An example: w riting test cases without a test framework  Class Definition  Consider a class that handles and manages a Fibonacci array  Initialized the class  Fibonacci f = new Fibonacci ();  Extend to a certain length  f. extend (length)  Provide the number at certain index  f. get(index)  Get Length  f.getLength()  Provide a range of numbers  f. getRange(start, end)

13 An example: writing automatic test cases without a test framework public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 50){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != ){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } … } Not Enough! What happened if an exception is thrown

14 An example: writing automatic test cases without a test framework public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 50){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != ){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } … } Potential test interference!

15 Common Operations  Comparison between actual results and expected results  Report error messages  Catch Exceptions  Re-initialize the class to be tested

16 Unit test framework  xUnit  Created by Kent Beck in 1989  This is the same guy we mentioned in XP, design patterns  The first one was sUnit (for smalltalk)  Junit  The most popular xUnit framework  There are about 70 xUnit frameworks for corresponding languages  Never in the annals of software engineering was so much owed by so many to so few lines of code Martin Fowler

17 State machine of JUnit Initialization Of Test Class setUp try{ testMethod } catch {…} tearDown Why? Try and Catch?

18 Writing a Test Case public class FibonacciTest extends TestCase { protected Fibonacci fTest; protected static int groudtruth[] = {0, 1,..., }; public void setUp(){ fTest = new Fibonacci(); public testInitialize(){ assertEquals(“Length of fTest after initialization”, 0, fTest.getLength()); public testExtend(){ fTest.extend(50); assertEquals(“…”, 50, fTest.getLength()); public testGet(){ fTest.extend(50); assertEquals(“…”, groudtruth[49], fTest.get(50)); } … }

19 Benefits of using test framework  Clear structure of testing  Each test method for one feature  Common initialization to setup  Assertion methods  You can always define new assertion methods  Reduced Interference  Try catch to run all test methods  Always do re-initialization in setup

20 Demo  Writing a JUnit testcase for a class in your project  Run it as a JUnit test case

21 Tips of writing test cases  Bottom Up  If you know that obj.m1 -> obj.m2, test obj.m2 first  Why?  You may not understand the error in m1 if you don’t know whether m2 is working or not  Externalize data  Externalize expected results to files and load them at the test case constructor  Why?  Easier to maintain if you want to reuse them in several places

22 Tips of writing test cases  Use development Database  Do not use real database for testing  Why? Obvious  Do enough logging in the code during testing  Why?  Help debugging  Testing for exceptions  How? public testException(){ try{ do(invalid); fail(); }catch(xxxException e){} }

23 Assertions public void testCapacity() { // a test method …. assertTrue(fFull.size() == 100+size); //assertion } If assertion fails: Assertion failed: myTest.java:150 (expected true but was false) Not so good! Try: assertEquals(100+size, fFull.size()); //expected value first Assertion failed: myTest.java:150 (expected 102 but was 103) Better! Try: assertEquals(“list length”, 100+size, fFull.size()); Assertion failed: myTest.java:150 (list length expected 102 but was 103)

24 Assertions  Extend TestCase and write your own assertions  AssertStringContains, AssertArrayEquals, …  Use fail(String) to fail a test with certain message  Feel free to change the fixture  Each test will re-setup public void testRemoveAll() { fFull.removeAllElements(); fEmpty.removeAllElements(); assertTrue(fFull.isEmpty()); assertTrue(fEmpty.isEmpty()); }

Tear down  Consider the following test code void setUp() { File f = open(“foo”); File b = open(“bar”); } void testAAA() { use f and b } void testBBB(){ use f and b } Problems? void setUp() { File f = open(“foo”); File b = open(“bar”); } void testAAA() { try { use f and b } finally { clean&close f, b } void testBBB() { try { use f and b } finally { clean&close f, b } Better?

26 Tear down  Consider the following test code void setUp() { File f = open(“foo”); File b = open(“bar”); } void testAAA() { use f and b } void testBBB(){ use f and b } void tearDown{ clean&close f, b } Problems? void setUp() { File f = open(“foo”); File b = open(“bar”); } … void tearDown{ try{ clean&close f }catch{ … } the same for b }

27 Tear down  Be careful about tear down  If tear down is not complete, a test failure may affect the following test cases  Recover the changes done to global data that are not well handled by the setup  Database, files, network, global variables  Clean resources  Caution of exceptions in tear down itself