1 CS115 Class 15: Testing Due today –User Manual –Software Inspections –Review: Testing Next time –Review No Silver Bullet

2 French Guyana, June 4, 1996 $800 million software failure

3 Mars Climate Orbiter –The 125 million dollar Mars Climate Orbiter is assumed lost by officials at NASA. The failure responsible for loss of the orbiter is attributed to a failure of NASA’s system engineer process. The process did not specify the system of measurement to be used on the project. As a result, one of the development teams used Imperial measurement while the other used the metric system of measurement. When parameters from one module were passed to another during orbit navigation correct, no conversion was performed, resulting in the loss of the craft.

4

5 More BSOD Embarrassments

6

7 Economic Impact NIST study –On CNN.com - April 27,

8 Basic Definitions Error-mistake or deviation from correct value Failure-unacceptable system behavior Defect/fault-flaw in any part of system that may cause a failure

9 Testing Objective Find defects/faults –Find them effectively as many as possible –Find them efficiently as many as possible given testing time

10 Testing scope Unit Integration System Acceptance Alpha/Beta

11 Test approach White box Black box

12 Software Development Today ProgrammerTester Decision Maker Why do we have this structure?

13 Typical Scenario (1) ProgrammerTester Decision Maker “I’m done.” “It doesn’t #$%& compile!” “OK, calm down. We’ll slip the schedule. Try again.”

14 Typical Scenario (2) ProgrammerTester Decision Maker “I’m done.” “It doesn’t install!” “Now remember, we’re all in this together. Try again.”

15 Typical Scenario (3) ProgrammerTester Decision Maker “I’m done.” “It does the wrong thing in half the tests.” “Let’s have a meeting to straighten out the spec.” “No, half of your tests are wrong!”

16 Typical Scenario (4) ProgrammerTester Decision Maker “I’m done.” “It still fails some tests we agreed on.” “Try again, but please hurry up!”

17 Typical Scenario (5) ProgrammerTester Decision Maker “I’m done.” “Yes, it’s done!” “Oops, the world has changed. Here’s the new spec.”

18 Software Development Today ProgrammerTester Decision Maker Why do we have this structure?

19 Standard Testing Questions How shall we generate/select test cases? Did this test execution succeed or fail? How do we know when to stop testing?

20 Summary Testing is hard –If done manually, also very expensive and boring Use inspections! –Will save more time in testing and debugging A number of techniques can make testing effective –Randomized testing –Exhaustive testing on small examples –Regression testing with nightly build

21 Back to Design Testing has a profound impact on design –Because some designs are easier to test Design software so it can be tested!

22 Principles of Testability Avoid unpredictable results –No unnecessary non-deterministic behavior Design in self-checking –At appropriate places have system check its own work Asserts –May require adding some redundancy to the code Have a test interface Minimize interactions between features –Number of interactions can easily grow huge –Rich breeding ground for bugs

23 JUnit Test automation

24 Benefits of JUnit Free - Automatic check of expected vs. actual Simple to use – quick to write and run Tests are written in Java

25 Design of JUnit Built around Command pattern –each method under test is represented by a separate test method

26 JUnit Sample Unit Test write tests for all methods in a class import junit.framework.TestCase public class ShoppingCartTest extends TestCase { protected void setUp() {…} protected void tearDown() {…} public void testEmpty() {…} public void testAddItem() {…} }

27 How to create a test suite import junit.framework.TestSuite; public class EcommerceTestSuite { public static Test suite() { TestSuite suite = new TestSuite(); suite.addTestSuite(ShoppingCartTest.class); suite.addTest(CreditCardTestSuite.suite()); } /** * Runs the test suite using the textual runner. */ public static void main(String[] args) { junit.textui.TestRunner.run(suite()); }

28 A GUI for running the test suite java junit.swingui.TestRunner EcommerceTestSuite

29 Initialization called before every test case method protected void setUp() { cart = new ShoppingCart(); book1 = new Product("Pragmatic Unit Testing", 29.95); cart.addItem(book1); }

30 Clean up called after every test case method protected void tearDown() { // set to null any references to big objects

31 Tests the empty method public void testEmpty() { cart.empty(); assertEquals(0, cart.getItemCount()); }

32 Tests addItem method public void testAddItem() { Product book2 = new Product("Pragmatic Project Automation", 29.95); cart.addItem(book2); double expectedBalance = book1.getPrice() + book2.getPrice(); assertEquals(expectedBalance, cart.getBalance()); assertEquals(2, cart.getItemCount()); }

33 Output of JUnit >java junit.textui.TestRunner TestServer 1) testAddItem AssertionFailedError: expected: but was:

34 JUnit assertXXX assertEquals(x,y) //.equals, or values assertFalse(boolean expr) assertNotNull(obj ref) assertNotSame(x, y) // using == assertNull() assertSame() assertTrue() fail() public void testPassNullsToConstructor(){ try{ Server s = new Server(null, null); fail(“Expected IllegalArgumentException”); } catch (IllegalArgumentException expected){} }

35 JUnit Sequence New test case for every test method For each test method –create new object no sharing of instance variables between tests –setUp() –test –tearDown()

36 JUnit Practices One function check per test method –usually one assert per test sometimes more –failure aborts entire method Many test per TestCase Can group multiple TestCase into a TestSuite

37 JUnit Recommendations Add JUnit tests in your code –run regularly (eg before SVN commit) Benefits: –catch errors early easier to debug, easier to fix –reduce cost of integration testing –reduce risk of large, untested code base of late slips in schedule (no time to recover) increase confidence, reduce stress

38 More details... JUnit home: Tutorial Also (thanks Stan) –

39 “Best Practices” Session By now, you have now developed some expertise in your particular specialization –(tester, coder, documenter, facilitator) Group by specialization to discuss –what knowledge you’ve gained –what works, what doesn’t –tips to help other teams Short (5min) presentation using doc camera