1. 1 Some Software Testing Quotes: “ Testing proves a programmer’s failure. Debugging is the programmer’s vindication.” -Boris Beizer, “The most likely way for the world to be destroyed, most experts agree, is by accident. That's where we come in; we're computer professionals. We cause accidents.” Nathaniel S. Borenstein, “All programmers are optimists. Perhaps this modern sorcery especially attracts those who believe in happy endings and fairy godmothers. Perhaps the hundreds of nitty frustrations drive away all but those who habitually focus on the end goal. Perhaps it is merely that computers are young, programmers are younger, and the young are always optimists. But however the selection process works, the result is indisputable: 'This time it will surely run' or 'I just found the last bug'.” -Frederick P. Brooks http://www.wired.com/software/coolapps/news/2005/11/69355 Time to cover: 30-40 minutes

2. 2 Software Testing ( all UWO notes, not from the text ) Intro to Testing Unit Testing Drivers Stubs

3. 3 Introduction to Testing What is the purpose of testing? Older view: “Testing is done to show that the system works” —Tend to “go easy” on program —Programmer use same logic to test as they did to design or code —Therefore some bugs do not get caught More modern and useful view: “Testing is done to uncover bugs” —We purposely take attitude of trying to break program —Hence, more bugs discovered —Result: a better system We will take the more modern view!

4. 4 Test Cases and Test Suites Test Case: A set or sequence of inputs used to test a program, along with expected output. Example: test case for an automated day planner: —A sequence of commands to add appointments, move around the calendar, etc, with the expected output of each command Similar to use cases, however: Use cases tend to: —Concentrate on features of user interface —Show common patterns of use —Be used as a basis for design Test cases tend to: —Concentrate on program correctness —Test obscure or problematic patterns of use —Be chose in order to try to break program Test Suite: set of test cases Example: set of test cases to automate day planner, each testing for a different set of requirements

5. 5 Good Test Cases and Test Suites A good test case is one that we think is likely to uncover a bug A good test suite contains good test cases and tests the requirements thoroughly. So: The better our test suite is, and The more of it our software passes, The more confidence we can have in our software

6. 6 Failures, Faults and Errors We often use the informal term bug In fact, this can mean several different things Sometimes it is more useful to use precise terminology Failure: what the program does wrong (crashing, wrong answer) Fault: incorrect code causing the failure (less than should be greater than) Error: mistake that programmer made leading to the fault (assumed that less than was correct) We will still often use bug when it is unambiguous

7. 7 Can We Find all Bugs? 80% of the failures are caused by 20% of the faults 20% of faults are easy to find 20% of the failures are caused by 80% of the faults Less frequent and therefore harder to find May be just as bad, if not worse than other errors Some serious failures may be very hard to find Must push system in unexpected / unusual ways in order to find bugs In large systems, it is likely we will never find all the bugs Therefore for (e.g.) airplane software, nuclear power plant software, etc. developers must devote much time and money to: Avoiding, finding, eliminating as many bugs as possible Building in failsafe checks to alleviate effects of faults

8. 8 Testing vs. Debugging Testing: running test cases, finding failures Testing often can be done without looking at the code Can be automated or partially automated —By putting a test suite in a shell script or makefile Debugging: finding and correcting faults Need to work with the code Use a debugger Can’t be automated (of course!)

9. 9 Levels of Testing Unit Testing: testing individual functions or modules Done by programmer, Usually use drivers and stubs System Testing: test the whole system Uses real inputs to the whole system Planned well in advance In between those comes: integration testing: make sure the modules compile together and pass basic tests Should be easy if: —The detailed design was good —Everyone assumed the same interfaces between modules We will study each of these levels in more detail System testing by client or evaluator usually referred to as acceptance testing Tells whether program is acceptable to the people you are doing it for

10. 10 Building Good Test Cases Two basic strategies: Look at requirements, construct cases which correspond directly to these —Called functional testing, “black box testing” —Can be written in parallel with coding Look at finished code, try to execute paths through it, see what code has been missed, think up test cases which test missed code. —Called structural testing, “clear box testing” —Tools exist to help with this We usually need and use both strategies

11. 11 Unit Testing Lets imagine that: Each person has finished writing the code of their modules Each person has got their modules compiling Now, we just put everything together without testing the individual modules After some interface problems, the whole system compiles We go to test it What happens if: A failure happens right away; and it takes a while to trace it The fault is traced to person X’s code —Person X has to fix —Everyone else stands around and waits for Person X to fix the fault The fault is fixed, the test now succeeds Another failure happens, and it takes a while to trace it The fault is traced to person Y’s code —Person Y has to fix it —Everyone else stands around and waits for Person Y to fix the fault In other words: system testing is long and frustrating

12. 12 Unit Testing: Motivation Now let’s imagine that: Each person has finished writing the code of their module(s) Each person has got them compiling Each person does some separate testing on their modules before the system is integrated What happens is: A similar pattern!… However, most basic, obvious bugs have already been eliminated They were eliminated while everyone was working on their modules at the same time Less frustration, less waiting around for other people In other words: doing unit testing first makes system testing much smoother when we get to it

13. 13 Unit Testing: Questions How do we select test cases for the units (modules, objects)? Both structural and functional methods will work We will get to these topics a bit later How to test one module in isolation at all, without having working implementation We will address this question first

14. 14 Drivers In particular: how to test a module which is low in the module hierarchy, without the main program? Answer: a driver Driver: a simple main program which exists only in order to test a low-level function or module To test a module using a driver: Write the driver Compile the driver together with the module Test the driver + module as if it is the program you wanted to test Typical simple driver for single function: Prompts user for input or reads it from command-line arguments Calls the function, reports its result, terminates Slightly more complex driver for module: Reads simple commands from user (e.g. one command per function in module) Calls function corresponding to command, reports results Terminates on quit command

15. 15 Simple Driver: Example Function to be tested (in CalendarImplementation.c): printCalendar(int y, int x) Function for printing out a calendar for year y, month m Main program of driver (CalendarDriver.c): Interprets the first two command-line arguments as y and m Calls function on y and m, then terminates To Test: Compile CalendarImplementation.c, CalendarDriver.c Issue command CalendarDriver 2002 10 to test Oct 2002 —Should display something like: —Run on test cases to check output correctness QUESTION: What test cases would you supply to CalendarDriver to thoroughly test the module you have written?

16. 16 More Complex Driver: Example Module to be tested (StackImplementation.c): Integer stack ADT, with functions push, pop, printStack, etc. Main program of driver (StackDriver.c) Has one Stack variable s Reads and interprets simple user command —User command push n causes driver to call push(s,n); printStack(s) —User command pop causes driver to call n=pop(s); printf(“Popped %d\n”, n); printStack(s) —Etc.. User: push 42 System: 42 User: push 56 System: 42 56 User: pop System: Popped 56 46

17. 17 Stubs Now: how to test a module which is high in the module hierarchy, without any of the modules it calls? Answer: stubs Stub: a simple fake function which stands in for another function, without much of the ability of the real function To test a module using stubs: Write a main program (this may be the real main purpose for a driver) Write stubs for the functions you need to Compile main program + module + stubs Test the resulting program as if it is the program you wanted to test Typical simple stub for a single function Always prints out the same message; e.g. —Function warpDrive being called More complex stub for a whole module A simple, slow implementation of the module which doesn’t have all the required functionality

18. 18 Simple Stub: Example Function to be “stubbed out” (in CalendarImplementation.c): Function for printing out a calendar for year y, month m Stub: Function which always prints a message of the form Main program initially compiled together with stub When higher-level functions tested, every time they call the stubbed-out function, this message will appear; e.g. After real CalendarImplementation.c is available, stub will be replaced by the real thing Calendar for YYYY/ MM goes here Calendar for 2002/10 goes here

19. 19 More Complex Stub: Example Module to be tested: StackImplementation.c, will all usual stack functions Stub module: Simple array implementation of a stack with 2 elements Will crash or complain if we try to put more than 2 elements in it However, this may be enough to test basic functionality of higher-level modules Main program initially compiled together with stub module and then tested After real implementation of StackImplementation.c is available; stub will be replaced by the real thing

20. 20 Functions with Random Behaviour So far: Examples for stubs were useful for functions that exhibit repeatable behavior What if this is not the case? Example: a game program has functions that choose randomly from a number of alternatives In this case, a stub might Always pick a particular alternative Read an alternative from a file or request one from the user Make a simple random choice, unbiased by other game factors

21. 21 Functions with Random Behaviour: Example A game program has a module that manipulates a deck of cards, DeckImplementation.c Contains functions: void shuffle (Deck d); CardList deal (Deck d, int ncards); void cut (Deck d); Say, DeckType.h contains definitions” #define DECKSIZE 52 typedef int Deck[DECKSIZE]; typedef int CardList[DECKSIZE]; QUESTION: How might we write stubs for the above functions, perhaps to test a gameplay module?