Neil Ghani Software testing

2 Introduction In a perfect world all programs fully verified testing thus redundant Back in the real world We are human so Make mistakes Can’t be bothered doing it properly Code for speed not accuracy So in this lecture we look at testing But not evaluation

3 Testing and evaluation Evaluation Qualitative decisions Is it a good system? HCI, purpose of system, use of resources, design You will be asked to (briefly) evaluate your system Testing Correctness decisions Does the system correctly perform its function Sins of omission – doesn’t do something it should Sins of commission – does do something it shouldn’t

4 Limits of software testing “ Testing is the process of executing a program with the intent of finding errors.” Glenn Myers “Good” testing will find bugs i.e. the program not doing what it should testing can prove the presence of bugs never their absence worst person to test code is the developer gives easy tests to avoid breaking it best person to test your code is someone else they will break it

5 Testing Is never complete (1) Module A takes integer input possible inputs If we test each possible input at one input millisecond, it would take 49 days to test this module So we need to choose good tests! Is never complete (2) We cannot replicate all situations in which software will be used We cannot predict everything a user will do with it Is never given enough time Testing is not proving code correct It is finding faults and fixing them (and then retesting) This takes time

6 Dynamic testing Static Analysis Hand execution: Reading the source code Walk-Through (informal presentation to others) Code Inspection (formal presentation to others) Automated Tools checking for Syntactic and semantic errors, e.g. compilers Dynamic Analysis Black-box testing (Test the specification) White-box testing (Test the code)

7 Black-box Testing Focus: I/O behavior. If for any given input, we can predict the output, then the module passes the test. Almost always impossible to generate all possible inputs ("test cases") Reduce number of test cases by equivalence partitioning Think about what input each module takes Divide input into equivalence classes i.e partition possible input data into sets of data Usually valid and invalid Choose one test cases for each equivalence class If module works for this input, works for all inputs of this class

8 Examples Method accepts one positive integer Two equivalence classes – valid and invalid Valid set – all positive integers Invalid set – all negative integers Test cases – 5 (valid), -1 (invalid) Method accepts one integer in range equivalence classes Valid data – any number between Invalid – too low – any number less than 1 Invalid – too high – any number greater than 10 Test cases – 5 (valid), 0 (invalid too low), 12 (invalid too high)

9 Examples Module takes a legitimate string i.e. no punctuation, spaces or digits Valid class Legitimate strings, e.g. ‘ian’, ‘programming’, ‘project’ Invalid classes ‘ian,’ – punctuation after ‘,ian’ – punctuation before ‘ian’s’ – punctuation in the middle ‘ian5s’ – digits anywhere ‘’ – empty string (valid?) For each input we need to think about what input is valid and what is not (and why) Also need invalid to check error-handling

10 Black box testing (contd) We also have to consider output E.g. module G takes a person’s age and returns Yes if the person is old enough to buy alcohol No, if the person is not old enough to buy alcohol Valid data are numbers greater than equal to 0 Invalid data are numbers less than 0 But some valid data will return yes, other no We need to test both Equivalence classes Valid + old enough – greater than or equal to 18 Valid + not old enough – Invalid – less than 0 018

11 Testing Questions for each variable What groups of data are valid What groups of data are invalid For each class, do we expect module to treat some data differently i.e. to give different outputs If yes, split class into sub-classes For some languages also need to treat different types of data as invalid e.g. input is one integer Invalid might also include strings, lists, etc

12 Examples Note: duplication tells us nothing! e.g. module takes a legitimate string valid class legitimate strings, e.g. ‘ian’, ‘programming’, ‘project’ testing ‘ian’ and ‘notian’ tells us nothing also worth testing boundary values e.g. module takes a integer in range of many errors happen at boundaries test 1, 10, 11, 0

13 Examples Can combine testing in some cases module A removes punctuation and digits from strings input – ‘st7in,d,s!?<>ss;’ expected output – ‘stindsss’ test is removal of all non-characters But not all test remove punctuation and enter word into bst No! unless you have already tested them separately

14 Test oracle We use test cases to specify a test oracle Set of test cases, each specifying input and predicted output for each equivalence class E.g. inputpredicted output 5 -1 error Predicted output used to test actual output Also need to do for combinations of input e.g. input5 (valid)5 (valid) -1 (invalid) 5 (valid) 5 (valid) -1 (invalid) -1 (invalid) –1 (invalid)

15 For your report Structure of test case Aim: “This test case tests the ability of the module to …… It assumes that input can only be integers because…” Input: the data used Predicted output: what you expect to happen if the module works correctly Actual output: What you actually got (the result) Conclusion: Did it work? If not why not and what (if anything) did you do to correct it? What does this tell us about the module?

16 Example Structure of test case Aim: “This test case tests the ability of the module to remove punctuation from a string. Assumes punctuation characters are all non-alphanumeric characters” Input: “”!ia?n%” Predicted output: “ian” Actual output: (1) “ian” (2) “ia” Conclusion: (1) The module correctly removes punctuation from the input string” Or (2) “The input string was printed out but the final letter was missing. The error was fixed by correcting the while loop and re-tested as test case X”

17 “Adequate” set of test cases Test the interface of the components all public methods Is one test case per method enough? Usually no! So, how many do we need? Depends on method More complex methods need more testing Methods that take more input need more testing But avoid duplication….. Testing the same thing twice tells us nothing

18 Automating software testing Manual software testing is time consuming Software testing has to be repeated after every change (regression testing) Write test drivers that can run automatically and produce a test report public boolean myTest() { int value1; int value2; int actual_result, expected_result; value1 = 5; value2 = 5; expected_result = 10; actual_result = Integer sum(value1, value2); return ( actual_result == expected_result ); }

19 Black box testing Tells us what happens outside module Tells us nothing about what goes on inside Module input output

20 White box testing our goal is to ensure that all statements and conditions have been executed at least once... Tests the code itself loop < 20x

21 White-box Testing Four types of white-box testing (1) Statement Testing Test single statements (assignments, etc) SumOfScores = 0.0; System.out.println(SumOfScores);

22 Loop Testing while (x > 0){ out.write(x); read(x);} Cause execution of the loop to be skipped completely. Set x to minus 1 Exception: For loops Loop to be executed exactly once Enter –1 as x Loop to be executed more than once Try different positive values of x Loop to be executed n times (n-1), n, and (n+1) passes through the loop White-box Testing (Continued)

23 White box testing Branch Testing (Conditional Testing) Make sure that each possible outcome from a condition is tested at least once Here, two test cases i=true, i=false if(i == ‘true’) System.out.println(‘True’); else if (i == ‘false’) System.out.println(‘False); And every condition is covered if(i < 0) System.out.println(‘True’); else if (i >0) System.out.println(‘False); What about i == 0???

24 Branch testing Path testing Test all ways (paths) an input can go through the program Define test cases that test each decision

25 Good design makes it easier Features that help/influence testing: observability the results are easy to see, simple data returned best distinct output is generated for each input incorrect output is easily identified decomposability software modules can be tested independently good cohesion! simplicity - no complex architecture and logic stability Fixing the design before you start coding! understandability program is easy to understand no kludges!

26 Testing vs. fault tolerance design and implementation Testing happens before software is released Finding errors and fixing them What about after release? Hardware faults, data transmission faults, poor user input, …. Fault tolerant design Assume that errors occur during execution and try to recover from them (also during execution) For complex and important systems we need both!

27 For this course Black-box testing is necessary White-box testing Similar documentation to before Is not necessary But will gain extra marks Also interface testing Need test cases Usually based on use cases Can people use it?