1. EECE 310: Software Engineering Testing / Validation

2. What will we learn ? Testing – Black-box testing – Glass-box testing – Integration testing

3. val·i·date 1.To declare or make legally valid. 2.To mark with an indication of official sanction. 3.To establish the soundness of; corroborate. Can we do any of these with software?

4. Java’s License READ THE TERMS OF THIS AGREEMENT AND ANY PROVIDED SUPPLEMENTAL LICENSE TERMS (COLLECTIVELY "AGREEMENT") CAREFULLY BEFORE OPENING THE SOFTWARE MEDIA PACKAGE. BY OPENING THE SOFTWARE MEDIA PACKAGE, YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCESSING THE SOFTWARE ELECTRONICALLY, INDICATE YOUR ACCEPTANCE OF THESE TERMS BY SELECTING THE "ACCEPT" BUTTON AT THE END OF THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL THESE TERMS, PROMPTLY RETURN THE UNUSED SOFTWARE TO YOUR PLACE OF PURCHASE FOR A REFUND OR, IF THE SOFTWARE IS ACCESSED ELECTRONICALLY, SELECT THE "DECLINE" BUTTON AT THE END OF THIS AGREEMENT.

5. Java’s License 5. LIMITATION OF LIABILITY. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL SUN OR ITS LICENSORS BE LIABLE FOR ANY LOST REVENUE, PROFIT OR DATA, OR FOR SPECIAL, INDIRECT, CONSEQUENTIAL, INCIDENTAL OR PUNITIVE DAMAGES, HOWEVER CAUSED REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF OR RELATED TO THE USE OF OR INABILITY TO USE SOFTWARE, EVEN IF SUN HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. …

6. Java’s License 2. RESTRICTIONS. … Unless enforcement is prohibited by applicable law, you may not modify, decompile, or reverse engineer Software. You acknowledge that Software is not designed, licensed or intended for use in the design, construction, operation or maintenance of any nuclear facility. Sun disclaims any express or implied warranty of fitness for such uses.

7. Software Validation Process designed to increase our confidence that a program works as intended For complex programs, guarantees are very unusual This is why typical software licenses don’t make any claims about their program working

8. Increasing Confidence Testing Run the program on set of inputs and check the results Verification Argue formally or informally that the program always works as intended Analysis / Code reviews Poor programmer’s verification [arguable!]: examine the source code to increase confidence that it works as intended

9. Code Reviews Inspect other’s code for bugs (manually) Why code reviews alone are not enough ? – Ad-hoc techniques, no standard way of doing it – Not reproducible easily – too much reliance on human factors – Errors can occur even when all standard coding practices are followed (corner cases) – Often, things that are difficult for one person are also difficult for another (Knight & Leveson study)

10. Formal Verification Tests the correctness of a piece of code without executing it – Requires detailed formal specifications of behavior – Can reveal the presence of hard-to-find bugs Theorem-proving -Can prove correctness in all cases (inductive) -Requires significant expert intervention Model-Checking -Proves correctness up to a finite depth bound -Can be completely automated, but not always scalable Formal Verification

11. Testing and Fishing Using some successful tests to conclude that a program has no bugs, is like concluding there are no fish in the lake because you didn’t catch one! flickr cc: gsankary

12. EECE 310 - 12 Testing Testing increases confidence that a program meets its specification Testing can never be exhaustive for non-trivial programs So pick a partition of the input space for testing Systematic testing depends on partitioning 1. Partition the set of possible behaviors of the system 2. Choose representative samples from each partition 3. Make sure we covered all partitions How do you identify suitable partitions? That’s what testing is all about!!! Methods: black box, glass box,...

13. Goals of Testing Repeatable: If you find an error, you’ll want to repeat the test to show others If you correct an error, repeat the test to check you did fix it Systematic: Select test sets that cover the range of behaviors of program Select test sets that are representative of program’s real uses Documented: Keep track of what tests were performed, and their results Estimate how much more testing needs to be performed

14. What will we learn ? Testing – Goals of Testing – Black-box testing – Glass-box testing

15. Naïve Testing Consider the specification of the sqrt procedure ? How will we test it ? When are we done ? float sqrt (float x, float epsilon) throws InvalidValueException { // EFFECTS: Throws InvalidValueException // if x < 0 or 0.00001 ≥ epsilon or epsilon ≥ 0.001. // Else returns y such that x-epsilon  y 2  x+epsilon float sqrt (float x, float epsilon) throws InvalidValueException { // EFFECTS: Throws InvalidValueException // if x < 0 or 0.00001 ≥ epsilon or epsilon ≥ 0.001. // Else returns y such that x-epsilon  y 2  x+epsilon

16. Black-box Testing Generate test cases from the specification only (don’t look at code) Advantages: – Avoids making the same assumptions as the programmer – Test data is independent of the procedure’s implementation – Results can be interpreted without knowing implementation details – Forces you to write a good specification

17. Test cases for black box testing Paths through the spec – E.g., choose test cases that cover each part of the ‘requires’, ‘modifies’ and ‘effects’ clauses Boundary conditions – Choose test cases that are at or close to boundaries for ranges of inputs – Test for aliasing errors (e.g., two parameters referring to the same object) Invalid input – The program should degrade gracefully, without experiencing data loss – The program should throw an exception without propagating the error further

18. Black Box Testing: Example Paths through the spec: – “x  0” means “x>0 or x = 0”, so test both “paths” – It is not always possible to choose tests to cover the effects clause… can’t choose test cases for “x-epsilon=y 2” or “y 2 =x+epsilon” if the algorithm always generates positive errors, can’t even generate y 2 < x Boundary conditions: – As “x  0” choose: 1, 0, as values for x – As “0.00001 < epsilon < 0.001” choose: 0.000011, 0.00001, 0.0000099, 0.0011, 0.001, 0.00099, as values for epsilon – Also, try very large & very small values for x Invalid input: – Negative values for x and epsilon – Values for epsilon > 0.001, values for epsilon < 0.00001 float sqrt (float x, float epsilon) throws InvalidValueException{ // EFFECTS: Throws InvalidValueException // if x < 0 or 0.00001 ≥ epsilon or epsilon ≥ 0.001. // Else returns y such that x-epsilon  y 2  x+epsilon float sqrt (float x, float epsilon) throws InvalidValueException{ // EFFECTS: Throws InvalidValueException // if x < 0 or 0.00001 ≥ epsilon or epsilon ≥ 0.001. // Else returns y such that x-epsilon  y 2  x+epsilon

19. Aliasing Static void appendVector(Vector v1, Vector v2) throws NullPointerException { // EFFECTS: if v1 or v2 is null, throws NullPointerException // else removes all elements of v2 and appends them // in reverse order to the end of v1 if (v1 == null) throw new NullPointerException(“Vector.appendVector”); while (v2.size ( ) > 0 ) { v1.addElement( v2.lastElement( ) ); v2.removeElementAt(v2.size() – 1 ); } }

20. Black Box Testing: Abs Program Can you come up with black box test cases for the program shown below ? int abs(int x) { // returns: x returns -x // otherwise => returns x int abs(int x) { // returns: x returns -x // otherwise => returns x

21. Black Box Testing: Abs Program Can you come up with black box test cases for the program shown below ? – Will this error be detected by the test suite ? int abs(int x) { // returns: x returns -x // otherwise => returns x if (x < -5) return –x; else return x; } int abs(int x) { // returns: x returns -x // otherwise => returns x if (x < -5) return –x; else return x; }

22. What will we learn ? Testing – Goals of Testing – Black-box testing – Glass-box testing

23. Glass Box testing Examine the code and test all paths – Because black box testing can never guarantee we exercised all the code Path completeness: – A path is a sequence of statements in the code – A test set is path-complete if each path through the code is exercised by at least one case in the test set Not the same as saying each statement in the code is reached!! (Why ?)

24. Glass Box Testing: Example There are four paths in the following code So we need at least 4 test-cases. Example: – x=3, y=2, z=1 – x=3, y=2, z=4 – x=2, y=3, z=2 – x=2, y=3, z=4 Are the above tests sufficient ? int maxval(int x,int y,int z) { // EFFECTS: returns maximum // value of the three inputs if (x > y) { if (x > z) return x else return z } else { if (y > z) return y else return z } } int maxval(int x,int y,int z) { // EFFECTS: returns maximum // value of the three inputs if (x > y) { if (x > z) return x else return z } else { if (y > z) return y else return z } } x > y x > z y > z

25. Weaknesses of path completeness Path completeness is insufficient Consider the test case x=4, y=1, z=2. This is path complete. The program performs correctly on this test case but the program is still wrong !! Path completeness is usually infeasible – How many paths are there through this program segment (for any a[i])? – Loops are problematic. Try: test 0, 1, 2, n-1, and n iterations, (n is the max number of iterations possible) for (j=0, i=0; i<100; i++) if a[i]=true then j=j+1 for (j=0, i=0; i<100; i++) if a[i]=true then j=j+1 int maxval (int x, y, z) { /* EFFECTS: returns maximum value of the three inputs */ return z; } int maxval (int x, y, z) { /* EFFECTS: returns maximum value of the three inputs */ return z; }

26. How to approximate Path Completeness ? 1. Loops: – With variable amounts of iteration 0, 1, 2 iterations All possible ways to exit loop for each case 2. For each statement where an exception could be raised – Test case that attempts to cause the exception 3. Straight-line code All paths through the code THESE ARE ALL HEURISTICS. THERE IS NO SILVER BULLET.

27. Path Completeness Approximation: Example Test NPException: – factorial(-1) Paths through the loop: – 0 iterations (n = 1) returns 1 – 1 iteration (n = 2) returns 1 (Bug found !) 2 iterations (n = 3) returns 2 (Bug found !) The program has a bug  the loop condition should be (i <= n) not (i < n) int factorial( int n ) { int result = 1;int i; if (n < 0) throw new NPException(“fact”); for (i = 1; i < n; i++) { result = result * i; } return result; }

28. Triangle Example Assume that the triangle testing procedure was as follows. Can you come up with white-box tests for it ? String triangle (int x, int y, int z) { if ( (x + y <= z) || (x + z <=y) || (y + z <=x) ) return “Invalid”; if (x == y || x == z) if (x==y && y==z) return “Equilateral”; else return “Isoceles”; return “Scalene”; }

29. Condition Completeness Path completeness is alone not enough in the triangle case as the same path can result from different combinations of conditionals Try to exercise all combinations of conditional statements in the code – Aim for condition completeness rather than path completeness – Ensure that side-effects of conditionals are covered

30. Group Activity static boolean isPalindrome (String s) throws NullPointerException // EFFECTS: If s is null throws NullPointerException, // else returns true if s reads the same forward // and backward, else returns false. // E.g., “deed” and “a” are both palindromes. Can you come up with black box test cases ? Can you come up with glass box test cases ? How much overlap is there among them ?

31. Black-box testing cases reasoninputexpected output Testing all paths of the specification Test for the null argumentnull NullPointerException Make it return true and false “deed”TRUE “seed”FALSE Testing boundary conditions empty string“” one character string“d”TRUE Invalid inputs Can you think of any ?

32. Glass-box testing cases ReasonInputExpected output Test for s.length( ) when s is nullnullNullPointerException Not executing the loop“d”TRUE Returning false in the first iteration“seed”FALSE Returning true after the first iteration“zz”TRUE Returning false in the second iteration“abca”FALSE Returning true after the second iteration“deed”TRUE One more input of odd length“abc”FALSE

33. Putting all the test cases together ReasonInputExpected output Test for s.length( ) when s is null & testing for null argument (bb) nullNullPointerException Empty string (bb)“”TRUE Not executing the loop & one character string (bb) “d”TRUE Returning true after the first iteration“zz”TRUE One more input of odd length“abc”FALSE Returning false in the first iteration & make it return false (bb) “seed”FALSE Returning true after the second iteration & make it return true (bb) “deed”TRUE Returning false in the second iteration“abca”FALSE

34. Summary so far: Testing Testing is vital for finding bugs in programs and writing robust code Testing can be black box or white box – Black box Testing : Looks only at specifications – Glass Box Testing: Looks at code, aim for complete path completeness – though not possible always – Neither is sufficient to guarantee finding all errors

35. Unit testing each unit is tested separately to check it meets its specification Integration testing units are tested together to check they work together. Two strategies: Integration testing is hard: much harder to identify equivalence classes problems of scale tends to reveal specification errors rather than integration errors Integration Testing Bottom up for this dependency graph, the test order is: 1) d 2) e and r 3) q 4) p p q r ed Top down for this structure chart the order is: 1) test a with stubs for b, c, and d 2) test a+b+c+d with stubs for e…k 3) test whole system b a efg c hi d jk

36. System-level testing Other types of tests facility testing - does the system provide all the functions required? volume testing - can the system cope with large data volumes? stress testing - can the system cope with heavy loads? endurance testing - will the system continue to work for long periods? usability testing - can the users use the system easily? security testing - can the system withstand attacks? performance testing - how good is the response time? storage testing - are there any unexpected data storage issues? configuration testing - does the system work on all target hardware? installability testing - can we install the system successfully? reliability testing - how reliable is the system over time? recovery testing - how well does the system recover from failure? serviceability testing - how maintainable is the system? documentation testing - is the documentation accurate, usable, etc. operations testing - are the operators’ instructions right? regression testing - repeat all testing every time we modify the system!

37. Automated Testing Ideally, testing should be automated tests can be repeated whenever the code is modified (“regression testing”) takes the tedium out of extensive testing makes more extensive testing possible Will need: test driver - automates the process of running a test set sets up the environment makes a series of calls to the unit-under-test (UUT) saves results and checks they were right generates a summary for the developer test stub(s) - simulates part of the program called by the unit-under-test checks whether the UUT set up the environment correctly checks whether the UUT passed sensible input parameters to the stub passes back some return values to the UUT (according to the test case) (stubs could be interactive - ask the user to supply return values)

38. Summary so far Testing: a way to validate program correctness Partition the testing space -- using specification: black box testing -- using the code: glass box testing A large numbers of other, non-functional tests to validate programs integration testing other system tests (e.g., endurance, security, usability, volume) Automated testing

39. To do before next class 1. Read in the textbook: – Chapter sections 10.1, 10.2, – Chapter 4 – Exercises 4.1 to 4.4 and 10.1