1. Topic 13Summer 2003 1 ICS 52: Introduction to Software Engineering Lecture Notes for Summer Quarter, 2003 Michele Rousseau Topic 13 Partially based on lecture notes written by Sommerville, Frost, Van Der Hoek, Taylor & Tonne. Duplication of course material for any commercial purpose without the written permission of the lecturers is prohibited

2. Topic 13Summer 2003 2 Today’s Objectives l Testing continued l How to use a Test Matrix to organize test cases l More examples of test coverage l Formal Specification based testing

3. Topic 13Summer 2003 3 Recap: Choosing Test Cases l Identify what to test functionality from requirements (black box) classes, methods, statements (white box) l Determine set of all inputs l Identify one or more bases l For each basis, define equivalence partitions l For each eq. partition, choose test cases l Use combinations of eq. partitions when appropriate l For each test input, determine correct output l Use a matrix to organize your test cases Rows: test cases Columns: equivalence partitions

4. Topic 13Summer 2003 4 Example 1float homeworkAverage(float[] scores) { 2 float min = 99999; 3 float total = 0; 4 for (int i = 0 ; i < scores.length ; i++) { 5 if (scores[i] < min) 6 min = scores[i]; 7 total += scores[i]; 8 } 9 total = total – min; 10 return total / (scores.length – 1); 11}

5. Topic 13Summer 2003 5 Testing Matrix Test case (input) Basis (subdomain) Equivalence Partitions Expected output Notes

6. Topic 13Summer 2003 6 Testing Matrix - homeworkAverage 1 Test case (input) Basis: Array length Expected output Notes EmptyOneSmallLarge ()x0.0 (87.3)x87.3 (90,95,85)x92.5 (80,81,82,83, 84,85,86,87, 88,89,90,91) x86.0

7. Topic 13Summer 2003 7 Testing Matrix - homeworkAverage 2 Test case (input) Basis:Position of minimum Expected output Notes FirstMiddleLast (80,87,88,89)x88.0 (87,88,80,89)x88.0 (99,98,0,97,96)x97.5 (87,88,89,80)x88.0

8. Topic 13Summer 2003 8 Testing Matrix - homeworkAverage 3 Test case (input) Basis:Number of minimaExpected output Notes OneSeveralAll (80,87,88,89)x88.0 (87,86,86,88)x87.0 (99,98,0,97,0)x73.5 (88,88,88,88)x88.0

9. Topic 13Summer 2003 9 Security Dialog Example

10. Topic 13Summer 2003 10 Testing Matrix - Security Dialog 1 Test case (input) Choice of Button (Black Box) Expected output Notes Ok ButtonCancel Button Press Ok x See Security Dialog #2 Enter ID, Press Cancel x Text Boxes are cleared, dialog remains Don’t Enter ID, Press Cancel x Text Boxes are cleared, dialog remains

11. Topic 13Summer 2003 11 Testing Matrix - Security Dialog 2 Test case (input) Basis: Correctness of Input, with OK Button Press Expected output Notes Valid ID, Correct PW Valid ID, Incorrect PW Invalid. ID. Test1, 77775555 x Proceed to Main Screen Test1, 7777555 x Audible Beep, Dialog Remains Tes1, 77775555x Audible Beep, Dialog Remains

12. Topic 13Summer 2003 12 Another Structural Testing Example // In a linked list class Node getSecondElement() { 1 Node head = getHead(); 2 if (head == null) 3 return null; 4 if (head.next == null) 5 return null; 6 return head.next.node; } 12 3 4 5 6

13. Topic 13Summer 2003 13 Basis _______________________Test Case (Input) Expected Output Notes Testing Matrix - getSecondElement (White) All-Statement (white box) 1 2 3 4 5 6 null list x x xnull (A  null) x x x x null (A  B  x x x x B null) (A  B  x x x x B C  null)

14. Topic 13Summer 2003 14 Basis _______________________Test Case (Input) Expected Output Notes Testing Matrix-homeworkAverage 3(White) All statements (White box) 123,4,5,6,7,8,11,4,5,8,11,4,13,14,15 Scores[30,30] x“all scores are the same” 30

15. Topic 13Summer 2003 15 Test Oracles l Where does “expected output” come from? l Critical to testing l Difficult to create systematically l Typically done with a lot of guesswork great dependence on the intuition of testers l Formal specifications make it possible to automate oracles A test oracle is a mechanism for deciding whether a test case execution failed or succeeded

16. Topic 13Summer 2003 16 Formal Verification l Techniques for proving consistency between two software descriptions subject to assumptions of proof system only as good as formal specification to prove consistency of specification to prove correctness of implementation does not show other qualities, especially the subjective ones has not been shown to scale up to large-scale software systems only informal techniques for validating against user needs

17. Topic 13Summer 2003 17 Verification with Formal Specifications

18. Topic 13Summer 2003 18 Testing with Formal Specifications

19. Topic 13Summer 2003 19 Specification-based testing should augment implementation-based testing, not replace it Specification-based testing should augment implementation-based testing, not replace it Formal Specification-Based Testing l Formalize functional testing for formal specifications l Extend implementation-based techniques to be applicable to formal specifications l Dependent on the specification language Model-based specifications: test cases selected to structurally cover specification Algebraic specifications: test cases selected to test against axioms or axioms tested by interpretation on test data State-based specifications: test cases selected to cover operations’ state changes

20. Topic 13Summer 2003 20 Something to think about: Testing Process/Integration l It is generally accepted that no single technique is sufficient to verify and validate software l How do you choose? strengths and weaknesses integration issues incremental issues