1. © Andrew IrelandSoftware Design F28SD2 Software Design (F28SD2): Life-Cycle Perspective - Part 2 Andrew Ireland School of Mathematical & Computer Sciences Heriot-Watt University Edinburgh

2. © Andrew IrelandSoftware Design F28SD2 Outline A strategy for dynamic analysis - testing Designing test cases – specification and construction of test case data

3. © Andrew IrelandSoftware Design F28SD2 Testing (Dynamic Analysis) Testing involves system operation, i.e. code execution. Testing that does not exploit the internal structure of the code is known as specification-based, functional, black-box, behavioural, requirement-based, … Testing that does exploit the internal structure of the code is known as structure-based, structural, white-box, glass- box, coverage based, … Testing will typically involve the construction of additional code to facilitate the testing process.

4. © Andrew IrelandSoftware Design F28SD2 A Classification of Test Case Design Techniques The aim of design techniques is to provide a systematic basis for developing effective test cases, i.e. test cases that will provide a high yield in terms of defect detection. Specification-basedStructure-basedExperience-based Equivalence analysisStatement coverageError guessing Boundary value analysis Decision coverage …Condition/Decision coverage ……

5. © Andrew IrelandSoftware Design F28SD2 A Strategy for Dynamic Analysis 1.Use specification and experience-based techniques to design initial test cases. 2.Use coverage metrics to determine the effectiveness of the initial test cases. 3.Use structure-based techniques to increase coverage if judged necessary. Note: Both positive (desirable behaviours) and negative (undesirable behaviours) test cases should be used.

6. © Andrew IrelandSoftware Design F28SD2 Error Guessing Error guessing is driven by experience and will typically cover both functional and structural perspectives. To illustrate, given a software component to test that exploits dynamic storage allocation, then an experienced tester will typically focus upon the correct use of resource allocation and deallocation, e.g. seek out memory leakage defects. By its very nature, error guessing is hard to teach. However, check lists designed around language features and related defects can play a useful role in the transfer of experience.

7. © Andrew IrelandSoftware Design F28SD2 Structural/Functional Balance While internal knowledge can greatly simplify the task of dynamic testing - over reliance on the actual code should be avoided, i.e. risk of designing test cases that demonstrate that the code does what the code does! The primary source of test cases should be the functional specification. The process of designing tests based upon functional specification will typically expose many defects before coding gets underway.

8. © Andrew IrelandSoftware Design F28SD2 Equivalence Partitioning A technique that partitions the space of possible program inputs (and potentially outputs) into a finite set of equivalence classes – a set of data values for which the program will perform the same function For example, consider a program that accepts numbers, and doubles the odd numbers – this suggests partitioning the inputs into two equivalences classes, i.e. odd and even

9. © Andrew IrelandSoftware Design F28SD2 Equivalence Classes (EC) Equivalence partitioning can significantly prune the space of possible test cases - assuming the equivalence partitioning is performed correctly! 0 8 … 4 6 2 1 9 … 5 7 3 Odd EC Even EC 14

10. © Andrew IrelandSoftware Design F28SD2 Valid and Invalid ECs Parts of the input space which a program should accept give rise to Valid Equivalence Classes Parts of the input space which a program should accept give rise to Invalid Equivalence Classes 0 -4 … -2 -3 1 5 … 3 4 2 Positive Not positive

11. © Andrew IrelandSoftware Design F28SD2 Some Guidance for Partitioning Ranges: if a requirement specifies a range then three ECs are required, one valid and two invalid. Note that there are always implementation dependent boundaries, e.g. 16-bit integer gives rise to a partition bounded by 32767 and -32768. Size: if a requirement specifies a size then three ECs are required, one valid and two invalid, e.g. a four character password gives rise to a valid EC which contains all four character alpha numeric sequences, and two invalid ECs containing less-than and greater- than four character alpha numeric sequences Sets: if a requirement specifies a set then two equivalence classes are required, one valid (member) and one invalid (non-member).

12. © Andrew IrelandSoftware Design F28SD2 Boundary Value Analysis

13. © Andrew IrelandSoftware Design F28SD2 Test Case Effectiveness When have we tested enough? Coverage metrics provide one of the most widely used approaches to judging the effectiveness of testing. A coverage metric (CM) represents the ratio of metric items executed at least once (EM) to the total number of metric items (TM): CM = EM/TM For example, if the code under test contains 1000 statements (TM), are your test cases execute 900 of the statements (EM), then you have achieved 90% statement coverage, i.e. CM = 0.9 Coverage metrics are provide a criteria for specifying and constructing structural test cases...

14. © Andrew IrelandSoftware Design F28SD2 Statement Coverage Definition: Every statement is executed at least once. Advantage: understandable & relatively simple to achieve 100% coverage. Disadvantage: Even with 100%, statement coverage is weak, e.g. 1. int* ptr = NULL; 2. if (flag) ptr = &result; 3. *ptr = 666; 100% statement coverage only requires that the execution path including flag set to true is achieved. Note that if flag is set to false, then the code crashes! That is, statement coverage misses the bug! statementsflag lines 1, 2, 3true

15. © Andrew IrelandSoftware Design F28SD2 Decision Coverage Definition: Every statement and every decision (if-statement, while- statement, etc) is executed at least once. Advantage: easy to understand and relatively simple to achieve 100% coverage and overcomes the deficiency of statement coverage. Disadvantage: 100% coverage does not mean that the deeper logical structure of the decision point will be adequately explored, e.g. if (condition1 || condition2) statement1; else statement2; case(condition1 || condition2)branch 1true 2false Note: no guarantee that the decision point will be tested when condition2 is true …

16. © Andrew IrelandSoftware Design F28SD2 Condition/Decision Coverage Definition: Every statement and every branch is executed at least once, with each condition within each branch taking on all possible values at least once. Advantage: Addresses the deficiency of decision coverage to some extent, i.e. explores the deeper structure of the decision point to some extent. Disadvantage: Can still leave untested combinations of conditions, e.g. if (condition1 && (condition2 || condition3))... casecondition1condition2condition3branch 1true 2false

17. © Andrew IrelandSoftware Design F28SD2 Multiple Condition Coverage

18. © Andrew IrelandSoftware Design F28SD2 Multiple Condition Coverage if (condition1 && (condition1 || condition2)) … casecondition1condition2condition3branch 1true 2 falsetrue 3 falsetrue 4 false 5 true false 6 truefalse 7 truefalse 8

19. © Andrew IrelandSoftware Design F28SD2 Modified Condition/Decision Coverage (MC/DC) Definition: Every condition within a decision is executed to shown that it can independently effect the outcome of the decision. Advantage: A compromise requiring fewer cases than multiple condition coverage, i.e. minimum of N+1 and a maximum of 2N cases where N denotes the number of Boolean operands involved. Disadvantage: Time consuming to calculate. No real gains when N is small.

20. © Andrew IrelandSoftware Design F28SD2 MC/DC Coverage if (condition1 && (condition1 || condition2)) … casecondition1condition2condition3branch 2true falsetrue 3 falsetrue 4 false 6 truefalse Cases 2 and 6 show independence of condition1 Value of condition1 changed Values of condition2 and condition3 fixed Both true and false branches executed

21. © Andrew IrelandSoftware Design F28SD2 MC/DC Coverage if (condition1 && (condition1 || condition2)) … casecondition1condition2condition3branch 2true falsetrue 3 falsetrue 4 false 6 truefalse Cases 2 and 4 show independence of condition2 Value of condition2 changed Values of condition1 and condition3 fixed Both true and false branches executed

22. © Andrew IrelandSoftware Design F28SD2 MC/DC Coverage if (condition1 && (condition1 || condition2)) … casecondition1condition2condition3branch 2true falsetrue 3 falsetrue 4 false 6 truefalse Cases 3 and 4 show independence of condition3 Value of condition3 changed Values of condition1 and condition2 fixed Both true and false branches executed

23. © Andrew IrelandSoftware Design F28SD2 MC/DC Coverage if (condition1 && (condition1 || condition2)) … casecondition1condition2condition3branch 2true falsetrue 3 falsetrue 4 false 6 truefalse Cases 2 and 6 show independence of condition1 Cases 2 and 4 show independence of condition2 Cases 3 and 4 show independence of condition3

24. © Andrew IrelandSoftware Design F28SD2 Test Case Specifications to Test Cases Construct test cases for the following if-statement using Condition/Decision coverage: if !(closed) && ((n < max) || flag) … Where max denotes the constant 100, closed, flag are Boolean variables and n is an integer variable. Test case specification: Test cases, e.g. TC1: closed == false, n == 99, flag true TC2: closed == true, n == 100, flag == false caseclosed(n < max)flagbranch TC1falsetrue TC2truefalse

25. © Andrew IrelandSoftware Design F28SD2 Summary Learning outcomes: –A strategy for testing –An understanding of the complementary nature of specification-based and structure-based testing methods –Equivalence partitioning & boundary value analysis –Coverage metrics for structure-based testing Recommended reading: –“Software Testing: An ISTQB-ISEB Foundation Guide” (2nd Edition). BCS, 2010. –“Software Testing in the Real World”, E. Kit, Addison-Wesley, 1995. –“Black Box Testing”, B. Beizer. Wiley & Sons, 1995. –“Applicability of modified condition/decision coverage to software testing'', J.J. Chilensky and S.P. Miller. Software Engineering J. 1994.

26. © Andrew IrelandSoftware Design F28SD2 Summary Recommended reading: –“Software Testing: An ISTQB-ISEB Foundation Guide” (2nd Edition). BCS, 2010. –“Software Testing in the Real World”, E. Kit, Addison-Wesley, 1995. –“Black Box Testing”, B. Beizer. Wiley & Sons, 1995. –“Applicability of modified condition/decision coverage to software testing'', J.J. Chilensky and S.P. Miller. Software Engineering J. 1994.