1. Lecture 5 Test Design Techniques

2. 5.1 The Test Development Process 5.2 Categories of Test Design Techniques 5.3 Specification based or Black Box Techniques 5.4 Structure based or White Box Techniques

3. 5.1 The Test Development Process Developing test material can be split into two distinct stages: –Defining “what” needs to be tested –Defining “how” the system should be tested This process can vary from organisation to organisation, can be very formal or very informal with little documentation The more formal, the more repeatable the tests, but it does depend on the context of the testing being carried out The Test Development Process consists of the following steps : 1.Defining test conditions 2.Specifying test cases 3.Specifying test procedures 4.Developing a Test execution schedule

4. Test Execution Schedule Test Conditions, Cases, Procedures and Schedule How Test Cases Test Cases Sourced Documentation What Test Condition Priority WhenHow Test Procedure Specification Manual Test Script Automated Test Script or Test Procedure Specifications

5. 5.2 Categories of Test Design Techniques 5.2 Categories of Test Design Techniques Definition of Black Box Testing What is black box testing? Definition of White Box Testing What is white box testing? Black, White and Experience based (comparison)

6. Black and White Box Testing Definition of Black Box Testing Black-box testing : Testing, either functional or non-functional, without reference to the internal structure of the component or system.

7. What is Black Box Testing? –testing without knowing the internal workings of the code –WHAT a system does, rather than HOW it does it –typically used at System Test phase, although can be useful throughout the test lifecycle –also known as specification based testing –Applies for Functional and Non-Functional testing Black and White Box Testing Input Output If Output = Expected result then pass

8. Black and White Box Testing Definition of White Box Testing white-box testing: Testing based on an analysis of the internal structure of the component or system.

9. Black and White Box Testing What is White Box Testing? –testing based upon the structure of the code –typically undertaken at Component and Component Integration Test phases by development teams –also known as structural or glass box testing or structure based testing Input Output

10. Dynamic black-box testing Dynamic black-box testing is testing without having an insight into the details of the underlying code. –Dynamic, because the program is running –Black-box, because testing is done without knowledge of how the program is implemented. Sometimes referred to as behavioral testing. Requires an executable program and a specification (or at least a user manual). Test cases are formulated as a set of pairs –E.g., (input, expected output)

11. Test data: Inputs which have been devised to test the system. Test cases: Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification. Test Data and Test Cases

12. Test-to-pass and test-to-fail Test-to-pass: –assures that the software minimally works, –does not push the capabilities of the software, –applies simple and straightforward test cases, –does not try to “break” the program. Test-to-fail: –designing and running test cases with the sole purpose of breaking the software. –strategically chosen test cases to probe for common weaknesses in the software.

13. Black-box testing Characteristics of Black-box testing: –Program is treated as a black box. –Implementation details do not matter. –Requires an end-user perspective. –Criteria are not precise. –Test planning can begin early.

14. Black, White and Experienced based Black (Specification Based) Based on requirements From the requirements, tests are created Specification Models can be used for systematic test case design Techniques Equivalence Partitioning Boundary Value Analysis Decision Tables State Transition Testing Use Case Testing White (Structure Based) Based on code and the design of the system The tests provide the ability to derive the extent of coverage of the whole application Techniques Statement coverage Branch Coverage Decision Coverage Experience (Black box) Techniques Error Guessing Exploratory Testing Based on the knowledge of the tester Using past experienced use & intuition to “guess” where errors may occur

15. 5.3 Specification based or Black Box Techniques Equivalence Partitioning Boundary Value Analysis Decision Table Testing State Transition Testing Use Case Testing Other black box test techniques

16. 5.3.1 Equivalence Partitioning aim is to treat groups of inputs as equivalent and to select one representative input to test them all Best shown in the following example…. –If we wanted to test the following IF statement: –‘IF VALUE is between 1 and 100 (inclusive) (e.g. VALUE >=1 and VALUE <= 100) THEN …..’ –We could put a range of numbers as shown in the next slide through test cases Black Box Test Techniques

17. Equivalence Partitioning 1100 99 87 65 48 37 1953 IF Value >= 1 AND Value <= 100 THEN …. IN RANGE 0 101 OUT OF RANGE 1000 Black Box Test Techniques

18. the numbers fall into a partition where each would have the same, or equivalent, result i.e. an Equivalence Partition (EP) or Equivalence Class EP says that by testing just one value we have tested the partition (typically a mid-point value is used). It assumes that: 1. if one value finds a bug, the others probably will too 2. if one doesn't find a bug, the others probably won't either Equivalence Partitioning ( EP) Black Box Test Techniques

19. in EP we must identify Valid Equivalence partitions and Invalid Equivalence partitions where applicable (typically in range tests) the Valid partition is bounded by the values 1 and 100 plus there are 2 Invalid partitions Equivalence Partitioning Black Box Test Techniques

20. 101 1000 1100 99 87 65 48 37 1953 IF Value >= 1 AND Value <= 100 THEN …. 0 ‘VALID’ PARTITION ‘INVALID’ PARTITION Equivalence Partitioning Black Box Test Techniques Happy Path Negative Scenario

21. Time would be wasted by specifying test cases that covered a range of values within each of the three partitions, unless the code was designed in an unusual way There are more effective techniques that can be used to find bugs in such circumstances (such as code inspection) EP can help reduce the number of tests from a list of all possible inputs to a minimum set that would still test each partition Equivalence Partitioning Black Box Test Techniques

22. If the tester chooses the right partitions, the testing will be accurate and efficient If the tester mistakenly thinks of two partitions as equivalent and they are not, a test situation will be missed Or on the other hand, if the tester thinks two objects are different and they are not, the tests will be redundant EP can be used for all Levels of Testing EP is used to achieve good input and output coverage, knowing exhaustive testing is often impossible It can be applied to human input, input via interfaces to a system, or interface parameters in integration testing Equivalence Partitioning Black Box Test Techniques

23. 5.3.2 Boundary Value Analysis Boundary Value Analysis (BVA) uses the same analysis of partitions as EP and is usually used in conjunction with EP in test case design As with EP, it can be used for all Test levels BVA operates on the basis that experience shows us that errors are most likely to exist at the boundaries between partitions and in doing so incorporates a degree of negative testing into the test design BVA Test cases are designed to exercise the software on and at either side of boundary values Black Box Test Techniques

24. For our previous example of ‘Value >= 1and Value <= 100’ Value = 0 (invalid) Value = 1 (valid) Value = 2 (valid) Value = 99 (valid) Value = 100 (valid) Value = 101 (invalid) Boundary Value Analysis ( 6 values as test cases are tested ) Black Box Test Techniques

25. find the boundary and then test one value above and below it ALWAYS results in two test cases per boundary for valid inputs and three tests cases per boundary for all inputs ( 2 valid and 1 invalid) inputs should be in the smallest significant values for the boundary (e.g. Boundary of ‘a > 10.0’ should result in test values of 10.0, 10.1 & 10.2) only applicable for numeric (and date) fields Boundary Value Analysis Black Box Test Techniques

26. 5.3.3 Decision Table Testing Table based technique where –Inputs to the system are recorded –Outputs to the system are defined Inputs are usually defined in terms of actions which are Boolean (true or false) Outputs are recorded against each unique combination of inputs Using the Decision Table the relationships between the inputs and the possible outputs are mapped together As with State Transition testing, an excellent tool to capture certain types of system requirements and to document internal system design. As such can be used for a number of test levels Especially useful for complex business rules Black Box Test Techniques

27. Decision Table Structure Test 1Test 2Test 3 Input 1TTF Input 2TTF Input 3TDON’T CAREF Input 4TFT Response 1YYN Response 2YNY Response 3NYN Inputs / Actions Output / Response Each column of the table corresponds to a business rule that defines a unique combination of conditions that result in the execution of the actions associated with that rule The strength of Decision Table testing is that it creates combinations of conditions that might not otherwise have been exercised during testing Black Box Test Techniques

28. Decision Table Example Test 1Test 2Test 3 > 55 yrs oldFTT SmokerFTF Exercises 3 times a week +TFT History of Heart AttacksFTF InsureYNY Offer 10% DiscountNNY Offer 30% DiscountYNN What will be the out come of the following business rules? Joe is a 22 year old non smoker who goes to the gym 4 times / week and has no history of heart attacks in his family ( first business rule :test 1) Kevin is 62 year old non smoker who swims twice a week and plays tennis. He has no history of heart attacks in his family (third business rule: Test 3) Black Box Test Techniques

29. 5.3.4 State Transition Testing 5.3.4 State Transition Testing State Transition Testing uses the following terms: 1.state diagram: A diagram that depicts the states that a component or system can assume, and shows the events or circumstances that cause and/or result from a change from one state to another. [IEEE 610] 2.state table: A grid showing the resulting transitions for each state combined with each possible event, showing both valid and invalid transitions. 3.state transition: A transition between two states of a component or system. 4.state transition testing: A black box test design technique in which test cases are designed to execute valid and invalid state transitions. Also known as N-switch testing. An excellent tool to capture certain types of system requirements and to document internal system design. As such can be used for a number of test levels Often used in testing: –Screen dialogues –Web site transitions Black Box Test Techniques

30. State Transition Diagram State A State B Event/Action etc Starting State End State Transition Between start and end states Event from outside the system Action triggered by Event Black Box Test Techniques

31. State Transition Example Simplified Car Gears Reverse Neutral 1 st Gear 2 nd Gear 3 rd Gear Change Up/ Accelerate Change Up/ Accelerate Change Up/ Accelerate Change Down/ Decelerate Change Down/ Move Back Black Box Test Techniques Change Up/ Accelerate Change Down/ Decelerate Change Down/ Decelerate

32. State Transition - Switch Coverage Switch Coverage is a method of determining the number tests based on the number “hops” between transitions. Sometimes known as Chow 0-Switch Coverage (1 hop) R  N N  R N  1 1  N 1  2 2  1 2  3 3  2 1-Switch Coverage (2 hops) R  N  1 R  N  R N  R  N N  1  2 N  1  N 1  N  R Etc. These hops can be used to determine the VALID tests Black Box Test Techniques

33. State Transition - State Table While Switch testing helps determine the valid tests, we also need to look for the invalid tests. Invalid tests are those identified by a null output in this case Changing down from reverse Changing up from 3rd Black Box Test Techniques Change UpChange Down RAcc / NeutralNull NAcc/1 st GearDec / Reverse 1 st Acc/ 2 nd GearDec / Neutral 2 nd Acc / 3 rd gearDec / 1st Gear 3 rd NullDec / 2 nd Gear

34. State Transition – Another example for a Theatre Show reservation Black Box Test Techniques Show Options provided Request Show Options Show selected Choose Show Change Mind/ Return to Options Show Reservation Made Reserve Show Show Reservation Paid For Pay for Show Ticket Received Issue Ticket Cancelled Reservation Cancel reservation Cancel reservation (return ticket)/Issue Refund Cancel reservation/ Issue Refund 2' 6 5 4 32 1 5' 4' 6'

35. State Transition – Another type of State Table Black Box Test Techniques Current State Event/ Next State Event/ Next State ABCDEF SS1 12 213 3 4 4ES B 4321 SS C EDFA

36. 5.3.4 Use Case Testing 5.3.4 Use Case Testing Uses Cases are method of describing requirements Structured approach to requirements design Usually has one flow (though may also have alternate flows) Each use case is a process flow or scenario Particularly useful in determining tests for (Functional) Systems Test and for UAT Also good at detecting defects in the integration of interfaces The main parts of a Use Case are: –ActorsUsers of the systems –Pre conditionsWhat are the starting requirements for the use case –Post conditionsThe state the system will end up in once completed Black Box Test Techniques

37. Use Case Example ( as requirements representation or text which are the specific business scenarios) Precondition: customer must have an account Postcondition: due to transactions, balance increased or decreased Tester can extract 6 test cases as positive scenarios. Negative scenarios are application dependent

38. Syntax testing ( critical item testing ) —test cases are prepared to exercise the rule governing the format of data in a system (e.g. a Zip or Postal Code, a telephone number) Random testing ( using random generator) —test cases are selected, possibly using a pseudo- random generation algorithm, to match an operational profile Black Box Test Techniques are more commonly used than the white Box Test Techniques Note: mainly Black Box Test Techniques are more commonly used than the white Box Test Techniques Other Black Box Test Techniques

39. 5.4 Structure Based or White Box Test Techniques ( source code is available for tester ) Statement Testing Decision Testing Assessing Completeness (Coverage) Other White Box test techniques The next examples that will be shown are quite simplistic and, as such, are possible to determine manually. However, in a real life scenario, the complexity and scale of code means that this would normally be undertaken with the assistance of a test tool ( automated)

40. –aim is to display that all executable statements have been run at least once –coverage measurement is: number of statements executed total number of statements White Box Test Techniques Statement Testing 5.4.1 Statement Testing

41. 1. Read vehicle 2. Read colour 3. If vehicle = ‘Car’ Then 4. If colour = ‘Red’ Then 5. Print “Fast” 6. End If 7. End If 12 7 34 5 6 Example 1 White Box Test Techniques Statement Testing Control Flow Graph

42. 1.Read A 2. If A > 40 Then 3.A = A * 2 4. End If 5. If A > 100 Then 6. A = A – 10 7. End If 13 7 6425 Example 2 White Box Test Techniques Statement Testing

43. 1. Read bread 2. Read filling 3. If bread = ‘Roll’ Then 4.If filling = ‘Tuna’ Then 5. Price = 1.50 6.Else 7. Price = 1.00 8. End If 9. Else 10. Price = 0.75 11. End If 124/65 7 10 11 8 3/9 Example 3 White Box Test Techniques Statement Testing

44. –Aim is to demonstrate that all Decisions have been run at least once –Coverage measurement is: the number of Decisions executed the total number of Decisions –Note that Decision testing is related to Branch testing –Decisions are lines of code where there are two or more alternative routes based on a decision that has to be made, i.e. IF/THEN/ELSE –Branches are where there are one of two or more alternative routes from a line of code but also include JUMPs and GO TOs White Box Test Techniques Decision Testing 5.4.2 Decision Testing

45. 1. Read vehicle 2. Read colour 3. If vehicle = ‘Car’ Then 4. If colour = ‘Red’ Then 5. Print “Fast” 6. End If 7. End If 12 7 34 5 6 Example 1 White Box Test Techniques Decision Testing

46. 13 7 6425 Example 2 White Box Test Techniques 1.Read A 2. If A > 40 Then 3.A = A * 2 4. End If 5. If A > 100 Then 6. A = A – 10 7. End If Decision Testing

47. 1. Read bread 2. Read filling 3. If bread = ‘Roll’ Then 4.If filling = ‘Tuna’ Then 5. Price = 1.50 6.Else 7. Price = 1.00 8. End If 9. Else 10. Price = 0.75 11. End If 124/65 7 10 11 8 3/9 Example 3 White Box Test Techniques Decision Testing

48. Note that in this Technique, the Ifs and associated Elses are wrapped up as One Node. The final example requires three test cases to execute all of the possible decision branches from the two branch points (3/9 & 4/6). The left hand side would be exercised by an input of “Cheese Sandwich”, the centre line by “Cheese Roll” and the right hand side by “Tuna Roll”. Corresponding outputs would be 0.75, 1.00 and 1.50. For this example, the number of test cases is the same for Branch / Decision testing. ‘While a single test case is sufficient to test every line of code in this module (for example, use a=6 and b=3 as input), it is apparent that this level of coverage will miss testing many paths. Thus, statement coverage, while a beginning, is generally not an acceptable level of testing. Comments

49. White Box Test Techniques Statement and Decision Testing The pseudo-code can be expressed in Flow-chart format 1.Read A 2. If A > 40 Then 3.A = A * 2 4. End If 5. If A > 100 Then 6. A = A – 10 7. End If What path is required to provide 100% Statement coverage? Answer : a, b, c, d, g

50. 5.4.3 Assessing Completeness (Coverage) 5.4.3 Assessing Completeness (Coverage) 1. Read bread 2. Read filling 3. If bread = ‘Roll’ Then 4.If filling = ‘Tuna’ Then 5. Price = 1.50 6.Else 7. Price = 1.00 8. End If 9. Else 10. Price = 0.75 11. End If We already know: ( see next slide ) Statement Coverage = 3 (#5,7,10) Number of Statements = 7 the number of Statements covers ALL EXECUTABLE Statements, not just the ‘perform an action’ ones. So we include our IFs and WHILEs but exclude any ELSEs and ENDIFs (#all except 6,8,9,11) Number of Branches = 4 (T and F of each IF) (#3 and #4) White Box Test Techniques For Global Assessment

51. Read Bread Read Filling If bread = ‘Roll’ If Filling = ‘Tuna’ Print 1.00 EGP Print 1.5 EGP Print 0.75 EGP END If Y YN 1 2 3 4 5 7 9 10 11 END If 6 8 Number of Branches = 4 = Y&N of Node (3) + Y&N of Node (4) Statement Coverage = 3 = Node (5) + Node (7) + Node (10) Assessing Completeness (Coverage)-contn N

52. Assessing Completeness (Coverage) Based on the following partial test set: What is the test Statement Coverage and Decision Coverage? ( for the two cases together ) Therefore Statement Coverage = 6/7 (85%) And Decision Coverage = 3/4 (75%) See next slide for details 1. Read bread 2. Read filling 3. If bread = ‘Roll’ Then 4.If filling = ‘Tuna’ Then 5. Price = 1.50 6.Else 7. Price = 1.00 8. End If 9. Else 10. Price = 0.75 11. End If BreadFilling RollTuna SandwichHam White Box Test Techniques For Partial Assessment

53. In this example we know that there are 7 EXECUTABLE statements (1, 2, 3, 4, 5, 7, and 10) and there are 2 decision points that each require 2 test paths – therefore 4 paths. With the test set of “Tuna Roll” and “Ham Sandwich” how complete is our statement test coverage and how complete is our decision test coverage? “Roll Tuna” tests 3 executable statements (1, 2, 3, 4, 5) “Sandwich Ham” tests 3 executable statements (1, 2, 3, 10) Using both will test 6 unique executable statements (all except line 7) Since there are 7 executable statements, this test set only tests 85% (6/7) of the executable statements. This is the test statement coverage. (Executable statement 7 is not tested). “Roll Tuna” tests 2 decisions points (3-right and 4-right) “Sandwich Ham” tests 1 decision point (3-left) Using both will test 3 unique decisions (3-right, 3-left and 4-right) Since there are 4 paths, this test set only tests 75% (3/4) of the decisions. (Decision 4-left is not tested). Comments

54. Choosing test Techniques How do you choose the right technique? –Type of system –Standards –Customer or contractual requirements –Level of risk –Type of risk –Testing objectives –Documentation available –Knowledge / skills of the testers –Time and budget –Development processes Pick the right techniques for the right situation