1. Fundamental Programming 310201 1 Fundamental Programming Testing

2. Fundamental Programming 310201 2 Review  our goal : we aim to develop simple designs that work, and that do what users want  programmers spend 75% of the time maintaining existing programs  in lectures, we’ve mentioned a number of principles and practices that can help us to achieve this goal…

3. Fundamental Programming 310201 3 Design Principles  design principles introduced in lectures:  keep it simple  take care when selecting variable names  use comments to describe your designs  use sub-programs to simplify a design  test logic using tracing and walkthroughs  produce test cases while developing designs  validate user input

4. Fundamental Programming 310201 4 Good Coding Practices  some good coding practices introduced:  declare each variable on a new line  initialise variables  indent if-blocks, else-blocks and while-blocks  align braces  use comments to explain logic  use test cases to verify the program’s logic  using tracing and walkthroughs to find bugs

5. Fundamental Programming 310201 5 Testing  in this class, we look at strategies and techniques for testing programs - we cover:  the why and when of testing  testing pseudocode designs  testing programs  we test programs using different sets of input data – test cases  in the previous lecture we saw an example of validating user input  a few more things on input data validation…

6. Fundamental Programming 310201 6 Input Data Validation  the program that follows continues looping until the user enters a valid exam mark – between 0 and the number of marks in the exam…

7. Fundamental Programming 310201 7 Input Data Validation Example write “Number of marks in exam ==> “ read NbrMarks set StudentMark to -1 while StudentMark NbrMarks write “Student’s mark ==> “ read StudentMark if StudentMark NbrMarks then write “ ERROR: Enter a value between 0 and ” write NbrMarks write NewLine :

8. Fundamental Programming 310201 8 A C++ Implementation #include void main (void) { int NbrMarks = 0; floatStudentMark = 0; cout "; cin >> NbrMarks; StudentMark = -1; while ((StudentMark NbrMarks)) { cout "; cin >> StudentMark; if ((StudentMark NbrMarks)) { cout << " ERROR: Enter a value between 0 and "; cout << NbrMarks; cout << endl; }

9. Fundamental Programming 310201 9 Input Data Validation  a dialog to test the logic looks like this… Number of marks in exam ==> 20 Student’s mark ==> -8 ERROR: Enter a value between 0 and 20 Student’s mark ==> 28 ERROR: Enter a value between 0 and 20 Student’s mark ==> 18

10. Fundamental Programming 310201 10 Input Data Validation  but what if the user enters a value with the wrong data type?  depending on the compiler, you may find that:  in the first case, the program will not accept any further input  in the second case, the program goes into a endless loop displaying “ERROR: Enter a mark…” messages Number of marks in exam ==> 23.5...? Number of marks in exam ==> a...?

11. Fundamental Programming 310201 11 Input Data Validation  it is possible to add code to guard against the user entering the wrong data type  this is a technical issue that is of no interest in this course  in this course, we will always assume that the user enters a value of the required type  do not waste time trying to develop code that will handle incorrect data types  all of the testing performed in this course will involve values of the correct type

12. Fundamental Programming 310201 12 Why Test Designs?  on some systems (collections of inter-related programs), up to 50% of the development costs are spent on testing the system - why?  the reliability of some systems can be a matter of life-and-death - examples: space systems, military systems, medical systems  reliability of commercial and administrative systems can be costly in terms of: direct loss (incorrect invoicing), development costs, system acceptance, legal fees, damaged reputation…etc

13. Fundamental Programming 310201 13 When To Test Designs?  it is a mistake to leave testing to last  testing is a cultural thing - one needs to be committed to building reliable programs  we can:  test design specifications  test pseudocode designs  test programs  test documentation  in this course, we are mainly concerned with testing pseudocode designs and programs

14. Fundamental Programming 310201 14 Testing Pseudocode Designs  we test that a design can deal with the full range of cases it is expected to handle  a test case consists of a set of data inputs  test cases most likely to expose errors in the design tend to lie at the edge of the range of possible input values

15. Fundamental Programming 310201 15 Tracing For Logic Errors  in the handout for this lecture you will find a pseudocode design for a program to to obtain the lowest & highest mark in a set of marks  this design has logic errors - one is reasonable easy to find - one is more subtle  trace the logic in the design – see if you can find the logic errors in five minutes

16. Fundamental Programming 310201 16 Activity Break

17. Fundamental Programming 310201 17 Using Test Cases To Find Errors  if you have not found the errors, consider the following test case:  what output would you expect in this case?  what output will the program produce in this case? Number of marks in Exam ==> 20 Student Mark ==> 15 Another Mark ? [Y/N] ==> N

18. Fundamental Programming 310201 18 Using Test Cases To Find Errors  for this case, the expected output is:  however, the program will produce: Number of marks in Exam ==> 20 Student Mark ==> 15 Another Mark ? [Y/N] ==> N Highest student mark: 15 Lowest student mark: 15 Highest student mark: 20 Lowest student mark: 0

19. Fundamental Programming 310201 19 Using Test Cases To Find Errors  you may have found that the design has initialisation errors:  LowestStudentMark is initially set to 0  HighestStudentMark is initially set to the maximum number of marks in the test  having fixed these errors, the design still has another logic error  what is the expected outputs, and actual outputs, for the following test cases…

20. Fundamental Programming 310201 20 Using Test Cases To Find Errors Number of marks in Exam ==> 10 Student Mark ==> 9 Another Mark ? [Y/N] ==> Y Student Mark ==> 6 Another Mark ? [Y/N] ==> N Number of marks in Exam ==> 10 Student Mark ==> 3 Another Mark ? [Y/N] ==> Y Student Mark ==> 7 Another Mark ? [Y/N] ==> N

21. Fundamental Programming 310201 21 Activity Break

22. Fundamental Programming 310201 22 Subtle Logic Error! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  this logic works fine for the following case:  maximum marks in test: 10  marks: 9, 6

23. Fundamental Programming 310201 23 Tracing A Test Case... : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects before entering loop: HighestStudentMark: 0 LowestStudentMark: 10 StudentMark: ? (undefined)

24. Fundamental Programming 310201 24 First Loop! : ==> ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 0 LowestStudentMark: 10 StudentMark: 9

25. Fundamental Programming 310201 25 First Loop! : ( if required, update highest or lowest mark… ) ==>if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 0 LowestStudentMark: 10 StudentMark: 9

26. Fundamental Programming 310201 26 First Loop! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then ==> set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 0 LowestStudentMark: 10 StudentMark: 9

27. Fundamental Programming 310201 27 First Loop! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark ==> :  value of data objects: HighestStudentMark: 9 LowestStudentMark: 10 StudentMark: 9

28. Fundamental Programming 310201 28 Second Loop! : ==> ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 9 LowestStudentMark: 10 StudentMark: 6

29. Fundamental Programming 310201 29 Second Loop! : ( if required, update highest or lowest mark… ) ==>if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 9 LowestStudentMark: 10 StudentMark: 6

30. Fundamental Programming 310201 30 Second Loop! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark ==> else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 9 LowestStudentMark: 10 StudentMark: 6

31. Fundamental Programming 310201 31 Second Loop! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then ==> set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 9 LowestStudentMark: 10 StudentMark: 6

32. Fundamental Programming 310201 32 Second Loop! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark ==> :  value of data objects: HighestStudentMark: 9 LowestStudentMark: 6 StudentMark: 6

33. Fundamental Programming 310201 33 Subtle Logic Error! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  this logic does not work for the second case:  maximum marks in test: 10  marks: 3, 7

34. Fundamental Programming 310201 34 Tracing A Test Case... : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects before entering loop: HighestStudentMark: 0 LowestStudentMark: 10 StudentMark: ? (undefined)

35. Fundamental Programming 310201 35 First Loop! : ==> ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 0 LowestStudentMark: 10 StudentMark: 3

36. Fundamental Programming 310201 36 First Loop! : ( if required, update highest or lowest mark… ) ==>if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 0 LowestStudentMark: 10 StudentMark: 3

37. Fundamental Programming 310201 37 First Loop! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then ==> set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 0 LowestStudentMark: 10 StudentMark: 3

38. Fundamental Programming 310201 38 First Loop! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark ==> :  value of data objects: HighestStudentMark: 3 LowestStudentMark: 10 StudentMark: 3

39. Fundamental Programming 310201 39 Second Loop! : ==> ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 3 LowestStudentMark: 10 StudentMark: 7

40. Fundamental Programming 310201 40 Second Loop! : ( if required, update highest or lowest mark… ) ==>if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 3 LowestStudentMark: 10 StudentMark: 7

41. Fundamental Programming 310201 41 First Loop! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then ==> set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :  value of data objects: HighestStudentMark: 3 LowestStudentMark: 10 StudentMark: 7

42. Fundamental Programming 310201 42 Second Loop! : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark ==> :  value of data objects: HighestStudentMark: 7 LowestStudentMark: 10 StudentMark: 7

43. Fundamental Programming 310201 43 Subtle Logic Error!  if time, take another couple of minutes to fix the problem…

44. Fundamental Programming 310201 44 Activity Break

45. Fundamental Programming 310201 45 Possible Solution  a simple solutions is to remove the “else”  instead of: : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark else if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :

46. Fundamental Programming 310201 46 Possible Solution  it’s: : ( if required, update highest or lowest mark… ) if StudentMark > HighestStudentMark then set HighestStudentMark to StudentMark if StudentMark < LowestStudentMark then set LowestStudentMark to StudentMark :

47. Fundamental Programming 310201 47 Testing Pseudocode Designs  to test a pseudocode design, we check that the logic can deal with the full range of cases it is expected to handle  two approaches can we taken when developing test cases:  block-box testing: simply concentrate on the range of data inputs the design must handle  white-box testing: look at loops and branches in logic; formulate test cases to fully exercise the design - eg, a test case to exercise all branches; others to exercise only some branches

48. Fundamental Programming 310201 48 Documenting A Test Case  it’s not enough for testing to be done, it must be seen to be done:  to reduce professional liability risk  to document tests to perform on program(s)  to document tests to perform on future versions of the program  the Study Guide describes a table that can be used to document a test case  the basic idea is shown on the next slide (see Study Guide Book 1 for details)

49. Fundamental Programming 310201 49 Label Prompt P1 Maximum number of marks in test: P2 Student mark: P3 Another mark? [Y/N]: Prompt Input P110 P29 P3Y P26 P3N Output Highest student mark: 9 Lowest student mark: 6

50. Fundamental Programming 310201 50 Testing Pseudocode Designs  there’s an even simpler, cost-effective way to test a design - a walk-through  a walk-through is similar to a trace, but it is a less-formal way of following a design’s logic  a walk-through of the logic is presented to a group of peers (fellow-developers/students):  very effective !  highly recommended !

51. Fundamental Programming 310201 51 Testing Programs  when developing a program, there are four types of error you will encounter:  syntax - programming language grammar error  linker - error linking machine code components  logic - false reasoning about the task  run-time – errors when running the program (e.g., a division by 0)  only logic errors arise in pseudocode designs  syntax and linker errors are relatively easy to resolve

52. Fundamental Programming 310201 52 Testing Programs  some techniques to test programs are:  walk-throughs - same as for pseudocode designs  unit testing  validation testing  system testing  final acceptance of the program is through a formal acceptance test in which the client performs tests and signs a document to testify to the correctness of the program

53. Fundamental Programming 310201 53 Unit Testing  test distinct parts of a program/system – e.g. a get valid student ID sub-program  a test harness (special testing program) is developed to exercise the unit  testing can be automated - a series of inputs, and expected outputs, can be stored in files; files can be extended to add new test cases  user input can by simulated by the harness  the harness can be used to test future versions of the unit

54. Fundamental Programming 310201 54 Validation Testing  all testing described so far is verification testing - testing that we have built the program right ; in validation testing, we test we have built the right program  beta testing is one style of validation testing; the program is released to users for testing; alpha testing is in-house testing  useability testing is another form of validation testing - bring users into labs and observing the way they use the program

55. Fundamental Programming 310201 55 System Testing  tests that are of little interest to users  the focus of interest here is the robustness, stability and performance of the system:  recovery (from a failure) testing  security testing (against unauthorised access)  stress testing (heavy data or usage loads)  performance testing (test response times)  configuration testing (new operating system, say)

56. Fundamental Programming 310201 56 Debugging Programs  we mention debugging here simply because it is an important technique for detecting and removing bugs (logic and run-time errors)  the debugger in a development environment allows us to:  trace a program - line by line  set breakpoints  examine the value of variables  set the value of variables  set watches on variables  evaluate expressions

57. Fundamental Programming 310201 57 Summary  programs are tested using test cases  walk-throughs can be very cost-effective  unit testing is the process of testing sub- programs - it can be automated  verification testing tests that we build the system right  validation testing tests that we build the right system

58. Fundamental Programming 310201 58 Summary  during system testing we test robustness, stability and performance of a system  debugging is the process of detecting and removing bugs