1. Com1040 Systems Design and Testing Part II – Testing (Based on A.J. Cowling’s lecture notes) LN-Test1+2: Introduction to Testing Marian Gheorghe ©University of SheffieldCom1040 - Testing

2. Introduction to testing Testing types Unit testing Specification-based testing AJC’s lecture notes at https://www.dcs.shef.ac.uk/~ajc/campus_only/teach/com1030/notes/index.html#testnotes MG’s slides at: http://www.dcs.shef.ac.uk/~marian/teaching/com1040/com1040.html G.J. Myers, The Art of Software Testing, 1979 M. Roper, Software Testing, 1994 Testing -Summary

3. The Testing Problem - Triangle Example The aim of this program is to classify triangles. The program accepts three positive whole numbers as lengths of the sides of a triangle. The program classifies the triangle into one of the following groups: Equilateral: all the sides have equal lengths (return 1) Isosceles: two sides have equal length, but not all three (return 2) Scalene: all the lengths are unequal (return 3) Impossible: the three lengths cannot be used to form a triangle, or form only a flat line (return 4) (it appears in Myers’ book) Find test cases which adequately runs or abruptly breaks it down. An example…

4. Java Solution int triangle(int a, int b, int c) { int mx, x, y; mx = a; x = b; y = c; if (mx < b) {x = mx; mx = b;} if (mx < c) {y = mx; mx = c;} if (mx >= x + y) {return 4; // impossible} if (a == b && b == c) {return 1; // equilateral} if (a == b || b == c || a == c) {return 2; // isosceles} return 3; // scalene }

5. Java Solution & Test int triangle(int a, int b, int c) { int mx, x, y; mx = a; x = b; y = c; if (mx < b) {x = mx; mx = b;} if (mx < c) {y = mx; mx = c;} if (mx >= x + y) {return 4; // impossible} if (a == b && b == c) {return 1; // equilateral} if (a == b || b == c || a == c) {return 2; // isosceles} return 3; // scalene } White box: code coverage (3,5,3)

6. Java Solution & Test int triangle(int a, int b, int c) { int mx, x, y; mx = a; x = b; y = c; if (mx < b) {x = mx; mx = b;} if (mx < c) {y = mx; mx = c;} if (mx >= x + y) {return 4; // impossible} if (a == b && b == c) {return 1; // equilateral} if (a == b || b == c || a == c) {return 2; // isosceles} return 3; // scalene } White box: code coverage (3,5,3)

7. Java Solution & Test int triangle(int a, int b, int c) { int mx, x, y; mx = a; x = b; y = c; if (mx < b) {x = mx; mx = b;} if (mx < c) {y = mx; mx = c;} if (mx >= x + y) {return 4; // impossible} if (a == b && b == c) {return 1; // equilateral} if (a == b || b == c || a == c) {return 2; // isosceles} return 3; // scalene } White box: code coverage (3,5,3)

8. Java Solution & Test int triangle(int a, int b, int c) { int mx, x, y; mx = a; x = b; y = c; if (mx < b) {x = mx; mx = b;} if (mx < c) {y = mx; mx = c;} if (mx >= x + y) {return 4; // impossible} if (a == b && b == c) {return 1; // equilateral} if (a == b || b == c || a == c) {return 2; // isosceles} return 3; // scalene } White box: code coverage (3,5,3)

9. Java Solution & Test int triangle(int a, int b, int c) { int mx, x, y; mx = a; x = b; y = c; if (mx < b) {x = mx; mx = b;} if (mx < c) {y = mx; mx = c;} if (mx >= x + y) {return 4; // impossible} if (a == b && b == c) {return 1; // equilateral} if (a == b || b == c || a == c) {return 2; // isosceles} return 3; // scalene } White box: code coverage (3,5,3)

10. Java Solution & Test int triangle(int a, int b, int c) { int mx, x, y; mx = a; x = b; y = c; if (mx < b) {x = mx; mx = b;} if (mx < c) {y = mx; mx = c;} if (mx >= x + y) {return 4; // impossible} if (a == b && b == c) {return 1; // equilateral} if (a == b || b == c || a == c) {return 2; // isosceles} return 3; // scalene } White box: code coverage (3,5,3) Correct!

11. Motivation for Software Testing Testing part of any scientific and engineering activity Software testing needed – errors are unavoidable: psychology of testing - limited capacity and performances of the human memory layers short term medium term long term – history of software Ariane 5 failure, 1996: http://www.ima.umn.edu/~arnold/disasters/ariane5rep.html http://www.ima.umn.edu/~arnold/disasters/ariane5rep.html Genomic data, Science 2007

12. Errors in Software Processes Errors may occur in any stage of a software project: requirements capture: misinterpreted or incomplete aspects, incorrect statements etc specifications: wrong/inadequate notations, missing reqs, unnecessary components etc design: wrongly reflected specifications, wrong diagrams etc code: syntax, logical, I/O errors, format etc

13. Definitions of Testing Hetzel: Any activity aimed at evaluating an attribute or capability of a program or system. Myers: The process of executing a program with the intent of finding errors. IEEE: The process of exercising or evaluating a system or system component by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results.

14. Testing Processes Test case Test set Test script Test execution Test analysis Test suite Anatomy of a test case: What are the parts of a test case? a description of input condition(s) a description of expected results Where do ‘‘expected results’’ come from?

15. Brief History of Software Testing* Debugging, up to 1956 Demonstration, 1957-1978 Destruction, 1979-1982 Evaluation, 1983 -… Focus on shorter development cycles, after 1990 *Some details and examples follow from http://www.cise.ufl.edu/class/cen6070/lecture_notes.html - University of Florida

16. Performing Testing When might testing guarantee an error-free program? When branch, condition, and loop coverage are achieved When dataflow testing is utilized When path and compound condition coverage are achieved When all combinations of all possible input and state variable values are covered (None of the above ?)

17. Performing Testing When Might Testing Guarantee an Error-Free Program? When branch, condition, and loop coverage are achieved When dataflow testing is utilized When path and compound condition coverage are achieved When all combinations of all possible input and state variable values are covered = exhaustive testing (None of the above)

18. Exhaustive testing Example: A module has 2 input parameters. Word size is 32 bits. Testing is completely automated: 100 nanoseconds are required for each test case. Question: How long would it take to test this module exhaustively, i.e., covering every possible combination of input values?

19. Exhaustive Testing - Answers Short Answer: too long… Long Answer: 2 64 X 100 X 10 -9         > 57,000 years! 3600 X 24 X 365

20. Exhaustive Testing - Answers Short Answer: too long… Long Answer: 2 64 X 100 X 10 -9         > 57,000 years! 3600 X 24 X 365

21. Properties of Testing Methods Construction of a test set of reasonable size Any bigger test set should show better achievements Relevant aspects of the product are covered Catch adequate faults

22. Various classifications Random testing – generate random test sets Implementation-based testing (white box, glass box – start from code) Specification-based testing (black box – use the, formal or informal, specification) Hybrid testing – mixture of the above Other classifications consider the amount of code (unit vs integration, system testing), performance (stress testing), usability (acceptance testing) etc. We focus on (1) white box unit testing and black box integration testing utilising category-partition method Testing types

23. You have learned about JUnit (Com1020 – 2 nd lecture) public class MyClass { public static double myMeth(Params) { … some code } … } public class TestMyClass { @Test public void fTestSomeFunctionality() { assertEquals(val, MyClass.myMeth(someVals), errApp); } … } White box unit testing Current class, method Testing suite

24. Through assertEquals(val, MyClass.myMeth(someVals), errApp); results returned by a method, myMeth(), of a class, MyClass, (in a certain stage of its development) are compared with predefined values, val, for some given inputs, someVals. Reminds pre- and post-conditions utilised by formal methods, including Z, to specify correct behaviour Problem: how do we choose val and someVals JUnit testing principle

25. Test every unit of code – java method: As complete as possible code coverage Using category-partition to identify suitable inputs – remember (3,5,3) for the “triangle” problem Generate suitable test sets Aim of unit testing for Crossover

26. Starting from specifications May apply to unit as well as integration and system testing Specifications may be informal – plain English or (semi)formal – diagrams or specific notations Z may be suitable for black box unit testing X-machines for black box system testing: appropriate sequence of correct functions Two more concepts, related to data set selection, apply: equivalence partition and boundary conditions Specification-based methods

27. Equivalence partition: Data set identification Exhaustive testing is not possible Identify in the set of input data “equivalent” elements – leading to similar behaviour; ex in triangle program (3,5,3), (30,50,30), (4,6,4) show the same behaviour and result, whereas (3,7,3) doesn’t! why? Enough to use one representative per equivalent class

28. Example Mail-order firm wants to use a computer system to handle customer accounts. In particular, at the end of each month, they require the system to print out a statement for each customer, detailing their purchases during the month together with their current credit rating. The system must also send out invoices to customers with outstanding payments, and should not send anything to customers who bought nothing in the past month, and owe nothing for earlier items. We need to partition: customers set, their balances, orders set, addresses set

29. Example cont Mail-order firm wants to use a computer system to handle customer accounts. In particular, at the end of each month, they require the system to print out a statement for each customer, detailing their purchases during the month together with their current credit rating. The system must also send out invoices to customers with outstanding payments, and should not send anything to customers who bought nothing in the past month, and owe nothing for earlier items. Customer partition: no customers, one customer, more than one - 3 Balance partition: customers with negative, positive, exact – 3 Orders partition: no orders, one page, more than one – 3 Addresses partition: 1 st class, 2 nd class, Eu, rest of the world - 4

30. Example - analysis Mail-order firm wants to use a computer system to handle customer accounts. In particular, at the end of each month, they require the system to print out a statement for each customer, detailing their purchases during the month together with their current credit rating. The system must also send out invoices to customers with outstanding payments, and should not send anything to customers who bought nothing in the past month, and owe nothing for earlier items. If independent cases then 3  3  3  4 equivalence classes, but Empty set of customers leads to one test No need to consider 3  3  4 cases for one and many customers So, 3  3  4 + 1 + 1 would suffice

31. Conclusions Concepts related to testing introduced Need of testing showed Testing capabilities discussed White and black box testing defined and white box unit testing illustrated