1. Testing Verification and the Joy of Breaking Code
SENG 301

2. Learning Objectives By the end of this lecture, you should be able to:
Distinguish between validation and verification
Explain why exhaustive testing is impractical
Distinguish between black box and white box testing
Provide a rationale for Test-Driven Development (TDD)

3. Logic of Software Testing
Software testing is the process of identifying differences between how the system behaves vs. how the system was specified to behave.

4. /. Adds two numbers together, returns the result
/** Adds two numbers together, returns the result */ addTwo(int first, int second) => int addTwo(0, 2) => 2 addTwo(1, 2) => 3 Is this code bug free?

5. Logic of Software Testing
Software testing is the process of identifying differences between how the system behaves vs. how the system was specified to behave. Thus, testing cannot tell us that the system is “bug free”, it can only tell us if bugs exist.  Even if your system passes all the tests, it does not mean that it is “bug free.”

6. Logic of Testing Blind spots Robustness Reliability of code

7. Verification vs. Validation
Verification: Are we building the functionality correctly?
Static verification: code review and inspection
Dynamic verification: testing
Validation: Are we building the desired functionality?
Static validation: “Here’s a sketch of the interface, does it look right?”
Dynamic validation: user acceptance testing

8. /. Adds two numbers together, returns the result
/** Adds two numbers together, returns the result */ int addTwo(int first, int second) { return first + 2; } We should be able to discover this defect through static verification analysis (i.e. inspecting the code).

9. Example
Monopoly rules: “If the player lands on or passes Go in the course of his or her turn, he or she receives $200 from the Bank.” Code: gives player $200 once s/he is on a square after Go, but if s/he lands on Go, no money is given. (i.e. follows the board) Verification: this is incorrect Validation: “I didn’t want Monopoly, I wanted the Game of Life!”

10. Validation Ensuring that things follow the specification
Confirming with clients/users that
Functionality is adequate
Interface is usable
Non-functional requirements are met
Can be done before, during and after development
Continual efforts in this regard
Capture changes in requirements
Ensure there are no “surprises” at the end

11. Verification
Several different levels: unit testing, integration testing, functional/system testing, acceptance testing, regression testing, beta testing [we will return to this on Tuesday] Test case Test suite Test coverage

12. /** Adds two positive numbers together, returns the result */
int addTwoPositiveNums(int first, int second) {
1 if (first < 0 || second < 0)
throw new Exception(“Argument out of range”);
3 if (first == 2)
return 1 + second;
5 else
return first + second; }
Test case: defining specific inputs (+ preconditions), along with the expected results
e.g. addTwoPositiveNums(1,2) => 3 (expected)
e.g. addTwoPositiveNums(0,0) => 0 (expected)
e.g. addTwoPositiveNums(-1, 0) => Exception (expected)
e.g. addTwoPositiveNums(3,4) => 7 (expected)
I can tell you that all of these tests pass – they all check out true with the code that’s above

13. /** Adds two positive numbers together, returns the result */
int addTwoPositiveNums(int first, int second) {
1 if (first < 0 || second < 0)
throw new Exception(“Argument out of range”);
3 if (first == 2)
return 1 + second;
5 else
return first + second; }
Test case: defining specific inputs (+ preconditions), along with the expected results
e.g. addTwoPositiveNums(1,2) => 3 (expected)
e.g. addTwoPositiveNums(0,0) => 0 (expected)
e.g. addTwoPositiveNums(-1, 0) => Exception (expected)
e.g. addTwoPositiveNums(3,4) => 7 (expected)
Test suite: a related set of tests

14. addTwoPositiveNums(1,2) => 3 (expected)
/** Adds two positive numbers together, returns the result */
int addTwoPositiveNums(int first, int second) {
1 if (first < 0 || second < 0)
throw new Exception(“Argument out of range”);
3 if (first == 2)
return 1 + second;
5 else
return first + second; }
addTwoPositiveNums(1,2) => 3 (expected)
addTwoPositiveNums(0,0) => 0 (expected)
addTwoPositiveNums(-1, 0) => Exception (expected)
addTwoPositiveNums(3,4) => 7 (expected)
Test coverage: a metric that describes how much code is actually tested (e.g. lines of code, functions, etc.). “Code coverage”

15. Terminology
In performing a test (i.e. executing a test case), an actual result will occur.
A difference from the expected result is called a failure
The root cause of the failure is called a fault (i.e. bug)
Just because a fault has occurred, we may not actually see a failure until later.
e.g. your MakeChange() function results in a failure, but we don’t see consequences of the failure until later when we CHECK_DELIVERY()

16. /. Adds two positive numbers together, returns the result
/** Adds two positive numbers together, returns the result */ int addTwoPositiveNums(int first, int second) { 1 if (first < 0 || second < 0) 2 throw new Exception(“Argument out of range”); 3 if (first == 2) 4 return 1 + second; 5 else 6 return first + second; }
Input: first
Input: second
Expected result
Actual result 1 2 3 -1
Exception 4 7

17. Exhaustive Testing
int addTwoPositiveNums(int first, int second) We could, in principle, test every possible input, exhaustively testing everything from from Int32.MinValue through Int32.MaxValue The challenge is that practically, this would take a long, long time (in the order of thousands of years!)—just for this addition function  impractical

18. Selective Testing (Test Case Selection)
Consequently, we use selective testing (i.e. handful)
We will explore strategies to select good test cases on Thursday; but generally, you want to pick test cases that are
likely to find bugs, or are a really big deal if there’s a bug
Concentrating your time and effort where it matters
Natural consequence of all this: we can’t guarantee that code is bug- free, we can only prove the existence of bugs
Note: this is not some kind of “alternative facts” thing

19. Practical: What do you do when you find a problem?
You need to determine the cause of failure. What are possibilities for what the root cause could be?
Code defect
Test defect
Incorrect expected result
What should you do when you discover a defect?
Fix it
Note it (e.g. in a bug repository)
Fixing a bug immediately can have immediate, negative downstream effects (i.e. regression bugs)
e.g. another piece of code relies on the original bug (SimCity example)
In large systems, organizations need to determine the risk of fixing the bug!

20. White-box testing vs. Black-box testing
White-box testing (glass-box testing):
We can see the internals (i.e. code)
We can debug the system at runtime
This is the kind of testing you’re doing for A2
Black-box testing
Testers only consider inputs and outputs (i.e. without considering source code)
Tests focus strictly on requirement specs
Testers interact only through the defined user interfaces
 This is the user’s perspective

21. Example Test Case from a Test Plan
Test 2: Washer check
Purpose: Check that vending machine responds correctly to a washer being entered, normal case
Setup:
Machine contains 1 quarter, 1 loonie, 1 dime, 1 nickel; 1 Coke
Machine is on and initialized
Current value should be 0
Action:
Insert a washer
Expected result:
Current value displayed should be 0; washer should be returned; no other change should occur
Teardown:
None

22. Test Plans
You can (should) plan tests long before you even begin to write a line of code
How: looking at specifications to
Understand what functionality matters
Determine what inputs to use and what results to expect
Why: blind spots
It is easy to assume one’s own code is correct
Test cases written after you’ve written your code contain the same, false assumptions

23. Example Test Case from a Test Plan
Test 1: Insert quarter happy path
Purpose: Check that vending machine responds correctly to a quarter being entered, normal case
Setup:
Machine contains 1 quarter, 1 loonie, 1 dime, 1 nickel; 1 Coke
Machine is on and initialized; door is locked
Current value should be 0
Action:
Insert valid Canadian quarter
Expected result:
Current value displayed should be 0.25; coin should not be returned; no other change should occur
Teardown:
Remove the inserted quarter

24. Observe
The test scripts from A1 & A2 are essentially small test plans If you have the test scripts that we used to grade your A1 before you began, it would have been easier to write the code You could have written similar test scripts yourself based on the specification, and then you could write code until those test scripts pass! This (essentially) is known as Test-Driven Development (TDD). Rather than writing tests after (or while) code is being written, one writes the tests in advance, and using these tests to “drive” development.

25. How do we know what our interface should be like if we don’t try to use it? We don’t. Writing the test before the code makes us think about our design from the code user’s perspective, leading to a usable API. - Lasse Koskela

26. Learning Objectives You should now be able to:
Distinguish between validation and verification
Explain why exhaustive testing is impractical
Distinguish between black box and white box testing
Provide a rationale for Test-Driven Development (TDD)

27. /. Adds two numbers together, returns the result
/** Adds two numbers together, returns the result */ int addTwo(int first, int second) { return second + 1; } addTwo(1, 1) => 2 addTwo(1, 2) => 3 Is this code bug free?