Download presentation
Presentation is loading. Please wait.
Published byGregory Oliver Modified over 6 years ago
1
Testing Verification and the Joy of Breaking Code
SENG 301
2
Learning Objectives By the end of this lecture, you should be able to:
Distinguish between validation and verification Explain why exhaustive testing is impractical Distinguish between black box and white box testing Provide a rationale for Test-Driven Development (TDD)
3
Logic of Software Testing
Software testing is the process of identifying differences between how the system behaves vs. how the system was specified to behave.
4
/. Adds two numbers together, returns the result
/** Adds two numbers together, returns the result */ addTwo(int first, int second) => int addTwo(0, 2) => 2 addTwo(1, 2) => 3 Is this code bug free?
5
Logic of Software Testing
Software testing is the process of identifying differences between how the system behaves vs. how the system was specified to behave. Thus, testing cannot tell us that the system is “bug free”, it can only tell us if bugs exist. Even if your system passes all the tests, it does not mean that it is “bug free.”
6
Logic of Testing Blind spots Robustness Reliability of code
7
Verification vs. Validation
Verification: Are we building the functionality correctly? Static verification: code review and inspection Dynamic verification: testing Validation: Are we building the desired functionality? Static validation: “Here’s a sketch of the interface, does it look right?” Dynamic validation: user acceptance testing
8
/. Adds two numbers together, returns the result
/** Adds two numbers together, returns the result */ int addTwo(int first, int second) { return first + 2; } We should be able to discover this defect through static verification analysis (i.e. inspecting the code).
9
Example Monopoly rules: “If the player lands on or passes Go in the course of his or her turn, he or she receives $200 from the Bank.” Code: gives player $200 once s/he is on a square after Go, but if s/he lands on Go, no money is given. (i.e. follows the board) Verification: this is incorrect Validation: “I didn’t want Monopoly, I wanted the Game of Life!”
10
Validation Ensuring that things follow the specification
Confirming with clients/users that Functionality is adequate Interface is usable Non-functional requirements are met Can be done before, during and after development Continual efforts in this regard Capture changes in requirements Ensure there are no “surprises” at the end
11
Verification Several different levels: unit testing, integration testing, functional/system testing, acceptance testing, regression testing, beta testing [we will return to this on Tuesday] Test case Test suite Test coverage
12
/** Adds two positive numbers together, returns the result */
int addTwoPositiveNums(int first, int second) { 1 if (first < 0 || second < 0) throw new Exception(“Argument out of range”); 3 if (first == 2) return 1 + second; 5 else return first + second; } Test case: defining specific inputs (+ preconditions), along with the expected results e.g. addTwoPositiveNums(1,2) => 3 (expected) e.g. addTwoPositiveNums(0,0) => 0 (expected) e.g. addTwoPositiveNums(-1, 0) => Exception (expected) e.g. addTwoPositiveNums(3,4) => 7 (expected) I can tell you that all of these tests pass – they all check out true with the code that’s above
13
/** Adds two positive numbers together, returns the result */
int addTwoPositiveNums(int first, int second) { 1 if (first < 0 || second < 0) throw new Exception(“Argument out of range”); 3 if (first == 2) return 1 + second; 5 else return first + second; } Test case: defining specific inputs (+ preconditions), along with the expected results e.g. addTwoPositiveNums(1,2) => 3 (expected) e.g. addTwoPositiveNums(0,0) => 0 (expected) e.g. addTwoPositiveNums(-1, 0) => Exception (expected) e.g. addTwoPositiveNums(3,4) => 7 (expected) Test suite: a related set of tests
14
addTwoPositiveNums(1,2) => 3 (expected)
/** Adds two positive numbers together, returns the result */ int addTwoPositiveNums(int first, int second) { 1 if (first < 0 || second < 0) throw new Exception(“Argument out of range”); 3 if (first == 2) return 1 + second; 5 else return first + second; } addTwoPositiveNums(1,2) => 3 (expected) addTwoPositiveNums(0,0) => 0 (expected) addTwoPositiveNums(-1, 0) => Exception (expected) addTwoPositiveNums(3,4) => 7 (expected) Test coverage: a metric that describes how much code is actually tested (e.g. lines of code, functions, etc.). “Code coverage”
15
Terminology In performing a test (i.e. executing a test case), an actual result will occur. A difference from the expected result is called a failure The root cause of the failure is called a fault (i.e. bug) Just because a fault has occurred, we may not actually see a failure until later. e.g. your MakeChange() function results in a failure, but we don’t see consequences of the failure until later when we CHECK_DELIVERY()
16
/. Adds two positive numbers together, returns the result
/** Adds two positive numbers together, returns the result */ int addTwoPositiveNums(int first, int second) { 1 if (first < 0 || second < 0) 2 throw new Exception(“Argument out of range”); 3 if (first == 2) 4 return 1 + second; 5 else 6 return first + second; } Input: first Input: second Expected result Actual result 1 2 3 -1 Exception 4 7
17
Exhaustive Testing int addTwoPositiveNums(int first, int second) We could, in principle, test every possible input, exhaustively testing everything from from Int32.MinValue through Int32.MaxValue The challenge is that practically, this would take a long, long time (in the order of thousands of years!)—just for this addition function impractical
18
Selective Testing (Test Case Selection)
Consequently, we use selective testing (i.e. handful) We will explore strategies to select good test cases on Thursday; but generally, you want to pick test cases that are likely to find bugs, or are a really big deal if there’s a bug Concentrating your time and effort where it matters Natural consequence of all this: we can’t guarantee that code is bug- free, we can only prove the existence of bugs Note: this is not some kind of “alternative facts” thing
19
Practical: What do you do when you find a problem?
You need to determine the cause of failure. What are possibilities for what the root cause could be? Code defect Test defect Incorrect expected result What should you do when you discover a defect? Fix it Note it (e.g. in a bug repository) Fixing a bug immediately can have immediate, negative downstream effects (i.e. regression bugs) e.g. another piece of code relies on the original bug (SimCity example) In large systems, organizations need to determine the risk of fixing the bug!
20
White-box testing vs. Black-box testing
White-box testing (glass-box testing): We can see the internals (i.e. code) We can debug the system at runtime This is the kind of testing you’re doing for A2 Black-box testing Testers only consider inputs and outputs (i.e. without considering source code) Tests focus strictly on requirement specs Testers interact only through the defined user interfaces This is the user’s perspective
21
Example Test Case from a Test Plan
Test 2: Washer check Purpose: Check that vending machine responds correctly to a washer being entered, normal case Setup: Machine contains 1 quarter, 1 loonie, 1 dime, 1 nickel; 1 Coke Machine is on and initialized Current value should be 0 Action: Insert a washer Expected result: Current value displayed should be 0; washer should be returned; no other change should occur Teardown: None
22
Test Plans You can (should) plan tests long before you even begin to write a line of code How: looking at specifications to Understand what functionality matters Determine what inputs to use and what results to expect Why: blind spots It is easy to assume one’s own code is correct Test cases written after you’ve written your code contain the same, false assumptions
23
Example Test Case from a Test Plan
Test 1: Insert quarter happy path Purpose: Check that vending machine responds correctly to a quarter being entered, normal case Setup: Machine contains 1 quarter, 1 loonie, 1 dime, 1 nickel; 1 Coke Machine is on and initialized; door is locked Current value should be 0 Action: Insert valid Canadian quarter Expected result: Current value displayed should be 0.25; coin should not be returned; no other change should occur Teardown: Remove the inserted quarter
24
Observe The test scripts from A1 & A2 are essentially small test plans If you have the test scripts that we used to grade your A1 before you began, it would have been easier to write the code You could have written similar test scripts yourself based on the specification, and then you could write code until those test scripts pass! This (essentially) is known as Test-Driven Development (TDD). Rather than writing tests after (or while) code is being written, one writes the tests in advance, and using these tests to “drive” development.
25
How do we know what our interface should be like if we don’t try to use it? We don’t. Writing the test before the code makes us think about our design from the code user’s perspective, leading to a usable API. - Lasse Koskela
26
Learning Objectives You should now be able to:
Distinguish between validation and verification Explain why exhaustive testing is impractical Distinguish between black box and white box testing Provide a rationale for Test-Driven Development (TDD)
27
/. Adds two numbers together, returns the result
/** Adds two numbers together, returns the result */ int addTwo(int first, int second) { return second + 1; } addTwo(1, 1) => 2 addTwo(1, 2) => 3 Is this code bug free?
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.