1. Testing Your Program: Input Space Partitioning
CS 1110
Introduction to Programming
Spring 2017

2. Software is Everywhere
Software is a key ingredient in many of the devices and systems
Software defines the behaviors of the systems.
We use a lot of software
CS 1110

3. Software Failures
(1997) Ariane 5 explosion: Exception-handling bug forced self-destruct on maiden flight (64-bit to 16-bit conversion), causing $370 millions
(1999) NASA’s Mars lander: Crashed due to a unit integration fault
(2003) Northeast blackout: The alarm system in the energy management system failed due to a software error and operators were not informed of the power overload in the system – affected 40 million people in 8 US states, 10 million people in Ontario, Canada
(2012) Knight Capital’s trading software: Faults in a new trading algorithm causes $440 millions
(2014) Dropbox’s outage was due to a fault in a maintenance script
(2016) Information lost after clicking the back button while using TurboTax web software
Evidence shows huge lost due to software failures
Why? Is it because programmers don’t test their software? Is it because programmers don’t test them properly? Is it because programmers don’t test them enough?
One thing for sure, we need to test software to avoid/minimize failures
This lecture will introduce you one common, simple, systematic approach to test software (i.e., your programs)
Before we look at how we can test our programs, let’s agree on some terminologies
CS 1110

4. Terminologies
Software failures: external, incorrect behaviors with respect to the requirements or other descriptions of the expected behavior
Example: A patient gives a doctor a list of symptoms (failures)
Three kinds of failures we’ve seen
Syntax failure
Runtime failure
Logical failure
Software faults: static defects in the software
Example: The doctor tries to diagnose the root cause (fault)
Faults in software ≈ design mistakes in hardware.
CS 1110

5. An Example in Python
# Return index of the first occurrence of a letter in string,
# Otherwise, return -1
def get_index_of(string, letter):
index = -1
for i in range(1, len(string)):
if string[i] == letter:
index = i
return index
# Test1: inputs “python”, “z”
print(get_index_of("python", "z")) # expected: -1, actual: -1
# Test2: inputs “python”, “z”
print(get_index_of("python", "p")) # expected: 0, actual: -1
Fault: Should start at 0, not 1
Test1
Error: i is 1, not 0, on the first iteration
Failure: none
Test2
Error: i is 1, not 0, on the first iteration -- Error propagates to the variable index
Failure: index is -1 at the return statement
For simplicity, this example assumes a function accept a letter of size 1
CS 1110

6. Overview: Testing What is software testing? Why do we test software?
When should we test software?
Who should test software?
How do we test software?
Input space partitioning
When do we stop testing? Good enough?
CS 1110

7. Here! Test This!
My first “testing” assignment A stack of computer printouts – and no documentation
MicroSteff – big
software system
for the mac
V Jan/2007
1.44 MB
MF2-HD
DataLife
Verdatim
Big software program
Jan/2011
CS 1110

8. What is Software Testing?
Testing = process of finding input values to check against a software (today’s discussion)
Debugging = process of finding a fault given a failure (out of today’s scope)
Test case consists of test values and expected results
Test values
(inputs)
Program
Actual results
Expected results vs
Test is a process of finding input values that cause the software to fail
To determine if the software fails, given the test inputs, we compare
the actual results from running program with
the expected results (= the result that will be produced when executing the test if the program satisfies it intended behavior)
Testing is fundamentally about choosing finite sets of values from the input domain of the software being tested
Give the test inputs, compare the actual results with the expected results
CS 1110

9. Why do We Test Software? Goal of testing
Not to prove correctness, but to increase our confidence in correctness
Improve quality
Reduce overall software development cost
Preserve customer satisfaction
Get good grades in this course (and any other courses)
What fact does each test try to verify?
Benefits
You are not biased that your code works
You will better understand what you need to build
You will get insights on how to built it
- Know what to check and whether your program handle that properly
- Plan how much testing will be enough
- Plan the budget
CS 1110

10. When should We Test Software?
Testing is the most time consuming and expensive part of software development. Not testing is even more expensive.
If we have too little testing effort early, the cost of testing increases
Planning for testing after development is prohibitively expensive
Cost of late testing 60 50 40 30 20 10
Requirements
Fault origin (%)
Fault detection (%)
Unit cost (X)
Assume $1000 unit cost, per fault, 100 faults
$6K
$13K
$20K
$360K
$250K
$100K
Integration
Test
System Test
Post-
Deployment
Design
Program/Unit
test
[source: wiki]
An early start to testing reduces the cost and time to rework and produce error-free software that is delivered to the client. However in Software Development Life Cycle (SDLC), testing can be started from the Requirements Gathering phase and continued till the deployment of the software. It also depends on the development model that is being used. For example, in the Waterfall model, formal testing is conducted in the testing phase; but in the incremental model, testing is performed at the end of every increment/iteration and the whole application is tested at the end.
Testing is done in different forms at every phase of SDLC:
During the requirement gathering phase, the analysis and verification of requirements are also considered as testing.
Reviewing the design in the design phase with the intent to improve the design is also considered as testing.
Testing performed by a developer on completion of the code is also categorized as testing.
Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002
CS 1110

11. Who should Test Software?
CS 1110 students
Software designers
Software developers/programmers
Software testers
Project managers
End users
Basically, everyone involved
CS 1110

12. How do We Test Software ? Acceptance testing (equivalence classes)
Boundary testing (corner cases)
Exception testing
CS 1110

13. How do We Test Software ? Input Space Partitioning (ISP) – Choose Inputs
Testing is fundamentally about choosing finite sets of values from the input domain of the software being tested
Input space partitioning describes the input domain of the software
Input domain contains all the possible values that the input parameters can have
Input parameters can be
Parameters to a function
Data read from a file
Global variables
User level inputs
Testing is all about choosing test input values to check against the software,
If we choose test values randomly, our tests may not cover all scenarios
we may extensively test our programs and still leave out some scenarios
To systematically create meaningful tests, we can use a technique called input space partitioning
Note: many testing techniques are available but we will only look at a common, simple one
Domain for each input parameter is partitioned into regions
At least one value is chosen from each region to create a meaningful test
Each value is assumed to be equally useful for testing
CS 1110

14. ISP: Choosing Test Inputs
Identify inputs
Find their characteristics
Each characteristic allows the testers to define a partition.
Partition each characteristic and identify values
The partition must cover the entire domain; not overlap
Two approaches
Interface-based (simpler)
Develop characteristics from individual input parameters
Can be partially automated in some situations
Functionality-based (harder)
Develop characteristics from a behavior view
May result in better tests, or fewer tests that are as effective
Choose tests by combining values of inputs
How does ISP work?
Identify inputs, parameters, or other categorization  then find characteristics in the inputs
Partition each characteristic, must not overlap  each partition represents each characteristic
Choose combinations of values to create tests
CS 1110

15. How do We Test Software ? Input Space Partitioning (ISP) – Compare Results
Give the test inputs, compare the actual results with the expected results
If the actual results == expected result, the program passes the test
Otherwise, the program fails the test
The test inputs reflect the characteristics of the input parameters
The characteristics of the inputs signifies the kinds of faults/bugs in program
The kinds of faults/bugs tell what to fix
CS 1110

16. ISP: Example 1 (Interface-based) Choose Test Inputs
# Return index of the first occurrence of a letter in string,
# Otherwise, return -1
def get_index_of(string, letter):
Identify inputs
Two parameters: string , letter
Find their characteristics
C1 = string is empty
C2 = letter is empty
How to partitioning
How to choose a value from each region
CS 1110

17. ISP: Example 1 (Interface-based) Choose Test Inputs
Partition each characteristic and identify values
Choose tests by combining values of test inputs
(assume choosing letter = "p")
Characteristic P1 P2
C1 = string is empty
True
possible values: ""
False
possible values: "python"
C2 = letter is empty
possible values: "p" or "z"
More blocks/partitions means more tests
Test
string
letter
T1 (C1=True, C2=True)
""
T2 (C1=True, C2=False)
"p"
T3 (C1=False, C2=True)
"python"
T4 (C1=False, C2=False)
CS 1110

18. ISP: Example 1 (Interface-based) Compare Results
Specify the expected result of the given test inputs
Test
string
letter
Expected result
T1 (C1=True, C2=True)
"" -1
T2 (C1=True, C2=False)
"p"
T3 (C1=False, C2=True)
"python"
T4 (C1=False, C2=False)
Put a test case T4 in Python
print(get_index_of("python", "p")) # expected: 0
If actual result == expected result, the program passes the test
Otherwise, the program fails the test
Notice that we don’t even have to look at the code.
We only rely on the specification (i.e., know what the program does) to come up with test inputs,
With the given test inputs, we can identify the expected output
Then use the test inputs, run the program  compare the actual output and the expected outputs
We don’t even have to look at the code to come up with the test cases
CS 1110

19. ISP: Example 1 (Functionality-based) Choose Test Inputs
# Return index of the first occurrence of letter in string,
# Otherwise, return -1
def get_index_of(string, letter):
Identify inputs
Two parameters: string , letter
Find their characteristics
C1 = number of occurrence of letter in string
C2 = letter occurs first in string
C3 = letter occurs last in string
No need to look at the code to come up with the test cases. The specification is sufficient.
CS 1110

20. ISP: Example 1 (Functionality-based) Choose Test Inputs
Partition each characteristic and identify values
Possible values
Characteristic P1 P2 P3
C1 = number of occurrence of letter in string 1
> 1
C2 = letter occurs first in string
True
False
C3 = letter occurs last in string C P1 P2 P3 C1
"intro to python", "z"
or "intro to python", ""
or "", "z"
"intro to python", "r"
"intro to python", "o" C2
"intro to python", "i" C3
"intro to python", "n"
CS 1110

21. ISP: Example 1 (Functionality-based) Choose Test Inputs
Choose tests by combining values of test inputs
Choose a characteristic, then choose test values for the characteristic C P1 P2 P3 C1
T1("intro to python", "z")
T2("intro to python", "")
T3("", "z")
T4("intro to python", "r")
T5("intro to python", "o") C2
T6("intro to python", "i") C3
T7("intro to python", "n")
CS 1110

22. ISP: Example 1 (Interface-based) Compare Results
Specify the expected result of the given test inputs
Test
string
letter
Expected result T1
"intro to python"
"z" -1 T2
"" T3 T4
"r" 3 T5
"o" 4 T6
"i" T7
"n" 14
print(get_index_of("intro to python", "z")) # expected: -1 …
print(get_index_of("intro to python", "i")) # expected: 0
print(get_index_of("intro to python", "n")) # expected: 14
CS 1110

23. When do We Stop Testing? Time constraint Coverage
(Assignments) due date
Coverage
Test cases cover all characteristics and partitions
Characteristics and partitions determine the quality of test cases
The quality of test cases determine how thorough the program is tested
How do we know when to stop testing? How can we tell if we test enough?
CS 1110

24. Note and Tip
I didn’t fail the test. I just found 100 ways to do it wrong. – Benjamin Franklin
Did you know?
There is an actual tactic called “rubber duck debugging” where programmers verbally explain a broken code to a rubber duck in hopes of finding the solution by describing the problem
CS 1110