Testing Your Program: Input Space Partitioning CS 1110 Introduction to Programming Spring 2017

Software is Everywhere Software is a key ingredient in many of the devices and systems Software defines the behaviors of the systems. We use a lot of software CS 1110

Software Failures (1997) Ariane 5 explosion: Exception-handling bug forced self-destruct on maiden flight (64-bit to 16-bit conversion), causing $370 millions (1999) NASA’s Mars lander: Crashed due to a unit integration fault (2003) Northeast blackout: The alarm system in the energy management system failed due to a software error and operators were not informed of the power overload in the system – affected 40 million people in 8 US states, 10 million people in Ontario, Canada (2012) Knight Capital’s trading software: Faults in a new trading algorithm causes $440 millions (2014) Dropbox’s outage was due to a fault in a maintenance script (2016) Information lost after clicking the back button while using TurboTax web software Evidence shows huge lost due to software failures Why? Is it because programmers don’t test their software? Is it because programmers don’t test them properly? Is it because programmers don’t test them enough? One thing for sure, we need to test software to avoid/minimize failures This lecture will introduce you one common, simple, systematic approach to test software (i.e., your programs) Before we look at how we can test our programs, let’s agree on some terminologies CS 1110

Terminologies Software failures: external, incorrect behaviors with respect to the requirements or other descriptions of the expected behavior Example: A patient gives a doctor a list of symptoms (failures) Three kinds of failures we’ve seen Syntax failure Runtime failure Logical failure Software faults: static defects in the software Example: The doctor tries to diagnose the root cause (fault) Faults in software ≈ design mistakes in hardware. CS 1110

An Example in Python # Return index of the first occurrence of a letter in string, # Otherwise, return -1 def get_index_of(string, letter): index = -1 for i in range(1, len(string)): if string[i] == letter: index = i return index # Test1: inputs “python”, “z” print(get_index_of("python", "z")) # expected: -1, actual: -1 # Test2: inputs “python”, “z” print(get_index_of("python", "p")) # expected: 0, actual: -1 Fault: Should start at 0, not 1 Test1 Error: i is 1, not 0, on the first iteration Failure: none Test2 Error: i is 1, not 0, on the first iteration -- Error propagates to the variable index Failure: index is -1 at the return statement For simplicity, this example assumes a function accept a letter of size 1 CS 1110

Overview: Testing What is software testing? Why do we test software? When should we test software? Who should test software? How do we test software? Input space partitioning When do we stop testing? Good enough? CS 1110

Here! Test This! My first “testing” assignment A stack of computer printouts – and no documentation MicroSteff – big software system for the mac V.1.5.1 Jan/2007 1.44 MB MF2-HD DataLife Verdatim Big software program Jan/2011 CS 1110

What is Software Testing? Testing = process of finding input values to check against a software (today’s discussion) Debugging = process of finding a fault given a failure (out of today’s scope) Test case consists of test values and expected results Test values (inputs) Program Actual results Expected results vs Test is a process of finding input values that cause the software to fail To determine if the software fails, given the test inputs, we compare the actual results from running program with the expected results (= the result that will be produced when executing the test if the program satisfies it intended behavior) Testing is fundamentally about choosing finite sets of values from the input domain of the software being tested Give the test inputs, compare the actual results with the expected results CS 1110

Why do We Test Software? Goal of testing Not to prove correctness, but to increase our confidence in correctness Improve quality Reduce overall software development cost Preserve customer satisfaction Get good grades in this course (and any other courses) What fact does each test try to verify? Benefits You are not biased that your code works You will better understand what you need to build You will get insights on how to built it - Know what to check and whether your program handle that properly - Plan how much testing will be enough - Plan the budget CS 1110

When should We Test Software? Testing is the most time consuming and expensive part of software development. Not testing is even more expensive. If we have too little testing effort early, the cost of testing increases Planning for testing after development is prohibitively expensive Cost of late testing 60 50 40 30 20 10 Requirements Fault origin (%) Fault detection (%) Unit cost (X) Assume $1000 unit cost, per fault, 100 faults $6K $13K $20K $360K $250K $100K Integration Test System Test Post- Deployment Design Program/Unit test [source: wiki] An early start to testing reduces the cost and time to rework and produce error-free software that is delivered to the client. However in Software Development Life Cycle (SDLC), testing can be started from the Requirements Gathering phase and continued till the deployment of the software. It also depends on the development model that is being used. For example, in the Waterfall model, formal testing is conducted in the testing phase; but in the incremental model, testing is performed at the end of every increment/iteration and the whole application is tested at the end. Testing is done in different forms at every phase of SDLC: During the requirement gathering phase, the analysis and verification of requirements are also considered as testing. Reviewing the design in the design phase with the intent to improve the design is also considered as testing. Testing performed by a developer on completion of the code is also categorized as testing. Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002 CS 1110

Who should Test Software? CS 1110 students Software designers Software developers/programmers Software testers Project managers End users Basically, everyone involved CS 1110

How do We Test Software ? Acceptance testing (equivalence classes) Boundary testing (corner cases) Exception testing CS 1110

How do We Test Software ? Input Space Partitioning (ISP) – Choose Inputs Testing is fundamentally about choosing finite sets of values from the input domain of the software being tested Input space partitioning describes the input domain of the software Input domain contains all the possible values that the input parameters can have Input parameters can be Parameters to a function Data read from a file Global variables User level inputs Testing is all about choosing test input values to check against the software, If we choose test values randomly, our tests may not cover all scenarios we may extensively test our programs and still leave out some scenarios To systematically create meaningful tests, we can use a technique called input space partitioning Note: many testing techniques are available but we will only look at a common, simple one Domain for each input parameter is partitioned into regions At least one value is chosen from each region to create a meaningful test Each value is assumed to be equally useful for testing CS 1110

ISP: Choosing Test Inputs Identify inputs Find their characteristics Each characteristic allows the testers to define a partition. Partition each characteristic and identify values The partition must cover the entire domain; not overlap Two approaches Interface-based (simpler) Develop characteristics from individual input parameters Can be partially automated in some situations Functionality-based (harder) Develop characteristics from a behavior view May result in better tests, or fewer tests that are as effective Choose tests by combining values of inputs How does ISP work? Identify inputs, parameters, or other categorization  then find characteristics in the inputs Partition each characteristic, must not overlap  each partition represents each characteristic Choose combinations of values to create tests CS 1110

How do We Test Software ? Input Space Partitioning (ISP) – Compare Results Give the test inputs, compare the actual results with the expected results If the actual results == expected result, the program passes the test Otherwise, the program fails the test The test inputs reflect the characteristics of the input parameters The characteristics of the inputs signifies the kinds of faults/bugs in program The kinds of faults/bugs tell what to fix CS 1110

ISP: Example 1 (Interface-based) Choose Test Inputs # Return index of the first occurrence of a letter in string, # Otherwise, return -1 def get_index_of(string, letter): Identify inputs Two parameters: string , letter Find their characteristics C1 = string is empty C2 = letter is empty How to partitioning How to choose a value from each region CS 1110

ISP: Example 1 (Interface-based) Choose Test Inputs Partition each characteristic and identify values Choose tests by combining values of test inputs (assume choosing letter = "p") Characteristic P1 P2 C1 = string is empty True possible values: "" False possible values: "python" C2 = letter is empty possible values: "p" or "z" More blocks/partitions means more tests Test string letter T1 (C1=True, C2=True) "" T2 (C1=True, C2=False) "p" T3 (C1=False, C2=True) "python" T4 (C1=False, C2=False) CS 1110

ISP: Example 1 (Interface-based) Compare Results Specify the expected result of the given test inputs Test string letter Expected result T1 (C1=True, C2=True) "" -1 T2 (C1=True, C2=False) "p" T3 (C1=False, C2=True) "python" T4 (C1=False, C2=False) Put a test case T4 in Python print(get_index_of("python", "p")) # expected: 0 If actual result == expected result, the program passes the test Otherwise, the program fails the test Notice that we don’t even have to look at the code. We only rely on the specification (i.e., know what the program does) to come up with test inputs, With the given test inputs, we can identify the expected output Then use the test inputs, run the program  compare the actual output and the expected outputs We don’t even have to look at the code to come up with the test cases CS 1110

ISP: Example 1 (Functionality-based) Choose Test Inputs # Return index of the first occurrence of letter in string, # Otherwise, return -1 def get_index_of(string, letter): Identify inputs Two parameters: string , letter Find their characteristics C1 = number of occurrence of letter in string C2 = letter occurs first in string C3 = letter occurs last in string No need to look at the code to come up with the test cases. The specification is sufficient. CS 1110

ISP: Example 1 (Functionality-based) Choose Test Inputs Partition each characteristic and identify values Possible values Characteristic P1 P2 P3 C1 = number of occurrence of letter in string 1 > 1 C2 = letter occurs first in string True False C3 = letter occurs last in string C P1 P2 P3 C1 "intro to python", "z" or "intro to python", "" or "", "z" "intro to python", "r" "intro to python", "o" C2 "intro to python", "i" C3 "intro to python", "n" CS 1110

ISP: Example 1 (Functionality-based) Choose Test Inputs Choose tests by combining values of test inputs Choose a characteristic, then choose test values for the characteristic C P1 P2 P3 C1 T1("intro to python", "z") T2("intro to python", "") T3("", "z") T4("intro to python", "r") T5("intro to python", "o") C2 T6("intro to python", "i") C3 T7("intro to python", "n") CS 1110

ISP: Example 1 (Interface-based) Compare Results Specify the expected result of the given test inputs Test string letter Expected result T1 "intro to python" "z" -1 T2 "" T3 T4 "r" 3 T5 "o" 4 T6 "i" T7 "n" 14 print(get_index_of("intro to python", "z")) # expected: -1 … print(get_index_of("intro to python", "i")) # expected: 0 print(get_index_of("intro to python", "n")) # expected: 14 CS 1110

When do We Stop Testing? Time constraint Coverage (Assignments) due date Coverage Test cases cover all characteristics and partitions Characteristics and partitions determine the quality of test cases The quality of test cases determine how thorough the program is tested How do we know when to stop testing? How can we tell if we test enough? CS 1110

Note and Tip I didn’t fail the test. I just found 100 ways to do it wrong. – Benjamin Franklin Did you know? There is an actual tactic called “rubber duck debugging” where programmers verbally explain a broken code to a rubber duck in hopes of finding the solution by describing the problem CS 1110