1 Phase Testing

Janice Regan, For each group of units Overview of Implementation phase Create Class Skeletons Define Implementation Plan (+ determine subphases) Define Coding Standards For each unit Implement Methods in class/es Code review Unit test Create integration Test plan Create unit Test plans Release unit for integration Integration Testing System Testing Create system Test plan

Janice Regan, Testing  Goal: Deliberately trying to cause failures in a software system in order to detect any defects that might be present  Test effectively: uncovers as many defects as possible  Test efficiently: find largest possible number of defects using fewest possible tests

Janice Regan, Selection ofTest Case  Selection of test cases based on two strategies:  Black box testing strategy  Select test cases by examining expected behaviour and results of software system  Glass (white) box testing strategy  Select test cases by examining internal structure of code

Janice Regan, Select Set of Test Cases using Glass (White) Box Testing Strategy Goal: Select test cases, i.e., determine values of input, that will cause flow of execution to exercise as much (greatest coverage) of your method as possible  Test as much of your method as possible with the minimum number of test cases.  Compromise between extent of test coverage and time needed to complete the tests  More robust testing → longer test phase → more expensive testing  Less robust testing → shorter test phase → less expensive testing

Janice Regan, Glass (White) Box Testing  We treat the software system as a glass box  Can see inside, i.e., we knows about  Source code (steps taken by algorithms)  Internal data  design documentation  We can select test cases using our knowledge of the course code, hence test cases can exercise all aspects of each algorithm and data  More thorough but more time-consuming than black box testing

Janice Regan, Glass (White) Box Testing  Focus: Thoroughness (Coverage)  Test as much as is feasible  Base test design on looking in at the actual implementation  Various types of glass-box testing e.g.:  Statement testing  Branch testing  Path testing

Janice Regan, Glass (White) Box Testing  Statement testing:  Test all statements at least once  Branch Testing:  Make sure that each possible outcome from a condition is tested at least once  Path testing:  Make sure a fair # of paths in the program are executed at least once  Regarding loops: choose cases that will  Cause execution of the loop to be skipped completely  Cause loop to be executed exactly once  Cause loop to be executed more than once

Janice Regan,  Select test cases (i.e., input values) that will cause each statement in your method to be executed at least once.  When performed (when your test cases are executed), your testing will achieve “100% statement coverage”.  Can miss problems when your method contains selection statements and loops Statement coverage

Janice Regan,  Useful if your method contains selection statements  Select test cases (i.e., input values) that will cause each branch in your method to be executed at least once (if your method has selection statement(s), i.e., if statement(s)).  When performed (when your test cases are executed), your testing will achieve “100% branch coverage”  Can miss problems when the method contains loops Branch coverage

Janice Regan, Select test cases (i.e., input values) that will cause each path in your method to be executed at least once if your method has iterative statement(s), i.e., loop(s)). We will see in this lecture that it is not feasible to perform testing that achieve 100% path coverage (way too many test cases). To make up for what could be missed when path coverage testing is performed, Black-box testing strategy is also used to select additional test cases Often, Black-box testing strategy is also used to select additional test cases no matter which type of coverage is considered Path coverage

Janice Regan, Determine what to test which method of which class 2. Create a flowchart Break the code into blocks Represent the blocks in your flowchart 3.Depending on the type of code your method contains, selecting test cases to satisfy your goal can be done in three different ways: Procedure

Janice Regan, Glass-Box Testing Example /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); }

Janice Regan, Glass-Box Testing: choosing code blocks /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } Where 5 corresponds to the empty “else” statement of the “if” statement # 3. Need for 5

Janice Regan, Finding test cases: Statement coverage  Find a minimal set of test cases that satisfy statement coverage  For statement coverage each line of the code must be executed.  If each line is executed all paths through the logic diagram will be taken  To get started, examine the method an break the code into logical blocks or units.  Then create a flowchart, showing the flow of execution between blocks

Janice Regan, Flowchart: Statement coverage /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } Corresponding flowchart for statement coverage Each number on the diagram represents a corresponding block of code F

Janice Regan,  What test cases (input values) would be necessary to achieve statement coverage for this example?  We are finding the mean of a series of positive numbers. The simplest case would be to find the of one positive number (the number itself). We would need an input file "input1" containing 1 positive number. Find Test cases: Statement coverage

Janice Regan, Test case1S: 1 data value (> 0) in file /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c, f d e e, i g h j k l F d Lower case letters show order of execution

Janice Regan,  What additional test cases (input values) would be necessary to achieve 100% statement coverage for this example?  We are finding the mean of a series of positive numbers. The path that has not been executed is executed only if the sum of all numbers included in the mean is <=0. Since all numbers in the mean are positive this means that no numbers are included in the mean  Therefore, also need a test case using an empty input file "input2" as input Find Test cases: Statement coverage

Janice Regan, Test case 2S: Empty data file /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c d e F Lower case letters show order of execution Paths traversed shown as dotted lines

Janice Regan, Statement Coverage  Test case id – 1S  coverage: Start Exit  Inputs: score file containing 1 score > 0.0  Test case id – 2S  coverage: Start Exit  Inputs: empty score file  So, to achieve 100% statement coverage (test all statements), we would need two test cases, i.e., two set of input values

Janice Regan, Both test cases: all statements covered /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c, g f, j h i k l m e F Paths traversed shown as red or dotted black lines All lines of code are covered, Not all paths are covered

Janice Regan, Test Case Example  Test id - 1S  Test purpose: Unit test function FindMean( ) using Glass Box testing strategy (statement coverage: Start Exit) (Note that FindMean( ) is a function, i.e., not a method of a class. If we are testing a method, we would state the class that is under testing as well.)  Requirement # 567  Inputs: score file “score” containing 1 score, value = 34.7 (> 0.0)  Testing procedure:  Open score file “score.dat” the call FindMean( … ) (part of unit test driver)  Evaluation: no step required  Expected behaviours and results: output “The mean score is 34.7” displayed on screen  Actual behaviours and results:

Janice Regan,  Selecting our test cases based on statement coverage will not allow us to achieve 100% coverage when performing testing because  our function contains selection statements (branches, ifs).  Selecting test cases based on statement coverage may result in a set of test cases that miss some critical aspect of our function associated with the branches it includes. So…  What type of test cases (input values) could we come up with if we were to base our selection of test cases on covering branches of our function (not just statements) Are we done?

Janice Regan, Glass-Box Testing Example: Branches /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } Corresponding flowchart Each number on the diagram represents a corresponding block of code Each letter labels a particular branch of execution a b Data file must contain at least 1 value c e negative or 0 score g Positive score d f Total score > 0 i k h j Total score <= 0 l (Reached whether flow of control follows branch d (then f) or branch e (then g) ) (Data set must now be empty )

Janice Regan, Test case1B /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } j a b e g h d f c i k l

Janice Regan, Branch Coverage  Test case id – 1B  coverage:  Start-a-b-d-f-h-b-e-g-h-c-i-k-Exit  Inputs: score file containing 2 scores  score 1 > 0.0  score 2 <= 0.0  We are finding the mean of a series of positive numbers. The path that has not been executed is executed only if the sum of all numbers included in the mean is <=0. Since all numbers in the mean are positive this means that no numbers are included in the mean  Therefore, also need a test case using an empty input file "input2" as input

Janice Regan, Test case 2B: Empty data file /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c d e f g h ij k l

Janice Regan, Branch Coverage  Test case id – 2B  coverage:  Start-a-c-j-l-Exit  Inputs: score file containing 0 scores

Janice Regan,  Selecting our test cases based on branch coverage will not allow us to achieve 100% coverage when performing testing  our test cases based on branch coverage will result in a set of test cases that would still miss any potential error that could occur when the loop is executed many times (e.g. 13 times). So…  Let see what type of test cases (input values) we could come up with if we were to base our selection of test cases on covering (testing) paths of our function

Janice Regan, Glass-Box Testing: choosing code blocks /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } Where 5 corresponds to the empty “else” statement of the “if” statement # 3. Need for 5

Janice Regan, Path Coverage  Test case id – 1P -> do while loop 0 time  coverage: Start Exit  Test case id – 2P -> do loop once  Inputs: "score1P": a score file with 1 score > 0.0  coverage: Start Exit  Test case id – 3P -> do loop once  coverage: Start Exit  Inputs: "score1N: a score file with 1 score <= 0.0

Janice Regan, Test case 1P: Empty data file /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c d e f g h ij k l

Janice Regan, Test case 2P /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } j a b e g h d f c i k l

Janice Regan, Test case 3P /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } j a b e g h d f c i k l

Janice Regan, Path Coverage  Test case id – 4P -> do loop many (3) times  one possible coverage: Start Exit  Inputs: "score3": a score file with 3 scores  score 1 > 0.0  score 2 <= 0.0  score 3 > 0.0

Janice Regan, Path Coverage  Test case id – 5P -> do loop many (13) times  one possible coverage: Start Exit  Inputs: "score13": a score file with 13 scores  score 1 > 0.0  score 2 > 0.0  …  score 12 <= 0.0  score 13 > 0.0  Test case id – etc…

Janice Regan, Test case 4P, 5P /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } j a b e g h d f c i k l

Janice Regan,  Considering the various paths the flow of execution can take in our function is the most extensive (closest to 100%) coverage we can achieve in our testing of FindMean( )  And this makes total sense considering the types of statements we find in your function  Therefore, since our function is composed of iterative statements, considering the paths the flow of execution can take when selecting our test cases (i.e., input values) is bound to give us the best coverage when we perform our testing of this function

Janice Regan, Redundant test cases  When you have selected your set of test cases (here, the set of test cases using path coverage), ensure that there are no redundant test cases within the set  Want the smallest set of test cases that gives adequate testing coverage  Avoid multiple test cases with the same coverage

Janice Regan, Coverage of proposed tests  1S -- coverage: Start Exit  2S -- coverage: Start Exit  1B -- coverage: Start-a-b-d-f-h-b-e-g-h-c-i-k-Exit  2B -- coverage: Start-a-c-j-l-Exit  1P -- coverage: Start Exit  2P -- coverage: Start Exit  3P -- coverage: Start Exit  4P -- coverage: Start Exit  5P -- Start Exit

Janice Regan, Set of proposed tests  1P -- coverage: Start Exit  2P -- coverage: Start Exit  3P -- coverage: Start Exit  4P -- coverage: Start Exit  And perhaps  5P -- Start Exit