Computer Science 2212a/b - UWO1 Structural Testing Motivation The gcov Tool An example using gcov How does gcov do it gcov subtleties Further structural testing measures Flowcharts Structural testing and flowcharts Minimal test suites Path coverage Testing loops Time to cover: minutes

Computer Science 2212a/b - UWO2 Structural Testing: Motivation Consider a typical programming project: We have designed code based on requirements We have also built in some additional features Additional features interact with required features in various ways We need to test code of additional features We don’t remember what all of them are Because new features are not mentioned in requirements, testing based on requirements will not help us Consider a function to sort an array of integers Requirements state only that resulting array must preserve original elements and be sorted Many different algorithms exist for doing this A test suite which tests one algorithm thoroughly may not test another

Computer Science 2212a/b - UWO3 Structural Testing: Motivation continued In both these cases, the essential problem is: How do we know we have tested all of our code? For all we know, some code may never have been executed by any test that we have run User may do something that executes this untested code It will therefore be run for the first time by the user Code run for the first time often has bugs! Have we tested all of our code? Tools exist to help us answer this question These tools generally called “code coverage tools” Help us to see what percentage of code has been “covered” by some test case These tools usually targeted to individual programming languages Good free code coverage tools exist for C Seems only commercial tools exist for Java We will look at one for C

Computer Science 2212a/b - UWO4 gcov: A Structural Testing Tool gcov: GNU Project coverage utility Based on earlier utility tcov GNU Project: —Supplies free software —Created gcc compiler for C —Most Linux utility programs come from GNU Project To use gcov (overview): Compile your C program using gcc with special switches Run your program normally While running, your program will write information to special “log files” After running test cases, run gcov gcov will generate a coverage report listing which lines have been executed We will look at each of these steps in more detail

Computer Science 2212a/b - UWO5 Normal Compilation of a Program In the above diagram, Solid lines = inputs Dashed lines = outputs Normally, we just: Compile a program from source code Execute it on some selected test cases Look at the output for correctness

Computer Science 2212a/b - UWO6 Compilation Using gcov Compilation / execution with gcov is basically the same However, extra switches to gcc cause extra things to happen Switches: ­fprofile­arcs ­ftest­coverage

Computer Science 2212a/b - UWO7 Compilation Using gcov continued When we compile with the extra switches, gcc does the following extra things: Generates “map file” of the blocks of code in our source files Generates object code which counts the number of times each block of code has been executed “Block of code”: Any sequence of statements such that executing one of them guarantees we must execute the next one Example (program wordcount.c): Instead of just “gcc wordcount.c”, we say gcc ­fprofile­arcs ­ftest­coverage wordcount.c gcc generates map file wordcount.gcno Exact format of the “map file” is not important to us

Computer Science 2212a/b - UWO8 Running a gcov-Compiled Program Run program as normal on test cases First time program is run, a coverage data file appears (with extension.gcda) Coverage data file contains information about how many times each line of code has been executed Example: After running wordcount, file wordcount.gcda is created Coverage data is updated every subsequent time program is run Coverage data is cumulative; e.g. Line 42 executed 5 times on first run; Line 42 executed 10 times on second run; Therefore, coverage data stored in.gcda file shows 15 executions of line 42

Computer Science 2212a/b - UWO9 Getting Information from gcov Run gcov with source file name as argument (e.g. gcov wordcount.c) gcov writes some statistics to screen, creates a file with extension “.gcov” (e.g. wordcount.c.gcov).gcov file contains: On right: source code On left: number of times each line has been executed If lines have never been executed,.gcov file shows ##### to highlight it

Computer Science 2212a/b - UWO10 Example wordcount Program count the number of lines, words, chars in input file Original Source file: #include #define TRUE 1 #define FALSE 0 main() { int nl=0, nw=0, nc=0; int inword = FALSE; char c; c = getchar(); while (c != EOF) { ++nc; if (c == ‘\n’) {++nl;} if (c==‘ ‘ || c==‘\n’ || c ==‘\t’) {inword = FALSE;} else if (!inword) {inword = TRUE; ++nw;} c = getchar(); } printf(“%d lines, %d words, %d chars\n”, nl, nw, nc); }

Computer Science 2212a/b - UWO11 wordcount Example: Compiling and Running We compile with: gcc ­fprofile­arcs ­ftest­coverage ­o wordcount wordcount.c We get: Executable in wordcount System files wordcount.gcno We run wordcount We give it as input two empty lines (two carriage returns) File wordcount.da is created We now run gcov wordcount.c gcov tells us: 86.67% of 15 source lines executed in file wordcount.c Creating wordcount.c.gcov.

Computer Science 2212a/b - UWO12 -: 0:Source:wordcount.c -: 0:Graph:wordcount.gcno -: 0:Data:wordcount.gcda -: 0:Runs:1 -: 0:Programs:1 -: 1:#include -: 2:#define TRUE 1 -: 3:#define FALSE 0 1: 4:main() { 1: 5: int n1=0, nw=0, nc=0; 1: 6: int inword=FALSE; -: 7: char c; -: 8: 1: 9: c=getchar(); 4: 10: while (c != EOF) { 2: 11: ++nc; 2: 12: if (c=='\n') 2: 13: {++n1;} 4: 14: if (c==' ' || c=='\n' || c=='\t') 2: 15: {inword=FALSE; } #####: 16: else if (!inword) #####: 17: {inword=TRUE; ++nw; } 2: 18: c=getchar(); -: 19: } 1: 20: printf("%d lines, %d words, %d chars\n", -: 21: n1, nw, nc); -: 22:}

Computer Science 2212a/b - UWO13 wordcount Example: Explanation of the Output The two lines marked with # have never been executed The other lines have been executed For instance: The initial c = getchar() has been executed once (1 in left margin) The ++nc has been executed twice (2 in left margin) —This is because each carriage return counts as 1 character The declaration lines: int nl=0, nw=0, nc=0; int inword = FALSE; are marked as executed because they contain initialization code The declaration line char c; is not marked at all because it is just a declaration

Computer Science 2212a/b - UWO14 wordcount Example: Carrying On We now run wordcount again, this time giving just the line Zippy as input We now run gcov wordcount.c gcov tells us: % of 15 source lines executed in file wordcount.c Creating wordcount.c.gcov.

Computer Science 2212a/b - UWO15 Wordcount Example: New.gcov File -: 0:Source:wordcount.c -: 0:Graph:wordcount.gcno -: 0:Data:wordcount.gcda -: 0:Runs:2 -: 0:Programs:1 -: 1:#include -: 2:#define TRUE 1 -: 3:#define FALSE 0 2: 4:main() { 2: 5: int n1=0, nw=0, nc=0; 2: 6: int inword=FALSE; -: 7: char c; -: 8: 2: 9: c=getchar(); 12: 10: while (c != EOF) { 8: 11: ++nc; 8: 12: if (c=='\n') 3: 13: {++n1;} 11: 14: if (c==' ' || c=='\n' || c=='\t') 3: 15: {inword=FALSE; } 5: 16: else if (!inword) 1: 17: {inword=TRUE; ++nw; } 8: 18: c=getchar(); -: 19: } 2: 20: printf("%d lines, %d words, %d chars\n", -: 21: n1, nw, nc); -: 22:}

Computer Science 2212a/b - UWO16 wordcount Example: New Output We have given the system an input file of 6 characters on this run (5 letters + carriage return) Along with the 2 characters we gave on the previous run, the runs of wordcount have read 8 characters Hence, ++nc line has been executed 8 times in total Counts on other lines have similarly been updated We have now executed the case where we have a non­whitespace character Therefore, the lines else if (!inword) { inword = TRUE; ++nw; } have been executed Therefore, 100% of the lines of the program have now been executed

Computer Science 2212a/b - UWO17 Measuring the Usefulness of a Test Suite A good structural test suite will execute most lines of code It may not be possible to execute all lines Executing a large percentage may be acceptable Hence, measure of code covered (% of lines executed) is a good measure of the usefulness of such a test suite However, with gcov: As we keep executing test cases, the.gcda file keeps getting updated Therefore, to measure goodness of a test suite, what we would like to do is: Reset all counts in.gcda file to 0 Run test suite Run gcov and look at % of lines covered To reset all counts in the.gcda file: Just remove the file! Your compiled code will: Assume this is the first time it has been run again Create the.gcda file anew

Computer Science 2212a/b - UWO18 How Does gcov Do It? Each block in code numbered gcc inserts some code into your program at beginning of main, beginning of each block, end of main At beginning of main: —Program zeros out a big array During execution: —Every time block n executed, entry n of array incremented At end of main: —If.gcda file does not exist, program creates it —Otherwise, program reads.gcda file, increments counts in array, writes out.gcda file again Does similar processing for calls to exit() as at end of main Map files map source line number to block gcov program matches block execution counts in.da file with source lines

Computer Science 2212a/b - UWO19 gcov Subtleties If we compile and link many source files with the “gcov options”: Each source file gets a.gcno file (the map file) When we run the program, each source file gets a.gcda file (the coverage data file) Thus, we can/must get a separate report from gcov on each separate source file Useful if we are interested in coverage of each module of our program Example: We have a main program main.c and two modules, StackImplementation.c and IO.c On compilation, we get main.gcno, StackImplementation.gcno and IO.gcno Every time we run the program, files main.gcda, StackImplementation.gcda, and IO.gcda are updated We can say gcov StackImplementation.c —Get a report on StackImplementation.c separately —Useful if that is the module we are interested in

Computer Science 2212a/b - UWO20 #include int main() { int x; int y; int z=0; printf("\nEnter x:"); scanf ("%d", &x); printf("\nEnter y:"); scanf("%d", &y); z = x+y; if (z>x) printf("\nHi"); else printf("\nBye"); if (z>y) printf ("\nYes"); else printf("\nNo"); printf("\nAll Done"); return 0; } Let’s try it out on the gaul network: ls gcc -fprofile-arcs -ftest-coverage junk2.c

Computer Science 2212a/b - UWO21 gcov, Lines and Statements Consider the following code: scanf(“%d”, &x); /* Read x from terminal */ if (x >= 0) { pos = 1; } printf(“%d %d \n”, x, pos); We can execute 100% of lines in this code only if we input a positive number for x sometime Now consider the following code: scanf(“%d”, &x); /* Read x from terminal */ if (x >= 0) {pos = 1;} printf(“%d %d\n”, x, pos); We can execute 100% of lines in this code even if we input only negative numbers for x

Computer Science 2212a/b - UWO22 gcov, Lines and Statements continued Reason: In second code fragment, the sub­statement pos = 1 of the if is on the same line as the if We have executed the if in the sense that we have evaluated its condition (x>=0) Line is considered executed even when only some of the code on it is executed Moral: to ensure all statements executed: Must put each statement on a separate line

Computer Science 2212a/b - UWO23 Is gcov Enough? gcov is OK for a simple analysis of the thoroughness of a test suite It tells us what percentage of the lines of code we have executed However, a test suite executing 100% of lines may still not catch some problems Example: consider again the code scanf(“%d”, &x); /* Read x from terminal */ if (x>=0) { pos = 1; } printf(“%d %d\n”, x, pos); If all tests in our test suite set x to a positive number, we can achieve 100% line coverage However, we have still not tested the case in which x is negative or zero That case might be handled incorrectly —For instance, maybe pos is supposed to be 1 if x is positive, and 0 otherwise To talk about stronger measures of thoroughness, it is useful to have a graph of the program control flow

Computer Science 2212a/b - UWO24 int main() { int n1, n2, n3, ans; printf("\n\nEnter a value for n1:"); scanf("%d",&n1); printf("\n\nEnter a value for n2:"); scanf("%d",&n2); printf("\n\nEnter a value for n3:"); scanf("%d",&n3); ans = getnum(n1, n2, n3); printf("\nThe number was: %d\n”,ans); return 0; } int getnum(int num1,int num2,int num3) { if (num1>num2) return num1; else if (num2>num3) return num2; else { if (num1==num2) return num3; else return num1+num2; } Consider the code to the right and answer the following questions: gcov would count 16 lines, which lines do you think get counted? What % coverage would the following test case give: n1  5, n2  4, n3  3 Can we get 100% coverage? If so, list each of the test cases you would use to get that coverage, give the minimal number of test cases in your answer needed to get 100% coverage. Junit info Review Questions