Verification and Validation (Part 2) John Morris Computer Science/ Electrical and Computer Engineering 28-Aug-151 A hard day’s work ensuring that some Japanese colleagues understand why Auckland is The City of Sails! Credit: Several slides in this presentation were adapted from originals written by Stephen M. Thebaut, University of Florida

Validation or Verification? Validation Building the right software Make sure it’s what the user wants Verification Building the software right Make sure it works Accurate, complete specification essential! 28-Aug-152

Testing Aim Locate and repair defects Axiom No matter how much testing you do, you can’t be sure that there isn’t an error waiting to bite you! Testing only reveals the presence of defects, it never proves their absence!! 28-Aug-153

Testing The alternative? Formal verification Uses formal logic to prove that software is correct Currently: Prohibitively expensive Little automated support Mainly manual techniques Error prone Only feasible when cost of failure is extreme Usually when failure leads to loss of life Air and space craft control Medical systems Nuclear plants 28-Aug-154

Testing - Motivation Definitely the least glamorous part of software development  Possibly the most expensive! If not carried out thoroughly! Estimates of the economic cost of software failure produce astronomic numbers US: $59.5 billion in ~10% of projects are abandoned entirely Including some very large ones 28-Aug-155

Exhaustive Testing Inefficient, naive? Never forget the KISS principle An automated test system can do a very large numbers of tests in a reasonable time and do them while you’re designing the next test! Analysis needed is trivial whereas Analysis for a more efficient test regime may be quite complex and error-prone It’s easy to convince someone that an exhaustively tested system is reliable 28-Aug-156

Equivalence classes – formal definition A set of equivalence classes is a partition of a set such that Each element of the set is a member of exactly one equivalence class For a set, S, and a set of equivalence classes, C i U C i = S C i  C j =  (null set) unless I = j 28-Aug-157

Equivalence classes – formal definition A set of equivalence classes is a partition of a set such that The elements of an equivalence class, C, are classified by an equivalence relation, ~ If a  C and b  C, then a ~ b The equivalence relation is Reflexivea ~ a Transitiveif a ~ b and b ~ c, then a ~ c Symmetric if a ~ b then b ~ a A Representative of each class is an arbitrary member of the class They’re all ‘equivalent’ – so choose any one! 28-Aug-158

Equivalence classes – verification Equivalence relation In the verification context, the elements of the set are the sets of input values for a function under test eg we are verifying a function f( int a, int b ) The 2-tuples (1,1), (1,2), (1,3).. (and many more!) are the elements of the set of all possible inputs for f The equivalence relation is “behaves the same way under testing” One common interpretation of this is: “follows the same path through the code” 28-Aug-159

Black Box and White Box Verification Black Box Specification is available but no code White Box Code is available and can be analyzed Equivalence classes are derived from rules in the specification and the code 28-Aug-1510

Generating the Equivalence Classes Outputs Find equivalence classes that cover outputs also! Same general rules apply as for inputs One representative of each class plus Boundaries Null output eg No items in a report – does the user want a confirming report anyway? Just one output eg Reports often have header and trailer sections - are these correctly generated for a short (<1 page) report? 28-Aug-1511 Never neglect the null case! It’s very easy to neglect at specification stage Required behaviour may be ‘obvious’ No need to write it down! It will require coding Experienced programmers know that it’s a very common source of error!

Coverage in White Box Testing Black Box testing will not usually cover all the special cases required to test data structures Often, the functional goals of the specification could be met by one of several data structures Specification may deliberately not prescribe the data structure used Allows developers to choose one meeting performance goals Permits substitution of an alternative with better performance ( vs non functional specifications) Coverage defines the degree to which white box testing covers the code Measurement of completeness of testing 28-Aug-1512

Coverage in White Box Testing Usually, at least some white box coverage goals will have been met by executing test cases designed using black-box strategies How would you know if this were the case or not? In simple modules, which don’t use internal data structures, black box classes may be adequate This is not the general case though! Various coverage criteria exist Every statement at least once Every branch taken in true and false directions Every path through the code 28-Aug-1513

Coverage in White Box Testing Coverage criteria Logic coverage Statement: each statement executed at least once Branch: each branch traversed (and every entry point taken) at least once Condition: each condition True at least once and False at least once Branch/Condition: both Branch and Condition coverage Compound Condition: all combinations of condition values at every branch statement covered (and every entry point taken) Path: all program paths traversed at least once 28-Aug-1514

Pseudocode and Control Flow Graphs input(Y) if (Y<=0) then Y = −Y end while (Y>0) do input(X) Y = Y-1 end 28-Aug-1515 “nodes” “edges”

Statement Coverage Statement Coverage requires that each statement is executed at least once Simplest form of logic coverage Also known as Node Coverage What is the minimum number of test cases required to achieve statement coverage for the program segment given next? 28-Aug-1516

Pseudocode and Control Flow Graphs input(Y) if (Y<=0) then Y = −Y end while (Y>0) do input(X) Y = Y-1 end 28-Aug-1517 “nodes” “edges”

Branch coverage Branch Coverage requires that each branch will have been traversed, and that every program entry point will have been taken, at least once Also known as Edge Coverage 28-Aug-1518

Branch Coverage – Entry points Why include “…and that every program entry point will have been taken, at least once.” Not common in HLLs ( eg Java) now Common in scripting languages Any language that allows a goto and a statement label! 28-Aug-1519

Branch Coverage (cont’d) Relationship between Statement and Branch Coverage? Possibilities: 1.None. 2.Statement Coverage subsumes Branch Coverage (“statement => branch”). 3.Branch Coverage subsumes Statement Coverage (“branch => statement”).

“statement => branch” ? Minimum input sets required for Statement Coverage? Minimum input sets required for Branch Coverage?

“branch => statement” ???

Normally, YES. But not in the presence of Dead Code Dead Code is not reachable via any executable program path.

Condition Coverage A branch predicate may have more than one condition input(x,y); if ((y<=0) || (x=0) ){ y = -y; } while ( (y>0) && (! EOF) )do { input(x); y = y-1; }

Condition Coverage Condition Coverage requires that each condition will have been True at least once and False at least once What is the relationship between Branch and Condition Coverage?

Condition Coverage (cont’d) if (A || B) s1; else s2; ABBranch test 1TFtrue test 2FFfalse ABBranch test 3TFtrue test 4FTtrue Branch Coverage + Condition Coverage Check each condition

Branch/Condition Coverage Branch/Condition Coverage requires that both Branch AND Condition Coverage be achieved Branch/Condition Coverage subsumes both Branch Coverage and Condition Coverage

Compound Condition Coverage What if the compiler generates code that masks the evaluation of conditions? That is, suppose if ((A) || (y/x=5) ) { … } is compiled so that if A is true, y/x=5 will not be evaluated

Compound Condition Coverage Compound Condition Coverage requires that all combinations of condition values at every branch statement be covered and that every entry point be taken at least once Also know as Multiple Condition Coverage Subsumes Branch/Condition Coverage, regardless of the order in which conditions are evaluated

Compound Condition Coverage (cont’d) input(X,Y) if ((Y<=0) || (X==0) ) Y = -Y Possible condition values: TT, TF, FT, FF In general, how many different combinations of condition values must be considered when a branch predicate has n conditions?

Path Coverage Path Coverage requires that all program paths be traversed at least once Often described as the “strongest” form of logic coverage? Is it stronger than Compound Condition Coverage? Path Coverage is usually impossible when loops are present How many test cases would be required to cover all paths in the example below?)

Path Coverage for (i=0;i<30;i++) { input(X,Y); if (Y<=0) { if (X<=0) Y = -X; else Y = -Y; } else Y = X+Y; } repeat 30 times

Path Coverage 3 paths 3 X 3 = 9 paths

Path Coverage (cont’d) repeat 30 times 3 X 3 X…X 3 = 3 paths 30

Path Coverage Various strategies have been developed for identifying useful subsets of paths for testing when Path Coverage is impractical: Loop Coverage Basis Paths Coverage Dataflow Coverage

Loop Coverage Loop Coverage requires that the body of loops be executed 0, 1, 2, t, max and max+1 times Rationale 0: Is some action taken in the body that must also be taken when the body is not executed? 1: Lower bound on number of times body may be executed 2: Check loop re-initialization. t: Check typical number of iterations. max: Check upper (valid) bound on number of times body may be executed. max+1: If the maximum can be exceeded, what behavior results?

Coverage tools Simple statement based coverage metrics can be checked by coverage tools Sometimes called profilers Tool augments the code by adding additional code to collect statistics – mainly counts of individual statement executions At the end of a run of tests, lists of unexecuted statements are readily produced 28-Aug-1537 input(a); if ( a < 0 ) { print(“error”); } else x = f(a); input(a); inc_count( 1002 ); if ( a < 0 ) { print(“error”); inc_count( 1004 ); } else { x = f(a); inc_count( 1006 ); }

Statement Coverage tools Mechanical confirmation that every statement has been executed once is clearly a valuable first step to our goal of demonstrating that a system has been thoroughly tested  If a statement hasn’t been executed, it’s a potential timebomb! Can be augmented for compound condition coverage by parsing code and identifying multiple conditions However, complex or subtle interactions between input parameter values will not necessarily be caught if relying on statement coverage alone! Path coverage is needed Correct equivalence class analysis should produce this manually (and therefore at considerable cost!) 28-Aug-1538

Dataflow Coverage 28-Aug-1539

Procedure – Module Verification Steps Obtain precise specification Should include definitions of exception or illegal input behaviour For each input of module Determine equivalence classes ( inc special cases) Choose representatives Determine expected outputs Repeat for outputs Many output equivalence classes are probably covered by input equivalence classes Build test table 28-Aug-1540

Procedure – Module Verification Steps Write test program Tests in tables is usually the best approach Easily maintained Test programs are part of the deliverables for a system They need to be run when any change is made To ensure that code that worked isn’t broken by a fix of something else!! or worse a new feature added! Tables are easily augmented When you discover the case that you didn’t test for! 28-Aug-1541