1 Testing and Maintainability. 2 Bibliography Fagan, M.E. Design and Code Inspections to Reduce errors in Program Development, IBM Journal 3:182-211 (1976).

1 Testing and Maintainability

2 Bibliography Fagan, M.E. Design and Code Inspections to Reduce errors in Program Development, IBM Journal 3:182-211 (1976). Beizer, B. Software Testing Techniques Second Edition, Van Nostrand Reinhold 1990 G. Gordon Schulmair et al. Handbook of Software Quality AssuranceVan Nostrand Reinhold 1992 Drexel University Course on Testing S. Kirani, W. T. Tsai, “Method Sequence Specification and Verification of Classes”, Journal of Object-Oriented Programming, October, 1994.

3 Course Goal Show why is testing needed Teach how to test systems Teach how to inspect documents Explain what is needed for maintainability

4 What Will NOT Discuss Science Philosophy Interesting Software Problems Correct Use of Language We Will Discuss The Customer Perspective Software Life Cycle Production of better systems

5 Customer perspective When does the Consumer “see” the system What bothers the consumer What happens with half working systems What if damage is done What if revenue is withheld What does the customer want

6 Software Life Cycle Successive approximations Old Model Requirements Requirements analysis HLD + LLD Code Integration

7 Software Life Cycle What is missing –The different testing levels –The positive feedback –Exact development All come under “testing” Why plan and what is planned

8 What is Testability Maintainability The 5-Nines Model Gudel Theorem and Testing So why should we?

9 Godel (Geodel) Theorem Gödel's proof, which states that within any rigidly logical mathematical system there are propositions (or questions) that cannot be proved or disproved on the basis of the axioms within that system and that, therefore, it is uncertain that the basic axioms of arithmetic will not give rise to contradictions. This proof has become a hallmark of 20th-century mathematics, and its repercussions continue to be felt and debated.

10 What Is There For Me Better Product ==> Company’s Success Continuous Appreciation (Company and Customer) Personal success Shorter Development ==> New Products Early Solution ==> Shorter & Better Development

11 Shorter & Better Development? Problems are found in every phase Divide into 3 main Phases Development (up to integration) System Test Delivered System (problems from the field) Problem elimination: Cheapest the earlier found

12 Your Way to Create Quality What is the quality of your development? - How many errors are discovered - After delivery/or when you think - - you have completed. How long does it take to “clean” What is the final productivity? Where can we shorten development cycle?

13 Low Quality Issues Job Security The Feedback Factory Low Availability Low Reliability Customers hate it US Car Industry, Motorola, Elscint No new Products, Sales Down etc.

14 Solution..... DEVELOP WITH NO BUGS! Impossible CLASSICAL WATER FALL MODEL Rigid passage from Phase to Phase The Whirlpool Model No rigid passage but exit and entry criteria Multiple phases in parallel

15 Software Testing Software testing is a critical element of software quality assurance and represents the ultimate review of: –specification –design –coding Software life-cycle models (e.g., waterfall) frequently include software testing as a separate phase that follows implementation!

16 Software Testing (Cont’d) Contrary to life-cycle models, testing is an activity that must be carried out throughout the life-cycle. It is not enough to test the end product of each phase. Ideally, testing occurs during each phase.

17 Software Testing Terminology Error: A measure of the difference between the actual and the ideal. Fault: A condition that causes a system to fail in performing its required function. Failure: The inability of a system or component to perform a required function according to its specifications. Debugging: The activity by which faults are identified and rectified.

18 Software Testing Myths If we were really good at programming, there would be no bugs to catch. There are bugs because we are bad at what we do. Testing implies an admission of failure. Testing is a punishment for our errors.

19 Software Testing Myths (Cont’d) All we need to do is: –concentrate –use structured programming –use OO methods –use a good programming language –...

20 Software Testing Reality Human beings make mistakes, especially when asked to create complex artifacts such as software systems. Studies show that even good programs have 1-3 bugs per 100 lines of code. People who claim that they write bug-free software probably haven’t programmed much.

21 Goals of Testing Discover and prevent bugs. The act of designing tests is one of the best bug preventers known. (Test, then code philosophy) The thinking that must be done to create a useful test can discover and eliminate bugs in all stages of software development. However, bugs will always slip by, as even our test designs will sometimes be buggy.

22 Phases in a Testers Mental Life Testing is debugging. The purpose of testing is to show that the software works. The purpose of testing is to show that the software doesn’t work. The purpose of testing is to reduce the risk of failure to an acceptable level.

23 Testing Isn’t Everything Other methods for improving software reliability are: –Inspection methods: Walkthroughs, formal inspections, code reading. –Design style: Criteria used by programmers to define what they mean by a “good program”. –Static analysis: Compilers take over mundane tasks such as type checking. –Good Programming Languages and Tools: Can help reduce certain kinds of bugs (e.g., Lint).

24 Testing Versus Debugging The purpose of testing is to show that a program has bugs. The purpose of debugging is to find the faults that led to the program’s failure and to design and implement the program changes that correct the faults. Testing is a demonstration of failure or apparent correctness. Debugging is a deductive process.

25 Testing Versus Debugging (Cont’d) Testing proves a programmer’s failure. Debugging is a programmer’s vindication. Testing can be automated to a large extent. Automatic debugging is still a dream. Much of testing can be done without design knowledge (by an outsider). Debugging is impossible without detailed design knowledge (by an insider).

26 Function Versus Structure Functional Testing: –Program is treated as a black box. –Program is subjected to inputs, and its outputs are verified for conformance to specified behavior. –Implementation details do not matter. –Takes a user’s point of view. –In principle, can detect all bugs in an infinite amount of time.

27 Function Versus Structure (Cont’d) Structural Testing: –Aims at exercising the different control and data structures used in the program. –Criteria are precise as they are based on program structures (i.e., are quite precise). –Looks at implementation details. –Takes a developer’s point of view. –Is inherently finite but cannot detect all faults.

28 Designer Versus Tester The designer and the tester can be completely separated if only functional testing is performed. In structural testing the tester, like the designer, has intimate knowledge of the structure of the program.

29 Designer Versus Tester (Cont’d) The more the tester knows about the design, the more likely he will eliminate useless tests (functional differences handled by the same code). Testers that have design knowledge may have the same misconceptions as the designer.

30 Designer Versus Tester (Cont’d) Lack of design knowledge may help the tester to develop test cases that a designer would never have thought of. Lack of design knowledge may result in inefficient testing and blindness to missing functions and strange cases.

31 Small Versus Large Systems For small systems with one user, quality assurance may not be a major concern. As systems scale up in size and number of users, quality assurance becomes more of a concern. As systems dramatically scale up in size (e.g., millions of lines of code), our quality criteria may have to change as exhaustive testing may not be economically possible. –E.g., a 75% code coverage may be acceptable.

32 Programs and their Environment A program’s environment is the hardware and systems software required to make it run. Programmers should learn early in their careers that it is not smart to blame the environment for bugs. Bugs in the environment are rare because most bugs have been found over a long period of usage by a large number of users.

33 Myths About Bugs Benign Bug Hypothesis: Bugs are nice, tame, and logical. Bug Locality Hypothesis: A bug discovered within a component affects only that component’s behavior. Control Bug Dominance: Most bugs are in the control structure of programs. Corrections Abide: A corrected bug remains correct.

34 Myths About Bugs (Cont’d) Silver Bullets: A language, design method, environment grants immunity from bugs. Sadism Suffices: All bugs can be caught using low cunning and intuition. (Only easy bugs can be caught this way.)

35 Testing Paradox The “Pesticide Paradox” Regardless of testing quality some problems stay! Eliminating “easy” bugs allow complexity to grow and we get to the “complexity barrier”. Our ability to understand those complex states and conditions!

36 Test Cases Test cases are formal procedures: –inputs are prepared –outcomes are predicted –tests are documented –commands are executed –results are observed All of these steps are subject to error. Tests may have bugs themselves.

37 Levels of Testing Unit Testing: A unit is the smallest testable piece of software (e.g., function). Component Testing: A component is an integrated aggregate of one or more units (e.g., module). Integration Testing: This testing is used to demonstrate that a combination of successfully tested components has bugs.

38 Levels of Testing (Cont’d) System Testing: A system is a very large component. System testing includes testing for: –performance –security –system recovery from failure Acceptance Testing: performed at customer site to ensure customer of software viability

39 Test types Unit Testing Subsystem Tests System Tests Integration Acceptance Tests

40 Testing Oracles A testing oracle is any program, process, or body of data that specifies the expected outcome of a set of tests. Oracles are often defined as a set of input/expected outcome pairs. A necessary condition for good testing No “oracle” means kids testing

41 Sources of Testing Oracles Regression Test Suites: Test software using the test suites developed for previous versions of the same software. Purchased Test Suites: Highly standardized software (compilers, mathematical routines) often have commercially available test suites. Existing Program: A working, trusted, program that is being re-hosted to a new language or O/S.

42 Is Complete Testing Possible? NO. Complete testing is both practically and theoretically impossible for non-trivial software. (see Gudel’s theorem)

43 Complete Functional Testing A complete functional test would consist of subjecting a program to all possible input streams. Even if the program has an input stream of 10 characters, it would require 2 80 tests. At 1 microsecond/test, exhaustive functional testing would require more time than twice the current estimated age of the universe!

44 Complete Structural Testing One should design enough tests to ensure that every path is executed at least once. What if the loops never terminate? Even if loops terminate, the number of paths may be too large.

45 How About Correctness Proofs? Requirements are specified in a formal language. Each program statement is used in a step of an inductive proof. In practice, such proofs are time consuming and expensive. There are formal languages (B,Z) for which an automatic tool allowing both writing and proving correctness (Find J-Edit on the web).

46 How About Correctness Proofs? (Cont’d) Proving correctness is lengthy but possible. Completeness of a specification is, in general, a provably unsolvable problem. When not using automatic tools usually proofs have bugs.

47 A Theory of Program Testing A program is a function: A program specification is the function: Input domain is D, output range is R. Proving program correctness is the process of demonstrating that:

48 A Theory of Program Testing (Cont’d) A program P is correct with respect to the specification S if the program behavior matches the specified behavior for all possible inputs in D. This is impossible to achieve for any reasonably complex program.

49 A Theory of Program Testing (Cont’d) Software Testing is the process of demonstrating program correctness by selecting a subset of set D called T such that: –T is a finite subset of D. –If S(t) is incorrect for some t in T, P(t) must be incorrect also. –If S(t) is correct for some t in T, P(t) must be correct also.

50 A Theory of Program Testing (Cont’d) A testing strategy is a set of rules that outline: –the criteria by which T is selected (test selection criteria) –the properties of the program that must be exercised (test coverage criteria)

51 Goodenough and Gerhardt Test Selection Criteria Every branching condition in the program must be represented by a test case. Every potential termination condition in the program (e.g., overflow) must have a test case. Every condition in the specification that is relevant to the correct operation of the program must have a test case. Test cases must be independent.

52 Can reveal the presence of faults NOT their absence. A successful test is a test which discovers one or more faults. Only validation technique for non- functional requirements. Should be used in conjunction with static verification. Program Testing

53 Defect Testing The objective of defect testing is to discover defects in programs. A successful defect test is a test which causes a program to behave in an anomalous way. Tests show the presence not the absence of defects.

54 Only exhaustive testing can show a program is free from defects. However, exhaustive testing is impossible. Tests should exercise a system’s capabilities rather than its components. Testing old capabilities is more important than testing new capabilities. Testing typical situations is more important than boundary value cases. Testing Priorities

55 Test data: Inputs which have been devised to test the system. Test cases: Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification Test Data and Test Cases

56 Testing Effectiveness In an experiment, black-box testing was found to be more effective than structural testing in discovering defects.

57 Black-box Testing Approach to testing where the program is considered as a “black-box”. The program test cases are based on the system specification. Test planning can begin early in the software process.

58 Black-box Testing

59 Equivalence Partitioning

60 Partition system inputs and outputs into “equivalence sets”: –If input is a 5-digit integer between 10,000 and 99,999, equivalence partitions are < 10,000, 10,000 - 99,999 > 10,000. Choose test cases at the boundary of these sets: –00000, 9999, 10000, 99999, 10000 Equivalence Partitioning

61 Equivalence Partitions

62 Search Routine Specification procedure Search (Key : INTEGER ; T: array 1..N of INTEGER; Found : BOOLEAN; L: 1..N) ; Pre-condition -- the array has at least one element 1 <= N Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, 1 >= i >= N, T (i) = Key ))

63 Inputs which conform to the pre-conditions. Inputs where a pre-condition does not hold. Inputs where the key element is a member of the array. Inputs where the key element is not a member of the array. Search Routine - Input Partitions

64 Testing Guidelines (Arrays) Test software with arrays which have only a single value. Use arrays of different sizes in different tests. Derive tests so that the first, middle and last elements of the array are accessed. Test with arrays of zero length (if allowed by programming language).

65 Search Routine - Input Partitions

66 Search Routine - Test Cases

67 Sometime called white-box testing. Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases. Objective is to exercise all program statements (not all path combinations). Structural Testing

68 White-box Testing

Binary search (Java)

Binary search flow graph

71 Pre-conditions satisfied, key element in array. Pre-conditions satisfied, key element not in array. Pre-conditions unsatisfied, key element in array. Pre-conditions unsatisfied, key element not in array. Input array has a single value. Input array has an even number of values. Input array has an odd number of values. Binary Search - Equivalence Partitions

72 Binary Search Equivalence Partitions

73 Binary Search - Test Cases

74 Path Testing- Structural Testing Family of test techniques based on judicious test paths selection Basis for other techniques Most applicable to new code Requires full knowledge of software structure (white testing!) Reminder: we test since we assume wrong coding

75 Flowgraphs Graphical representation of a program’s control structure Made of –Process blocks (one enter-one exit) –Decision –Junctions Do process A If A=B Yes No 12

76 Flowgraphs (cont.) –Case Statement (very similar to decision) 2 3 N 1.

77 Definitions Process Block: a sequence uninterrupted by –decisions and case statements Decision: A point in a program where control flow can diverge –(A case statement is a multiple decision point) Junctions: A point in a program where control can merge

78 Flowgraphs Versus Flowcharts Flowcharts are a detailed graphical representation of a program –too many details –difficult for control and flow understanding –due to complexity not used anymore

79 Flowgraphs Hide computational details (a single box) Emphasize the control structure Allow understanding control and flow Careful use reveals bugs without further testing (NOT ALL!) Allow test path selection There are extensions for data

80 Example –1 input x,y – z:= x+y – v:= x-y – if z>= 0 goto sam –5 joe: z:= z-1 – sam: z:=z+v – for u=0 to z – v(u),u(v):= (z+v)*u – if v(u)=0 goto joe –10 z=z-1 – if z=0 goto ell – u:=u+1 – next u – v(u-1):=v(u+1) +u(v-1) –15 ell: v(u+u(v)):= u+v – if u=v goto joe – if u>v then u:=z – z:=u – 19 end

81 Flowchart 1235 6u= 0 8+ 10121314 1715 z>0 joe sam Y N 8 loop v(u)=0 Y joe z=0 N N u=z N loop Y ell u=vu>v 18 N Y Y joe 8+ xxx END Y

82 Flowgraph P1P2 P3 P5P6P7P8 P10P9 z>0 joe sam Y N loop v(u)=0 Y joe N N u=z N loop Y ell u=vu>v P11 N Y Y joe P4 xxx END z= 0 Y

83 Flowgraph (cont.) Length of process is irrelevant Algorithm complexity is irrelevant Path analysis is independent of decision nature Tests depend on details as well as on path Need for a simple path finding but be able to understand detail

84 Example (cont.) As the path selection is independent of process we have: 1 2 345678 910111213

85 Example (cont.) Path is defined by [1,3,4,5,6,7,8,9,10,1112, 13,2] 1 and 2 are reserved for start and end Every possibility via a loop is a different path! Path understanding needs cross reference

86 Example Cross Reference Junction Source Next Junction Condition 1 begin 3 2 end Exit no outlink 3 if (z>0) 4 false 5 true 4 joe 5 5 sam 6 6 loop 7 7 if(v(u)=0) 4 true 8 false 8 if(z=0) 9 false 10 true 9 if(u=z) 6 false 10 true 10 ell 11 11 if(u=v) 4 true 12 false 12 if(u>v) 13 true 13 2 end

87 Path Selection Criteria How many paths inside a loop? What is complete testing 1-Exercise every path from entry to exit 2-Exercise every statement or instruction at least once 3-Exercise every branch and case statements at least once (1) includes (2) & (3) ? –Is it practical?

88 Path Selection Criteria Are (2) and (3) equivalent? Example: Correct Incorrect if(x>0) x=x+a; if(x>0) x=x+a; x=x+a; else x=x+a; ret = dosomething(); ret = dosomething (); –(2) would not reveal the incorrect version if not all branches are executed! (when?)

89 Static analysis cin>>choice; while (choice !=5) {switch (choice) { case 1 : x=x+a; break; case 2: x=x+2*a; break; case 3: x=x+3*a; break; case 4: x=x+4*a; break; case 5: x=x+5*a; break; default: break;} cout << “x= “<< x << “\n”; cin choice;}

90 Static Analysis Compiler cannot determine whether a piece of code is or is not reachable Path analysis does tell you! INSPECTIONS DO FIND THEM!

91 Path Testing Criteria Examples show (2) & (3) alone are insufficient Structured languages imply (3) contains (2). However, it can be always ruined! Path testing (P inf ) execute ALL possible control flow path (entry -> exit). Statement Testing (P 1 ) execute all statements at least once (some places call it node-testing) Branch Testing (P 2 ) executes enough tests too ensure every branch alternative is tested at least once (also called link coverage testing) Different P’s mean selection is needed. 100% coverage is the goal

92 What to consider Risk proportional at least to not tested parts When are High Probability (usual working path) paths tested? Logical errors and fuzzy thinking are inversely proportional to utilization probability (SW eng. Usually select the wrong ones!) “Elegant Code” usually tested much better! “Simple” stuff is dismissed by “how can it go wrong”

93 What to Consider As P 1 Does not guaranty P 2 choices are P 2 and above Common sense requires at least P 1 and P 2 Select enough paths to cover more … Use simple paths with small increments over the preceding paths Ensure paths are possible, not from code but from design!

94 Considerations (cont.) Multi fold “if” statements are tricky in order to ensure path through every branch build a table which correlates if branching to paths investigate whether only one outcome is possible when executing the condition (a “do not care” is usually superfluous) example: –if (a!!b) ret=dosub(); – else ret dosub(); two processes with same condition should be collapsed unless non orthogonal –if (a==b) ret = aa(); – else ret = bb() ; –if (a==b) ret = cc(); – else ret = dd() ;

95 Summary Pick simple paths Pick functional paths first. Paths which do not add functionality are suspicious investigate Add ALL paths needed for 100% coverage. Favor short paths over long ones, avoid loops if possible Do not follow rules without thinking!

96 Path Products and Regular Expressions Motivation A formal way to count paths A formal way to let you cast any program issue into a flowgraph This is the formal way EE analyze circuits and programs are the same Most automatic tools use this formal method and yield metrics based on it.

97 Basic Concepts A link connects two decision points –Example –Traversing links gives us a path product like “abcd” –Example possible paths: –ac, abc, abbc, abbbc, abbbbc …. –Using “+” as an or we can denote the set of paths as: –X = ac+abc+abbc+abbbc+ abbbbc yielding a set of paths –X is a path expression made of path products 12a 123 a c b

98 Basic Concepts X= ab+abc+abcd; Y= uv +w ==> XY= abuv+abcuv+abcduv+abw+abcw+abcdw X=ab; XX= abab = X 2 X n =X (n-1) X ; Definition X 0 = 1 ABC = (AB)C = A(BC) Product is associative but not commutative

99 Basic Concepts Parallel paths –a and b are parallel –a+b and c+d are parallel –eacf+eadf+ebcf+ebdf are a set of parallel paths –Assume X+Y+d is a set of parallel paths between 1 and 2 g connects 2 and 3 and another set is: U+V+W+h+I+j connects 3 and 4 then the set of all paths is:(X+Y+d)g(U+V+W+h+I+j) e a c f bd

100 Basic Concepts So finally we get: :(X+Y+d)g(U+V+W+h+I+j) =(Xg+Yg+dg)(U+V+W+h+I+j)= XgU+Ygu+dgU+………….+dgj As expected (X+Y)+Z = X+(Y+Z) = X+Y+Z From the definition the distributive law holds. A(B+C)=AB+AC; (c+d)f=cf+df

101 Basic Concepts As a+a denotes the same path we have the absorption rule: X+X=X Example X=a+aa+abc+abcd +def X+a = a +a +…+def=X  Loops are a sum like:  a i  Define a * =  a i when i can go to infinity  Special case: a + means the loop has to be executed at least once! ==> a * a = aa * = a + ; X * X = XX * = X +  Definition: X n =  n a i

102 Consequences X n +X m = X n n>m X n X m =X (n+m) X n X * = X * X n = X * X n X + = X + X n = X + X * X + = X + X * = X + X 0 = 1 (No path at all) 1 + +1 = 1 * = 1 = 1 n = 1 n An empty set of paths is defined as 0 or  Arithmetic –X+0 = 0+X = X –X0 = 0X = 0 –0* = 1+0+0 2 +….= 1

103 Links and Paths - Reduction Algorithm 1 get path product of all serial links 2 combine parallel paths and get their sum 3 denote self loops by X * where X denotes the loop path Algorithm loop –4 Select node for removal (not start or end) replace by all path products and equivalent links

104 Links and Paths - Reduction Algorithm Algorithm loop (cont.) –5 Combine remaining serial links by getting their path product (step 1) –6 Combine all parallel links by adding their paths (step 2) –7 repeat step 3 for all self loops –8 If graph does not consist yet of a single link between start and end repeat steps 4 through 7 until it does.

105 Path Finding Example 13 978 265 10 4 fh m gi l j bc k a ed 13 978 2654 fh m g i il j bc k a ed

106 Path Finding Example 13 78 2654 f m g i ilh j bc kh a ed 13 978 2654 fh m g i il j bc k a ed

107 Path Finding Example 13 78 2654 f m g i ilh j bc kh a ed 13 8 2654 mf g i ilh jf bc kh a ed

108 Path Finding Example 132654 mfi ilh gjf bc gkh a ed 13 8 2654 mf g i ilh jf bc kh a ed

109 Path Finding Example 13265 mfi ilh bgjf b(c+gkh) a ed 132654 mfi ilh gjf bc gkh a ed 132654 mfi ilh gjf bc+gkh a ed

110 Path Finding Example 1326 mfi ilhd a e 13265 mfi ilh bgif*b(c+gkh) a ed 13265 mfi ilh a ed bgif*b(c+gkh)

111 Path Finding Example 1326 mfi ilhd a e bgif*b(c+gkh) 1326 a (ihld)*ebgif*b(c+gkh) (ihld)*imf 126 (ihld)*eabgif*b(c+gkh) (ihld)*imf bgif*b(c+gkh)

112 Path Finding Example Removing node 6 we finally get: a(bgif)*b(c+gkh)d((ilhd*imf(bgjf)*b(c+gkh )d)*(ihld)*e

113 Back to Earth The maximum number of different paths algorithm: –weight a path by the number of different paths possible –weight a loop by the maximum no. of revolutions –Assume A and B are paths with weights W A and W B respectively

114 Expression Weight Parallel Path A+B => W=W A +W B Series Path AB => W=W A W B Loop A n =>  0 n W a j Assume we have this path formula: –a(b+c)d(e(fi)*fgi(m+l)k)*e(fi)*fgh –Each link weight is 1 –The outer loop can be taken exactly 4 times and the inner loop 0-3

115 Expression Weight Inserting these values we get: –1(1+1)(1(1*1) 3 1*1*1(1+1)1) 4 1(1*1) 3 = –1*2(1*(1+1 1 +1 2 +1 3 )*1*1*1*2) 4 4= –2(4*2) 4 4 = 8 5 =32768 Absolute maximum, remember we are not sure all paths are possible

116 Getting the Minimum # It is estimated only for structured programs It may well be an upper limit to the minimum Assumptions: –minimum needs only once through a loop –series connections weight is maximum of the two links involved –parallel links behave as before

117 Getting The Minimum Start from: a(b+c)d(e(fi)*fgi(m+l)k)*e(fi)*fgh –Inserting the numbers we get : –1(2)1(1)1(1)1(2)1(1)(1)1... –Using the max of series products we get 2! –As this is structured it is the absolute minimum.

118 Example Exponentiation Algorithm 6 5 7 3 4 1 2 1 scanf(“%d %d”,&x, &y); 2 if (y < 0) pow = -y; else pow = y; 3 z = 1.0; 4 while (pow != 0) { z = z * x; pow = pow - 1; 5 } 6 if (y < 0) z = 1.0 / z; 7 printf (“%f”,z);

119 Bubble Sort Algorithm 1234567 1 for (j=1; j<N; j++) { last = N - j + 1; 2 for (k=1; k<last; k++) { 3 if (list[k] > list[k+1]) { temp = list[k]; list[k] = list[k+1]; list[k+1] = temp; 4 } 5 } 6 } 7 print(“Done\n”);

120 Paths A path through a program is a sequence of statements that starts at an entry, junction, or decision and ends at another (possible the same), junction, decision, or exit. A path may go through several junctions, processes, or decisions, one or more times. Paths consist of segments. The smallest segment is a link. A link is a single process that lies between 2 nodes.

121 Paths (Cont’d) The length of a path is the number of links in a path. An entry/exit path or a complete path is a path that starts at a routine’s entry and ends at the same routine’s exit. Simplifying Assumption: All routines have a single entry and single exit.

122 Paths (Cont’d) Complete paths are useful for testing because: –It is difficult to set up and execute paths that start at an arbitrary statement. –It is difficult to stop at an arbitrary statement without changing the code being tested. –We think of routines as input/output paths.

123 Common Sense Strategies Statement and branch coverage have been used for over two decades as a minimum mandatory unit test requirement for new code developed at IBM and other companies. Insisting on statement and branch coverage is based on common sense rather than theory.

124 Common Sense Strategies (Cont’d) It makes sense to use branch coverage because software has a high density of conditional branches, loop, etc. (25% in most PLs) It is better to leave out untested code than to include it in a product release.

125 Which Paths? You must pick enough paths to achieve statement and branch coverage. Question: What is the fewest number of paths to achieve statement and branch coverage? Answer: Unask the question. –It is better to take many simple paths than a few complicated ones. –There is no harm in taking paths that will exercise the same code more than once.

126 Example of P1 and P2 Coverage 10 34562 987 1 m abcde i h g f jkl TTF F

127 Branch and Statement Coverage Question: Does every decision have a T (true) and a F (false) in its column? Answer: Yes implies branch coverage. Question: Is every link covered at least once? Answer: Yes implies statement coverage.

128 Guidelines Select paths as small variations of previous paths. Try to change one thing in each path at a time.

129 Loop Testing Loop testing is an aspect of path testing aimed at exposing bugs that typically occur in loops. Loop bugs can be divided into two major categories: –End point bugs –Initialization bugs Every loop has a control variable: –simple (e.g., an index in a for loop) –complicated and not necessarily numerical.

130 Single Loop Testing (Minimum Cases) –Bypassing the loop altogether: A set of input values that will cause an immediate exit. –One pass through the loop: Many initializations are caught by this test. –Two passes through the loop before exiting: Some initialization problems will surface after two loops. –A typical number of passes through the loop. –The maximum allowable number of passes. –One less than the maximum allowable number of passes.

131 Nested Loop Testing (Minimum Cases) Set all but one loop to a typical value and run through the single loop cases for that loop. Repeat for all loops. Do minimum values for all loops simultaneously. Set all but one loop to the minimum value and run through the single loop cases for that loop. Repeat for all loops. Do maximum looping values for all loops simultaneously.

132 Effectiveness of Path Testing About 65% of all bugs can be caught in unit testing. Unit testing is dominated by path testing methods. Statement and branch testing dominates path testing.

133 Effectiveness of Path Testing (Cont’d) Studies show that path testing catches 50% of all bugs caught during unit testing. –About 33% of all bugs. Path testing is more effective for unstructured code than for code that follows structured programming. Experienced programmers can bypass drawing flowgraphs by doing path selection on the source.

134 Limitations of Path Testing Path testing as a sole testing technique is limited: –Interface mismatches and errors are not caught. –Not all initialization errors are caught by path testing. –Specification errors are not caught.

135 Path Predicates Every path corresponds to a succession of true or false values for the predicates traversed on that path. A Path Predicate Expression is a Boolean expression that characterizes the set of input values that will cause a path to be traversed. Multiway branches (e.g., case / switch statements) are treated as equivalent if then else statements.

136 Input Values to Path Predicate Expressions Any set of input values that satisfies ALL of the conditions of the path predicate expression will force the routine through that path. If there is no such set of inputs, the path is not achievable.

137 Example X1,X2,X3,X4,X5,X6 if (X5 > 0 || X6 < 0) /* predicates A,B */... if(X1 + 3 * X2 + 17 >= 0)/* predicate C */... if(X3 == 17)/* predicate D */... if(X4 - X1 >= 14 * X2)/* predicate E */... Path Predicate Expression is: (A+B)CDE

138 Input Vector The input vector of a routine is the set of input parameters to that routine along with any global variables used in that routine.

139 Process for Creating a Path Expression Write down the predicates for the decisions you meet along a path. The result is a set of path predicate expressions. All of these expressions must be satisfied to achieve a selected path.

140 Process (In)dependent Predicates A predicate whose truth value cannot/can change as a result of the processing is said to be Process Independent/Dependent, respectively. If all the variables on which a predicate is based are process independent, the predicate must be process independent. Process dependence of a predicate does not always follow from dependence of the input variables on which the predicate is based.

141 Correlated Predicates A pair of predicates whose outcomes depend on one or more variables in common are said to be Correlated Predicates. Every path through a routine is achievable only if all predicates in that routine are uncorrelated.

142 Example of Correlated Predicates E.g., X,Y... /* no changes to X and Y here */ if(X == Y) if(X + Y == 8) To satisfy the first predicate we may have to pick values for X,Y that will force the truth value for the second predicate.

143 Path Sensitization The act of finding a set of solutions to the path predicate expression is called path sensitization.

144 Example: Uncorrelated & Independent Because the predicates are uncorrelated and independent … 4 binary decisions means = 16 possible paths. lA4C672 B D9 l m jkh g T F T i bac def F T F

145 Example: Correlated & Independent Paths abdeg and acdfg seem to provide coverage, but neither of these paths is achievable. Only 2 paths are achievable: abdfg and acdeg. lA4A62 a b cf e g F T T F d

146 Test Outcomes The outcome of test is what we expect to happen as a result of the test. Test outcomes include anything we can observe in the computer’s memory that should have (not) changed as a result of the test. Since we are not “kiddie testing” we must predict the outcome of the test as part of the test design process.

147 Testing Process run the test observe the actual outcome compare the actual outcome to the expected outcome.

148 Questions About Test Outcomes Question: If the predicted and actual outcomes match, can we say that the test has been passed? Answer: No! The desired outcome could have been achieved for the wrong reason. (coincidental correctness)

149 Questions About Test Outcomes Question: Assume that we ran a covering set of tests and achieved the desired outcomes for each case. Can we say that we’ve covered all branches? Answer: No! The desired outcome could have been reached by the wrong path! –Path instrumentation is necessary to confirm that the outcome was achieved by the intended path.

150 Path Instrumentation All instrumentation methods are a variation on a theme of an interpretive trace. An interpretive trace program executes every statement in order and records: –the intermediate values of all calculations –the statement labels traversed –...

151 Path Instrumentation (Cont’d) If we run the tested routine under a trace, then we have all the information we need to confirm: –the outcome of the test –whether the outcome was achieved by the intended path.

152 Two Detailed Examples Of Path Testing

153 /* ABS This program function returns the absolute value of the integer passed to the function as a parameter. INPUT: An integer. OUTPUT: The absolute value if the input integer. */ 1 int ABS(int x) 2 { 3 if (x < 0) 4 x = -x; 5 return x; 6 } Using Path Testing to Test Function ABS Consider the following function:

154 The Flowgraph for ABS /* ABS This program function returns the absolute value of the integer passed to the function as a parameter. INPUT: An integer. OUTPUT: The absolute value if the input integer. */ 1 int ABS(int x) 2 { 3 if (x < 0) 4 x = -x; 5 return x; 6 }

155 Test Cases to Satisfy Path Testing Coverage for ABS Complete path testing of ABS is theoretically possible but not practical. ABS takes as its input any integer. There are many integers (depending on the maximum size of an integer for the language) that could be input to ABS making it impractical to test all possible inputs to ABS.

156 Test Cases to Satisfy Statement Testing Coverage for ABS PATHSPROCESS LINKS TEST CASES abCd INPUT OUTPUT abc  A Negative Integer, x -x adc   A Positive Integer, x x

157 /* COUNT This program counts the number of characters and lines in a text file. INPUT: Text File OUTPUT: Number of characters and number of lines. */ 1 main(int argc, char *argv[]) 2 { 3 int numChars = 0; 4 int numLines = 0; 5 char chr; 6 FILE *fp = NULL; 7 Example: Using Path Testing to Test Program COUNT Consider the following program:

158 8 if (argc < 2) 9 { 10 printf(“\nUsage: %s ”, argv[0]); 11 return (-1); 12 } 13 fp = fopen(argv[1], “r”); 14 if (fp == NULL) 15 { 16 perror(argv[1]); /* display error message */ 17 return (-2); 18 } Program COUNT (Cont’d)

159 19 while (!feof(fp)) 20 { 21 chr = getc(fp); /* read character */ 22 if (chr == ‘\n’) /* if carriage return */ 23 ++numLines; 24 else 25 ++numChars; 26 } 27 printf(“\nNumber of characters = %d”, numChars); 28 printf(“\nNumber of lines = %d”, numLines); 29 } Program COUNT (Cont’d)

160 The Flowgraph for COUNT The junction at line 12 and line 18 are not needed because if you are at these lines then you must also be at line 14 and 19 respectively. 2

161 Test Cases to Satisfy Path Testing Coverage for COUNT Complete path testing of COUNT is impossible because there are an infinite number of distinct text files that may be used as inputs to COUNT.

162 Test Cases to Satisfy Statement Testing Coverage for COUNT 2

163 Test Cases to Satisfy Statement Testing Coverage for COUNT PATHSPROCESS LINKS TEST CASES abcdefghijkl INPUT OUTPUT ab  None“Usage: COUNT ” agc    Invalid Input Filename Error Message aghdj kli    Input File with one character and no Carriage Return at the end of the line Number of characters = 1 Number of lines = 0 aghd efli    Input file with no characters and one carriage return Number of characters = 0 Number of lines = 1

164 Test Cases to Satisfy Branch Testing Coverage for COUNT PATHSDECISIONS TEST CASES 8141922 INPUT OUTPUT abT None“Usage: COUNT ” agcFT Invalid Input Filename Error Message aghdjkliFFT, F FInput File with one character and no Carriage Return at the end of the line Number of characters = 1 Number of lines = 0 aghdefliFFT, F TInput file with no characters and one carriage return Number of characters = 0 Number of lines = 1

165 Summary The object of path testing is to execute enough tests to assure that statement and branch coverage has been achieved. Select paths as deviation from the normal paths. Add paths as needed to achieve coverage. Add paths to cover extreme cases for loops and combinations of loops.

166 Summary (Cont’d) Find path-sensitizing input data sets for each selected path. Use instrumentation (manual or using tools) to verify paths. Document all tests and expected test results. A test that reveals a bug has succeeded, not failed.

167 Data-Flow Testing Data-flow testing uses the control flowgraph to explore the unreasonable things that can happen to data (i.e., anomalies). Consideration of data-flow anomalies leads to test path selection strategies that fill the gaps between complete path testing and branch or statement testing.

168 Data-Flow Testing (Cont’d) Data-flow testing is the name given to a family of test strategies based on selecting paths through the program’s control flow in order to explore sequences of events related to the status of data objects. E.g., Pick enough paths to assure that: –Every data object has been initialized prior to its use. –All defined objects have been used at least once.

169 Data Object Categories (d) Defined, Created, Initialized (k) Killed, Undefined, Released (u) Used: – (c) Used in a calculation – (p) Used in a predicate

170 (d) Defined Objects An object (e.g., variable) is defined when it: –appears in a data declaration –is assigned a new value –is a file that has been opened –is dynamically allocated –...

171 (k) Killed Objects An object is killed when it is: –released (e.g., free) or otherwise made unavailable (e.g., out of scope) –a loop control variable when the loop exits –a file that has been closed –...

172 (u) Used Objects An object is used when it is part of a computation or a predicate. A variable is used for a computation (c) when it appears on the RHS (sometimes even the LHS in case of array indices) of an assignment statement. A variable is used in a predicate (p) when it appears directly in that predicate.

173 Data-Flow Anomalies A data-flow anomaly is denoted by a two character sequence of actions. E.g., –ku: Means that an object is killed and then used. –dd: Means that an object is defined twice without an intervening usage.

174 Example E.g., of a valid (not anomalous) scenario where variable A is a dpd: A = C + D; if(A > 0) X = 1; else X = -1; A = B + C;

175 Two Letter Combinations for d k u dd: Probably harmless, but suspicious. dk: Probably a bug. du: Normal situation. kd: Normal situation. kk: Harmless, but probably a bug. ku: Definitely a bug. ud: Normal situation (reassignment). uk: Normal situation. uu: Normal situation.

176 Single Letter Situations A leading dash means that nothing of interest (d, k, u) occurs prior to the action noted along the entry-exit path of interest. A trailing dash means that nothing of interest happens after the point of action until the exit.

177 Single Letter Situations -k: Possibly anomalous: –Killing a variable that does not exist. –Killing a variable that is global. -d: Normal situation. -u: Possibly anomalous, unless variable is global. k-: Normal situation. d-: Possibly anomalous, unless variable is global. u-: Normal situation.

178 Data-Flow Anomaly State Graph U K AD k,u,d u u d k d d,k u,k state of variable action anomalous state

179 Data-Flow Anomaly State Graph with Variable Redemption DU K DD DK KU u u k u d d d dd d k k k u k k u u

180 Static vs Dynamic Anomaly Detection Static Analysis is analysis done on source code without actually executing it. E.g., Syntax errors are caught by static analysis.

181 Static vs Dynamic Anomaly Detection (Cont’d) Dynamic Analysis is analysis done as a program is executing and is based on intermediate values that result from the program’s execution. E.g., A division by 0 error is caught by dynamic analysis. If a data-flow anomaly can be detected by static analysis then the anomaly does not concern testing. (Should be handled by the compiler.) C # allows use only of defined by value data

182 Anomaly Detection Using Compilers Compilers are able to detect several data- flow anomalies using static analysis. E.g., By forcing declaration before use, a compiler can detect anomalies such as: –-u –ku Optimizing compilers are able to detect some dead variables (never used).

183 Is Static Analysis Sufficient? Questions: Why isn’t static analysis enough? Why is testing required? Could a good compiler detect all data-flow anomalies? Answer: No. Detecting all data-flow anomalies is provably unsolvable.

184 Static Analysis Deficiencies Current static analysis methods are inadequate for: –Dead Variables: Detecting unreachable variables is unsolvable in the general case. –Arrays: Dynamically allocated arrays contain garbage unless they are initialized explicitly. (-u anomalies are possible)

185 Static Analysis Deficiencies (Cont’d) –Pointers: Impossible to verify pointer values at compile time. –False Anomalies: Even an obvious bug (e.g., ku) may not be a bug if the path along which the anomaly exists is unachievable. (Determining whether a path is or is not achievable is unsolvable.)

186 Data-Flow Modeling Data-flow modeling is based on the control flowgraph. Each link is annotated with: –symbols (e.g., d, k, u, c, p) –sequences of symbols (e.g., dd, du, ddd) that denote the sequence of data operations on that link with respect to the variable of interest.

187 Control Flowgraph Annotated for X and Y Data Flows 1 3 4 5 6 7 28910111213 Z? ENDYYU,V? U,Z? ELL SAM LOOPB(U)? dcc JOE Z? 1INPUT X,Y Z:= X+Y Y:= X-Y 3IF Z>=0 GOTO SAM 4JOE: Z:=Z-1 5SAM: Z:=Z+V U:=0 6LOOP B(U),Q(V):=(Z+V)*U 7IF B(U)=0 GOTO JOE Z:=Z-1 8IF Z=0 GOTO ELL U:=U+1 9UNTIL U=z B(U-1):=B(U+1)+Q(V-1) 10 ELL: B(U+Q(V)):=U+V 11IF U=V GOTO JOE 12IF U>V THEN U:=Z 13 YY:Z:=U 2END

188 Control Flowgraph Annotated for Z Data Flows 134567 28910111213 Z? ENDYYU,V? U,Z? ELL SAM LOOPB(U)? d JOE Z? p p cd c p d c pp p 1INPUT X,Y Z:= X+Y Y:= X-Y 3IF Z>=0 GOTO SAM 4 JOE: Z:=Z-1 5 SAM: Z:=Z+V U:=0 6LOOP B(U),Q(V):=(Z+V)*U 7IF B(U)=0 GOTO JOE Z:=Z-1 8IF Z=0 GOTO ELL U:=U+1 9UNTIL U=z B(U-1):=B(U+1)+Q(V-1) 10 ELL: B(U+Q(V)):=U+V 11IF U=V GOTO JOE 12IF U>V THEN U:=Z 13 YY:Z:=U 2END

189 Definition-Clear Path Segments A Definition-clear Path Segment for variable X is a connected sequence of links such that X is defined on the first link and not redefined or killed on any subsequent link of that path segment.

190 Definition-Clear Path Segments for Variable Z (Cont’d) 134567 28910111213 Z? ENDYYU,V? U,Z? ELL SAM LOOPB(), U? d JOE Z? p p cd c p d c pp p

191 Non Definition-Clear Path Segments for Variable Z (Cont’d) 134567 28910111213 Z? ENDYYU,V? U,Z? ELL SAM LOOPB(), U? d JOE Z? p p cd c p d c pp p

192 Simple Path Segments A Simple Path Segment is a path segment in which at most one node is visited twice. –E.g., (7,4,5,6,7) is simple. Therefore, a simple path may or may not be loop-free.

193 Loop-free Path Segments A Loop-free Path Segment is a path segment for which every node is visited at most once. –E.g., (4,5,6,7,8,10) is loop-free. –path (10,11,4,5,6,7,8,10,11,12) is not loop-free because nodes 10 and 11 are visited twice.

194 du Path Segments A du Path is a path segment such that if the last link has a use of X, then the path is simple and definition clear.

195 Data-Flow Testing Strategies All du Paths (ADUP) All Uses (AU) All p-uses/some c-uses (APU+C) All c-uses/some p-uses (ACU+P) All Definitions (AD) All p-uses (APU) All c-uses (ACU)

196 All du Paths Strategy (ADUP) ADUP is one of the strongest data-flow testing strategies. ADUP requires that every du path from every definition of every variable to every use of that definition be exercised under some test All du Paths Strategy (ADUP).

197 Example: pow(x,y) 158916142 a b c d e f g h i

198 Example: pow(x,y) du-Path for Variable x /* pow(x,y) This program computes x to the power of y, where x and y are integers. INPUT: The x and y values. OUTPUT: x raised to the power of y is printed to stdout. */ 1 void pow (int x, y) 2 { 3 float z; 4 int p; 5 if (y < 0) 6 p = 0 – y; 7 else p = y; 8 z = 1.0; 9 while (p != 0) 10 { 11 z = z * x; 12 p = p – 1; 13 } 14 if (y < 0) 15 z = 1.0 / z; 16 printf(z); 17 } 158916142 a b c d e f g h i

199 Example: pow(x,y) du-Path for Variable y /* pow(x,y) This program computes x to the power of y, where x and y are integers. INPUT: The x and y values. OUTPUT: x raised to the power of y is printed to stdout. */ 1 void pow (int x, y) 2 { 3 float z; 4 int p; 5 if (y < 0) 6 p = 0 – y; 7 else p = y; 8 z = 1.0; 9 while (p != 0) 10 { 11 z = z * x; 12 p = p – 1; 13 } 14 if (y < 0) 15 z = 1.0 / z; 16 printf(z); 17 } 158916142 a b c d e f g h i

202 /* COUNT This program counts the number of characters and lines in a text file. INPUT: Text File OUTPUT: Number of characters and number of lines. */ 1 main(int argc, char *argv[]) 2 { 3 int numChars = 0; 4 int numLines = 0; 5 char chr; 6 FILE *fp = NULL; 7 Example: Using du-Path Testing to Test Program COUNT Consider the following program:

203 8 if (argc < 2) 9 { 10 printf(“\nUsage: %s ”, argv[0]); 11 return (-1); 12 } 13 fp = fopen(argv[1], “r”); 14 if (fp == NULL) 15 { 16 perror(argv[1]); /* display error message */ 17 return (-2); 18 } Program COUNT (Cont’d)

204 19 while (!feof(fp)) 20 { 21 chr = getc(fp); /* read character */ 22 if (chr == ‘\n’) /* if carriage return */ 23 ++numLines; 24 else 25 ++numChars; 26 } 27 printf(“\nNumber of characters = %d”, numChars); 28 printf(“\nNumber of lines = %d”, numLines); 29 } Program COUNT (Cont’d)

205 The Flowgraph for COUNT The junction at line 12 and line 18 are not needed because if you are at these lines then you must also be at line 14 and 19 respectively. 182 11 14171922232426

206 du-Path for argc

207 du-Path for argc

208 du-Path for argv[]

209 du-Path for argv[]

210 du-Path for numChars

213 du-Path for numLines

216 du-Path for chr

217 du-Path for chr

218 du-Path for fp

219 du-Path for fp

220 du-Path for fp

221 All Uses Strategy (AU) AU requires that at least one path from every definition of every variable to every use of that definition be exercised under some test. Hence, at least one definition-clear path from every definition of every variable to every use of that definition be exercised under some test. Clearly, AU < ADUP.

222 All p-uses/Some c-uses Strategy (APU+C) APU+C requires that for every variable and every definition of that variable include at least one definition-free path from the definition to every predicate use. If there are definitions of the variable that are not covered by the above prescription, then add computational-use test cases to cover every definition.

223 All c-uses/Some p-uses Strategy (ACU+P) ACU+P requires that for every variable and every definition of that variable include at least one definition-free path from the definition to every computational use. If there are definitions of the variable that are not covered by the above prescription, then add predicate-use test cases to cover every definition.

224 All Definitions Strategy (AD) AD requires that for every variable and every definition of that variable include at least one definition-free path from the definition to a computational or predicate use. AD < ACU+P and AD < APU+C.

225 All p-uses (APU) All c-uses (ACU) APU is the same as APU+C without the C requirement. APU < APU+C. ACU is the same as ACU+P without the P requirement. ACU < ACU+P.

226 Relative Strength of Data-Flow Testing Strategies All Paths All du Paths All Uses All c-uses/ Some p-uses All p-uses/ Some c-uses All Definitions All c-usesAll p-uses Cannot Compare

227 Effectiveness of Strategies Ntafos compared Random, Branch, and All uses testing strategies on 14 Kernighan and Plauger programs. Kernighan and Plauger programs are a set of mathematical programs with known bugs that are often used to evaluate test strategies. Ntafos conducted two experiments:

228 Results of 2 of the 14 Ntafos Experiments Random Branch All Uses Strategy Mean Number of Test Cases Percentage of Bugs Found 35 3.8 11.3 93.7 91.6 96.3 Random Branch All Uses Strategy Mean Number of Test Cases Percentage of Bugs Found 100 34 84 79.5 85.5 90.0

229 Data-Flow Testing Tips Resolve all data-flow anomalies. Try to do all data-flow operations on a variable within the same routine (i.e., avoid integration problems). Use strong typing and user defined types when possible.

230 Data-Flow Testing Tips (Cont’d) Use explicit (rather than implicit) declarations of data when possible. Put data declarations at the top of the routine and return data objects at the bottom of the routine.

231 Testability Tips WASTE MEMEORY FOR CLARITY Avoid bit storage and bit sharing! Use strong typing and explicit declarations (what about c++ templates) Bias the design so all data is defined and tested the same way! Use simple IF (N decisions 2 n paths all to be tested!)

232 Testability Tips Do not write what you cannot test Do not add “features” or extras if you do not intend to test them –Extras: not needed generalization, hooks, additional functionality, some hidden gimmick usually an accidental addition

233 Summary Data are as important as code. Define what you consider to be a data-flow anomaly. Data-flow testing strategies span the gap between all paths and branch testing.

234 Summary AU has the best payoff for the money. It seems to be no worse than 3 times the number of required test cases for branch testing, but the results are much better. Path testing with Branch Coverage and Data-flow testing with AU is a very good combination.

235 Method Sequence Specification An object accessing the services of other objects must know the proper interface protocol of the objects. The Method Sequence Specification technique documents the correct order in which methods can be invoked. Regular Expressions are used to represent method sequence specifications.

236 Correct Object Behavior The result of any method execution depends on previously executed methods. Correct object behavior is possible only if methods are invoked in a well-defined sequence. The method sequence specification is important! it specifies the correct sequences in which objects must receive messages.

237 Stack Example The objects of a Stack class should receive a pop() message only after receiving a push() message. Some ordering rules are due to implementation issues: –E.g., objects must receive a constructor message before receiving any other message.

238 Method Sequence Methods(C) is the set of all public methods defined in class C. –For a Stack class the set of methods can be: Methods(Stack) = {push,pop,isempty,isfull,top} A method sequence S of C is a finite sequence of methods M of C, (M0.M1. ….Mn), where Mi (0 <= i <=N) are in set M. –E.g., for the Stack class, a method sequence is: (push.top.pop.isempty)

239 Regular Definition of Method Sequence Specifications Regular expressions are used to represent Method Sequence Specifications for a class C over the alphabet ( ) consisting of methods from Methods(C).

240 Example of a Regular Definition of a Method Sequence Specification

241 Specification of a Simple Bank Account Class Interface methods of the Account class: – = {Create,Deposit,Open,Withdraw,Close,Delete} –The regular expression of the method sequence specification for class Account is: The regular expression is equivalent to the set: –{(Create.Open.Deposit.Withdraw.Close.Delete), (Create.Open.Deposit.Deposit.Withdraw.Close.Delete), …}

242 Applications of Method Sequence Specification Ensuring the conformance of the implementation of a class with the corresponding method sequence specification. Constructing a run-time verification system for a class to ensure the correct method invocation sequence for each object of the class. Generating test cases for testing the implementation of a class.

243 Verification of Implementation The method sequence specification of a class specifies the correct sequence usage for all objects of the class. But the implementation of a class may contain method invocation sequences that are inconsistent with the method sequence specification. It is important to identify and correct all such inconsistencies.

244 Verification of Implementation (Cont’d) We can use manual inspection or static analysis of the implementation to identify inconsistencies. Note that pointers, polymorphism, and dynamic binding in OO programs complicates the control-flow analysis. –For dynamic binding of methods to messages, the static analysis must consider all possible methods that can be bound to a message.

245 Verification of Implementation (Example) Account Object Client A Client B Client B 1.Start 2.Open 3.Deposit4.foo 6.bar 5.Close 7.Withdraw (Open.Deposit.Close.Withdraw) This is an invalid sequence! Each such invalid method sequence may be a potential fault. foo() creates the new object Client B. bar() implicitly gets its address so now no knowledge of the exact object is known!

246 Run-time Verification System What if all the invalid method sequences are not identified using inspection and static analysis? For safety-critical systems it may be necessary to identify all invalid method sequences during execution. The run-time verification system helps identify and recover from invalid method invocations.

247 Run-time Verification System (Cont’d) Each object maintains an access pointer to the method sequence specification of its corresponding class. For each method invocation, a check is made for sequence consistency (if the sequence is in the regular language) with respect to the stored sequence specification. If the method invoked is not in correct sequence, an exception is raised and handled.

248 Implementing a Run-time Verification System Account Class Method Dictionary NameExecutable Deposit Withdraw... - - Sequence Specification Open.Deposit.(…) Account Object 1 State Dictionary Pointer Sequence Pointer Sequence Verifier Account Object 2 State Dictionary Pointer Sequence Pointer Sequence Verifier

249 Efficient Algorithm for Consistency Checking We must be able to verify whether one regular expression is the same as another. Each regular expression defines a finite automata (FA). An FA accepts a set of strings and all of the acceptable strings form a language L. If L1 and L2 are the languages represented by two regular expressions, then the two regular expressions are equal if L1=L2.

250 Algorithm for Consistency Checking (Cont’d) One can verify that L1=L2 by constructing a finite automaton, D1, that satisfies L1 L2. – is the exclusive-or operator. If D1 accepts only an empty string, then L1=L2.

251 Test Case Generation from Sequence Specification In an OO program, testing a class corresponds to testing the methods supported by the class. –Step 1: Individual methods must be tested using the white-box techniques we discussed. –Step 2: Method interactions (protocols) must be tested for correctness. We must define coverage criteria for this...

252 Test Case Generation from Sequence Specification (Cont’d) All-node coverage criterion: –(M1.M2.M3) and (M1.M2.M4) All-edge coverage criterion: –(M1.M2.M3.M3.M4) and (M1.M2.M4.M4.M3) S0S1S2 S3 S4 M1M2 M3 M4 M3 M4

253 Test Case Generation from Sequence Specification (Cont’d) For each method in the sequence, data input of each parameter and the expected output must be determined. If a method invokes methods of other objects, then during testing test stubs can be used to supply return values.

254 From STDs to Regular Expressions Sequence Transition Diagrams (STD) are often constructed for each class during the design phase, these STDs can be used for generating the method sequence RE. –Just use the events and ignore the states. –There exist several efficient algorithms for converting STDs to Regular Eexpressions (RE) [Hopcroft,Ullman 79].

255 From STDs to R.Es (Example) Start Empty Account Operational Account Null Account Open() Deposit() Close() Deposit() Withdraw() (Open.Deposit.(Deposit|Withdraw)*.Close)

256 Inheritance and Method Sequence Specification Child classes inherit the method sequence specification from their parent class. We next present rules to verify consistency between the method sequence specifications of the parent and child classes for three kinds of inheritance: –Specialization inheritance (New Methods) –Refinement inheritance (Changed Methods) –Implementation inheritance (Methods Exclusion)

257 Specialization Inheritance The child class can enhance itself with new methods, in addition to all the inherited methods from the parent class. The inherited methods are not changed (overridden).

258 Specialization Inheritance (Cont’d) Let RE(C1) and RE(C2) be two regular expressions defining the method sequence specifications for classes C1 and C2, where C2 inherits from C1. Let S be the new methods of C2. Let NRE = e-instantiates(RE(C2),S). –Replace symbols of S in RE(C2) with epsilon. –E.g., if RE=M0.M1.(M2|M3)* and S={M1,M3}, then e-instantiates(RE,S)=M0.(M2)* C2 is consistent with C1 iff Lang (RE(C1)) = Lang(NRE)

259 Specialization Inheritance (Example) Let CheckingAcc be a class that specializes class Account. CheckingAcc has 2 new methods: –S = {Balance, Report} The method sequence specifications of Account and CheckingAcc are:

260 CheckingAcc and Account are Consistent NRE = e-instantiates(RE(CheckingAcc),S) = (Deposit.(Deposit|Withdraw)*) = Lang(RE(Account))

261 Refinement Inheritance The child class modifies the semantics of some of the inherited methods from the parent class. The modification in the inherited methods can be either in the: –method signature –method behavior (overriding)

262 Refinement Inheritance (Cont’d) Let RE(C1) and RE(C2) be two regular expressions defining the method sequence specifications for classes C1 and C2, where C2 inherits from C1. S = Ref U N –Ref = refined methods in C2 –N = new methods in C2 NRE = e-instantiates(RE(C2),S) C1w/oRef = e-instantiates(RE(C1),Ref). C2 is consistent with C1 iff Lang(RE(C1w/oRef)) = Lang(NRE)

263 Refinement Inheritance (Example) Let CheckingAcc be a class that specializes class Account. CheckingAcc has 2 new and 1 refined methods: –N = {Balance, Report} –Ref = {Withdraw} –S = {Balance, Report, Withdraw} The method sequence specifications of Account and CheckingAcc are:

264 CheckingAcc and Account are Consistent NRE = e-instantiates(RE(CheckingAcc),S) = (Deposit.(Deposit)*) = Lang(RE(Accountw/oRef))

265 Implementation Inheritance Using inheritance for reuse by excluding some of the methods of a previously defined class. Some methods of the parent class are excluded in the child class public interface.

266 Implementation Inheritance (Cont’d) The rule that checks the consistency between parent and child class is similar to the one described in specialization inheritance. The only difference is some of the parent methods are not inherited and, therefore, these methods must not be considered in the parent class regular expressions when applying the consistency rule.

267 Document testing Review history Why Reviews fail (90-10) Classical methods (review, walk-through) The Japanese method Fagan’s Method

268 Issues with Audits Ownership - I have written it… - It is my baby… - nobody can do better... What is there for me? It is difficult to perform but...

269 The project’s Gain Dissemination of subject matter knowledge Teaching of good practices Ensures uniform coverage Two are better than one….

270 Company and Personal Gain Company - High Quality Product - Lower Development Cost - Second Sale - Faster Development Cycle The Japanese success!

271 For the Developer Developing only once Time to learn additional skills Good professional name Advancement

272 The Product Documentation Requirements Requirements Analysis High Level and Low Level Design Code Install (Really a part of code but…) Test Plans and Test Cases Project’s management and user documentation not treated here

273 Walkthrough What is it? (Testing, teaching or?) Why is it done? Are problems discovered? Is there any mechanism to ensure fixes? So it is…..

274 Classical Review Why is it performed? Size (document) Focus (usually try to find solution, lengthy discussions) Length (in hours) Efficiency Termination Participants’ Knowledge

275 Testing All types of testing are after the fact Required changes are costly (usually anything after code requires a lot of effort) TESTING IS TOO LATE TO ELLIMINATE PROBLEMS It finds only problems you are looking for!

276 The Inspection Method A full process - Requirements for quantity - understanding - preparation - roles - problem tracking - measurement - Process feedback

277 Method History Os-360 A HW person to lead SW development? Method inherits from HW development Corrects its self Tested in many SW development areas Improvement directly coupled to Investment

278 What is a Process Informal process –low on objectives –varying quality –no feedback? Formal process (Ob En Ex Fu) –measurement List of differences

279 Definitions Objectives list of products and/or services that must be delivered by the process [it includes constraints, timing, and no. of error etc.]. Entry/Exit Criteria For each objective the items, constraints standards etc. that must be present in order to start [entry] or stop [exit] the process. Function Operations that must be performed in order to transform the inputs into outputs full complying with the objectives

280 Resulting Quality Process = know what and how to perform Execution = actual performance Process x Execution = Product Quality Assume each can be graded between 1 and 10  maximum is 100 Only maximum is acceptable!  Highest Quality

281 Possible Process Inadequacy No process description or standard No training No acceptance Entry and Exit criteria do not match expectations –No documented agreement and no measurements Inadequate method for process control and change

282 Quality Affecting Changes Design Changes (including enhancements and late changes) 3 times more error prone that original 1 out of about 6 code corrections is defective In the field –30% design ; 20% coding; 20% wrong corrections ; Other (testing errors, wrong understanding, not-care!)

283 Testing Hidden Factory We can decrease development effort by 45% function % time spent Verify Function Find &Fix Planned Tests Activity Actual Tests Activity 65 35 30 70

284 Fagan Inspection Process Measure Product against Exit Criteria Detect ALL deviations Rework until Exit Criteria met Improve development process What is new? –Measurements –Process Improvements

285 Definitions Metric Size –NCSS (Non Commentary source statements) are the executable code statements and their declarations. Macros and called routines are counted only once –For documents this definition refers to lines of texts and diagrams. It is obvious that amount of text lines per FI session differs from NCCS

286 Definitions Quality Metric –Defect an instance of non conformance with requirements or exit criteria –In FI we have three defect classes Major A condition which can cause operational failure or produce an unexpected result Minor A condition of bad workmanship that is undesirable but would not cause product failure Investigate A questionable condition that from known facts cannot be pronounced a defect. In itself such a problem points to process deficiency

287 Major Defect In Code –A condition in code (or its documentation) that could cause a malfunction not returning memory not checking user’s input not assigning a default switch case no interrupt reactivation not telling the user what format the data should have

288 Major Defect In Design –An expression or action which if implemented as is will cause a malfunction (operational failure) Using non existent information not assigning a default switch case not enough memory allocated for variable or disk space not using old known conditions (re-inventing the wheel)

289 Major Defect In Requirements –An expression or action which if implemented as stated would cause erroneous results or an action which would need further clarification at later development phase Conflicting requirements (different locations!) “Forgetting” some states Not specifying sizes like converging criteria, maximal sum magnitude, total no. of users, time constraint per action etc. Very general requirements

290 Minor Defects Code & Design –a design or coding direction that is in violation of conventions (local!). In itself it does not result in an error however, if not corrected will cause hardship in maintenance wrong explanation or comment very complicated if statement meaningless variable names unnecessary long calling sequences huge not needed allocations

291 Test Plans and test cases Major –Incorrect test or omission from test which can result in an inadequate testing or failure Minor –Confusing information that will lead to wrong testing

292 Fagan Inspection Process Goals Five levels according to results Goal Level Process Process execution 1-Ad Hoc Undefined Guided by feeling 2-60% defects Design & Code Informal moderator 10% prod. Inspections some training 3-90% defects Formal exit cri- Formal moderator 25% prod. teria Req.,Des. Full training Code, inspec.

293 FI Process Goals (cont.) Five levels according to results Goal Level Process Process execution 5-Same as 4 Same as 4 Same as 4 with Every phase Defect cause TWO fixes for defect free. removal in every defect 50% prod every phase 4-Defect Free Formal process All documents in customer definition. inspected use. Full change 40% prod Control

294 FI Free benefits Cross Training –Encourages wide spread knowledge of project –Eliminate “silo” effect –Wide spread knowledge improves responsiveness to customer Uniform use of projects standards Good ideas spread faster Continuous process improvement

295 2 Complementary Process Walkthrough Objective: communicate Design/code/rationale to others Process: Informal, adjusted to needs, driven by author, no entry or exit criteria used. No follow up as no lists are created Inspections Objective: Verify product meets exit criteria Process: specific 7 steps, managed by moderator, entry/exit criteria met! Defects list for follow-up and process improvement kept!

296 The FI Process Step Objective 1- Planning - Material meets Entry Criteria - Assign inspectors roles - Schedule next steps 2- Overview - Teach inspection team background/context/rationale 3- Preparation - Inspector’s learning phase - Prepare to fulfill role 4- Inspection - Find defects (no solutions!)

297 The FI Process Step Objective 5- Inspection analysis - analyze steps 1-5 - identify cause of systemic defects and recommend proc. changes 6- Rework - Fix all defects 7- Follow up - Verify all defects were fixed and all investigate cleared!

298 Inspector’s Roles Each Role has a function to perform! Moderator: Manages the team while playing an active role as an inspector Author: Creator of the inspected document Reader: Usually a peer of the author, rephrases the document and tells it to the team during the inspection meeting Tester: Considers the testability of the inspected document

299 Moderator Role Description Position Requirements –technical knowledge; can be the creator –works well with people, tactful diplomatic,and forceful in a balanced way. Functions –leads team through 7 steps –keeps team focused; encourages mutually supportive behavior; invites the “phantom inspector” (sum total > individual contribution, synergy) –performs as a moderator (prime) and inspector (secondary)

300 Moderator Checklist Planning –Material meets Entry Criteria –get agreement of qualified inspectors –arrange meeting places and schedule (Overview, Inspection, Analysis) –ensures inspection material gets to inspectors Overview –Ensures Author delivers a balanced overview which covers the background context etc.

301 Moderator Checklist Preparation –Learn Material to a level of a new owner –Identify areas where inquiry should be stressed –Prepare questions (from every role angle) –Note questionable areas (DO NOT PROCLAIM THEM AS DEFECTS!) Inspection –check preparation time of each inspector! –Decide whether preparation adequate

302 Moderator Checklist Inspection (cont.) –Ensure reader paraphrases all statements! –When defect found, record defect description, classify its type, write it down in defects list –Listen to team mood, avert any dysfunctional behavior, make it safe for everyone to play his role –Through synergy invite the ”phantom inspector” –At the end, if needed, schedule re-inspection

303 Moderator Checklist Inspection (cont.) –Do not allow discussion on “a better way to code” or what is the defect’s solution. Inspection Analysis –Evaluate, with team, this particular inspection process (so it can be improved) –identify, using this inspection, opportunities for process improvements –identify root cause of each major defect so development process can be improved

304 Moderator Checklist Rework: No activity Follow-up –Verify all defects were fixed –follow-up and seek resolution for any systemic defects noticed in inspection –Prepare inspection reports

305 Author Checklist Planning –Prepare inspection material according to entry criteria –Prepare an overview –Work with moderator to select appropriate team Overview Deliver it! Preparation –Prepare questions to ask in order to have a thorough exploration of the document –Fix all defects found

306 Reader’s Checklist Preparation –Learn the product well enough to continue its development –Prepare a script to paraphrase the document. Say in your words what each “sentence” does –Prepare questions to ask! Inspection –Deliver your script and questions –Identify defects

307 Tester’s Checklist Preparation –Learn the product well enough to maintain it –Prepare questions of the following kind: In case A does it mean B Are these all possible states, is this the correct range? Where is state X treated? How come this case has a different output? Why is this loop nested Why is difficult condition here, how can it be tested Where is this requirement dealt with? How can a test case be built for this functionality

308 Entry Criteria Exit criteria of the previous phase met Material meets entry criteria (according to Moderator and supervisor document is complete) Requirements are special(why)

309 Requirements Exit Criteria Expert in subject matter agrees Document is written according to standard Document is compiled according to subjects Level of detail permits further development Enough information in the document to permit understanding of subject matter All defects found fixed!

310 The 250 LOC per Inspection Session Preparation RateMajor defects found Loc/hour 50 150 Size of material 200 loc80 125 Loc/hour Inspection Rate

311 Inspection planning rates Code/Pseudocode Req/Design/User doc. Test Plans/Procedures Planning 15 min. 30 min. Overview 500 loc/hour 560 lines/hour Preparation 100 “ / “ 140 “ / “ Inspection 125 “ / “ 140 “ / “ Insp. Anal. 30 min. 45 min.

312 Inspection Team size Defects detected Number of Inspectors 3 4 5 6 18 16 14 12 10

313 Continuous Process Improvement Objective –Eliminate systemic defects and refine work procedure based upon measurement Function –Through the use of statistics and root cause analysis of defects found –Post mortem are useless after shipping the product

314 Testability Tips Covering paths always make functional sense Deeply nested loops or multiple entry loops do not prove genius usually the other way around! Any play with instruction modification, program status word, are from the devil! Easy simple coverage always beats elegance! Multiple entry routines re-entrant code are there but avoid them they bring with them horrible bugs

315 End? We tested documents, code & functionality are we complete? What about: –resources utilization –throughput and waiting time –multiple users

316 Computer Resources Memory –single task –multiple tasks –memory heap Disk I/O –channel bandwidth –disk size (partitions, max. no. of files, multiple tasks) –flashing versus buffered

317 Computer Resources Disk I/O (cont.) –memory mapped files –memory - Disk paging (swapping) CPU –Rate OS services –queues

318 Computer Resources OS services (cont.) –semaphores –stacks –physical paging –heap memory management –etc…...

319 What are typical problems Huge paging rate Queues and semaphores exhausted Disk full Sluggish System behavior Slow response time SYSTEMIC PROBLEMS (forgotten….) Usually these problems found at customer site!

320 How do we measure performance Per task (under maximum input rate) –response time –memory allocation over time –queue and semaphore utilization over time –steady State disk read/write –paging rate –cpu % utilization

321 Single User For the full system –create a test that executes all functionality in a loop –measure the above and see whether system is working right (i.e. it is not being suffocated by work) –look for errors in the different error logs –convince yourself that this is a typical user load

322 Multiple Users Start with 2 users doing the same things as before Vary testing procedure to ensure non synchronized tests Check vital system signs Record performance deterioration Increase number of users until performance is unacceptable

323 Multiple users Increase stress by reducing available resources Kill input feeding line any time (select times so that input chain is broken and watch for recovery or malfunction) After recovery check for zombies in data and tasks, check for bad files etc. Develop the correct simulators it will save time

324 Some numbers to remember Response to GUI input about 5 seconds Response to input stream greater than 1/input rate CPU utilization should not go above 50- 60% (queue theory) Channel utilization should be about 50-60% of capacity (queue theory)

325 System Quality Metric Obstacles –Size –Language –Subject ? The 6 Sigma Model –Hardware definition –Migration to software

326 Left Errors Estimation 6 Sigma is after the fact heavily depends on use!! Can we estimate how many more errors left? How long and how many more tests needed

327 Model Assumptions Many subsystems All equally prone to bugs Subsystem and system tests performed by a few testers in parallel Distribution of bugs found over time Distribution of bugs per test found

328 Errors left & no. of Tests From tail of bugs versus time estimate no. problem left From no. of bugs found per unit time of test estimate how long to continue testing From number of bugs left estimate using no. bugs per test found, the additional tests to be designed and performed

329 Testability A design issue Use and test for return codes Plan to have a log with all actions and return codes What is repeated more then once and is longer than 4 lines should become a function

330 Testability Reuse comes naturally when trying to minimize testing Never use complex if statements Functions should do only one thing! For control use hierarchical bosses Avoid “beautiful” complex things KIS is always better and safer

331 Testability Issues –Keeping a log needs maintenance and execution time –Having many pop-ups and announcements to the user is cumbersome –If execution time is an issue have the log feature optional upon special input. –Build into the system ability to check contents of disk and memory (again time may prohibit)

332 Testability Testability has to be built into the system Testing tools are available they do: –Check memory bounds and run away allocations –Measure test coverage –Indicate paths not gone through –Measure complexity –Some c++ and Java testing tools will reveal wrong definition methods –They will not understand the functionality

333 Maintainability Systems have problems Functionality is added Huge systems are built piece wise How do we minimize –Staff maintaining the code –Improve customer response –Add functionality without changing the whole system

334 It’s a Discipline Roots in Design, Coding and Testing Design –Reduce cohesion between classes and methods –Have many methods each does only one action –Inherit the most general attributes and methods –Templates are “sexy” use them with wisdom –Use explicit constructors and destructors –Private methods and data safe guard integrity

335 It’s a Discipline Design (cont.) –Needed information from other classes means redesign classes –Variable names and methods should be self explanatory –Always design data structures in the same way –Calling sequences should follow the same rules

336 It’s a Discipline Design (cont.) –Design for testability Clear paths for control and data Have logs and the ability to manipulate them Add additional test data collection activated by flags

1 Testing and Maintainability. 2 Bibliography Fagan, M.E. Design and Code Inspections to Reduce errors in Program Development, IBM Journal 3:182-211 (1976).

Similar presentations

Presentation on theme: "1 Testing and Maintainability. 2 Bibliography Fagan, M.E. Design and Code Inspections to Reduce errors in Program Development, IBM Journal 3:182-211 (1976)."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

1 Testing and Maintainability. 2 Bibliography Fagan, M.E. Design and Code Inspections to Reduce errors in Program Development, IBM Journal 3:182-211 (1976).

Similar presentations

Presentation on theme: "1 Testing and Maintainability. 2 Bibliography Fagan, M.E. Design and Code Inspections to Reduce errors in Program Development, IBM Journal 3:182-211 (1976)."— Presentation transcript:

Similar presentations

About project

Feedback