Software Testing: A Selection of Papers by Elaine Weyuker Peter Greenwood for EEE 599

Apr 98EEE Weyuker Presentation2 Elaine Weyuker PhD (CS) Rutgers U ‘77 Software Testing Researcher at AT&T Bell Labs, Murray Hill, NJ CS Professor Courant Institute of NYU 30+ papers, primarily on Software Testing

Apr 98EEE Weyuker Presentation3 The Papers Axiomatizing Software Test Data Adequacy (TSE Dec 86) The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software (TSE Sep 95, with Avitzer) Reliability Testing of Rule-Based Systems (Software Jun 96, with Avritzer and Ros)

Apr 98EEE Weyuker Presentation4 Goal of Presentation highlight some of Weyuker’s prolific work in software testing contrast the general and theoretical character of her early work with the specific and practical nature of her current work

Apr 98EEE Weyuker Presentation5 Axiomizing Software Test Data Adequacy (Dec 86) How much testing is enough to determine if a program is acceptable? –very important question –relatively little research activity (1986) –no industry standards test until testing time expired test until all cases in current test set pass

Apr 98EEE Weyuker Presentation6 When Should Testing Stop? Terminate when an adequacy criterion is met –a measure of how well the testing process has been performed –relates a test set to the program, the specification, or both –could relate a test set to the program’s operational profile

Apr 98EEE Weyuker Presentation7 Constraints of this Approach deals only with dynamic testing (running the program on test cases) does not involve any static testing (analysis of code, structure, etc.)

Apr 98EEE Weyuker Presentation8 Constraints (con’t) Weyuker’s paper deals only with program-based criteria –ignores specifications –most adequacy criteria are of this type –easier to automate

Apr 98EEE Weyuker Presentation9 Weyuker on Testing “The purpose of testing is to uncover errors, not to certify correctness.”

Apr 98EEE Weyuker Presentation10 The Axioms (1 & 2) Axiom 1: (Applicability) For every program, there exists a finite adequate test set. Axiom 2: (Nonexhaustive Applicability) There is a program P and a test set T such that P is adequately tested by T, and T is not an exhaustive test set.

Apr 98EEE Weyuker Presentation11 Axioms 3 & 4 Axiom 3: (Monotonicity) If T is adequate for P, and T is a subset of T’, then T’ is adequate for P. Axiom 4: (Inadequate Empty Set) The empty set is not adequate for any program.

Apr 98EEE Weyuker Presentation12 Axioms 5 & 6 Axiom 5: (Anti-extensionality) There are programs P and Q such that P is equivalent to Q, T is adequate for P, but T is not adequate for Q. Axiom 6: (General Multiple Change) There are programs P and Q which are the same shape, and a test set T such that T is adequate for P, but T is not adequate for Q.

Apr 98EEE Weyuker Presentation13 Axiom 7 Axiom 7: (Anti-decomposition) There exists a program P and a component Q such that T is adequate for P, T’ is the set of vectors of values that variables can assume on entrance to Q for some t of T, and T’ is not adequate for Q.

Apr 98EEE Weyuker Presentation14 Axiom 8 Axiom 8: (Anti-composition) There exist programs P and Q such that T is adequate for P and P(T) is adequate for Q, but P is not adequate for P;Q.

Apr 98EEE Weyuker Presentation15 Summary of Axiomatizing axioms are applied to 5 adequacy criteria –path, branch, statement, mutations, modified size path, branch, statement fail application and antidecompositon branch and statement also fail anticomposition

Apr 98EEE Weyuker Presentation16 Summary of Axiomatizing

Apr 98EEE Weyuker Presentation17 Criticism Zweben & Gourlay, On the Adequacy of Weyuker’s Test Data Adequacy Axioms (IEEE TSE Apr 89) –“The properties proposed by Weyuker are neither fundamental enough nor precise enough to serve as a useful formal system for discussing adequacy or comparing criteria”

Apr 98EEE Weyuker Presentation18 More Criticism Professor Norman Fenton, keynote address to 2nd QUALMS User Group Meeting, South Bank University, Jul 92 –“Elaine Weyuker … has presented a list of axioms … Measurement Theory reveals that some of these axioms are contradictory.”

Apr 98EEE Weyuker Presentation19 Dr. Weyuker’s Current Work

Apr 98EEE Weyuker Presentation20 Automatic Generation of Load Test Suites (Sep 95) develops 3 algorithms for automatic test case generation applies them to 5 real industrial software systems assess the results

Apr 98EEE Weyuker Presentation21 Failure Types Hard failure –complete loss of traffic ( ie cable break, power failure, router failure) Soft failure –degraded performance (ie activity burst) –important that this does not occur –telecoms designed with excess capacity –avoid permanent soft failure

Apr 98EEE Weyuker Presentation22 Goals a smoothly degrading system graceful recovery

Apr 98EEE Weyuker Presentation23 Load Testing Strategies system functionality has already been tested assess system performance under a known load black-box oriented based on system operational profile

Apr 98EEE Weyuker Presentation24 Typical Telecom Operational Profile number of different types of call average service demands and probability distribution of each call type average external arrival rate routing probabilities for each node this information is usually readily available

Apr 98EEE Weyuker Presentation25 Purpose of Algorithms tests resource allocation mechanisms associated with software systems system behavior under load is not measured by functionality testing each variable vector is a distinct system state

Apr 98EEE Weyuker Presentation26 Basis of Algorithms too many states to exhaustively test calculate probability of system being in each state (from operational profile) heuristically determine some threshold  generate a test case for those states where P(S) > 

Apr 98EEE Weyuker Presentation27 The Algorithms Deterministic State Testing (DST) –generate a test case for each state S where P(S) >  DST with State Transition Checking (TDST) –as for DST, but also generate test cases for every S’ reachable from S

Apr 98EEE Weyuker Presentation28 The Algorithms (con’t) DST with Length N Cycles (CDST) –as for DST, but also generate test cases for every S’ within a cycle of length N from S with a probability > 

Apr 98EEE Weyuker Presentation29 Target Systems Project 1:telephone call rerouter Project 2:800 number call traffic monitor Project 3:routing automator Project 4:transaction processing system Project 5: network interface software

Apr 98EEE Weyuker Presentation30 Testing Results automated test generation routines were developed for all five projects automated test results were presented for projects 1 & 2

Apr 98EEE Weyuker Presentation31 Project 1 Results telephone call rerouter software test results accurately predicted system performance testing revealed a serious architectural problem

Apr 98EEE Weyuker Presentation32 Project 2 Results 800 number call traffic monitor led to better requirements analysis used automated test case generation exclusively, replacing hand-crafting more test cases with more variety uncovered four potentially severe faults minimal additional cost

Apr 98EEE Weyuker Presentation33 Reliability Testing of Rule- Based Systems (Sep 96) applies same automatic test case generation algorithm to an industrial network surveillance system system was a reimplementation, so a large body of operational data was available from the original system system designed to run 24 hours/day; 365 days/year

Apr 98EEE Weyuker Presentation34 Approach system has very large state space 355 days of operational data from 93 switches were used to identify 857 states entered coverage required testing 843 states

Apr 98EEE Weyuker Presentation35 State Testing Requirements

Apr 98EEE Weyuker Presentation36 Findings historical data is useful for selecting test cases based on state probabilities selection algorithm must consider –reliability requirements –size of the (limited) state space –comprehensiveness of individual test cases

Apr 98EEE Weyuker Presentation37 Summary change of focus from theory to application extension from telecom to general application accomplished, respected, and prolific researcher in software testing and reliability