ISSTA 2002, Rome, Italy 1 Investigating the Use of Analysis Contracts to Support Fault Isolation in Object-Oriented Code Lionel Briand, Yvan Labiche, Hong Sun Software Quality Engineering Laboratory Systems and Computer Engineering Carleton University Ottawa, Canada

ISSTA 2002, Rome, Italy 2 Fault Isolation in OO Programs Diagnosis of failure is time-consuming in OO software because of the distribution of functionality and the complexity of methods interactions How can Analysis contracts (invariant, pre and post-conditions) be (re)used to instrument the code with assertions so as to help the isolation of faults?  What is an Analysis contract?  How can we measure the benefits of using contracts for fault isolation?

ISSTA 2002, Rome, Italy 3 Related Work [Meyer 92]: Design by Contract [Rosenblum 95]: Most effective assertions in C programs [Voas 95]: Testing with assertions –Needs very large numbers of assertions and executions [Binder 96]: Suggests the use of Built-in Test support to help testing [Baudry 01]: use of contracts to locate faults in classes –Class level testing –Unclear definition of contracts –Unclear measurement of the benefits (diagnosability)  Carefully designed case studies are needed  Precise, operational measure for diagnosability

ISSTA 2002, Rome, Italy 4 Analysis Contracts Contracts defined during Analysis: –Using OCL in the context of UML –Only for problem domain classes –Class invariants, method pre and post-conditions Elements used in contracts ElementsInvariantPreconditionPost-condition AttributeXXX Input parameterXX Output parameterX Return valueX AssociationXXX

ISSTA 2002, Rome, Italy 5 Additional Issues Oracles are expensive parts of the test drivers. Can they be substituted with assertions instrumenting contracts? Can contract assertions help detect additional failures? (I.e., improve observability over oracles alone) Why do certain faults remain undetected when using contract assertions? What are the issues related to instrumenting contracts in Java systems? How does the level of precision of contracts affect the results?

ISSTA 2002, Rome, Italy 6 Level of Precision in Analysis Contracts Problem: Contract instrumentation has a cost. Question: Can we simplify the instrumentation? Constraint: Be systematic. Three levels of detail in post-conditions: –Highest (as defined during Analysis) if A1 then B1 else if A2 then B2 else B3 –Intermediate (check the condition for the standard situation) if A1 then B1 else B2 or B3 –Lowest (ignore the conditions, check only the possible results) B1 or B2 or B3

ISSTA 2002, Rome, Italy 7 Diagnosability measure Exception raisedMethods analyzedMeasure at e ’s entrya, b, c3 during e, before the call to f e, a, b, c4 during e, after the call to f e, f, a, b, c5

ISSTA 2002, Rome, Italy 8 Contract Instrumentation Instrumentation effort depends on: –The language used for writing analysis contracts (i.e., OCL) –The tool and language used for the instrumentation programming language dependent Translate OCL contracts to code Javadoc assertions Instrument the code based on Javadoc assertions Instrumentation issues: –Access to attribute values => requires the addition of ‘get’ methods –Collections => requires to write search algorithms in assertions –Delegation => where to place assertions? –Contracts using methods that read data from a user or a device => requires to use assertions in the body of the method and split postcondition assertions

ISSTA 2002, Rome, Italy 9 Experimental Setting Case study: an Automated Teller Machine –21 classes, 2200 LOCs –with Analysis contracts (defined using OCL) Seeding of faults: 112 faults (OO and non-OO) –17 mutation operators (Java mutation operators by Kim et al) –Faults seeded in 11 classes (non-GUI, non-device interface, core classes) Test cases: category-partition Execution: –5 different ATM case studies: without contracts (only oracles), with contracts at three different levels of details, with contracts and oracles. –Diagnosability measure for each mutant in each test execution that kills it.

ISSTA 2002, Rome, Italy 10 Results 112 mutants –99 mutants killed by oracles –84 mutants killed by contracts (5 equivalent mutants killed) Using contracts (at any level of detail) significantly improves diagnosability (one order of magnitude)  Significant effort savings during debugging Using simple contracts is sufficient # of mutants consideredAverage diagnosability Oracle only ContractsHighest Intermediate Lowest741.37

ISSTA 2002, Rome, Italy 11 Results

ISSTA 2002, Rome, Italy 12 Conclusion Instrumented Analysis contracts are useful for fault isolation –Significant effort savings during debugging –Reuse of contracts defined during UML/OCL Analysis Other results (in technical report) –When combined with oracles, contracts help detect additional mutants (+5, 104/112 mutants detected) –Contracts are good substitute to hard-coded oracles for 84% of mutants  Experiments needed in realistic contexts with actual software developers  Better instrumentation tools (supporting OCL)

ISSTA 2002, Rome, Italy 13 Investigating the Use of Analysis Contracts to Support Fault Isolation in Object-Oriented Code Lionel Briand, Yvan Labiche, Hong Sun Software Quality Engineering Laboratory Systems and Computer Engineering Carleton University Ottawa, Canada

ISSTA 2002, Rome, Italy 14 Analysis Contract: Example context: Bank::initiateWithdrawl(cardNumber:int, PIN:int, serialNumber:int, from:int, amount:Money, newBalance:Money, availableBalance:Money):int pre: serialNumber > 0 and from = CHECKING or from = SAVINGS or from = MONEY_MARKET post: if not self.card->exists(_cardNumber = cardNumber) then result = Status.UNKNOWN_CARD else if PIN <> self._currentTransactionCard._PIN then result = Status.INVALID_PIN else if not self._currentTransactionAccount._available then result = Status.NO_SUCH_ACCOUNT else if from = SAVINGS then result = Status.CANT_WITHDRAW_FROM_ACCOUNT else if self._currenttransactionAccount._availableBalance.less(amount) then result = Status.INSUFFICIENT_AVAILABLE_BALANCE else if (MAXIMUM_WITHDRAWL_AMOUNT_PER_DAY.less(Money.add(self._currentTransactionCard. _withdrawlsToday, amount)) then result = Status.DAILY_WITHDRAWL_LIMIT_EXCEEDED else result = Status.SUCCESS and _currentTransactionAmount.equals(amount) and newBalance.getValue()=self._currentTransactionAcount._currentBalance.getValue()– amount.getValue() and availableBalance.getValue()=self._currentTransactionAccount. _availableBalance.getValue() – amount.getValue() endif