1. Ch6: Software Verification

2. 1 Decision table based testing  Applicability:  Spec. is described by a decision table.  Tables describe:  How combinations of inputs generate different outputs.  Advantages:  Tables are easy to understand  Supports a systematic derivation of tests

3. 2 Decision table based testing (contd..) “The word-processor may present portions of text in three different formats: plain text (p), boldface (b), italics (i). The following commands may be applied to each portion of text: make text plain (P), make boldface (B), make italics (I), emphasize (E), super emphasize (SE). Commands are available to dynamically set E to mean either B or I (we denote such commands as E=B and E=I, respectively.) Similarly, SE can be dynamically set to mean either B (command SE=B) or I (command SE=I), or B and I (command SE=B+I.)” Example

4. 3 Decision table based testing (contd..) P B I E SE E = B E = I SE = I SE = B SE = B + I pbibibib,i action * * * ** * * *** * * * * *

5. 4 Cause effect graphs  Based on a formal way of structuring complex input-output specifications  Transformation of inputs and outputs:  Transform into Boolean values.  Program transformation converted into a Boolean function.

6. 5 Cause effect graphs (contd..) B I P E E = B SE E = I SE = B SE = I SE = B + I b i p A N D O R A N D O R The AND/OR graph represents the correspondence between causes and effects

7. 6 Cause effect graphs (contd..) Constraints may be added to describe legal output a b c e a b c o a b r a b m at most one a b c i at least one one and only one requires masks

8. 7 Cause effect graphs (contd..) “Both B and I exclude P (i.e., one cannot ask both for plain text and, say, italics for the same portion of text.) E and SE are mutually exclusive.” B I P E E = B SE E = I SE = B SE = I SE = B + I m m b i p m m A N D O R A N D O R

9. 8 Cause effect graphs (contd..)  Complete coverage principle:  Generate all possible combinations of inputs & check if o/p occurs according to specs.  Consistency & completeness: o/p corresponding to i/p violates compatibility, or no i/p comb. Generates o/p.  May reduce the number by going backwards from outputs  OR node with true output: Generate only one test case with at least one input true. Output will be true if at least one variable is true.  AND node with false output: Use only one combination with one false output.  May highlight when admissible input violates graph’s consistency requirements  May show incompleteness

10. 9 Testing boundary conditions  Input domain partitioning:  Partition i/p domain in classes, assuming that the behavior is similar for data within a class.  Some typical programming errors occur at the boundary between different classes  X < y, x <=y.  Suggestion:  Test using values both within & at the boundaries.  Applies to both white and black-box testing

11. 10 The oracle problem  How to inspect the results of test executions to reveal failures?  Oracles are reqd. at each stage of testing.  Automated test oracles for running a large # of tests.  Oracles are difficult to design – no universal recipe.

12. 11 Testing in the large: Testing and Modularity  How to test large software systems?  If we apply the same techniques we used to test modules to test systems, extremely complex. Try enumerating paths through the whole system.  Software architecture of the system may be used to drive verification.  Test modules individually and then the whole system.  Benefits of modular testing:  Easier to localize errors, discover errors early, classify errors as to their scope.  Good design techniques enhance verifiability.

13. 12 Testing in the large: Testing and modularity  Module testing: Verifies if a given module has been implemented correctly according to expected ext. behavior.  Integration testing: Testing that occurs as the system is being gradually integrated.  System testing: Test the entire system, as a collection of all the modules, usually in the delivery environment.  Acceptance testing: System testing by a customer.

14. 13 Module testing  Modules often cannot be tested in isolation  May need to provide a temporary context in which the module might execute:  Any modules it calls.  Any non-local data structures it accesses.  Simulate self-calls.

15. 14 Module testing (contd..) Stub: Module may call other modules. Procedure that emulates the behavior, Has the same i/o behavior as the called module, but simple impl. May be a Look-up table. Driver: Module may get called by other modules, piece of code that simulates the behavior of the calling module. PROCEDURE UNDER TEST DRIVERSTUB CALL ACCESS TO NONLOCAL VARIABLES Harness or scaffolding: Support s/w needed for testing.

16. 15 Integration testing  Big-bang testing: test individual modules & then system:  No system testing, all inter-module dependencies resolved during testing, many interactions may lead to high complexity.  Incremental testing: Apply incrementality principle to integration testing:  Modules are developed and tested incrementally.  Easier to localize errors.  May identify collection of modules as autonomous subsystems.  Reduces the need for stubs & drivers for module testing.  Can be bottom-up, top-down.

17. 16 Integration testing (contd..)  Bottom-up aggregation:  Start aggregation and testing via USES hierarchy.  Need drivers to emulate calls.  Top-down aggregation:  Start from the top-level modules.  Use stubs to simulate lower level modules.  Top-down and bottom-up approaches may be used in conjunction with each other

18. 17 Integration testing (contd..) A B C D E If integration and test proceed bottom-up only need drivers Otherwise, if we proceed top-down only stubs are needed

19. 18 Integration testing (contd..) M 1M 2 2,12,2 MM M1 USES M2 and M2 IS_COMPOSED_OF {M2,1, M2,2} CASE 1 Test M1, providing a stub for M2 and a driver for M1 Then provide an implementation for M2,1 and a stub for M2,2 CASE 2 Implement M2,2 and test it by using a driver, Implement M2,1 and test the combination of M2,1 and M2,2 (i.e., M2) by using a driver Finally, implement M1 and test it with M2, using a driver for M1

20. 19 Separation of concerns in testing  In general, testing involves several phases, with different goals, performed by different people  Use principle of separation of concerns to design test cases for different qualities:  Performance, robustness, user friendliness.  A sample of different concerns  Overload testing: Test behavior under peak conditions.  Robustness: Test under unexpected conditions, erroneous user commands, power failure.  Regression testing: Test whether correctness & other qualities are maintained after mods.

21. 20 Testing object-oriented systems  General testing principles may be applied to object-oriented systems  Classes are similar to components in the traditional approach  Testing in the small:  Add a few methods for testing. Other classes may be stubbed in.  Testing in the large:  Inheritance, interactions, public interface, private implementation.

22. 21 Testing object-oriented systems (contd..)  New issues  Inheritance  Genericity  Polymorphism  Dynamic binding  Open problems still exist

23. 22 Testing object-oriented systems (contd..)  “ Flattening ” the whole hierarchy and considering every class as a totally independent component  Finding an ad-hoc way to take advantage of the hierarchy  A sample strategy:  A test that does not have to be repeated for any child class  A test that must be performed for child class X and all of its further children  A test that must be redone by applying the same input data, but verifying that the output is not (or is) changed  A test that must be modified by adding other input parameters and verifying that the output changes accordingly

24. 23 Testing object-oriented systems (contd..)  Black-box testing:  Test if class provides the functionality, ignore implementation. Essentially testing of public interface.  Complete coverage principle – test each method.  White-box testing:  Test hidden implementation. Ignores if public implementation. Apply coverage criteria to test cases.  Must perform both black and white box testing