1 T.Y. Chen Swinburne University of Technology, Australia T.H. Tse and Zhiquan Zhou The University of Hong Kong Semi-Proving: an Integrated Method Based on Global Symbolic Evaluation and Metamorphic Testing (speaker)

2 Presentation Outline  Conventional Program Testing and Proving  Metamorphic Testing  Our method: Semi-Proving  Summary.

3  Conventional Program Testing and Proving  Metamorphic Testing  Our method: Semi-Proving  Summary. Presentation Outline

4 Conventional Program Testing and Proving Given a bijective function f ; A Program: F_Sort (a 1, a 2,..., a n ), n  2 Output: (a 1 ’, a 2 ’,..., a n ’), such that 1. (a 1 ’, a 2 ’,..., a n ’) is a permutation of (a 1, a 2,..., a n ) 2. f (a 1 ’)  f (a 2 ’) ...  f (a n ’). Given a bijective function f ; A Program: F_Sort (a 1, a 2,..., a n ), n  2 Output: (a 1 ’, a 2 ’,..., a n ’), such that 1. (a 1 ’, a 2 ’,..., a n ’) is a permutation of (a 1, a 2,..., a n ) 2. f (a 1 ’)  f (a 2 ’) ...  f (a n ’).

5 Conventional Program Testing and Proving  Testing 1. Design test cases: e.g. (2, 6, 3) for n=3 2. Run: F_Sort (2, 6, 3) = (6, 3, 2) 3. Check: f (6) < f (3) < f (2) ? 1. Design test cases: e.g. (2, 6, 3) for n=3 2. Run: F_Sort (2, 6, 3) = (6, 3, 2) 3. Check: f (6) < f (3) < f (2) ?

6 Conventional Program Testing and Proving  Proving correctness 1. F_Sort terminates for any valid input; 2. The output is correct. 1. F_Sort terminates for any valid input; 2. The output is correct.

7 Conventional Program Testing and Proving  Proving properties F_Sort (a 1, a 2,..., a n ) = (a 1 ’, a 2 ’,..., a n ’) Permutation.

8 Metamorphic Testing  Metamorphic Testing Employing relationships between different executions Fact: different permutations will produce same output F_Sort (a 1, a 2, a 3 ) Fact: different permutations will produce same output F_Sort (a 1, a 2, a 3 ) F_Sort (a 3, a 1, a 2 ) = “ Metamorphic Relation ” ·

9 Metamorphic Testing Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)} Metamorphic Testing: 1. F_Sort (2, 6, 3) = (6, 3, 2) Metamorphic Testing: 1. F_Sort (2, 6, 3) = (6, 3, 2) No matter whether an oracle is available or not; Very useful when the oracle cannot be found. 2. F_Sort (3, 2, 6) = (6, 3, 2) | || | PASS

10 Metamorphic Testing Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)} Metamorphic Testing: 1. F_Sort (2, 6, 3) = (6, 3, 2) Metamorphic Testing: 1. F_Sort (2, 6, 3) = (6, 3, 2) 2. F_Sort (3, 2, 6) = (3, 6, 2) Failure. | || |

11  Conventional Program Testing and Proving  Metamorphic Testing  Semi-Proving: Verifying Metamorphic Relations  Summary. Presentation Outline

12 Semi-Proving: Verifying Metamorphic Relations  Objective: If the program does not satisfy a metamorphic relation on some inputs, locate these inputs; Otherwise prove the satisfaction of the metamorphic relation over all inputs.

13  Why called “Semi”? Proving necessary properties, which may not be sufficient for program correctness  Characteristics of Semi-Proving Multiple symbolic executions Testing and proving. Semi-Proving: Verifying Metamorphic Relations

14 double GetMid (double x1, double x2, double x3) {double mid; mid = x3; if (x2 < x3) if (x1 < x2) mid = x2; else { if (x1 < x3) mid = x1; } else if (x1 > x2) mid = x2; else if (x1 > x3) mid = x1; return mid; } double GetMid (double x1, double x2, double x3) {double mid; mid = x3; if (x2 < x3) if (x1 < x2) mid = x2; else { if (x1 < x3) mid = x1; } else if (x1 > x2) mid = x2; else if (x1 > x3) mid = x1; return mid; } Semi-Proving: Verifying Metamorphic Relations

15 SpecificationSpecification “GetMid (X, Y, Z)” returns the median of (X, Y, Z) E.g. GetMid (3, 4, 1): “3”. Semi-Proving: Verifying Metamorphic Relations

16  Verifying “GetMid” by Semi-Proving Identify a Metamorphic Relation GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) ) Semi-Proving: Verifying Metamorphic Relations any numbersany permutation Purpose: to verify

17  Basic concepts Transposition simple permutation that exchanges two elements (1, 2, 3) (1, 2, 3)  1 (1, 2, 3)  (1, 3, 2)  2  (2, 1, 3) Semi-Proving: Verifying Metamorphic Relations

18 A tuple (1, 2, 3) A permutation (2, 3, 1) (1, 2, 3) A tuple (1, 2, 3) A permutation (2, 3, 1) (1, 2, 3) (2, 3, 1)  11 (2, 1, 3)  22  Basic concepts Composition of Transpositions Semi-Proving: Verifying Metamorphic Relations

19  Result from Group Theory Any permutation of (X, Y, Z) can be achieved by compositions of transpositions (X, Z, Y) and (Y, X, Z). Semi-Proving: Verifying Metamorphic Relations

20 Semi-Proving: Verifying Metamorphic Relations  Purpose GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) ) Only need to verify: Any permutation. GetMid (X, Y, Z) = GetMid (X, Z, Y) GetMid (X, Y, Z) = GetMid (Y, X, Z)

21 Semi-Proving: Verifying Metamorphic Relations  Purpose GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) ) Only need to verify: GetMid (X, Y, Z) = GetMid (X, Z, Y) GetMid (X, Y, Z) = GetMid (Y, X, Z)

22  Global Symbolic Evaluation on GetMid (X, Y, Z) Execute all the possible paths. Semi-Proving: Verifying Metamorphic Relations

23 double GetMid (double x1, double x2, double x3) {double mid; mid = x3; if (x2 < x3) if (x1 < x2) mid = x2; else { if (x1 < x3) mid = x1; } else if (x1 > x2) mid = x2; else if (x1 > x3) mid = x1; return mid; } double GetMid (double x1, double x2, double x3) {double mid; mid = x3; if (x2 < x3) if (x1 < x2) mid = x2; else { if (x1 < x3) mid = x1; } else if (x1 > x2) mid = x2; else if (x1 > x3) mid = x1; return mid; } Semi-Proving: Verifying Metamorphic Relations

24 C1: (Y  X < Z) OR (Z < X  Y) Path Conditions C2: (X < Y < Z) OR (Z  Y < X) C3: (Y < Z  X) OR (X  Z  Y) Semi-Proving: Verifying Metamorphic Relations X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true

25 Semi-Proving: Verifying Metamorphic Relations ? GetMid (X, Z, Y) ? X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true

26 C4: (Z  X < Y) OR (Y < X  Z) C5: (X < Z < Y) OR (Y  Z < X) C6: (Z < Y  X) OR (X  Y  Z) PASS Semi-Proving: Verifying Metamorphic Relations ? GetMid (X, Z, Y) ? X when C4 is true = Z when C5 is true Y when C6 is true X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true

27 ? ? X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true X when C4 is true = Z when C5 is true Y when C6 is true  Contradiction PASS C1: (Y  X < Z) OR (Z < X  Y) & Semi-Proving: Verifying Metamorphic Relations GetMid (X, Z, Y) ? C4: (Z  X < Y) OR (Y < X  Z) C5: (X < Z < Y) OR (Y  Z < X) C6: (Z < Y  X) OR (X  Y  Z)

28 ? ? C4: (Z  X < Y) OR (Y < X  Z) C5: (X < Z < Y) OR (Y  Z < X) C6: (Z < Y  X) OR (X  Y  Z) X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true X when C4 is true = Z when C5 is true Y when C6 is true C1: (Y <= X < Z) OR (Z < X <= Y) & X=Y<Z OR Z<Y=X Semi-Proving: Verifying Metamorphic Relations ? GetMid (X, Z, Y)

29 ? ? ? C4: (Z  X < Y) OR (Y < X  Z) C5: (X < Z < Y) OR (Y  Z < X) C6: (Z < Y  X) OR (X  Y  Z) X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true X when C4 is true = Z when C5 is true Y when C6 is true C1: (Y <= X < Z) OR (Z < X <= Y) & Yes. X=Y PASS X=Y<Z OR Z<Y=X Semi-Proving: Verifying Metamorphic Relations GetMid (X, Z, Y)

30 ? X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true Semi-Proving: Verifying Metamorphic Relations GetMid (X, Z, Y) verified

31 ? X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true Semi-Proving: Verifying Metamorphic Relations ConclusionConclusion GetMid (X, Z, Y)

32 ? X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true Semi-Proving: Verifying Metamorphic Relations ConclusionConclusion GetMid (X, Z, Y)

33 X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true Semi-Proving: Verifying Metamorphic Relations ConclusionConclusion GetMid (X, Z, Y)

34 X when C1 is true GetMid (X, Y, Z) =Y when C2 is true Z when C3 is true Semi-Proving: Verifying Metamorphic Relations ConclusionConclusion GetMid (X, Z, Y) Composition of transpositions GetMid (X, Y, Z) = GetMid ( Permute(X, Y, Z) ) GetMid (Y, X, Z) AnyAny.

35  Detecting Program Faults · Semi-Proving: Detecting Program Faults

36 double GetMid (double x1, double x2, double x3) {double mid; mid = x3; if (x2 < x3) if (x1 < x2) mid = x2; else { if (x1 < x3) mid = x1; } else if (x1 > x2) mid = x2; else if (x1 > x3) mid = x1; return mid; } double GetMid (double x1, double x2, double x3) {double mid; mid = x3; if (x2 < x3) if (x1 < x2) mid = x2; else { if (x1 < x3) mid = x1; } else if (x1 > x2) mid = x2; else if (x1 > x3) mid = x1; return mid; }

37 Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y) Semi-Proving: Detecting Program Faults | || | X when Y  X < Z ? | || | Y when (Z < Y  X ) OR (Y  Z AND X  Z) AND

38 Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y) Semi-Proving: Detecting Program Faults | X when Y  X < Z ? | Y when (Z < Y  X ) OR (Y  Z AND X  Z) AND  (Y=X<Z) OR (Y<X<Z)

39 Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y) Semi-Proving: Detecting Program Faults | X when Y  X < Z ? | Y when (Z < Y  X ) OR (Y  Z AND X  Z) AND  (Y=X<Z) OR (Y<X<Z) ?  failure Failure-causing input Can identify all the failure-causing inputs.

40 Summary  A proving technique: all the paths  A testing technique: failure-causing inputs selected path(s)  Characteristics Metamorphic relations Multiple symbolic executions Employing global symbolic evaluation and constraint solving.

41 Questions are welcome