Applications of Metamorphic Testing Chris Murphy University of Pennsylvania November 17, 2011

About Me Lecturer, University of Pennsylvania PhD-Computer Science, Columbia Univ, 2010 Advisor: Prof. Gail Kaiser Research: software testing, CS education Seven years experience in software development industry

Problem: When testing a piece of software, how can we know that we’ve created enough test cases?

Problem: When testing a piece of software, how can we create more test cases? Solution: We use properties of the software to create new test cases from existing ones (particularly those that have not failed). Result: This approach, known as metamorphic testing, is more effective at testing certain types of software than other approaches.

Today's Talk What is metamorphic testing? How is metamorphic testing used to find bugs in software? How can metamorphic testing be applied to applications that do not have test oracles? What are the open research questions related to metamorphic testing?

UPenn CIS 573 Software Engineering Graduate-level software engineering course Just over 100 students Focuses on software maintenance issues: – Testing – Formal verification – Debugging – Fault Localization – Refactoring

It's your first day of work as a software engineer at BloobleSoft. Your boss gives you 6,000 lines of code and a specification and says “find the bugs”. Where would you start?

Specification Code Test Case Generation Strategy Test Cases Test Cases Test Cases Test Cases

You've created 4,837 test cases. All of them pass. How do you know when you're done creating test cases?

Testing Requirements Test Cases Desired Adequacy Level Measurable Adequacy Criteria Acceptable? Coverage Level

Your test cases are achieving 100% coverage. None of them have found any bugs. Are those test cases useful?

Maybe those test cases can be used to create new test cases. The more test cases, the better. Right? This is the idea behind “metamorphic testing”. [Chen et al., HKUST TR CS-98-01, 1998]

A (really, really, really, really) simple example

Let's say you're testing a cosine function. (I know, I know...) How could we use this test case to create new test cases? You have a test case {45º, }, i.e. cos(45º) =

We know that the cosine function exhibits certain properties. That is, if we make certain changes to the input, we can predict the effect on the output. These are referred to as “metamorphic properties”.

What are the metamorphic properties of the cosine function? cos(x + 360º) = cos(x) cos(x + 180º) = -1 * cos(x) cos(x - 360º) = cos(x) That is, if we add 360 to the input, the output should not change.

Given our original test case {45°, }, we can create three follow-on test cases. Property: cos(x + 360º) = cos(x) Input: 45º + 360º = 405º Output: cos(45º) = Property: cos(x - 360º) = cos(x) Input: 45º - 360º = -315º Output: cos(45º) = Property: cos(-1 * x) = -1 * cos(x) Input: -1 * 45º = -45º Output: -1 * cos(45º) =

x f f(x) Initial test case {x, f(x)} t(x) f f(t(x)) Follow-on test case {t(x), f(t(x))} t A metamorphic property of a function f is a pair of functions (t, g) such that f(t(x)) = g(f(x)) for all inputs x g(f(x)) g =

but wait, isn’t that the same as…

Program invariants: -1 ≤ cos(x) ≤ 1 Algebraic properties: cos²(x) = 1 – sin²(x) Describe legal ranges/values of a function, but not how it should react when the input is changed. Describe the relationships between multiple functions, but not a single function.

simple categories of properties

abcdef Initial test case cebafd Permute 2a2b2c2d2e2f Multiply s sum s Add a+2b+2c+2d+2e+2f+2 s+12 2s sum s+g abcdef Include gabcde Exclude s-f sum

abcdef Initial test case #1 s sum ghijkl Initial test case #2 t sum Compose s+t sum abcdef ghijkl Combination of properties 2s+2t sum 2h2d2a2k2e2g 2i2c2l2f2b2j

Common Metamorphic Properties Additive: Increase (or decrease) numerical values by a constant Multiplicative: Multiply numerical values by a constant Permutative: Randomly permute the order of elements in a set Invertive: Create the “opposite” of a set Inclusive: Add a new element to a set Exclusive: Remove an element from a set Compositional: Compose a set [Murphy et al., SEKE’08]

Other Types of Properties Noise-based: include input values that will not affect the output Semantically Equivalent: create inputs that are have the same “meaning” as the original Heuristic: create inputs that are “close” to the original Statistical: create inputs that exhibit the same statistical properties

one more example

Consider a function that takes a set of Points (x-y coordinates) and calculates the total distance from the first to the last, via the rest. What are that function’s metamorphic properties?

Okay, I think I get it. But does it really work?!?!

In order to find bugs… But how can this be?!?! 1. The original test case must pass, even though there is a bug. 2. The follow-on test case must fail.

/* Return the smallest value in the array */ int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min; }

/* Return the smallest value in the array */ int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min; } Test case { {2, 1, 4, 3}, 1}PASS! 100% statement coverage! 100% branch coverage!

/* Return the smallest value in the array */ int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min; } Test case { {2, 1, 4, 3}, 1} Metamorphic property: If we permute the input, the output remains the same. Follow-on test case: { {4, 2, 3, 1}, 1}FAIL!

metamorphic testing in the real world

Bioinformatics [Chen et al., BMC Bioinf., 2009] Machine Learning [Xie et al., JSS, 2011] Network Simulation [Chen et al., FTDS, 2009] Computer Graphics [Guderlei et al., QSIC, 2007]

what types of applications is metamorphic testing good for?

Applications that deal primarily with numerical input and numerical output. Applications that do not have test oracles. Applications that use graph-based algorithms. Compilers. [Zhou et al., ISFST’04]

Specification Program Test Input Actual Output Expected Output Oracle

what if there is no oracle?

Machine Learning

Discrete Event Simulation

x f f(x) t(x) f f(t(x)) Actual t If f(t(x)) = g(f(x)) that does not mean that the output is correct. g =? But if f(t(x)) != g(f(x)) then one (or both) must be incorrect. g(f(x)) Expected

example: RapidMiner

RapidMiner is a suite of machine learning algorithms implemented in Java. That is: c = Classify(M(T), e) In its NaïveBayes implementation, a confidence level c is reported whenever it classifies an example e using a model M created from a training data set T.

We expect that if we modify T to include an extra instance of e, then the confidence level should double, since we are twice as certain about the classification. That is: Classify(M(T+e), e) = 2 * Classify(M(T), e) Our testing detected violations of this property, thus revealing a bug. [Murphy et al., ICST’09]

empirical study

Goal: Show that metamorphic testing is more effective than other techniques at finding bugs in applications without test oracles. Approach: Use mutation analysis to insert faults into the applications, and see how many are detected using various techniques.

Application domains investigated: 1. Machine Learning (C4.5, MartiRank, Support Vector Machines, PAYL) 2. Discrete Event Simulation (JSim) 3. Information Retrieval (Lucene) 4. Optimization (gaffitter)

Techniques investigated: 1. Metamorphic Testing 2. Runtime Assertion Checking 3. Partial Oracle

Experimental Results [Murphy et al., ISSTA’09]

can we do better?

That experiment used application- level metamorphic properties. What if we test at the function level, too? This is known as Metamorphic Runtime Checking. [Murphy et al., TR CUCS , 2009] And continuously conduct those tests while the software is running?

Experimental Results

research directions

When I run my test, I see that the metamorphic property is violated. Does that mean there's a bug? How can we know whether the metamorphic properties are sound? Well, not necessarily....

I've used the guidelines to identify as many metamorphic properties as I could. Does that mean that's all of them? How can we know whether the set of metamorphic properties is complete? Well, not necessarily....

I've used the guidelines to identify as many metamorphic properties as I could. Does that mean that's all of them? Could we detect (likely) metamorphic properties automatically? Well, not necessarily....

I have a function for which I expect that, if I double the input, the output should be doubled. Could I verify that property without actually executing the code? Can metamorphic properties be verified statically? Well, probably....

summary

Metamorphic testing is a method of creating new test cases from existing ones. It depends heavily on the software’s metamorphic properties, which are often numerical. Metamorphic testing is particularly effective at finding bugs in applications that do not have test oracles.

thanks!

Applications of Metamorphic Testing Chris Murphy University of Pennsylvania