MS thesis/project ideas for MS students in the Department of Computer Science at the University of Minnesota Duluth suggestions by Dr Andrew Brooks, Heller Hall /17/2014Dr Andrew Brooks - MS thesis/project ideas

background research article 10/17/20142 Kanewala, U.; Bieman, J.M., "Techniques for testing scientific programs without an oracle," Software Engineering for Computational Science and Engineering (SE-CSE), th International Workshop on, pp.48-57, 2013 doi: /SECSE Some presentation material drawn directly from article(s). Dr Andrew Brooks - MS thesis/project ideas

10/17/20143 A Taxonomy for Test Oracles, Douglas Hoffman, Software Quality Week (QW’98), 1998, 8pp. A gentle introduction to oracles: Some presentation material drawn directly from article(s). Chapter 6 Assertion facility, in JDK 1.4 Tutorial, Greg Travis, Manning Publications Company, A gentle introduction to assertions: Dr Andrew Brooks - MS thesis/project ideas

the difficulty of testing software Dr Andrew Brooks - MS thesis/project ideas4 Test case identifierTest case valuesExpected resultsActual results 1[2,7,0]11 2[0,7,2]10 How do we calculate the expected results? (and know that they are correct) The oracle is the mechanism used to generate expected results. 10/17/2014

Oracle methods used by Hoffman A human calculates the expected results. A separate program implementing the same algorithm is used to calculate the expected results. A simulation is used to calculate the expected results. A hardware simulator is used to calculate the expected results. An earlier version of the software is used to calculate the expected results. The same version of the software on a different hardware platform is used to calculate the expected results. A check is made on the consistency of generated values and end points. A sample of values can be checked against independently generated expected results from existing commercial or open-source software. 10/17/20145Dr Andrew Brooks - MS thesis/project ideas

the difficulty of testing scientific software 10/17/20146 “The existence of an oracle is often assumed in software testing.” “But in many situations, especially for scientific programs, oracles do not exist or they are too hard to implement.” Kanewala and Bieman, 2013 Scientists often write programs to discover answers that are previously unknown. Dr Andrew Brooks - MS thesis/project ideas

Metamorphic Testing (a way around the oracle problem) A metamorphic relation specifies how a particular change to the input of the program would change the output. 10/17/20147 failure Dr Andrew Brooks - MS thesis/project ideas

examples of metamorphic relations 10/17/20148Dr Andrew Brooks - MS thesis/project ideas

Is it possible to replicate an experiment? A report should contain enough detail so that another researcher can repeat the experiment and obtain similar results with similar conclusions. 10/17/2014Dr Andrew Brooks - MS thesis/project ideas9

MS thesis/project idea 1. First begin by replicating in part or in full one of the previous experiments on metamorphic testing. – References to several emprical studies are given by Kanewala and Bieman. 10/17/201410Dr Andrew Brooks - MS thesis/project ideas

MS thesis/project idea 1. Then tackle one of the problems described by Kanewala and Bieman. – automatic detection of metamorphic relations some recent work has been done by Kanewala and Bieman – identifying the most effective metamorphic relations – identifying the most effective combinations of metamorphic relations – understanding why some program mutations could not be killed using metamorphic testing 10/17/ deliberately introduced mistakes Dr Andrew Brooks - MS thesis/project ideas

Assertion Testing (a way around the oracle problem) Assertions are embedded into source code and evaluated when a test case is executed. 10/17/ An assertion is a predicate placed in a program to indicate that the developer thinks that the predicate is always true at that place. Dr Andrew Brooks - MS thesis/project ideas

13 package asserting; public class First { public static void main(String[] args) { int j = 10; assert j>10; } simple Java example Exception in thread "main" java.lang.AssertionError at asserting.First.main(First.java:7) What happens when you run this program? console 10/17/2014

Dr Andrew Brooks - MS thesis/project ideas14 Assertion basics (Greg Travis chapter) Preconditions should be true at the start of a method. Postconditions should be true at the end of a method. Invariants should always be true. – A speed should not be faster than light speed... 10/17/2014

MS thesis/project idea 2. First begin by replicating in part or in full one of the previous experiments on assertion testing. – References to several empirical studies are given by Kanewala and Bieman. 10/17/201415Dr Andrew Brooks - MS thesis/project ideas

MS thesis/project idea 2. Then tackle one of the problems described by Kanewala and Bieman. – automatic invariant detection can yield spurious invariants can ways be found to reduce or eliminate these? – the effectiveness of assertion checking seems all or nothing depending on the method can we understand why? – many more experiments are needed working with specifications to identify assertions rewriting program comments as assertions 10/17/201416Dr Andrew Brooks - MS thesis/project ideas