Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles Christian Murphy, Kuang Shen, Gail Kaiser Columbia.

Slides:



Advertisements
Similar presentations
Copyright © 2006 The McGraw-Hill Companies, Inc. Programming Languages 2nd edition Tucker and Noonan Chapter 18 Program Correctness To treat programming.
Advertisements

COMP 121 Week 9: AbstractList and ArrayList. Objectives List common operations and properties of Lists as distinct from Collections Extend the AbstractCollection.
Chapter 7 User-Defined Methods. Chapter Objectives  Understand how methods are used in Java programming  Learn about standard (predefined) methods and.
Dept. of Computer Science A Runtime Assertion Checker for the Java Modeling Language (JML) Yoonsik Cheon and Gary T. Leavens SERP 2002, June 24-27, 2002.
Type Checking.
1 CS2200 Software Development Lecture: Testing and Design A. O’Riordan, 2008 K. Brown,
Object Oriented Design An object combines data and operations on that data (object is an instance of class) data: class variables operations: methods Three.
Metamorphic Testing Techniques to Detect Defects in Applications without Test Oracles Christian Murphy Thesis Defense April 12, 2010.
1 Application of Metamorphic Testing to Supervised Classifiers Xiaoyuan Xie, Tsong Yueh Chen Swinburne University of Technology Christian Murphy, Gail.
Properties of Machine Learning Applications for Use in Metamorphic Testing Chris Murphy, Gail Kaiser, Lifeng Hu, Leon Wu Columbia University.
Software Testing and Quality Assurance
Chapter 8 Exceptions. Topics Errors and Exceptions try-catch throwing Exceptions Exception propagation Assertions.
Automatic System Testing of Programs without Test Oracles
The In Vivo Testing Approach Christian Murphy, Gail Kaiser, Ian Vo, Matt Chu Columbia University.
On Effective Testing of Health Care Simulation Software Christian Murphy, M.S. Raunak, Andrew King, Sanjian Chen, Christopher Imbriano, Gail Kaiser, Insup.
Applications of Metamorphic Testing Chris Murphy University of Pennsylvania November 17, 2011.
OOP #10: Correctness Fritz Henglein. Wrap-up: Types A type is a collection of objects with common behavior (operations and properties). (Abstract) types.
Copyright © 2006 The McGraw-Hill Companies, Inc. Programming Languages 2nd edition Tucker and Noonan Chapter 18 Program Correctness To treat programming.
Parameterizing Random Test Data According to Equivalence Classes Chris Murphy, Gail Kaiser, Marta Arias Columbia University.
Automatic Detection of Previously-Unseen Application States for Deployment Environment Testing and Analysis Chris Murphy, Moses Vaughan, Waseem Ilahi,
© 2006 Pearson Addison-Wesley. All rights reserved2-1 Chapter 2 Principles of Programming & Software Engineering.
1 Software Testing and Quality Assurance Lecture 5 - Software Testing Techniques.
Using Runtime Testing to Detect Defects in Applications without Test Oracles Chris Murphy Columbia University November 10, 2008.
1 Software Testing Techniques CIS 375 Bruce R. Maxim UM-Dearborn.
Subclasses and Subtypes CMPS Subclasses and Subtypes A class is a subclass if it has been built using inheritance. ▫ It says nothing about the meaning.
Handouts Software Testing and Quality Assurance Theory and Practice Chapter 9 Functional Testing
Your Interactive Guide to the Digital World Discovering Computers 2012.
Presenter: Shant Mandossian EFFECTIVE TESTING OF HEALTHCARE SIMULATION SOFTWARE.
Unit Testing & Defensive Programming. F-22 Raptor Fighter.
Introduction 01_intro.ppt
Verification and Validation Yonsei University 2 nd Semester, 2014 Sanghyun Park.
 Value, Variable and Data Type  Type Conversion  Arithmetic Expression Evaluation  Scope of variable.
1 Abstraction  Identify important aspects and ignore the details  Permeates software development programming languages are abstractions built on hardware.
1 Automatic Refinement and Vacuity Detection for Symbolic Trajectory Evaluation Orna Grumberg Technion Haifa, Israel Joint work with Rachel Tzoref.
Chapter 06 (Part I) Functions and an Introduction to Recursion.
Object-Oriented Program Development Using Java: A Class-Centered Approach, Enhanced Edition.
Low-Level Detailed Design SAD (Soft Arch Design) Mid-level Detailed Design Low-Level Detailed Design Design Finalization Design Document.
Programming Life Cycle Problem analysisunderstand the problem Requirements definition specify what program will do High- and low-level designhow it meets.
CS Data Structures I Chapter 2 Principles of Programming & Software Engineering.
CMP-MX21: Lecture 4 Selections Steve Hordley. Overview 1. The if-else selection in JAVA 2. More useful JAVA operators 4. Other selection constructs in.
1 Test Selection for Result Inspection via Mining Predicate Rules Wujie Zheng
© 2006 Pearson Addison-Wesley. All rights reserved2-1 Chapter 2 Principles of Programming & Software Engineering.
© 2006 Pearson Addison-Wesley. All rights reserved 2-1 Chapter 2 Principles of Programming & Software Engineering.
Principles of programming languages 6: Types Isao Sasano Department of Information Science and Engineering.
Function Overloading Two different functions may have the same name as long as they differ in the number or types of arguments: int max(int x, int y) and.
Lecture 5 functions 1 © by Pearson Education, Inc. All Rights Reserved.
CPSC 871 John D. McGregor Module 8 Session 1 Testing.
Data Design and Implementation. Definitions Atomic or primitive type A data type whose elements are single, non-decomposable data items Composite type.
PROGRAMMING TESTING B MODULE 2: SOFTWARE SYSTEMS 22 NOVEMBER 2013.
Refactoring Agile Development Project. Lecture roadmap Refactoring Some issues to address when coding.
PROGRAMMING PRE- AND POSTCONDITIONS, INVARIANTS AND METHOD CONTRACTS B MODULE 2: SOFTWARE SYSTEMS 13 NOVEMBER 2013.
1 Introduction 1. Why Data Structures? 2. What AreData Structure? 3. Phases of Software Development 4. Precondition and Postcondition 5. Examples.
Identifying “Best Bet” Web Search Results by Mining Past User Behavior Author: Eugene Agichtein, Zijian Zheng (Microsoft Research) Source: KDD2006 Reporter:
Chapter 1 Java Programming Review. Introduction Java is platform-independent, meaning that you can write a program once and run it anywhere. Java programs.
Array Size Arrays use static allocation of space. That is, when the array is created, we must specify the size of the array, e.g., int[] grades = new int[100];
 Simulation enables the study of complex system.  Simulation is a good approach when analytic study of a system is not possible or very complex.  Informational,
Object Design More Design Patterns Object Constraint Language Object Design Specifying Interfaces Review Exam 2 CEN 4010 Class 18 – 11/03.
Announcements Assignment 2 Out Today Quiz today - so I need to shut up at 4:25 1.
CPSC 372 John D. McGregor Module 8 Session 1 Testing.
Topics  Direct Predicate Characterization as an evaluation method.  Implementation and Testing of the Approach.  Conclusions and Future Work.
Principles of Programming & Software Engineering
Computer Simulation Henry C. Co Technology and Operations Management,
John D. McGregor Session 9 Testing Vocabulary
Principles of programming languages 8: Types
Object Oriented Programming
Accessible Formal Methods A Study of the Java Modeling Language
User-Defined Functions
Java Modeling Language (JML)
Assertions References: internet notes; Bertrand Meyer, Object-Oriented Software Construction; 4/25/2019.
Software Testing and QA Theory and Practice (Chapter 5: Data Flow Testing) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory and Practice.
Presentation transcript:

Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles Christian Murphy, Kuang Shen, Gail Kaiser Columbia University

Chris Murphy – Columbia University2 Problem Statement Some applications (e.g. machine learning, simulation) do not have test oracles that indicate whether the output is correct for arbitrary input Oracles may exist for a limited subset of the input domain, and gross errors (e.g. crashes) can be detected with certain inputs or other techniques However, it is difficult to detect subtle (computational) errors for arbitrary inputs

Chris Murphy – Columbia University3 Observation Even when there is no oracle in the general case (i.e. knowing the relationship between a particular input and its output), it may be possible to know relationships between sets of inputs and their corresponding outputs One such approach that takes advantage of this is “Metamorphic Testing” [Chen ’98]

Chris Murphy – Columbia University4 Metamorphic Testing An approach for creating follow-up test cases based on those that have not revealed any defects If input x produces output f(x), then the function’s “metamorphic properties” are used to guide a transformation function t, which is applied to produce a new test case, t(x) We can then predict the expected value of f(t(x)) based on the value of f(x) obtained from the actual execution

Chris Murphy – Columbia University5 Metamorphic Testing Example Consider a function std_dev(A[ ]) If we permute the elements in A, the result should stay the same If we multiply each element in A by 2, the result should be doubled These properties can be used to create a “pseudo-oracle” for the function

Chris Murphy – Columbia University6 Metamorphic Testing without an Oracle When a test oracle exists, we can know whether f(t(x)) is correct  Because we have an oracle for f(x)  So if f(t(x)) is as expected, then it is correct When there is no test oracle, f(x) acts as a “pseudo-oracle” for f(t(x))  If f(t(x)) is as expected, it is not necessarily correct  However, if f(t(x)) is not as expected, either f(x) or f(t(x)) (or both) is wrong

Chris Murphy – Columbia University7 Our Solution To address the problem of applications that have no test oracle, we use Metamorphic Testing at the function level To specify the metamorphic properties, we use extensions to the Java Modeling Language (JML) and a pre-processor called Corduroy To automate the process and address the need for initial input values, we check the properties at runtime

Chris Murphy – Columbia University8 Related Work Applying Metamorphic Testing to “non-testable programs” (applications without test oracles)  Chen IST’02, Chen ISSTA’02, Chan JWSR’07 Checking algebraic specifications at runtime  Sankar ISSTA’91, Nunes ICFEM’06 Checking program invariants at runtime  Flanagan FME’01, Hangal ICSE’02

Chris Murphy – Columbia University9 JML Basics Behavioral interface specification language that embraces the Design by Contract approach Developers can specify:  preconditions (“requires”)  postconditions (“ensures”)  arbitrary assertions Specifications can be checked dynamically using JML Runtime Assertion Checking tools

Chris Murphy – Columbia University10 Process Specify functions’ metamorphic properties using extension to JML Specify functions’ metamorphic properties using extension to JML Methods that check metamorphic properties Methods that check metamorphic properties Pure JML specifications that invoke test methods Pure JML specifications that invoke test methods Corduroy

Chris Murphy – Columbia University11 Metamorphic Properties in \result <= 1 && \result >= sine(x + 2 * PI) == -1 * sine(-x) == \result */ public double sine (double x) {... }

Chris Murphy – Columbia University12 Extension to JML: Array \result >= std_dev(\shuffle(A)) == std_dev(\multiply(A, 2)) == \result * 2 */ public double std_dev(double[] A) {... }

Chris Murphy – Columbia University13 Extensions to JML Array/Collection Functions  Shuffle/permute the order of the elements  Multiply each element by a constant  Add a constant to each element  Include a new element  Exclude an element Addressing non-determinism  Check in range of possible values  Check in a set of possible values

Chris Murphy – Columbia University14 Model Caller Function Caller Function Input(s) Callee Function Callee Function Output Check Metamorphic Properties Check Metamorphic Properties Output Input(s)

Chris Murphy – Columbia University15 average(\multiply(A, 2)) == \result * 2 */ public double average(double[] A) {... } protected boolean __metaTest_average (double[] A, double result) { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), result * 2); __metaTest_average(A, \result);

Chris Murphy – Columbia University16 Case Studies We investigated the WEKA and RapidMiner toolkits for Machine Learning in Java For WEKA, we tested four apps:  Naïve Bayes, Support Vector Machines (SVM), C4.5 Decision Tree, and k-Nearest Neighbors For RapidMiner, we tested one app:  Naïve Bayes

Chris Murphy – Columbia University17 Experimental Setup For each of the five apps, we specified 4-6 metamorphic properties of selected methods (based on our knowledge of the expected behavior of the overall application) Testing was conducted using data sets from UCI Machine Learning Repository Goal was to determine whether the properties held as expected

Chris Murphy – Columbia University18 Findings Discovered defects in WEKA k-NN and WEKA Naïve Bayes related to modifying the machine learning “model”  This was the result of a variable not being updated appropriately Discovered a defect in RapidMiner Naïve Bayes related to determining confidence  There was an error in the calculation

Chris Murphy – Columbia University19 Limitations Specifying and checking the properties was simplified by use of the tool, but still requires some knowledge of the app to determine the properties in the first place The approach works well for functions that take input and produce output, but in some case requires more complex properties than can be expressed in JML

Chris Murphy – Columbia University20 Future Work Reducing testing time by checking properties in parallel Implementations for other languages More empirical studies to determine effectiveness and applicability to other domains

Chris Murphy – Columbia University21 Summary We have presented a testing approach that uses metamorphic testing of individual functions in applications that do not have test oracles These are specified via an extension to the JML specification language We have also presented an implementation framework called Corduroy, which converts the specification of metamorphic properties into test methods

Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles Chris Murphy

Chris Murphy – Columbia University23 Categories of Metamorphic Properties Additive: Increase (or decrease) numerical values by a constant Multiplicative: Multiply numerical values by a constant Permutative: Randomly permute the order of elements in a set Invertive: Reverse the order of elements in a set Inclusive: Add a new element to a set Exclusive: Remove an element from a set ML apps such as ranking, classification, and anomaly detection exhibit these properties [Murphy SEKE’08]

Chris Murphy – Columbia University24 Specifying More Complex test_myFunc(x, \result) */ public double myFunc (int x) {... } private boolean test_myFunc (int x, double result) {... }

Chris Murphy – Columbia University25 Addressing Side Effects // holds result of last call to “average” private double average(\multiply(A, 2)) == value * 2 */ public void average(double[] A) {... value =... // no return value }

Chris Murphy – Columbia University26 Addressing Side Effects protected boolean __metaTest_average (double[] A) { double __value = value; // backup try { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), value * 2); } finally { value = __value; // restore } __value