Frank Xu, Ph.D. Gannon University Mining Decision Trees as Test Oracles for Java Bytecode Xu, W., Ding, T., Wang, H., Xu. D., Mining Test Oracles for Test.

Slides:



Advertisements
Similar presentations
Pat Langley Computational Learning Laboratory Center for the Study of Language and Information Stanford University, Stanford, California
Advertisements

Mining Specifications Glenn Ammons, Dept. Computer Science University of Wisconsin Rastislav Bodik, Computer Science Division University of California,
Introduction to White-box Testing
Frank Xu, Ph.D. Gannon University Automated Test-Input Generation Xu, W., Ding, T., Wang, H., Xu. D., Mining Test Oracles for Test Inputs Generated from.
Grey Box testing Tor Stålhane. What is Grey Box testing Grey Box testing is testing done with limited knowledge of the internal of the system. Grey Box.
A Regression Test Selection Technique for Aspect- Oriented Programs Guoqing Xu The Ohio State University
1 Program Slicing Purvi Patel. 2 Contents Introduction What is program slicing? Principle of dependences Variants of program slicing Slicing classifications.
Fall Semantics Juan Carlos Guzmán CS 3123 Programming Languages Concepts Southern Polytechnic State University.
1 Software Testing and Quality Assurance Lecture 9 - Software Testing Techniques.
A High Performance Application Representation for Reconfigurable Systems Wenrui GongGang WangRyan Kastner Department of Electrical and Computer Engineering.
PRESTO: Program Analyses and Software Tools Research Group, Ohio State University Regression Test Selection for AspectJ Software Guoqing Xu and Atanas.
Michael Ernst, page 1 Improving Test Suites via Operational Abstraction Michael Ernst MIT Lab for Computer Science Joint.
Parameterizing Random Test Data According to Equivalence Classes Chris Murphy, Gail Kaiser, Marta Arias Columbia University.
1 © 1998 HRL Laboratories, LLC. All Rights Reserved Development of Bayesian Diagnostic Models Using Troubleshooting Flow Diagrams K. Wojtek Przytula: HRL.
Winter Retreat Connecting the Dots: Using Runtime Paths for Macro Analysis Mike Chen, Emre Kıcıman, Anthony Accardi, Armando Fox, Eric Brewer
Copyright © 2006 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill Technology Education Copyright © 2006 by The McGraw-Hill Companies,
1 Software Testing and Quality Assurance Lecture 5 - Software Testing Techniques.
A Value-Based Approach for Quantifying Scientific Problem Solving Effectiveness Within and Across Educational Systems Ron Stevens, Ph.D. IMMEX Project.
Handouts Software Testing and Quality Assurance Theory and Practice Chapter 5 Data Flow Testing
State coverage: an empirical analysis based on a user study Dries Vanoverberghe, Emma Eyckmans, and Frank Piessens.
Automated Diagnosis of Software Configuration Errors
Comparing the Parallel Automatic Composition of Inductive Applications with Stacking Methods Hidenao Abe & Takahira Yamaguchi Shizuoka University, JAPAN.
Frank Xu, Ph.D. Gannon University Mining Decision Trees as Test Oracles for Java Bytecode Xu, W., Ding, T., Xu, D., Rule-based Test Input Generation From.
Presented By Dr. Shazzad Hosain Asst. Prof., EECS, NSU
1 A Static Analysis Approach for Automatically Generating Test Cases for Web Applications Presented by: Beverly Leung Fahim Rahman.
MBSat Satisfiability Program and Heuristics Brief Overview VLSI Testing B Marc Boulé April 2001 McGill University Electrical and Computer Engineering.
ISSTA 2002, Rome, Italy 1 Investigating the Use of Analysis Contracts to Support Fault Isolation in Object-Oriented Code Lionel Briand, Yvan Labiche, Hong.
Bug Localization with Machine Learning Techniques Wujie Zheng
Computer Science Department Data Structure & Algorithms Lecture 8 Recursion.
Grey Box testing Tor Stålhane. What is Grey Box testing Grey Box testing is testing done with limited knowledge of the internal of the system. Grey Box.
Summarizing the Content of Large Traces to Facilitate the Understanding of the Behaviour of a Software System Abdelwahab Hamou-Lhadj Timothy Lethbridge.
Yazd University, Electrical and Computer Engineering Department Course Title: Advanced Software Engineering By: Mohammad Ali Zare Chahooki 1 Machine Learning.
Summarizing “Structural” Testing Now that we have learned to create test cases through both: – a) Functional (blackbox)and – b) Structural (whitebox) testing.
Software Testing. Software testing is the execution of software with test data from the problem domain. Software testing is the execution of software.
CISC Machine Learning for Solving Systems Problems Presented by: Ashwani Rao Dept of Computer & Information Sciences University of Delaware Learning.
1 Test Selection for Result Inspection via Mining Predicate Rules Wujie Zheng
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University July 21, 2008WODA.
Xusheng Xiao North Carolina State University CSC 720 Project Presentation 1.
EXAMPLE 1 Standardized Test Practice SOLUTION Let ( x 1, y 1 ) = ( –3, 5) and ( x 2, y 2 ) = ( 4, – 1 ). = (4 – (–3)) 2 + (– 1 – 5) 2 = = 85 (
CASE/Re-factoring and program slicing
Java Basics Hussein Suleman March 2007 UCT Department of Computer Science Computer Science 1015F.
CISC Machine Learning for Solving Systems Problems Presented by: Suman Chander B Dept of Computer & Information Sciences University of Delaware Automatic.
Automatically detecting and describing high level actions within methods Presented by: Gayani Samaraweera.
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University 1 Extracting Sequence.
1 Test Coverage Coverage can be based on: –source code –object code –model –control flow graph –(extended) finite state machines –data flow graph –requirements.
Maikel Leemans Wil M.P. van der Aalst. Process Mining in Software Systems 2 System under Study (SUS) Functional perspective Focus: User requests Functional.
A PRELIMINARY EMPIRICAL ASSESSMENT OF SIMILARITY FOR COMBINATORIAL INTERACTION TESTING OF SOFTWARE PRODUCT LINES Stefan Fischer Roberto E. Lopez-Herrejon.
SOFTWARE TESTING AND QUALITY ASSURANCE. Software Testing.
CS223: Software Engineering Lecture 26: Software Testing.
Test Case Purification for Improving Fault Localization presented by Taehoon Kwak SoftWare Testing & Verification Group Jifeng Xuan, Martin Monperrus [FSE’14]
Behavior Recognition Based on Machine Learning Algorithms for a Wireless Canine Machine Interface Students: Avichay Ben Naim Lucie Levy 14 May, 2014 Ort.
Introduction to Software Testing (2nd edition) Chapter 5 Criteria-Based Test Design Paul Ammann & Jeff Offutt
Software Testing and QA Theory and Practice (Chapter 5: Data Flow Testing) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory and Practice.
Testing and Debugging UCT Department of Computer Science Computer Science 1015F Hussein Suleman March 2009.
10/23/ /23/2017 Presented at KDD’09 Classification of Software Behaviors for Failure Detection: A Discriminative Pattern Mining Approach David Lo1,
A Generic Approach to Big Data Alarms Prioritization
A Review of Software Testing - P. David Coward
Software Testing.
Handouts Software Testing and Quality Assurance Theory and Practice Chapter 4 Control Flow Testing
runtime verification Brief Overview Grigore Rosu
Eclat: Automatic Generation and Classification of Test Inputs
Chapter 9 Path Testing–Part 1
CSCE 747 Software Testing and Quality Assurance
MSIS 655 Advanced Business Applications Programming
Chapter 1 Introduction(1.1)
Mutation Testing The Mutants are Coming! Copyright © 2017 – Curt Hill.
Overview Functional Testing Boundary Value Testing (BVT)
Software Testing and QA Theory and Practice (Chapter 5: Data Flow Testing) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory and Practice.
Presentation transcript:

Frank Xu, Ph.D. Gannon University Mining Decision Trees as Test Oracles for Java Bytecode Xu, W., Ding, T., Wang, H., Xu. D., Mining Test Oracles for Test Inputs Generated from Java Bytecode, Proc. of the 37th Annual International Computer Software & Applications Conference, pp , Kyoto, Japan, July 2013 Mining Decision Trees as Test Oracles for Java Bytecode (Extended version of conference paper), Accepted by Journal of Systems and Software

Bio – Frank Xu Education Ph.D. in Software Engineering North Dakota State University M.S. in Computer Science Towson University B.S. in Computer ScienceSoutheast Missouri State University Working Experience GE Transportation, present, Consultant of Locomotive Remote Diagnostics Service Center Gannon University, present, Assistant Professor of Software Engineering, Director of Keystone Software Development Institute, University VA –Wise, , Assistant Professor of Software Engineering Swanson Health Products, 2005 ~ 2007, Sr. Programmer Analyst Volt Information Science Inc., 2004 ~ 2005, Software Engineer

Teaching Source: Student Evaluation Report

Research Source: Google scholar:

Mining Decision Trees as Test Oracles I NTRODUCTION R UNNING E XAMPLE T EST I NPUT G ENERATION M ODEL M INER E MPIRICAL S TUDY R ELATED W ORK C ONCLUSIONS

I NTRODUCTION

Exercise Implementing a method to solve Triangle problem

What is Triangle Problem?

How to test Triangle? String getTriangleType (int a, int b, int c){ if((a<b+c) && (b<a+c) && (c<a+b)){ if (a==b && b==c) return “Equilateral ”; else if (a!=b && a!=c &&b!=c) return “Scalene ”; else return “Isosceles” ; } else return “NotATriangle “; } String getTriangleType (int a, int b, int c){ if((a<b+c) && (b<a+c) && (c<a+b)){ if (a==b && b==c) return “Equilateral ”; else if (a!=b && a!=c &&b!=c) return “Scalene ”; else return “Isosceles” ; } else return “NotATriangle “; }

Control Flow Diagram

S UMMARY : T EST T RIANGLE S TEPS Source Code Control Flow Diagram Paths (based on coverage) assertEquals(“Isosceles ”, triangle.getTriangleType(7,7,7)) Junit Test cases assertEquals(“Isosceles ”, triangle.getTriangleType(6,6,8)) ….. Step 1 Step 2 Step 3

How to generate testing inputs automatically? E.g.,(7,7,7), (6,6,8)…. How to find expected results automatically for each inputs? Known as test oracle issue E.g., Equilateral, Isosceles... A UTO -G ENERATE T EST C ASES IS C HALLENGING assertEquals (“Equilateral”, triangle.getTriType(7,7,7)) assertEquals (“Isosceles ”, triangle.getTriType(6,6,8)) ….. ?

O UR A PPROACH TO S OLVE C HALLENGES Rule-based search method to generate inputs Seed value adjust seed values based on rules (5,7,8) for Isosceles Adjust input values a==b (7,7,8) (5,5,8) Using heuristic model for test oracle (expected results ) A new data mining approach to building a heuristic behavioral model (in the form of decision tree) A heuristic behavioral model represents the estimated expected results

Test Oracle Overview

R EVISIT : T RIANGLE P ROBLEM

Java is Complex Statement contains comparison and expression a <b+c (Java) Condition (a<b+c) && (b<a+c) && (c<a+b)

J AVA S IMPLER V ERSION Simplify Statement a <b+c (Java) [1] $i3=i1+i2 and [2] i0>=$i3 (Jimple) Simplify condition (a<b+c) && (b<a+c) && (c<a+b) (Java) Jimple if (a<b+c) { if (b<a+c) { if(c<a+b) … }}}

IDPath 1 2 3,,,..,,, IDabc ………… … Generate inputs Mine test oracle (7,7,7) Equilateral Path generation Generate CFG

H OW TO G ENERATE T EST I NPUTS IDabc … ………… … IDPath 1 2 3,,,

Search an input that make predicate [5]:i0>=$i3 to true a>=b+c (NotATriangle) Challenge: backtracking $i3 to input variables Recall $i3=i1+i2 Solution: Predicate Tree Recall Property 1 a>=b+c

Apply Rules to a Predicate Tree for Generating Test Inputs For a given seed value, we adjust the value to guide the execution path based on rules

IDPredicateExpected Evaluation Outcomes Advising Rules 1. i0 > i1(i0 > i1) = true(i0 ↑, i1) (i0, i1 ↓ ) 1. (i0 > i1) = false(i0 ↓,b) (i0, i1 ↑ ) 1. i0 == i1(i0==i1) = true(i0 ↓ D, i1)( i0, i1 ↑ D) 1. (i0== i1)= false(i0 ↑,i1) (i0, i1 ↑ ) (i0 ↓,i1) (i0, i1 ↓ ) 1. i2 = i0 + i1i2 ↑ (i0 ↑, i1) (i0, i1 ↑ ) 1. i2 ↓ (i0 ↓, i1) (i0, i1 ↓ ) 1. i2 = i0 - i1i2 ↑ (i0 ↑, i1) (i0, i1 ↓ ) 1. i2 ↓ (i0 ↓, i1)( i0, i1 ↑ ) 1. i2 = i0 * i1 (i0>0, i1>0) i2 ↑ (i0 ↑, i1) (i0, i1 ↑ ) 1. i2 ↓ (i0 ↓, i1) (i0, i1 ↓ ) 1. i2 = i0 / i1 (i0>0, i0 > 0) i2 ↑ (i0 ↑, i1) (i0, i1 ↓ ) 1. i2 ↓ (i0- ↓ i1)( i0, i1 ↑ ).. 1. s0>s1(s0 >s1) = true(s0[k] ↑, s1) (s0, s1[l] ↓ ) 1. (s0 > s1) = false(s0[k] ↓,s2) (s0, s1[l] ↑ )

M ODEL M INER IDabc … ………… …

Jimple Predicates and Attributes of Triangle Program Jimple Predicate Attribute of UUT i0 > = $i3a > = b + c i1 > = $i4b > = a + c i2 > = $i5c > = a + b i0 != i1a != b i1 != i2b != c i0 == i1a = b i0 == i2a = c i1 == i2b = c For a given test input generated by rule-based method, predicates produce a set of T or F values Input (a=7.b=7,v=7) f f f f f t t t

Covert Test Inputs Using Attributes Test input ID abc a1 a2a3a4a5a6a7a8 O fffffttt ftfttfff fftttfff tffttfff4 … fffttftf fffttfft ffftffff tffttfff4

C4.5 mining algorithm The key idea of the algorithm is to calculate the highest normalized information gain of attributes and then build a decision node that splits on the attributes Tool Weka 3: ID abc a1a2a3a4a5a6a7a8o fffffttt ftfttfff fftttfff tffttfff4 … fffttftf fffttfft ffftffff tffttfff4

E MPIRICAL S TUDY

T HREE S TUDY S UBJECTS Line of CodeNumber of Predicates JavaJimpleJava Jimple (Allow duplications) Attributes (No duplication) Triangle Next Date Vending Machine

G OAL OF E MPIRICAL S TUDIES Measure fault detection capability # mutants killed /#mutants *100%

Measure fault detection capability: Process Step 1: Implant mutants Step 2: Build a decision tree model Step 3: Find mismatches Find possible causes Step 4: Calculate fault detectability Mutation OperatorExamples CategoryIDTypeOriginalReplaced Arithmetic Operations 1Arithmetic Operator Replacement a + ba - b 2Arithmetic Operator Insertion b + c-b + c Relations3Relational Operator Replacement a != ba == b Conditions4Conditional Operator Replacement (a==b) && (b==c) (a==b) || (b==c) Constants5Constant Value Modifications = as = b Return Values 6Return Value Modificationreturn sreturn s’ Insert bug Faulty version Find mismatches Two possible causes -Found bugs -assertEquals(“Equilateral”, new Trianlge(7,7,7).getTriType()) -Model is not correct -assertEquals (“Isosceles”, new Trianlge(7,7,7).getTriType()) Two possible causes -Found bugs -assertEquals(“Equilateral”, new Trianlge(7,7,7).getTriType()) -Model is not correct -assertEquals (“Isosceles”, new Trianlge(7,7,7).getTriType())

ID # of Mutants # of Tests Executed Oracle Results # Mutants Discovered# Faults in Models SDUSDUSDU Triangle Problem Next Date Problem Vending Machine Total

R ELATED W ORK Lo et al. (Lo, Cheng, Han, Khoo, & Sun, 2009), Milea et al. (Milea, Khoo, Lo, & Pop, 2012) mines a set of discriminative features capturing repetitive series of events from program execution traces. These features are then used to train a classier to detect failures. Bowring et al. (Bowring, Rehg, & Harrold, 2004) models program executions as Markov models, and a clustering method for Markov models that aggregates multiple program executions into effective behavior classifiers. (Pacheco & Ernst, 2005) Pacheco and Ernst build an operational model from observations of the software running properly. The operation model includes object invariants and properties. The object invariants are the conditions hold on entry and exit of all public methods. Our approach generates and classifies inputs based on the internal structure of the UUT. Briand (Briand, 2008) has proposed the use of machine learning techniques - including decision trees - for the test oracle problem. The decision tree model he has proposed is manually built from software requirements.

C ONCLUSIONS The first attempt to mine decision tree models from auto-generated test inputs based on static analysis of Java bytecode Our empirical study indicates that using the mined test oracles, average 94.67% mutants are killed by the generated test inputs.

Thanks

Future Research Direction Requirements Engineering & Natural language Process Generating UML diagrams, e.g., Use case, Class diagram Validating SRS Deriving test cases from SRS Software Design & Social Networks Analysis Utilizing SSA for analyzing communication diagram, class diagram, and sequence diagram for improving the quality of the software Software Implementation & Big Data Mining repository for software quality assurance using Hadoop Software Testing & Mobile/Cloud Application Testing mobile applications and distributed applications

Build Variable Dependency Tree (VDT)