1. Frank Xu, Ph.D. Gannon University Mining Decision Trees as Test Oracles for Java Bytecode Xu, W., Ding, T., Xu, D., Rule-based Test Input Generation From Bytecode, Proc. of the 8th International Conference on Software Security and Reliability, pp. 108-117, San Francisco, CA, June 2014. Xu, W., Ding, T., Wang, H., Xu. D., Mining Test Oracles for Test Inputs Generated from Java Bytecode, Proc. of the 37th Annual International Computer Software & Applications Conference, pp. 27- 32, Kyoto, Japan, July 2013

2. Overview Introduction Test Oracles, Decision Trees How the research related to cyber security Objective Mining Decision Trees as Test Oracles Approach Test Input Generator Model Miner Empirical Study and Conclusions Related Work

3. I NTRODUCTION

4. How to test it? String getTriangleType (int a, int b, int c){ if((a<b+c) && (b<a+c) && (c<a+b)){ if (a==b && b==c) return “Equilateral ”; else if (a!=b && a!=c &&b!=c) return “Scalene ”; else return “Isosceles” ; } else return “NotATriangle “; } String getTriangleType (int a, int b, int c){ if((a<b+c) && (b<a+c) && (c<a+b)){ if (a==b && b==c) return “Equilateral ”; else if (a!=b && a!=c &&b!=c) return “Scalene ”; else return “Isosceles” ; } else return “NotATriangle “; }

5. S TEPS FOR W HITE - BOX T ESTING Source Code Control Flow Diagram Paths (based on coverage) assertEquals(“equalateral”, triangle.getTriangleType(7,7,7)) Junit Test cases assertEquals(“Isosceles ”, triangle.getTriangleType(6,6,8)) ….. Step 1 Step 2 Step 3

6. A UTO -G ENERATE T EST C ASES IS C HALLENGING assertEquals (“Equilateral”, triangle.getTriType(7,7,7)) assertEquals (“Isosceles ”, triangle.getTriType(6,6,8)) ….. ? Challenging 1:inputs generation Challenging 2: Oracle Problem

7. Related to Cyber Security Include Software security Network security Data security Common approach for software security Testing for security E.g., testing for web security Test cases are needed for all testing Test oracle is required

8. O UR A PPROACH Rule-based search method to generate inputs Seed value adjust seed values based on rules (5,7,8) for Isosceles Adjust input values a==b (7,7,8) (5,5,8) Using heuristic model for test oracle A new data mining approach to build a heuristic behavioral model (in the form of decision tree)

9. R EVISIT : T RIANGLE P ROBLEM

10. Java is Complex Statement contains comparison and expression a <b+c (Java) Condition (a<b+c) && (b<a+c) && (c<a+b)

11. J IMPLE Java simpler version Intermediate representation of bytecode Simplify Statement a <b+c (Java) [1] $d=b+c and [2] a <$d (Jimple) Simplify condition (a<b+c) && (b<a+c) && (c<a+b) (Java) Jimple if (a<b+c) { if (b<a+c) { if(c<a+b) … }}} www.sable.mcgill.ca/soot/

12. IDPath 1 2 3,,,..,,, 8 9 10 11 12 IDabc 1777 21173 38 19 41173 522 9 ………… … 1333 45 14523052 153147 16272822 Input Generator Model miner (7,7,7) Equilateral Path generator CFG Generator Input Generator Training data Validation data

13. I NPUT G ENERATOR

14. Search an input that make predicate [5]:i0>=$i3 to true a>=b+c (NotATriangle)

15. Apply Rules to a Predicate Tree for Generating Test Inputs 10 4 7

16. IDPredicateExpected Evaluation Outcomes Advising Rules 1. i0 > i1(i0 > i1) = true(i0 ↑, i1) (i0, i1 ↓ ) 1. (i0 > i1) = false(i0 ↓,b) (i0, i1 ↑ ) 1. i0 == i1(i0==i1) = true(i0 ↓ D, i1)( i0, i1 ↑ D) 1. (i0== i1)= false(i0 ↑,i1) (i0, i1 ↑ ) (i0 ↓,i1) (i0, i1 ↓ ) 1. i2 = i0 + i1i2 ↑ (i0 ↑, i1) (i0, i1 ↑ ) 1. i2 ↓ (i0 ↓, i1) (i0, i1 ↓ ) 1. i2 = i0 - i1i2 ↑ (i0 ↑, i1) (i0, i1 ↓ ) 1. i2 ↓ (i0 ↓, i1)( i0, i1 ↑ ) 1. i2 = i0 * i1 (i0>0, i1>0) i2 ↑ (i0 ↑, i1) (i0, i1 ↑ ) 1. i2 ↓ (i0 ↓, i1) (i0, i1 ↓ ) 1. i2 = i0 / i1 (i0>0, i0 > 0) i2 ↑ (i0 ↑, i1) (i0, i1 ↓ ) 1. i2 ↓ (i0- ↓ i1)( i0, i1 ↑ ).. 1. s0>s1(s0 >s1) = true(s0[k] ↑, s1) (s0, s1[l] ↓ ) 1. (s0 > s1) = false(s0[k] ↓,s2) (s0, s1[l] ↑ )

17. M ODEL M INER

18. Jimple Predicates and Attributes of Triangle Program Predicates JimpleSource Code i0 > = $i3a > = b + c i1 > = $i4b > = a + c i2 > = $i5c > = a + b i0 != i1a != b i1 != i2b != c i0 == i1a == b i0 == i2a == c i1 == i2b == c For a given input, predicates produce a set of T or F values Input (a=7.b=7,v=7) f f f f f t t t

19. Covert Inputs Using Attributes Test input ID abc a>=b+c b>=a+cc>=a+ba!=bb!=ca==ba==cb==c Output 1 777 fffffttt1 2 1173 ftfttfff4 3 8 19 fftttfff4 4 1173 tffttfff4 … 13 33 45 fffttftf2 14 523052 fffttfft2 15 3147 ffftffff3 16 272822 tffttfff4

20. C4.5 mining algorithm The key idea of the algorithm is to calculate the highest normalized information gain of attributes and then build a decision node that splits on the attributes Tool Weka 3: http://www.cs.waikato.ac.nz/ml/weka/ Test input ID abc a>=b+c b>=a+cc>=a+ba!=bb!=ca==ba==cb==c Output 1 777 fffffttt1 2 1173 ftfttfff4 3 8 1919 fftttfff4 4 73 tffttfff4 … 13 33 4545 fffttftf2 14 52305252 fffttfft2 15 31474747 ffftffff3 16 27282 tffttfff4

21. E MPIRICAL S TUDY

22. T HREE S TUDY S UBJECTS Unit Under Testing Line of CodeNumber of Predicates JavaJimpleJava Jimple (Allow duplications) Attributes (No duplication) Triangle2227388 Next Date4851919 Vending Machine8268214110

23. G OAL OF E MPIRICAL S TUDIES Measure fault detection capability # mutants killed /#mutants *100%

24. Measure fault detection capability: Process Step 1: Implant mutants Step 2: Build a decision tree model Step 3: Find mismatches Find possible causes Step 4: Calculate fault detectability Mutation OperatorExamples CategoryIDTypeOriginalReplaced Arithmetic Operations 1Arithmetic Operator Replacement a + ba - b 2Arithmetic Operator Insertion b + c-b + c Relations3Relational Operator Replacement a != ba == b Conditions4Conditional Operator Replacement (a==b) && (b==c) (a==b) || (b==c) Constants5Constant Value Modifications = as = b Return Values 6Return Value Modificationreturn sreturn s’ Insert bug Faulty version Find mismatches Two possible causes -Found bugs -assertEquals(“Equilateral”, new Trianlge(7,7,7).getTriType()) -Model is not correct -assertEquals (“Isosceles”, new Trianlge(7,7,7).getTriType()) Two possible causes -Found bugs -assertEquals(“Equilateral”, new Trianlge(7,7,7).getTriType()) -Model is not correct -assertEquals (“Isosceles”, new Trianlge(7,7,7).getTriType())

25. Mutation ID # of Mutants # of Tests Executed Oracle Results # Mutants DiscoveredFaults in Models SDUSDUSDU Triangle51123483 NextDate549495109151618320 Vending45465450 Total1508861428 R ESULTS OF FAULT DETECTION EVALUATION statement coverage (S), decision coverage (D), and un-reduced decision coverage (U),

26. C ONCLUSIONS The first attempt to mine decision tree models from auto-generated test inputs based on static analysis of Java bytecode Our empirical study indicates that using the mined test oracles, average 94.67% mutants are killed by the generated test inputs. Future plan Empirical study

27. Thanks

28. R ELATED W ORK Lo et al. (Lo, Cheng, Han, Khoo, & Sun, 2009), Milea et al. (Milea, Khoo, Lo, & Pop, 2012) mines a set of discriminative features capturing repetitive series of events from program execution traces. These features are then used to train a classier to detect failures. Bowring et al. (Bowring, Rehg, & Harrold, 2004) models program executions as Markov models, and a clustering method for Markov models that aggregates multiple program executions into effective behavior classifiers. (Pacheco & Ernst, 2005) Pacheco and Ernst build an operational model from observations of the software running properly. The operation model includes object invariants and properties. The object invariants are the conditions hold on entry and exit of all public methods. Our approach generates and classifies inputs based on the internal structure of the UUT. Briand (Briand, 2008) has proposed the use of machine learning techniques - including decision trees - for the test oracle problem. The decision tree model he has proposed is manually built from software requirements.

29. Future Research Direction Requirements Engineering & Natural language Process Generating UML diagrams, e.g., Use case, Class diagram Validating SRS Deriving test cases from SRS Software Design & Social Networks Analysis Utilizing SSA for analyzing communication diagram, class diagram, and sequence diagram for improving the quality of the software Software Implementation & Big Data Mining repository for software quality assurance using Hadoop Software Testing & Mobile/Cloud Application Testing mobile applications and distributed applications

30. Test Oracle Overview

31. Control Flow Diagram

32. Build Variable Dependency Tree (VDT)

33. Bio – Frank Xu Education Ph.D. in Software Engineering North Dakota State University M.S. in Computer Science Towson University B.S. in Computer ScienceSoutheast Missouri State University Working Experience GE Transportation, 2008- present, Consultant of Locomotive Remote Diagnostics Service Center Gannon University, 2008- Present, Assistant/Associate Professor of Software Engineering, Director of Keystone Software Development Institute University VA –Wise, 2007- 2008, Assistant Professor of Software Engineering Swanson Health Products, 2005 ~ 2007, Sr. Programmer Analyst Volt Information Science Inc., 2004 ~ 2005, Software Engineer

34. Teaching Software engineer in UML Requirement engineering Software Design Software Quality Assurance Software Maintenance Object-Oriented Modeling Personal Software Process Advance Programming in Java Mobile Application Programming Multi-Tiered Systems Dynamic Web Programming Operating System Discrete Math

35. Teaching Evaluation Source: Student Evaluation Report

36. Research

37. Publications and Citations Year200520062007200820092010201120122013 2014 Total Conference322114442 1 24 Journal 12112 1 8 Total312235562 2 32 Source: Google scholar: http://scholar.google.com/citations?user=9_I4ZUgAAAAJ&hl=en