1. Detecting Assumptions on Deterministic Implementations of Non-deterministic Specifications August Shi, Alex Gyori, Owolabi Legunsen, Darko Marinov 4/12/2016 ICST 2016 Chicago, Illinois CCF-1012759, CCF-1409423, CCF-1421503, CCF-1439957

2. Example Code and Test 2 public class Book { String author; String title; public Book(String author, String title) { this.author = author; this.title = title; } public String getStringRep() { JSONObject j = new JSONObject(); // JSONObject extends java.util.HashMap j.put("author", this.author); j.put("title", this.title); return j.toString(); // toString() iterates through entries } public class BookTest { @Test public void testGetStringRep() { Book b = new Book("A", "T"); assertEquals("{\"author\":\"A\",\"title\":\"T\"}", b.getStringRep()); } /** … This class makes no guarantees as to the order of the map … */ public class java.util.HashMap …

3. Non-deterministic Specifications A specification (spec) that allows multiple implementations with different outputs for a given input Good: Allow freedom of implementation Although specs are non-deterministic, underlying implementations are often deterministic Bad: Code that Assumes a Deterministic Implementation of a Non- deterministic Specification (ADINS) Such code can behave unexpectedly when run using a different underlying implementation that still satisfies the spec Such code can be a cause of flaky tests 3

4. Library NonDex (Non-Deterministic Explorer) A simple technique for detecting ADINS code 4 Test Method with Deterministic Spec calls outputs Method with Non-deterministic Spec calls outputs …

5. Library NonDex (Non-Deterministic Explorer) A simple technique for detecting ADINS code 5 NonDex Model calls … Method with Deterministic Spec calls Implementation 1 Implementation 2 Implementation n … outputs NonDex outputs Test

6. Library NonDex (Non-Deterministic Explorer) A simple technique for detecting ADINS code 6 … Method with Deterministic Spec Explore: Implementation 1 … Implementation n Test NonDex Model Implementation 1 Implementation 2 Implementation n … NonDex outputs calls outputs

7. Finding Non-deterministic Specs Searched Java Standard Library for method specs that are non-deterministic 1.Searched JavaDocs for these keywords: “order”, “deterministic”, “not specified” 2.Searched for methods that return an array Examined JavaDocs for all found methods Identified 31 method specs that are non-deterministic 7

8. Some Non-deterministic Specs java.util.Set.iterator(): “Returns an iterator over the elements in this set. The elements are returned in no particular order…” java.util.HashMap.entrySet(): “Returns a Set view of the mappings contained in this map…” “This class makes no guarantees as to the order of the map; in particular, it does not guarantee that the order will remain constant over time.” java.lang.Class.getDeclaredMethods(): “Returns an array containing Method objects reflecting all the declared methods of the class … The elements in the returned array are not sorted and are not in any particular order.” 8

9. 9 ClassMethodsCategory java.lang.ObjecthashCodeRandom java.util.HashMap*keySet, values, entrySetPermute java.util.ConcurrentHashMap*keySet, values, entrySet, keys, elementsPermute java.io.Filelist, listFiles, listRootsPermute java.lang.ClassgetClasses, getDeclaredMethods, …Permute java.lang.reflect.MethodgetParameterAnnotationsPermute java.lang.reflect.FieldgetDeclaredAnnotationsPermute java.text.DateFormatSymbolsgetAvailableLocales, getZoneStrings Permute, Extend java.text.BreakIteratorgetAvailableLocalesPermute java.text.DecimalFormatSymbolsgetAvailableLocalesPermute java.text.NumberFormatgetAvailableLocalesPermute java.text.DateFormatgetAvailableLocalesPermute * Non-determinism is actually in internal iterator, exposed to outside by these methods NonDex Models for Specs Permute: method returns array/collection whose order is not specified NonDex model explores different permutations of elements

10. Different Levels of Non-determinism Some assumptions of determinism may be “acceptable” For example, if two calls of iterator() are made on the same, unmodified Set, should the iteration orders be the same? We introduce four “levels” for NonDex to “Permute” ONE: assumes deterministic implementation, but shuffles once, potentially different than underlying implementation EQ: shuffles differently only when objects are not “equal” ID: shuffles differently only when object address is different, or object has been modified FULL: shuffles differently with every call 10

11. 01. Set s = new HashSet (); 02. s.add(1); s.add(2); 03. Integer[] a = s.toArray(); 04. // assertArrayEquals(a, new Integer[]{1, 2}); // FULL, ID, EQ, ONE can fail 05. 06. // assertArrayEquals(a, s.toArray()); // FULL can fail 07. 08. s.contains(1); 09. // assertArrayEquals(a, s.toArray()); // FULL can fail 10. 11. s.add(3); s.remove(3); 12. // assertArrayEquals(a, s.toArray()); // FULL, ID can fail 13. 14. Set t = new HashSet (); 15. t.add(1); t.add(2); 16. // assertArrayEquals(a, t.toArray()); // FULL, ID can fail 17. 18. Set u = new HashSet (); 19. u.add(3); u.add(4); 20. Integer[] b = u.toArray(); 21. // assertEquals(a[0] < a[1], b[0] < b[1]); // FULL, ID, EQ can fail 11

12. NonDex Model Exploration In our prototype, NonDex models use seeds to control which implementations to use Models use seeds for different java.util.Random calls that represent the different implementations We execute code with NonDex multiple times, each using different seeds, to explore different behaviors 12

13. Research Questions How many flaky tests are due to ADINS code? In open-source projects? In student submissions? How many seeds should be used to likely find all flaky tests due to ADINS code? How often are developers making “acceptable” assumptions about non-determinism? 13

14. Experimental Setup Tool (also called NonDex) is implemented for OpenJDK 8 Models implemented by modifying code in OpenJDK 8 Use modified NonDex JVM in place of the OpenJDK 8 JVM Evaluation projects: 195 open-source GitHub projects that build with Maven First run each project’s tests with 10 different seeds If a test with different behavior is found, run project’s tests with 100 different seeds 72 student submissions from Software Engineering I course Run each student submission’s tests with 100 different seeds 14

15. Open-source Project Results 15 FULLIDEQONE Flaky Tests Detected6054 Total Seeds Detecting4362/60003744/60003643/60003590/6000 Min Seeds Detecting8000 Max Seeds Detecting100

16. 16 ProjectTestClass#TestNameFULLIDEQONE reflectasmFieldAccessTest#testIndexSetAndGet48000 joda-timeTestDateTimeZone#testGetShortName3553 oryxTextUtilsTest#testJSONMap51526053 visualeeJPAExaminerTest#testFindAndSetAttributes…8556 commons-cliOptionGroupTest#testToString42000 commons-langMultilineRecursiveToStringStyleTest#boolArray100 easy-batchPrinterTest#testPrettyPrinting69735453 scribe-javaMapUtilsTest#shouldPrettyPrintMap979497 geoserver-managerGSLLayerEncoder21Test#testMetadata848171100 handlebars.javaTagTypeTest#collectSectionAndVars100 jscepDefaultCertStoreInspectorTest#example92945953 junitMethodSorterTest#testJvmMethodSorter100000 org-jsonTestSuite#testJSONStringerObject79778384 slf4jEventLoggerTest#testEventLogger100000 wsdoc CollectionTests#testBasicSets100969184 Subset of Flaky Tests Detected

17. Example Flaky Test (from commons-cli) 17 public class OptionGroupTest { public void testToString() { OptionGroup g1 = new OptionGroup(); g1.addOption(new Option(null, "foo", false, "Foo")); g1.addOption(new Option(null, "bar", false, "Bar")); if (!"[--bar Bar, --foo Foo]".equals(g1.toString())) { assertEquals("[--foo Foo, --bar Bar]", g1.toString()); }...}} public class OptionGroup... { Map om = new HashMap(); public OptionGroup addOption(Option option) { om.put(option.getKey(), option); return this;} public String toString() { StringBuilder buff = new StringBuilder(); Iterator iter = getOptions().iterator(); buff.append("["); while (iter.hasNext()) { /*... populate buff with the values in iter... */ } return buff.toString();}}

18. Student Submission Results Flaky tests mostly due to asserting result of method call is exactly equal to some String (similar beginning example) We also used Java PathFinder (JPF) to do systematic exploration Replace java.util.Random calls with JPF choice points Using JPF choice points did not detect any more flaky tests than using java.util.Random (with 100 seeds) 18 FULLIDEQONE Flaky Tests Found1108834 Total Seeds Detected8159/110006785/110002031/110001827/11000 Min Seeds Detected37000 Max Seeds Detected100 8178

19. Conclusion Non-deterministic specs give freedom to implementers, but code that assumes deterministic implementations (ADINS) leads to undesirable behavior We develop a technique NonDex to detect ADINS code We detected 60 flaky tests in open-source projects, and 110 flaky tests in student submissions We plan to further research how to better detect, debug, and repair ADINS code We plan on releasing NonDex, but are currently having issues with OpenJDK/Oracle licensing 19 August Shi: awshi2@illinois.edu

20. BACKUP 20

21. 21 class HashMap {... class HashIterator {... /* Rename hasNext() -> original_hasNext() */ /* Rename nextNode() -> original_nextNode() */ Iterator NonDex_iter; HashIterator() {... List original = new ArrayList<>(); while (original_hasNext()) original.add(original_nextNode()); NonDex.shuffle(original, (NonDex.level == ID) ? System.identityHashCode(HashMap.this) + modCount : (NonDex.level == EQ) ? HashMap.this.hashCode() : 0); NonDex_iter = original.iterator(); } public final boolean hasNext() { return NonDex_iter.hasNext(); } final Node nextNode() { if (modCount != expectedModCount) throw new ConcurrentModificationException(); current = NonDex_iter.next(); return current; }

22. 22 class NonDex { static int level; // FULL, ID, EQ, or ONE int seed =...; static Random full = new Random(seed); public static List shuffle(List l, int v) { int size = l.size(); Random rand = (level == FULL) ? full : // Full (level == ID) ? new Random(seed + v) : // Same object (level == EQ) ? new Random(seed + v) : // Equal object (level == ONE) ? new Random(seed); // Once for (int i = 0; i < size - 1; i++) { int s = rand.getNext(i, size); if (s == i) continue; T obj = l.get(i); l.set(i, l.get(s)); l.set(s, obj); } return l; }

23. Example Flaky Test (from scribe- java) 23 public class MapUtilsTest { @Test public void shouldPrettyPrintMap() { Map map = new HashMap<>(); map.put(1, "one"); map.put(2, "two"); map.put(3, "three"); map.put(4, "four"); assertEquals( "{ 1 -> one, 2 -> two, 3 -> three, 4 -> four }", MapUtils.toString(map)); } public class MapUtils { public static String toString(Map map) {... StringBuilder result = new StringBuilder(); for (Map.Entry entry : map.entrySet()) { result.append(String.format(", %s -> %s ", entry.getKey().toString(), entry.getValue().toString())); } return "{" + result.substring(1) + "}"; }