1. TECHNOLOGY DIVISION 1 GS Collections and Java 8 Functional, Fluent, Friendly & Fun! GS.com/Engineering Fall, 2014 Donald Raab Craig Motlin

2. TECHNOLOGY DIVISION 2 Agenda Introductions Lost and Found Streams The Iceberg –APIs –Fluency –Memory Efficiency –Method references are awesome Framework Comparisons

3. TECHNOLOGY DIVISION 3 What is GS Collections? Open source Java collections framework developed in Goldman Sachs –In development since 2004 –Hosted on GitHub w/ Apache 2.0 License github.com/goldmansachs/gs-collections GS Collections Kata –Internal training developed in 2007 –Taught to > 1,500 GS Java developers –Hosted on GitHub w/ Apache 2.0 License github.com/goldmansachs/gs-collections-kata

4. TECHNOLOGY DIVISION 4 1997 - Smalltalk Best Practice Patterns (Kent Beck) do: select: reject: collect: detect: detect:ifNone: inject:into: … (Dr. Seuss API) 2007 - Implementation Patterns (Kent Beck) Map List Set The collection iteration patterns disappeared All that remained were the types Paradise Lost

5. TECHNOLOGY DIVISION 5 Paradise Found Pattern in Smalltalk-80Pattern in Classic JavaPattern in GS Collections w/ Lambdas List result = new ArrayList<>(); for (Integer each : list) if (each > 50) result.add(each); select List result = new ArrayList<>(); for (Integer each : list) if (each <= 50) result.add(v); reject for (Integer each : list) if (each > 50) return true; return false; any satisfy for (Integer each : list) if (each <= 50) return false; return true; detect for (Integer each : list) if (each > 50) return each; return null; List result = new ArrayList<>(); for (Integer each : list) result.add(each.toString()); collect int result = 3; for (Integer each : list) result = result + each; all satisfy inject into list.detect(each -> each > 50); list.select(each -> each > 50); list.reject(each -> each > 50); list.anySatisfy(each -> each > 50); list.allSatisfy(each -> each > 50); list.collect(Object::toString); list.injectInto(3, Integer::sum); list detect: [:each | each > 50]. list select: [:each | each > 50]. list reject: [:each | each > 50]. list anySatisfy: [:each | each > 50]. list allSatisfy: [:each | each > 50]. list collect: [:e | e printString]. list inject: 3 into: [:x :y | x + y].

6. TECHNOLOGY DIVISION 6 Lazy by any other name select reject any satisfy detect collect all satisfy inject into GS Collections LazyIterable Integer result = list.asLazy().detectIfNone(e -> e > 50, () -> null); LazyIterable result = list.asLazy().select(e -> e > 50); LazyIterable result = list.asLazy().reject(e -> e > 50); boolean any = list.asLazy().anySatisfy(e -> e > 50); boolean all = list.asLazy().allSatisfy(e -> e > 50); LazyIterable result = list.asLazy().collect(Object::toString); Integer result = list.asLazy().injectInto(3, Integer::sum); filter any Match findAny map all Match reduce Java 8 Streams Integer result = list.stream().filter(e -> e > 50).findFirst().orElse(null); Stream result = list.stream().filter(e -> e > 50); Stream result = list.stream().filter(e -> e <= 50); boolean any = list.stream().anyMatch(e -> e > 50); boolean all = list.stream().allMatch(e -> e > 50); Stream result = list.stream().map(Object::toString); Integer result = list.stream().reduce(3, Integer::sum);

7. TECHNOLOGY DIVISION 7 Eager vs. Lazy Eager GS Collections select reject any satisfy detect collect all satisfy inject into Integer result = list.detect(e -> e > 50); MutableList result = list.select(e -> e > 50); MutableList result = list.reject(e -> e > 50); boolean any = list.anySatisfy(e -> e > 50); boolean all = list.allSatisfy(e -> e > 50); MutableList result = list.collect(Object::toString); Integer result = list.injectInto(3, Integer::sum); filter any Match findAny map all Match reduce Java 8 Streams Integer result = list.stream().filter(e -> e > 50).findFirst().orElse(null); List result = list.stream().filter(e -> e > 50).collect(Collectors.toList()); List result = list.stream().filter(e -> e <= 50).collect(Collectors.toList()); boolean result = list.stream().anyMatch(e -> e > 50); boolean result = list.stream().allMatch(e -> e > 50); List result = list.stream().map(Object::toString).collect(Collectors.toList()); Integer result = list.stream().reduce(3, Integer::sum);

8. TECHNOLOGY DIVISION 8 Java 8 Streams Great framework that provides feature rich functional API Lazy by default Supports serial and parallel iteration patterns Support for three types of primitive streams Extendable through Collector implementations Java 8 Streams is the tip of an enormous iceberg

9. TECHNOLOGY DIVISION 9 Iceberg dead ahead! Eager iteration patterns on Collections Covariant return types on collection protocols New Collection Types –Bag, SortedBag, BiMap, Multimap Memory Efficient Set and Map Primitive containers Immutable containers

10. TECHNOLOGY DIVISION 10 Ice is Twice as Nice Java 8GS Collections Stream vs. LazyIterable Interfaces59 Functional Interfaces46298 Object Container Interfaces1175 Primitive Container Interfaces0309 Stream vs. RichIterable API47109 Primitive Stream vs. Iterable API48 x 3 = 14438 x 8 = 304 LOC (Streams vs. GSC w/o code gen)~15k~400k

11. TECHNOLOGY DIVISION 11 More Iteration Patterns flatCollect partition makeString / appendString groupBy aggregateBy sumOf sumBy

12. TECHNOLOGY DIVISION 12 Futility of Utility Utility –Easy to extend with new behaviors without breaking existing clients API –Easy to discover new features –Easy to optimize –Easy to read from left to right –Return types are specific and easy to understand –Verb vs. gerund

13. TECHNOLOGY DIVISION 13 Joining vs. MakeString String joined = things.stream().map(Object::toString).collect(Collectors.joining(", ")); String joined = things.makeString(", ");

14. TECHNOLOGY DIVISION 14 SummingInt vs. SumOfInt int total = employees.stream().collect( Collectors.summingInt(Employee::getSalary)); long total = employees.sumOfInt(Employee::getSalary);

15. TECHNOLOGY DIVISION 15 GroupingBy vs. GroupBy Map > byDept = employees.stream().collect(Collectors.groupingBy( Employee::getDepartment)); Multimap byDept = employees.groupBy(Employee::getDepartment);

16. TECHNOLOGY DIVISION 16 GroupingBy/SummingBy vs. SumBy Map totalByDept = employees.stream().collect(Collectors.groupingBy( Employee::getDepartment, Collectors.summingInt(Employee::getSalary))); // Upcoming GS Collections 6.0 ObjectLongMap totalByDept = employees.sumByInt( Employee::getDepartment, Employee::getSalary);

17. TECHNOLOGY DIVISION 17 PartitioningBy vs. Partition Map > passingFailing = students.stream().collect(Collectors.partitioningBy( s -> s.getGrade() >= PASS_THRESHOLD)); PartitionList passingFailing = students.partition( s -> s.getGrade() >= PASS_THRESHOLD);

18. TECHNOLOGY DIVISION 18 How do they stack up?

19. TECHNOLOGY DIVISION 19 Agenda Introductions Lost and Found Streams The Iceberg –APIs –Fluency –Memory Efficiency –Method references are awesome Framework Comparisons

20. TECHNOLOGY DIVISION 20 Anagram tutorial http://docs.oracle.com/javase/tutorial/collections/algorithms/ Start with all words in the dictionary Group them by their alphagrams –Alphagram contains sorted characters –alerts  aelrst –stelar  aelrst Filter groups containing at least eight anagrams Sort groups by number of anagrams (descending) Print them in this format 11: [alerts, alters, artels, estral, laster, ratels, salter, slater, staler, stelar, talers]

21. TECHNOLOGY DIVISION 21 Anagram tutorial this.getWords().stream().collect(Collectors.groupingBy(Alphagram::new)).values().stream().filter(each -> each.size() >= SIZE_THRESHOLD).sorted( Comparator. >comparingInt(List::size).reversed() ).map(each -> each.size() + ": " + each).forEach(System.out::println);

22. TECHNOLOGY DIVISION 22 Anagram tutorial this.getWords().groupBy(Alphagram::new).multiValuesView().select(each -> each.size() >= SIZE_THRESHOLD).toSortedListBy(RichIterable::size).asReversed().collect(each -> each.size() + ": " + each).each(System.out::println);

23. TECHNOLOGY DIVISION 23 Anagram tutorial this.getWords().groupBy(Alphagram::new).multiValuesView().select(each -> each.size() >= SIZE_THRESHOLD).toSortedListBy(RichIterable::size).asReversed().collect(each -> each.size() + ": " + each).each(System.out::println); Type: MutableListMultimap

24. TECHNOLOGY DIVISION 24 Anagram tutorial this.getWords().groupBy(Alphagram::new).multiValuesView().select(each -> each.size() >= SIZE_THRESHOLD).toSortedListBy(RichIterable::size).asReversed().collect(each -> each.size() + ": " + each).each(System.out::println); Type: RichIterable >

25. TECHNOLOGY DIVISION 25 Anagram tutorial this.getWords().groupBy(Alphagram::new).multiValuesView().select(each -> each.size() >= SIZE_THRESHOLD).toSortedListBy(RichIterable::size).asReversed().collect(each -> each.size() + ": " + each).each(System.out::println); Type: RichIterable >

26. TECHNOLOGY DIVISION 26 Anagram tutorial this.getWords().groupBy(Alphagram::new).multiValuesView().select(each -> each.size() >= SIZE_THRESHOLD).toSortedListBy(RichIterable::size).asReversed().collect(each -> each.size() + ": " + each).each(System.out::println); Type: MutableList >

27. TECHNOLOGY DIVISION 27 Anagram tutorial this.getWords().groupBy(Alphagram::new).multiValuesView().select(each -> each.size() >= SIZE_THRESHOLD).toSortedListBy(RichIterable::size).asReversed().collect(each -> each.size() + ": " + each).each(System.out::println); Type: LazyIterable >

28. TECHNOLOGY DIVISION 28 Anagram tutorial this.getWords().groupBy(Alphagram::new).multiValuesView().select(each -> each.size() >= SIZE_THRESHOLD).toSortedListBy(RichIterable::size).asReversed().collect(each -> each.size() + ": " + each).each(System.out::println); Type: LazyIterable

29. TECHNOLOGY DIVISION 29 Parallel Lazy Iteration Stream addresses = people.parallelStream().map(Person::getAddress); ParallelListIterable addresses = people.asParallel(executor, batchSize).collect(Person::getAddress); http://www.infoq.com/presentations/java-streams-scala-parallel-collections

30. TECHNOLOGY DIVISION 30 Agenda Introductions Lost and Found Streams The Iceberg –APIs –Fluency –Memory Efficiency –Method references are awesome Framework Comparisons

31. TECHNOLOGY DIVISION 31 Comparing Maps

32. TECHNOLOGY DIVISION 32 Memory Optimizations Entry holds key, value, next, and hash. Better to put the keys and values in the backing array. Uses half the memory on average. But watch out for Map.entrySet(). –Leaky abstraction The assumption is that Maps are implemented as tables of Entry objects. –It’s now O(n) instead of O(1). –Use forEachKeyValue() instead.

33. TECHNOLOGY DIVISION 33 Comparing Sets

34. TECHNOLOGY DIVISION 34 Memory Optimizations HashSet is implemented by delegating to a HashMap. Entries are still a waste of space. Values in each (key, value) pair are a waste of space. Uses 4x the memory on average.

35. TECHNOLOGY DIVISION 35 Bad decisions from long ago

36. TECHNOLOGY DIVISION 36 Save memory with Primitive Collections

37. TECHNOLOGY DIVISION 37 List vs. IntList Java has object and primitive arrays –Primitive arrays have no behaviors Java does not have primitive Lists, Sets or Maps –Primitives must be boxed –Boxing is expensive Reference + Header + alignment

38. TECHNOLOGY DIVISION 38 Agenda Introductions Lost and Found Streams The Iceberg –APIs –Fluency –Memory Efficiency –Method references are awesome Framework Comparisons

39. TECHNOLOGY DIVISION 39 Lambdas and Method References We upgraded the Kata (our training materials) from Java 7 to Java 8 Some anonymous inner classes converted easily into Method References MutableList customerCities = customers.collect(Customer::getCity); Some we kept as lambdas MutableList customersFromLondon = customers.select(customer -> customer.livesIn("London"));

40. TECHNOLOGY DIVISION 40 Lambdas and Method References The method reference syntax is appealing Can we write the select example with a method reference? MutableList customersFromLondon = customers.select(Customer::livesInLondon); No one writes methods like this.

41. TECHNOLOGY DIVISION 41 Lambdas and Method References Now we use method references We used to use constants MutableList customerCities = customers.collect(Customer.TO_CITY); public static final Function TO_CITY = new Function () { public String valueOf(Customer customer) { return customer.getCity(); } };

42. TECHNOLOGY DIVISION 42 Lambdas and Method References The select example would have created garbage MutableList customersFromLondon = customers.select(new Predicate () { public boolean accept(Customer customer) { return customer.livesIn("London"); } });

43. TECHNOLOGY DIVISION 43 Lambdas and Method References So we created selectWith(Predicate2) to avoid garbage MutableList customersFromLondon = customers.selectWith(Customer.LIVES_IN, "London"); public static final Predicate2 LIVES_IN = new Predicate2 () { public boolean accept(Customer customer, String city) { return customer.livesIn(city); } };

44. TECHNOLOGY DIVISION 44 Lambdas and Method References The *With() methods work perfectly with Method References MutableList customersFromLondon = customers.selectWith(Customer::livesIn, "London"); This increases the number of places we can use method references.

45. TECHNOLOGY DIVISION 45 Framework Comparisons FeaturesGS CollectionsJava 8GuavaTroveScala Rich API  InterfacesReadable, Mutable, Immutable, FixedSize, Lazy Mutable, StreamMutable, FluentMutableReadable, Mutable, Immutable, Lazy Optimized Set & Map  (+Bag)  Immutable Collections  Primitive Collections  (+Bag, +Immutable)  Multimaps  (+Bag, +SortedBag)  (+Linked)(Multimap trait) Bags (Multisets)  BiMaps  Iteration StylesEager/Lazy, Serial/Parallel Lazy, Serial/Parallel Lazy, Serial Eager, Serial Eager/Lazy, Serial/Parallel (Lazy Only)

46. TECHNOLOGY DIVISION 46 Resources GS Collections on GitHub https://github.com/goldmansachs/gs-collections https://github.com/goldmansachs/gs-collections/wiki https://github.com/goldmansachs/gs-collections-kata GS Collections Memory Benchmark http://www.goldmansachs.com/gs-collections/presentations/GSC_Memory_Tests.pdf http://www.goldmansachs.com/gs-collections/presentations/GSC_Memory_Tests.pdf NY JUG Presentation, May 2014 http://www.goldmansachs.com/gs- collections/presentations/2014_05_19_NY_Java_User_Group.pdf http://www.goldmansachs.com/gs- collections/presentations/2014_05_19_NY_Java_User_Group.pdf Parallel-lazy Performance: Java 8 vs Scala vs GS Collections http://www.infoq.com/presentations/java-streams-scala-parallel-collections Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

47. TECHNOLOGY DIVISION 47 Learn more at GS.com/Engineering © 2014 Goldman Sachs. This presentation should not be relied upon or considered investment advice. Goldman Sachs does not warrant or guarantee to anyone the accuracy, completeness or efficacy of this presentation, and recipients should not rely on it except at their own risk. This presentation may not be forwarded or disclosed except with this disclaimer intact.