1. UC Berkeley Spark A framework for iterative and interactive cluster computing Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scott Shenker, Ion Stoica

2. Outline Background: Nexus project Spark goals Programming model Example jobs Implementation Interactive Spark

3. Nexus Background Rapid innovation in cluster computing frameworks Dryad Apache Hama Pregel Pig

4. Problem Rapid innovation in cluster computing frameworks No single framework optimal for all applications Want to run multiple frameworks in a single cluster »…to maximize utilization »…to share data between frameworks »…to isolate workloads

5. Solution Nexus is an “operating system” for the cluster over which diverse frameworks can run »Nexus multiplexes resources between frameworks »Frameworks control job execution

6. Nexus slave Nexus master Hadoop v20 scheduler Nexus slave Hadoop job Hadoop v20 executor task Nexus slave Hadoop v19 executor task MPI scheduler MPI job MPI execut or task Nexus Architecture Hadoop v19 scheduler Hadoop job Hadoop v19 executor task MPI execut or task

7. Nexus Status Prototype in 7000 lines of C++ Ported frameworks: » Hadoop (900 line patch) » MPI (160 line wrapper scripts) New frameworks: » Spark, Scala framework for iterative jobs & more » Apache+haproxy, elastic web server farm (200 lines)

8. Outline Background: Nexus project Spark goals Programming model Example job Implementation Interactive Spark

9. Spark Goals Support iterative jobs »Machine learning researchers in our lab identified this as a workload that Hadoop doesn’t perform well on Experiment with programmability »Leverage Scala to integrate cleanly into programs »Support interactive use from Scala interpreter Retain MapReduce’s fine-grained fault-tolerance

10. Programming Model Distributed datasets »HDFS files, “parallelized” Scala collections »Can be transformed with map and filter »Can be cached across parallel operations Parallel operations »Foreach, reduce, collect Shared variables »Accumulators (add-only) »Broadcast variables (read-only)

11. Example 1: Logistic Regression

12. Logistic Regression Goal: find best line separating two sets of points + – + + + + + + + + – – – – – – – – + target – random initial line

13. Serial Version val data = readData(...) var w = Vector.random(D) for (i <- 1 to ITERATIONS) { var gradient = Vector.zeros(D) for (p <- data) { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y gradient += scale * p.x } w -= gradient } println("Final w: " + w)

14. Spark Version val data = spark.hdfsTextFile(...).map(readPoint).cache() var w = Vector.random(D) for (i <- 1 to ITERATIONS) { var gradient = spark.accumulator(Vector.zeros(D)) for (p <- data) { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y gradient += scale * p.x } w -= gradient.value } println("Final w: " + w)

15. Spark Version val data = spark.hdfsTextFile(...).map(readPoint).cache() var w = Vector.random(D) for (i <- 1 to ITERATIONS) { var gradient = spark.accumulator(Vector.zeros(D)) for (p <- data) { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y gradient += scale * p.x } w -= gradient.value } println("Final w: " + w)

16. Spark Version val data = spark.hdfsTextFile(...).map(readPoint).cache() var w = Vector.random(D) for (i <- 1 to ITERATIONS) { var gradient = spark.accumulator(Vector.zeros(D)) data.foreach(p => { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y gradient += scale * p.x }) w -= gradient.value } println("Final w: " + w)

17. Functional Programming Version val data = spark.hdfsTextFile(...).map(readPoint).cache() var w = Vector.random(D) for (i <- 1 to ITERATIONS) { w -= data.map(p => { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y scale * p.x }).reduce(_+_) } println("Final w: " + w)

18. Job Execution Big Dataset Slave 4 Slave 3 Slave 2 Slave 1 Master R1 R2R3R4 aggregate update param param Spark

19. Job Execution Slave 4 Slave 3 Slave 2 Slave 1 Master R1 R2R3R4 aggregate update param param Master aggregate param Map 4 Map 3 Map 2 Map 1 Reduce aggregate Map 8 Map 7 Map 6 Map 5 Reduce param      SparkHadoop / Dryad

20. Performance 127 s / iteration first iteration 174 s further iterations 6 s

21. Example 2: Alternating Least Squares

22. Collaborative Filtering Predict movie ratings for a set of users based on their past ratings R = 1??45?3??35??35?5???14????2?1??45?3??35??35?5???14????2? Movies Users

23. Matrix Factorization Model R as product of user and movie matrices A and B of dimensions U×K and M×K RA = Problem: given subset of R, optimize A and B BTBT

24. Alternating Least Squares Algorithm Start with random A and B Repeat: 1.Fixing B, optimize A to minimize error on scores in R 2.Fixing A, optimize B to minimize error on scores in R

25. Serial ALS val R = readRatingsMatrix(...) var A = (0 until U).map(i => Vector.random(K)) var B = (0 until M).map(i => Vector.random(K)) for (i <- 1 to ITERATIONS) { A = (0 until U).map(i => updateUser(i, B, R)) B = (0 until M).map(i => updateMovie(i, A, R)) }

26. Naïve Spark ALS val R = readRatingsMatrix(...) var A = (0 until U).map(i => Vector.random(K)) var B = (0 until M).map(i => Vector.random(K)) for (i <- 1 to ITERATIONS) { A = spark.parallelize(0 until U, numSlices).map(i => updateUser(i, B, R)).collect() B = spark.parallelize(0 until M, numSlices).map(i => updateMovie(i, A, R)).collect() } Problem: R re-sent to all nodes in each parallel operation

27. Efficient Spark ALS val R = spark.broadcast(readRatingsMatrix(...)) var A = (0 until U).map(i => Vector.random(K)) var B = (0 until M).map(i => Vector.random(K)) for (i <- 1 to ITERATIONS) { A = spark.parallelize(0 until U, numSlices).map(i => updateUser(i, B, R.value)).collect() B = spark.parallelize(0 until M, numSlices).map(i => updateMovie(i, A, R.value)).collect() } Solution: mark R as broadcast variable

28. ALS Performance

29. Subseq. Iteration Breakdown 36% of iteration spent on broadcast

30. Outline Background: Nexus project Spark goals Programming model Example job Implementation Interactive Spark

31. Architecture Driver program connects to Nexus and schedules tasks Workers run tasks, report results and variable updates Data shared with HDFS/NFS No communication between workers for now Driver Workers HDFS user code, broadcast vars tasks, results Nexus local cache

32. Distributed Datasets Each distributed dataset object maintains a lineage that is used to rebuild slices that are lost / fall out of cache Ex: errors = textFile(“log”).filter(_.contains(“error”)).map(_.split(‘\t’)(1)).cache() HdfsFile path: hdfs://… HdfsFile path: hdfs://… FilteredFile func: contains(...) FilteredFile func: contains(...) MappedFile func: split(…) MappedFile func: split(…) CachedFile HDFS Local cache getIterator(slice)

33. Language Integration Scala closures are Serializable objects »Serialize on driver, load & run on workers Not quite enough »Nested closures may reference entire outer scope »May pull in non-Serializable variables not used inside »Solution: bytecode analysis + reflection Shared variables »Accumulators: serialized form contains ID »Broadcast vars: serialized form is path to HDFS file

34. Interactive Spark Modified Scala interpreter to allow Spark to be used interactively from the command line Required two changes: »Modified wrapper code generation so that each “line” typed has references to objects for its dependencies »Place generated classes in distributed filesystem Enables in-memory exploration of big data

35. Demo

36. Conclusions Spark provides two abstractions that enable iterative jobs and interactive use: 1.Distributed datasets with controllable persistence, supporting fault-tolerant parallel operations 2.Shared variables for efficient broadcast and imperative style programming Language integration achieved using Scala features + some amount of hacking All this is surprisingly little code (~1600 lines)

37. Related Work DryadLINQ »SQL-like queries integrated in C# programs »Build queries through operations on lazy datasets »Cannot have a dataset persist across queries »No concept of shared variables for broadcast etc Pig & Hive »Query languages that can call into Java/Python/etc UDFs »No support for caching a dataset across queries OpenMP »Compiler extension for parallel loops in C++ »Annotate variables as read-only or accumulator above loop »Cluster version exists, but not fault-tolerant

38. Future Work Open-source Spark and Nexus »Probably this summer »Very interested in getting users! Understand which classes of algorithms we can handle and how to extend Spark for others Build higher-level interfaces on top of interactive Spark (e.g. R, SQL)

39. Questions ? ? ?