Making Fly Parviz Deyhim http://bit.ly/sparkemr

Scalable, Highly-available and Secure Obsession Scalable, Highly-available and Secure

Scalable, Highly-available and Secure Obsession Scalable, Highly-available and Secure

Amazon Elastic MapReduce

Cost effective AWS wrapper What is EMR? Map-Reduce engine Integrated with tools Hadoop-as-a-service Massively parallel Cost effective AWS wrapper Integrated to AWS services EMR Is managed Hadoop Offering that takes burden of deploying and maintaining hadoop clusters away from developers. EMR uses Apache Hadoop mapreduce engine and integrates with variety of different tools.

Integration HDFS Amazon EMR

Integration HDFS Amazon EMR Amazon S3 Amazon DynamoDB

Integration Amazon EMR Data management Analytics languages Amazon S3 HDFS Amazon EMR Amazon S3 Amazon DynamoDB

Integration Amazon EMR Data management Analytics languages Amazon RDS HDFS Amazon EMR Amazon S3 Amazon DynamoDB

Integration Amazon EMR Data Pipeline Data management Analytics languages Integration Amazon RDS HDFS Amazon EMR Amazon S3 Amazon DynamoDB Data Pipeline Amazon RedShift

Integration Amazon EMR Data Pipeline Data management Analytics languages Integration Amazon RDS HDFS Amazon EMR Amazon S3 Amazon DynamoDB Data Pipeline Amazon RedShift

Amazon EMR Concepts Master Node Core Nodes Task Nodes

Core Nodes DataNode (HDFS) Amazon EMR cluster Master instance group Master Node We’ll start with Core nodes. Core nodes run TaskTracker and Datanode. Core nodes are very similar to traditional Hadoop salve nodes. They can process data with mappers and reducers and can also store data with HDFS or Datanode. HDFS HDFS Core instance group

Core Nodes Can Add Core Nodes: More CPU More Memory More HDFS Space Amazon EMR cluster Can Add Core Nodes: More CPU More Memory More HDFS Space Master instance group Master Node We’ll start with Core nodes. Core nodes run TaskTracker and Datanode. Core nodes are very similar to traditional Hadoop salve nodes. They can process data with mappers and reducers and can also store data with HDFS or Datanode. HDFS HDFS HDFS Core instance group

Core Nodes Can’t remove core nodes: HDFS corruption Amazon EMR cluster Master instance group Master Node We’ll start with Core nodes. Core nodes run TaskTracker and Datanode. Core nodes are very similar to traditional Hadoop salve nodes. They can process data with mappers and reducers and can also store data with HDFS or Datanode. HDFS HDFS HDFS Core instance group

Task Nodes No HDFS Provides compute resources: CPU Memory Amazon EMR cluster No HDFS Provides compute resources: CPU Memory Master instance group Master Node We’ll start with Core nodes. Core nodes run TaskTracker and Datanode. Core nodes are very similar to traditional Hadoop salve nodes. They can process data with mappers and reducers and can also store data with HDFS or Datanode. HDFS HDFS Core instance group

Task Nodes Can add and remove task nodes Amazon EMR cluster Master instance group Master Node We’ll start with Core nodes. Core nodes run TaskTracker and Datanode. Core nodes are very similar to traditional Hadoop salve nodes. They can process data with mappers and reducers and can also store data with HDFS or Datanode. HDFS HDFS Core instance group

Spark On Amazon EMR

Bootstrap Actions Ability to run or install additional packages/software on EMR nodes Simple bash script stored on S3 Script gets executed during node/instance boot time Script gets executed on every node that gets added to the cluster

Spark on Amazon EMR http://bit.ly/sparkemr Bootstrap action installs Spark on EMR nodes Currently on Spark 0.73 & upgrading to 0.8 very soon http://bit.ly/sparkemr

Why Spark on Amazon EMR? Deploy small and large Spark clusters in minutes EMR handles node recover in case of failures Integration with EC2 Spot Market, Amazon Redshift, Amazon Data pipeline, Amazon Cloudwatch and etc

Why Spark on Amazon EMR? Shipping Spark logs to S3 for debugging Define S3 bucket at cluster deploy time

Spark on EC2 Spot Market Bid on un-used EC2 capacity Spark is memory hungry. Bid on large memory instances with the fraction of the cost

Spark on EC2 Spot Market 1TB Memory Cluster Instance Type # of nodes On-demand Cost Spot Cost Cost/GB Of Memory Using Spot M1.xlarge 63 $31/h $4.41/h 0.44c/GB/h CC2.8xlarge 16 $39/h $4.64/h 0.46c/GB/h M2.4xlarge 15 $24/h $2.25/h 0.22c/GB/h

Spark on EC2 Spot Market Launch initial Spark cluster with core nodes Amazon EMR cluster Master instance group Master Node Launch initial Spark cluster with core nodes HDFS to store and checkpoint RDDs HDFS HDFS 32GB Memory

Spark on EC2 Spot Market Amazon EMR cluster Master instance group Master Node Add Task nodes in spot market to increase memory capacity HDFS HDFS 32GB Memory 256GB Memory

Spark on EC2 Spot Market Create RDDs from HDFS or Amazon S3 with: Amazon EMR cluster Create RDDs from HDFS or Amazon S3 with: sc.textFile OR sc.sequenceFile Run Computation on RDDs Master instance group Master Node HDFS HDFS 32GB Memory 256GB Memory Amazon S3

Spark on EC2 Spot Market Save the resulting RDDs to HDFS or S3 with: Amazon EMR cluster Master instance group Save the resulting RDDs to HDFS or S3 with: rdd.saveAsSequenceFile OR rdd.saveAsObjectFile Master Node HDFS HDFS 32GB Memory 256GB Memory Amazon S3 saveAsObjectFile

Spark on EC2 Spot Market Shutdown TaskNodes when your job is done Amazon EMR cluster Master instance group Master Node Shutdown TaskNodes when your job is done HDFS HDFS 32GB Memory

Elastic Spark With Amazon EMR

Autoscaling Spark Master Node Amazon EMR cluster HDFS 32GB Memory

Autoscaling Spark Amazon EMR cluster 32GB Memory 256GB Memory Master Node Amazon EMR cluster HDFS 32GB Memory 256GB Memory

Elastic Spark When to Scale? CPU bounded or Memory intensive? Depends on your job CPU bounded or Memory intensive? Probably both for Spark jobs Use CPU/Memory util. metrics to decide when to scale

Amazon EMR Cloudwatch metrics EMR integrates with Cloudwatch Provides many metrics. Examples: Load Metrics HDFS Metrics S3 Metrics

Amazon EMR Cloudwatch metrics

Basics on Cloudwatch Metrics Pick any Cloudwatch metrics Pick a threshold that you like to be notified if its breached Setup Cloudwatch Alarms based on your thresholds Receive SNS notification in forms of: Email SNS HTTP API Call

Basics on Cloudwatch Metrics Receive Email Notification Monitor With Cloudwatch Take Manual Action Such As Adding More Task Nodes

Basics on Cloudwatch Metrics Monitor With Cloudwatch HTTP API Calls Take Automated Actions

Spark Autoscaling Based on Load Setup Cloudwatch alarm on EMR “TotalLoad” metric Receive Email/SNS/HTTP notification Add more worker nodes by adding EMR task nodes

Spark Autoscaling Based Memory Spark needs memory Lost of it!! How to scale based on the memory usage?

Spark Metrics Spark 0.8 provides cluster metrics Source and Sink topology

Spark Metrics Spark Metric Sources (metrics.properties): worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Spark Metrics Spark Metric Sinks (metrics.properties): Package: org.apache.spark.metrics.sink ConsoleSink JmxSink CsvSink GangliaSink

Spark Metrics Spark Metric Sinks (metrics.properties): CloudwatchSink

Spark Metrics & Cloudwatch

Spark Metrics & Cloudwatch Monitor Spark metrics with Cloudwatch Setup Cloudwatch alarms and get notified if any metrics reached your threshold. Example: if JvmHeapUsed > 20G Receive notification and take manual or automated actions

Spark Streaming and Amazon Kinesis

Amazon Kinesis Kinesis

Amazon Kinesis CreateStream Creates a new Data Stream within the Kinesis Service PutRecord Adds new records to a Kinesis Stream DescribeStream Provides metadata about the Stream, including name, status, Shards, etc. GetNextRecord Fetches next record for processing by user business logic MergeShard / SplitShard Scales Stream up/ down DeleteStream Deletes the Stream

Amazon Kinesis Kinesis

Spark Streaming and Amazon Kinesis SparkStreaming Kinesis Receiver Extends NetworkReceiver Creates a single Receiver per shard and reads from Kinesis

Misc. New AWS instances provide enhanced networking in VPC Higher PPS Less Jitter Great CPU Power Suitable for Spark: Serialization and Shuffle

What Do You Like To See On Spark By Amazon? Send Feedbacks To: Parviz Deyhim parvizd@amazon.com @pdeyhim