Pyspark 최 현 영 컴퓨터학부

Distributed Processing with the Spark Framework API Spark Cluster Computing Spark Standalone YARN Mesos Storage HDFS

The Hadoop Distributed File System (HDFS)

Chapter Topics - The Hadoop Distributed File System !! Why HDFS? !! HDFS Architecture !! Using HDFS !! Conclusion !! Hands"On Exercise: Using HDFS

Big Data Processing with Spark Three key concepts Distribute data when the data is stored – HDFS Run computation where the data is – HDFS and Spark Cache data in memory – Spark

Chapter Topics - The Hadoop Distributed File System !! Why HDFS? !! HDFS Architecture !! Using HDFS !! Hands"On Exercise: Using HDFS

HDFS Basic Concepts HDFS is a filesystem written in Java Based on Google’s GFS Provides redundant storage for massive amounts of data Using readily-available, industry-standard computers HDFS Native OS filesystem Disk Storage

How Files Are Stored Very Large Data

Example: Storing and Retrieving Files (1)

Example: Storing and Retrieving Files (2)

Example: Storing and Retrieving Files (3)

Example: Storing and Retrieving Files (4)

HDFS NameNode Availability ! The NameNode daemon must be running at all times – If the NameNode stops, the cluster becomes inaccessible ! HDFS is typically set up for High Availability – Two NameNodes: Active and Standby Active Name Node Standby Name Node Small clusters may use ‘Classic mode’ One NameNode One “helper” node called the Secondary NameNode – Bookkeeping, not backup Secondary Name Node Name Node

Chapter Topics - The Hadoop Distributed File System !! Why HDFS? !! HDFS Architecture !! Using HDFS !! Hands"On Exercise: Using HDFS

HDFS NameNode Availability ! The NameNode daemon must be running at all times – If the NameNode stops, the cluster becomes inaccessible ! HDFS is typically set up for High Availability – Two NameNodes: Active and Standby Active Name Node Standby Name Node Small clusters may use ‘Classic mode’ One NameNode One “helper” node called the Secondary NameNode – Bookkeeping, not backup Secondary Name Node Name Node

hdfs dfs Examples (1) ! Copy file foo.txt from local disk to the user’s directory in HDFS $ hdfs dfs -put foo.txt foo.txt – This will copy the file to /user/username/foo.txt ! Get a directory listing of the user’s home directory in HDFS $ hdfs dfs -ls ! Get a directory listing of the HDFS root directory $ hdfs dfs –ls /

hdfs dfs Examples (2) ! Display the contents of the HDFS file /user/fred/bar.txt $ hdfs dfs -cat /user/fred/bar.txt ! Copy that file to the local disk, named as baz.txt $ hdfs dfs -get /user/fred/bar.txt baz.txt ! Create a directory called input under the user’s home directory $ hdfs dfs -mkdir input ! Delete the directory input_old and all its contents $ hdfs dfs -rm -r input_old

Example: HDFS in Spark ! Specify HDFS files in Spark by URI –hdfs://hdfs-host[:port]/path – Default port is 8020 > mydata = sc.textFile \ ("hdfs://hdfs-host:port/user/training/purplecow.txt") > mydata.map(lambda s: s.upper()).\ saveAsTextFile \ ("hdfs://hdfs-host:port/user/training/purplecowuc") Paths are relative to user’s home HDFS directory > mydata = sc.textFile("purplecow.txt") hdfs://hdfs-host:port/user/training/purplecow.txt

Chapter Topics - The Hadoop Distributed File System !! Why HDFS? !! HDFS Architecture !! Using HDFS !! Hands"On Exercise: Using HDFS

Hands"On Exercise: Using HDFS(1) ! Upload File $ cd ~/training_materials/sparkdev/data $ hdfs dfs -put weblogs /user/training/weblogs ! List the contents of your HDFS home directory now $ hdfs dfs –ls /user/training

Hands"On Exercise: Using HDFS(2) ! Create an RDD base on one of the files you uploaded to HDFS pyspark> logs=sc.textFile("hdfs://localhost/user/training/weblogs/2014-03-08.log") pyspark> logs.filter(lambda s: “.jpg” in s).saveAsTextFile(“hdfs://localhost/user/training/jpgs”) ! View result $ hdfs dfs -ls jpgs

Running Spark on a Cluster

Distributed Processing with the Spark Framework API Spark Cluster Computing Spark Standalone YARN Mesos Storage HDFS

Chapter Topics - Running Spark on a Cluster !! Overview !! A Spark Standalone Cluster !! Hands"On Exercise: Running the Spark Shell on a Cluster

Environment can run spark ! Locally - No distributed processing ! Locally with multiple worker threads ! On a cluster - Spark Standalone - Apache Hadoop YARN (Yet Another Resource Negotiator) - Apache Mesos

Why Run on a Cluster? Run Spark on a cluster to get the advantages of distributed processing Ability to process large amounts of data efficiently Fault tolerance and scalability Local mode is useful for development and testing Production use is almost always on a cluster

Cluster architecture

Spark Cluster Terminology

The Spark Driver Program

Starting the Spark Shell on a Cluster

Chapter Topics - Running Spark on a Cluster !! Overview !! A Spark Standalone Cluster !! Hands"On Exercise: Running the Spark Shell on a Cluster

Spark Standalone Daemons

Running Spark on a Standalone Cluster (1)

Running Spark on a Standalone Cluster (2)

Running Spark on a Standalone Cluster (3)

Running Spark on a Standalone Cluster (4)

Running Spark on a Standalone Cluster (4)

Hands"On Exercise: Running the Spark Shell on a Cluster ! Start the Spark Standalone Cluster $ sudo service spark-master start $ sudo service spark-worker start ! View the Spark Standalone Cluster UI

Chapter Topics - Running Spark on a Cluster !! Overview !! A Spark Standalone Cluster !! Hands"On Exercise: Running the Spark Shell on a Cluster

Hands"On Exercise: Running the Spark Shell on a Cluster ! Start the Spark Standalone Cluster $ MASTER=spark://localhost:7077 pyspark

Hands"On Exercise: Running the Spark Shell on a Cluster ! View the sc.master property pyspark> sc.master ! Excute a simple operation pyspark> sc.textFile(“weblogs/*”).count()