Problem-solving on large-scale clusters: theory and applications Lecture 4: GFS & Course Wrap-up.

Slides:



Advertisements
Similar presentations
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP 2003 Presented by Wenhao Xu University of British Columbia.
Advertisements

Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China
Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung
The google file system Cs 595 Lecture 9.
G O O G L E F I L E S Y S T E M 陳 仕融 黃 振凱 林 佑恩 Z 1.
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google Jaehyun Han 1.
The Google File System Authors : Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung Presentation by: Vijay Kumar Chalasani 1CS5204 – Operating Systems.
NFS, AFS, GFS Yunji Zhong. Distributed File Systems Support access to files on remote servers Must support concurrency – Make varying guarantees about.
The Google File System (GFS). Introduction Special Assumptions Consistency Model System Design System Interactions Fault Tolerance (Results)
Google File System 1Arun Sundaram – Operating Systems.
Lecture 6 – Google File System (GFS) CSE 490h – Introduction to Distributed Computing, Winter 2008 Except as otherwise noted, the content of this presentation.
CS 345A Data Mining MapReduce. Single-node architecture Memory Disk CPU Machine Learning, Statistics “Classical” Data Mining.
The Google File System. Why? Google has lots of data –Cannot fit in traditional file system –Spans hundreds (thousands) of servers connected to (tens.
Homework 2 In the docs folder of your Berkeley DB, have a careful look at documentation on how to configure BDB in main memory. In the docs folder of your.
The Google File System.
Google File System.
Northwestern University 2007 Winter – EECS 443 Advanced Operating Systems The Google File System S. Ghemawat, H. Gobioff and S-T. Leung, The Google File.
MapReduce : Simplified Data Processing on Large Clusters Hongwei Wang & Sihuizi Jin & Yajing Zhang
Case Study - GFS.
Google Distributed System and Hadoop Lakshmi Thyagarajan.
Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google∗
The Hadoop Distributed File System: Architecture and Design by Dhruba Borthakur Presented by Bryant Yao.
MapReduce.
1 The Google File System Reporter: You-Wei Zhang.
Google MapReduce Simplified Data Processing on Large Clusters Jeff Dean, Sanjay Ghemawat Google, Inc. Presented by Conroy Whitney 4 th year CS – Web Development.
CSC 456 Operating Systems Seminar Presentation (11/13/2012) Leon Weingard, Liang Xin The Google File System.
MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat.
MapReduce and Hadoop 1 Wu-Jun Li Department of Computer Science and Engineering Shanghai Jiao Tong University Lecture 2: MapReduce and Hadoop Mining Massive.
Distributed systems [Fall 2014] G Lec 1: Course Introduction.
MapReduce M/R slides adapted from those of Jeff Dean’s.
Data in the Cloud – I Parallel Databases The Google File System Parallel File Systems.
Introduction. Readings r Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn. 3 m Note: All figures from this book.
MapReduce and GFS. Introduction r To understand Google’s file system let us look at the sort of processing that needs to be done r We will look at MapReduce.
CENG334 Introduction to Operating Systems Erol Sahin Dept of Computer Eng. Middle East Technical University Ankara, TURKEY Network File System Except as.
Presenters: Rezan Amiri Sahar Delroshan
GFS : Google File System Ömer Faruk İnce Fatih University - Computer Engineering Cloud Computing
Eduardo Gutarra Velez. Outline Distributed Filesystems Motivation Google Filesystem Architecture The Metadata Consistency Model File Mutation.
GFS. Google r Servers are a mix of commodity machines and machines specifically designed for Google m Not necessarily the fastest m Purchases are based.
Distributed systems [Fall 2015] G Lec 1: Course Introduction.
HADOOP DISTRIBUTED FILE SYSTEM HDFS Reliability Based on “The Hadoop Distributed File System” K. Shvachko et al., MSST 2010 Michael Tsitrin 26/05/13.
Distributed File Systems Architecture – 11.1 Processes – 11.2 Communication – 11.3 Naming – 11.4.
Distributed Computing Systems CSCI 4780/6780. Scalability ConceptExample Centralized servicesA single server for all users Centralized dataA single on-line.
Presenter: Seikwon KAIST The Google File System 【 Ghemawat, Gobioff, Leung 】
Google File System Robert Nishihara. What is GFS? Distributed filesystem for large-scale distributed applications.
Silberschatz, Galvin and Gagne ©2009 Operating System Concepts – 8 th Edition, Lecture 24: GFS.
Next Generation of Apache Hadoop MapReduce Owen
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Presenter: Chao-Han Tsai (Some slides adapted from the Google’s series lectures)
1 Student Date Time Wei Li Nov 30, 2015 Monday 9:00-9:25am Shubbhi Taneja Nov 30, 2015 Monday9:25-9:50am Rodrigo Sanandan Dec 2, 2015 Wednesday9:00-9:25am.
Dr. Zahoor Tanoli COMSATS Attock 1.  Motivation  Assumptions  Architecture  Implementation  Current Status  Measurements  Benefits/Limitations.
1 CMPT 431© A. Fedorova Google File System A real massive distributed file system Hundreds of servers and clients –The largest cluster has >1000 storage.
Lecture 3 – MapReduce: Implementation CSE 490h – Introduction to Distributed Computing, Spring 2009 Except as otherwise noted, the content of this presentation.
Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung
Large-scale file systems and Map-Reduce
Google Filesystem Some slides taken from Alan Sussman.
Google File System CSE 454 From paper by Ghemawat, Gobioff & Leung.
Gregory Kesden, CSE-291 (Storage Systems) Fall 2017
Gregory Kesden, CSE-291 (Cloud Computing) Fall 2016
The Google File System Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung Google Presented by Jiamin Huang EECS 582 – W16.
MapReduce Simplied Data Processing on Large Clusters
The Google File System (GFS)
The Google File System (GFS)
The Google File System (GFS)
The Google File System (GFS)
CS 345A Data Mining MapReduce This presentation has been altered.
The Google File System (GFS)
THE GOOGLE FILE SYSTEM.
by Mikael Bjerga & Arne Lange
The Google File System (GFS)
Presentation transcript:

Problem-solving on large-scale clusters: theory and applications Lecture 4: GFS & Course Wrap-up

Today’s Outline File Systems Overview –Uses –Common distributed file systems GFS –Goals –System design –Consistency model –Performance Introduction to Distributed Systems –MapReduce as a distributed system

File System Uses Uses: –Persistence of data –Inter Process Communication –Shared Namespace of Objects Requirements: –Lots of files –Random Access –Permissions –Consistency

Common Distributed FSs Traditional: –NFS –SMB –AFS What about: –Kazaa –BitTorrent –Keberos

GFS Goals Common usage and environment patterns: –Regular Component Failure –Few, Large, Multi-GB files –Reads are streaming –Writes are appends –Control over client implementation What are the consequences on… –data caching? –latency and bandwidth balance? –consistency? –control of data flow?

GFS System Design System attributes and components: –Single master controls file namespace –Data broken in 64Meg “Chunks” –Chunks replicated over many “Chunk Servers” –Client talks directly to chunk servers for data From GFS paper

GFS Write flow Control flow 1.Client get chunk list from master 2.Master responds with primary/secondary chunk servers 3.Client starts pipelining data to chunk servers 4.Client asks primary chunk servers to notify finish 5.Primary chunk server signals write ordering to replicated servers 6.Chunk servers responds with successful commit 7.Client notified of good write From GFS paper

Consequence of design Questions: 1.Where are the bottlenecks? 2.What if a replica fails? 3.What if the primary fails? 4.What if the master fails? 5.Why do writes need to be ordered? From GFS paper How do you work around these issues?

GFS Consistency: terms Two new terms Consistent: All chunk servers have the same data Defined: The result of the “last write” is fully available A defined chunk is also a consistent chunk Questions: –Is data corrupt if it is inconsistent? –Is data corrupt if it is undefined? –Can applications use data in either state?

GFS Consistency: mutations Consistency for types of writes: –Single random write Consistent and Defined –Single append Consistent and Defined –Concurrent random write Consistent –Aborted write Inconsistent –Concurrent append Final location is defined

GFS Single Master handling Single Master = bottleneck = SPF 1.Master persists changes to multiple replicas 2.Can delegate to “shadow masters” 3.Naming done via DNS (easy failover) How does this work: 1.If the network is partitioned? 2.Over multiple data centers?

GFS Performance Here’s some random perf numbers –1MB replicates in about 80ms –342 node cluster 72 TB Avail, 55 Used 735 K Files, 22 K dead files, 992 K Chunks 13GB Chunk server Meta data 48MB Master Meta data Read rate: ~580 MB/s Write rate: ~2 MB/s Master ops: ~320 Op/s 100Mbit full duplex with Gigabit backbone ` Data from gfs paper

GFS Performance Cont’d More random perf numbers –227 node cluster 180 TB Avail, 155 Used 737 K Files, 232 K dead files, 1550 K Chunks 21GB Chunk server Meta data 60MB Master Meta data Read rate: ~380 MB/s Write rate: ~100 MB/s Master ops: ~500 Ops/s Data from gfs paper

And now … An overview of the concepts we’ve been alluding to all quarter: parallel and distributed systems

Parallel vs Distributed Computing Parallel computing –Dividing a problem into identical tasks to be executed at the same time on multiple machines or threads Distributed computing –Dividing a problem into (possibly identical) tasks to be executed on multiple machines or threads, but generally on machines separated by a network Parallel computing is often a limited form of distributed computing Is the MapReduce programming model parallel or distributed?

Requirements: Parallel Computing Requirement: minimal (to no) data dependencies! –Also nice: static data source Nice to have: minimal communication overhead –Replicating state –Coordinating / scheduling tasks (eg, administrative overhead)

Requirements: Distributed System See worksheet / activity

Distributed System Design (1 of 2) From studying large (but not necessarily distributed) systems, we know that distributed systems trade off one guarantee for another –Ergo, you need to know what you’re designing for, eg use-cases From studying MapReduce, we know that successful distributed system minimize data dependencies and administrative communication

Distributed System Design (2 of 2) From MapReduce & GFS, we know that a distributed system must assume that components fail –Network may not be reliable Latency can be high Bandwidth is not infinite Data may get corrupted –Attacks from malicious users –Hardware failures: motherboards, disks, and memory chips will fail –… etc …