Cloud Computing Platform as a Service The Google Filesystem

Slides:



Advertisements
Similar presentations
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP 2003 Presented by Wenhao Xu University of British Columbia.
Advertisements

Question Scalability vs Elasticity What is the difference?
Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung
The google file system Cs 595 Lecture 9.
THE GOOGLE FILE SYSTEM CS 595 LECTURE 8 3/2/2015.
G O O G L E F I L E S Y S T E M 陳 仕融 黃 振凱 林 佑恩 Z 1.
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google Jaehyun Han 1.
The Google File System Authors : Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung Presentation by: Vijay Kumar Chalasani 1CS5204 – Operating Systems.
GFS: The Google File System Brad Karp UCL Computer Science CS Z03 / th October, 2006.
The Google File System (GFS). Introduction Special Assumptions Consistency Model System Design System Interactions Fault Tolerance (Results)
Google File System 1Arun Sundaram – Operating Systems.
Lecture 6 – Google File System (GFS) CSE 490h – Introduction to Distributed Computing, Winter 2008 Except as otherwise noted, the content of this presentation.
The Google File System. Why? Google has lots of data –Cannot fit in traditional file system –Spans hundreds (thousands) of servers connected to (tens.
The Google File System and Map Reduce. The Team Pat Crane Tyler Flaherty Paul Gibler Aaron Holroyd Katy Levinson Rob Martin Pat McAnneny Konstantin Naryshkin.
1 The File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung (Google)
GFS: The Google File System Michael Siegenthaler Cornell Computer Science CS th March 2009.
Large Scale Sharing GFS and PAST Mahesh Balakrishnan.
The Google File System.
Google File System.
Northwestern University 2007 Winter – EECS 443 Advanced Operating Systems The Google File System S. Ghemawat, H. Gobioff and S-T. Leung, The Google File.
Case Study - GFS.
Google Distributed System and Hadoop Lakshmi Thyagarajan.
Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google∗
1 The Google File System Reporter: You-Wei Zhang.
CSC 456 Operating Systems Seminar Presentation (11/13/2012) Leon Weingard, Liang Xin The Google File System.
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung
The Google File System Ghemawat, Gobioff, Leung via Kris Molendyke CSE498 WWW Search Engines LeHigh University.
Homework 1 Installing the open source cloud Eucalyptus Groups Will need two machines – machine to help with installation and machine on which to install.
The Google File System Presenter: Gladon Almeida Authors: Sanjay Ghemawat Howard Gobioff Shun-Tak Leung Year: OCT’2003 Google File System14/9/2013.
The Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung
MapReduce and GFS. Introduction r To understand Google’s file system let us look at the sort of processing that needs to be done r We will look at MapReduce.
CENG334 Introduction to Operating Systems Erol Sahin Dept of Computer Eng. Middle East Technical University Ankara, TURKEY Network File System Except as.
Presenters: Rezan Amiri Sahar Delroshan
GFS : Google File System Ömer Faruk İnce Fatih University - Computer Engineering Cloud Computing
Eduardo Gutarra Velez. Outline Distributed Filesystems Motivation Google Filesystem Architecture The Metadata Consistency Model File Mutation.
GFS. Google r Servers are a mix of commodity machines and machines specifically designed for Google m Not necessarily the fastest m Purchases are based.
HADOOP DISTRIBUTED FILE SYSTEM HDFS Reliability Based on “The Hadoop Distributed File System” K. Shvachko et al., MSST 2010 Michael Tsitrin 26/05/13.
Presenter: Seikwon KAIST The Google File System 【 Ghemawat, Gobioff, Leung 】
Eduardo Gutarra Velez. Outline Distributed Filesystems Motivation Google Filesystem Architecture Chunkservers Master Consistency Model File Mutation Garbage.
Google File System Robert Nishihara. What is GFS? Distributed filesystem for large-scale distributed applications.
Silberschatz, Galvin and Gagne ©2009 Operating System Concepts – 8 th Edition, Lecture 24: GFS.
Google File System Sanjay Ghemwat, Howard Gobioff, Shun-Tak Leung Vijay Reddy Mara Radhika Malladi.
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Presenter: Chao-Han Tsai (Some slides adapted from the Google’s series lectures)
Dr. Zahoor Tanoli COMSATS Attock 1.  Motivation  Assumptions  Architecture  Implementation  Current Status  Measurements  Benefits/Limitations.
1 CMPT 431© A. Fedorova Google File System A real massive distributed file system Hundreds of servers and clients –The largest cluster has >1000 storage.
Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung
File and Storage Systems: The Google File System
Google File System.
GFS.
The Google File System (GFS)
Google Filesystem Some slides taken from Alan Sussman.
Google File System CSE 454 From paper by Ghemawat, Gobioff & Leung.
Gregory Kesden, CSE-291 (Storage Systems) Fall 2017
Gregory Kesden, CSE-291 (Cloud Computing) Fall 2016
The Google File System Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung Google Presented by Jiamin Huang EECS 582 – W16.
The Google File System (GFS)
The Google File System (GFS)
The Google File System (GFS)
The Google File System (GFS)
CENG334 Introduction to Operating Systems
CSE 451: Operating Systems Distributed File Systems
The Google File System (GFS)
Cloud Computing Storage Systems
THE GOOGLE FILE SYSTEM.
by Mikael Bjerga & Arne Lange
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google SOSP’03, October 19–22, 2003, New York, USA Hyeon-Gyu Lee, and Yeong-Jae.
The Google File System (GFS)
Presentation transcript:

Cloud Computing Platform as a Service The Google Filesystem Cs 595 Lecture 10

Paper to Read Read the paper on GFS: Evolution on Fast-Forward Also a link to a longer paper on GFS – original paper from 2003 I assume you are reading papers as specified in the class schedule

During the lecture, you should point out problems with GFS design decisions

Common Goals of GFS and most Distributed File Systems Performance Reliability Scalability Availability

GFS Design Considerations Component failures are the norm rather than the exception. File System consists of hundreds or even thousands of storage machines built from inexpensive commodity parts. Files are Huge. Multi-GB Files are common. Each file typically contains many application objects such as web documents. Append, Append, Append. Most files are mutated by appending new data rather than overwriting existing data. Co-Designing Co-designing applications and file system API benefits overall system by increasing flexibility

GFS Why assume hardware failure is the norm? The amount of layers in a distributed system mean failure on any could contribute to data corruption. Application OS Memory Disk Network Physical connections Power It is cheaper to assume common failure on poor hardware and account for it, rather than invest in expensive hardware and still experience occasional failure. Quantity and quality of components guarantee some are not functional at any given time. Constant monitoring, error detection, fault tolerance, and automatic recovery must exist in GFS

Initial Assumptions System built from inexpensive commodity components that fail Modest number of files: few million 100MB or larger files. Multi-GB files are common. Didn’t optimize for smaller files 2 kinds of reads: large streaming read (1MB) small random reads (batch and sort) Well-defined semantics: Master/slave, producer/ consumer and many-way merge. 1 producer per machine append to file. Atomic R/W with low overhead High sustained bandwidth chosen over low latency (difference?)

High bandwidth versus low latency Example: An airplane flying across the country filled with backup drives has very high bandwidth because it gets all data at destination faster than any existing network However – each individual piece of data had high latency

Interface GFS – familiar file system interface Files organized hierarchically in directories and identified by names Operations include: Create, delete, open, close, read, write Snapshot and record append Snapshot creates a copy of a file or directory at low cost/overhead Record append allows multiple users to append data to the same file concurrently.

Master/ChunkServers Single master, multiple chunkservers, all running Linux Each file divided into fixed-size chunks of 64 MB Typical file system block size: 4KB Chunks stored by chunkservers on local disks as Linux files Immutable and globally unique 64 bit chunk handle (name or number) assigned at creation Size of namespace: 264 Size of storage capacity: 264 x 64 MB Regulated by storage capacity of the master

Master/chunkServers Read or write chunk data specified by chunk handle and byte range Each chunk replicated on multiple chunkservers – default is 3 Why?

Master/ChunkServers Master maintains all file system metadata Namespace, access control info, mapping from files to chunks, location of chunks Controls garbage collection of chunks Reclaim physical storage when files are deleted Communicates with each chunkserver through HeartBeat messages Instructions and collect state Clients interact with master for metadata, chunksevers do the rest, e.g. R/W on behalf of applications No caching For client working sets too large, simplified coherence For chunkserver – chunks already stored as local files, Linux caches MFU in memory

Heartbeats What do we gain from Heartbeats? Not only do we get the new state of a remote system, this also updates the master regarding failures. Any system that fails to respond to a Heartbeat message is assumed dead. This information allows the master to update his metadata accordingly. This also queues the Master to create more replicas of the lost data.

Client Client translates offset in file into chunk index within file Send master request with file name/chunk index Master replies with chunk handle and location of replicas Client caches info using file name/chunk index as key Client sends request to one of the replicas (closest) Further reads of same chunk require no interaction Can ask for multiple chunks in same request

Master Operations Master executes all namespace operations Manages chunk replicas Makes placement decision Creates new chunks (and replicas) Coordinates various system-wide activities to keep chunks fully replicated Balance load Reclaim unused storage

Master - Justification Single Master – Simplifies design Placement, replication decisions made with global knowledge Doesn’t R/W, so not a bottleneck Client asks master which chunkservers to contact Clients then connect directly to chunkservers to access/manipulate data

Client – Simple read request Interactions for a simple read: The client translates the file name and byte offset of the data into a chunk index. The client then sends the master a request containing the filename and chunk index. The master replies with the chunk handle and location of replicas. The client sends a request to the corresponding chunkserver using the data gathered from the master.

Chunk Size - Justification 64 MB, larger than typical file system block size Replica stored as plain Linux file, extended as needed Reduces interaction of client with master R/W on same chunk only 1 request to master Mostly R/W large sequential files Likely to perform many operations on given chunk (keep persistent TCP connection) Reduces size of metadata stored on master

Chunk problems But – If small file – one chunk may be hot spot Can fix this with replication, stagger batch application start times

Metadata Three types: File and chunk namespaces Mapping from files to chunks Location of each chunk’s replicas All metadata in memory of master First two types stored in logs for persistence on master local disk and replicated remotely

Metadata Instead of keeping track of chunk location info Poll – which chunkserver has which replica Master controls all chunk placement/replacement Disks may go bad, chunkserver errors, etc.

Metadata - Justification In memory – fast Periodically scans state garbage collect if needed Re-replication if chunkserver failure Migration to load balance Master maintains < 64 B data for each 64 MB chunk File namespace < 64B To support larger file systems: Add extra RAM/Disks to the master (cheap)

Chunk size (again)- Justification 64 MB is large – think of typical size of email Why Large Files? METADATA! Every file in the system adds to the total overhead metadata that the system must store. More individual data means more data about the data is needed.

Operation Log Historical record of critical metadata changes Provides logical time line of concurrent ops Log replicated on remote machines Flush record to disk locally and remotely Log kept small – checkpoint when > size Checkpoint in B-tree form New checkpoint built without delaying mutations takes about 1 min for 2 M files Only keep latest checkpoint and subsequent logs

Snapshot Snapshot makes copy of file Used to create checkpoint or copies of huge data sets Newly created snapshot points to same chunks as source file After snapshot, client sends request to master to find lease holder

Shadow Master Master Replication Replicated for reliability Not mirrors, so may lag primary slightly (fractions of second) Shadow master reads operation log, applies same sequence of changes to data structures as the primary does

Shadow Master If Master fails: Start shadow instantly Read-only access to file systems even when primary master down If machine or disk fails, monitor outside GFS starts new master with replicated log Clients only use canonical name of master

Creation, Re-replication, Rebalancing Master creates chunk Place replicas on chunkservers with below-average disk utilization Limit number of recent creates per chunkserver New chunks may be hot Spread replicas across racks Re-replicate When number of replicas falls below goal Chunkserver unavailable, corrupted, etc. Replicate based on priority (fewest replicas) Master limits number of active clone operations

Creation, Re-replication, Rebalancing Rebalance Periodically moves replicas for better disk space and load balancing Gradually fills up new chunkserver Removes replicas from chunkservers with below-average free space

Consistency Model Why Append Only? Overwriting existing data is not state safe. We cannot read data while it is being modified. A customized ("Atomized") append is implemented by the system that allows for concurrent read/write, write/write, and read/write/write events.

Data Integrity Checksumming to detect corruption of stored data Impractical to compare replicas across chunkservers to detect corruption Chunk divided into 64KB blocks, each with 32 bit checksums Checksums stored in memory and persistently with logging

Data Integrity Before read, checksum If problem, return error to requestor and reports to master Requestor reads from replica, master clones chunk from other replica, delete bad replica Most reads span multiple blocks, checksum small part of it Checksum lookups done without I/O Checksum computation optimized for appends If partial corrupted, will detect with next read During idle, chunkservers scan and verify inactive chunks

Garbage Collection When a file is deleted, the file is renamed to a hidden name including the delete timestamp During regular scan of file namespace hidden files removed if existed > 3 days Until then can be undeleted When removed, in-memory metadata erased Orphaned chunks identified and erased HeartBeat message: chunkserver/master exchange info about files, master tells chunkserver about files it can delete

Garbage Collection Easy in GFS Everything else is garbage All chunks exist in file-to-chunk mappings of the master All chunk replicas are Linux files under designated directories on each chunkserver Everything else is garbage

Conclusions GFS – qualities essential for large-scale data processing on commodity hardware Component failures the norm rather than exception Optimize for huge files appended to Fault tolerance by constant monitoring, replication, fast/automatic recovery High aggregate throughput Separate file system control Large file size

GFS In the Wild - 2003 Has Google made any adjustments? Google currently has multiple GFS clusters deployed for different purposes. The largest currently implemented systems have over 1000 storage nodes and over 300 TB of disk storage. These clusters are heavily accessed by hundreds of clients on distinct machines. Has Google made any adjustments?