The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP 2003 Presented by Wenhao Xu University of British Columbia

Outline Background Assumption and Google Workloads Architecture System Interaction Master Relaxed Consistency Model Experiment Conclusion Discussion

Background In 2003, – 50+ GFS clusters – Each with thousands of storage nodes – Managing petabytes of data – GFS is under lots of applications and other systems in Google, such as bigtable, etc. Built on top of commodity machine and disks Tailored for Google workloads

Assumption & Google Workloads Failures occur everywhere – Disks, Network, Machines Store a modest number of large files – A few million files; Each typically 100MB or larger in size Read: Large streaming reads and small random reads – Typically read hundreds of KBs of an individual operation, more commonly 1MB or more – Small reads typically read a few KBs at arbitrary offsets, which are usually batched and sorted to avoid seeking back and forth Mutation: Mostly appending new data Concurrent clients Design flexibility: Co-designing the application and the file system API High availability: Replicas, Heartbeat, Fast recovery, checksums Large data chunk & No Client data cache Make appending fast Relaxed consistency model & Atomicity Record Append Customizable API & Dont need to conform to POSIX semantic! Throughput is more important

Architecture A Chunk Size: 64MB 64 bit handler Stored as a Linux file, lazy space allocation 3+ replicas Single Master Single point of Failures? Shadow masters, operation log Performance bottlenecks? Less involvment Simplicity: concurrency control, metadata management, etc. Global knowledge: make sophisticated placement and replication decisions.

System interaction – Mutation Data flow Leases – Primary replica Master is only involved in 1 & 2 Separated data flow and control flow – Employs network topology to transfer data – Easily to control the operation order

System Interaction - Heartbeat What is a heartbeat message like? – hey, I am still healthy. And I have chunks a, b, c …. Their versions are v1, v3, v9, … What is heartbeat for? – Chunkserver status – Chunk placement – Garbage collection – Stale Replica deletion – Lease extension

Master - Metadata Management Three major types – File and chunk namespaces – (File, chunk) mapping – Location of each chunks replicas All metadata is kept in the memory – Possibility: (<64B/chunk, 2 26 (64M) * 64 = 4G, a few millions of files) – Fast access & scanning Operation log – Store First two types of metadata – Any changes to the first two types of metadata are first written to the operation log – It is replicated on multiple remote machines and respond to a client operation only after flushing the corresponding log record to disk both locally and remotely – Checkpoints are made to avoid larger log and long starting up time for masters.

Master – Namespace Management & locking Namespace is organized into a lookup table – (file name, metadata) mapping – Does not support aliases for the same file or directory Mutation lock – Read lock of all parents directory – Write lock of mutation directory – E.g: /d1/d2/…/dn/leaf creation Read lock: /d1, /d1/d2, …, /d1/d2/…/dn Write lock: /d1/d2/…/dn/leaf

Master - Replica Creation and Placement Consideration – Balance disk space utilization – Performance: Limit the number of recent creations (hot pot) on each chunkserver – Availability: different racks; as far as possible – Maximize network utilization: as less switches as possible; exploit the aggregate bandwidth of multiple racks; – Multiple racks: Tradeoff: read bandwidth & write traffic Re-replication & rebalancing

Relaxed Consistency model Two state definitions: – Consistent: All the clients will always see the same data, regardless of which replicas they read from – Defined: Consistent & Clients will see what the mutation writes in its entirety The state after mutations depends on – whether the ops succeeds or fails – Whether there are concurrent mutations Move work to applications/clients – Error code check consistency – Application checksum check if it is defined – Identifier for record figure out duplicates – Retry File namespace mutation are atomic – Handled exclusively by the master

Atomic Record Appends Used heavily by Google applications Clients specify data – Not specify offset in files GFS appends it to the files – At least once atomically – Successfully written regions are defined – Does not guarantee that all replicas are identical Intervening regions are inconsistent

Experiments

Conclusion Gfs is designed for Google applications It is simple, but good enough for Google applications It is important for Google to store data for other applications.

Discussion (1/5) It is designed specifically to serve search needs. For what other applications this filesystem is suitable for. And for what applications, it is not optimal? Reply – Depends on: Major file size (small or large) Major file reads (streaming or not) Major writing modes (appending or not)

Discussion (2/5) When the Master decides on which replicas should a particular chunk goes in, shouldn't it consider the potential applications that is going to access that chunk ? – Reply: Not really. More replicas if it is hot spot. We don't want to see unbalanced disk usages. In the paper it mentioned that it "rebalances" for load balancing but a way of doing it is not mentioned. – Reply : Master knows the disk usages of all chunkservers In case of a shadow master is operating, it does not provide any mutation operations. But the processes that got lease from the primary before primary got crashed will continue on writing to the disks right ? So how can we keep track of those writes that is taken place while the shadow master is in operation ? – Reply : It can run until finished.

Discussion (3/5) What happens in GFS if the primary master is down? Will one of the shadow masters become a new primary master? If so, how to pick one from several shadow ones? – Reply : Maybe. Will the centralized master-server architecture degrade the GFS's scalability? – Reply : Seems not under Googles workload. GFS uses the "closest" client for replication. However, the term "closest" is never explained - is it based on network latency, or does it factor in network load on the closest chunkserver, does it factor in the system latency? How does one chunkserver figure out the closest server on the fly? – Reply : Deduced from IP addresss

Discussion (4/5) The concept of 'records' appears frequently in the paper. GFS can guarantee that a record is appended atomically, and can associate an application defined checksum with each record. This seems to imply that GFS keeps metadata for each record. How is this done? – Reply : The checksum is in the record data, not in the metadata. Clients cache chunk locations, so if one chunk server cannot contact the master it may unwittingly serve stale data to clients. Comments? – Reply : The master includes the chunk version number when it informs clients.

Discussion (5/5) What do you think about the performance gain of Concurrent appends to the same file operation and the relaxed consistency model under normal workload? (I believe this is only suits well to google). Since appending is key to GFS write performance in a multi-writer environment, it might be that GFS would give up much of its performance advantage even in large serial writes. What do you think about this? Reply: – It is a tradeoff between complexity between GFS and its clients. – I think it is good for performance. – It is tailored for googles workloads (appending).

Thanks for your attention!