The Google File System and Map Reduce

The Team Pat Crane Tyler Flaherty Paul Gibler Aaron Holroyd Katy Levinson Rob Martin Pat McAnneny Konstantin Naryshkin Elliot Pennington Chris Scalabrini

System Expectations System failures are the norm System will store some large (multi-gigabyte) files Expected Reads o Large Streaming (over 1MB) o Small Random (a few KB out of someplace random) Expected Writes o Long Appending Writes  Multiple Users Appending at Once o Middle of File Seldom Modified High Bandwidth > Low Latency

The Basic Layout One Master Server A bunch of Chunk Servers Clients

"Chunks" Chunks are pieces of data Identified by "chunk handle" Replicated across multiple Chunkservers to prevent loss o If number is low Master will reallocate more replicas o Reallocation prioritized 64MB o Why so big?  Leaves the Master alone for longer  Less metadata on the master because there are less chunks o Drawbacks  Many clients might want same chunk  Doesn't happen much Store as plain Linux files

The Master Server One master server to rule them all o Makes it simple o Big precautions needed to prevent Master from being a bottleneck Maintains all system Metadata o File and Chunk namespaces o Access control information o Mapping from files to Chunks o Locations for the replicas of every Chunk Also controls o Chunk lease management o Garbage collection o HeartBeat

The Chunkserver Commodity computers running Linux Failures expected Holds Chunks

The Client Responsibilities o Interact with Master to find where to read/write o Interact with Chunkservers to read/write Requests several chunks at once to reduce load on Master All data-bearing messages go directly to and from chunkservers The client does not cache data it reads or writes (nor does the chunkserver) o Files being read are too big for it to be useful o Now no worries about cache coherence

Metadata Storage and Updates All Metadata is kept in memory of the Master A log is kept of all changes, kept on Master disk and replicated on remote systems Master does not store persistent chunk data o Queries all chunk servers at startup to build its Metadata table o Queries new Chunkservers as they join the cluster o Master keeps itself up to date, since it controls all chunk placement o Also periodically scans background (HeartBeat)  Garbage Collection  Re-replication (chunkserver failure)  Chunk Migration

Operation Log Only persistent record of Metadata Time line to define order of concurrent operations Everything uniquely identified in this log Replicated across many Chunkservers No changes visible to client until new changes replicated Batches several files together before flushing Master uses the log file to recover its file system on startup

Log Checkpoints Taken when log grows to a certain size System recovers by playing newest records o Written in tree form to easily read back into memory Take a long time to do Made to not interfere with incoming requests o Done in a separate thread o When completed, written both locally and remotely

Leases and Mutations Client asks master which Chunkserver holds the lease on the chunk (chooses one if no lease) Master tells Clients o which Chunkserver to contact o locations of replicas and secondary replicas C3lient caches this After all the replicas acknowledge data sent write request to primary Primary forwards all data to secondary replicas Secondaries write Primary replies to client and reports errors

Data Flow Fully utilize each machine's bandwidth o Data pushed linearly along chain o Full outbound bandwidth used to transfer data as fast as possible Avoid network bottlenecks o Foward data to "closest" machine o "Distances" can be accurately estimated from IP addresses Minimize Latency to push through all the data o Pipeline data transfer over TCP connections o Once a chunkserver receives some data, it starts forwarding immediately.

Snapshots Copy of file or directory tree Process o Revoke outstanding leases o Logs operation to disk o Makes a copy  On the same Chunkserver  Has everybody refer to that  Nobody can tell the difference!

Namespace Management and Locking Many operations take a long time Don't want to make everybody wait Locking specific areas of namespace o Read lock on directory - prevents deleting, renaming, and snapshotting o Write lock on file name - prevents the same file from being written to twice at the same time Permits concurrent edits to same directory

Garbage Collection (1/2) Garbage collection used rather than immediate deletion o Advantages  Simple, Reliable,Chunkcreation, Uniform o Note that files are rarely deleted o Space is NOT reused right away! Steps to perform "Garbage Collection" o System logs deletion o File is hidden and name is changed o File has its time stamp set - this is used for removal o During regular scan deletes file if timestamp minus current time is older than a specified interval  Interval time is configurable  3 days is cited in paper

Garbage Collection (2/2) Also deletes "orphaned chunks" in scans o An "orphaned chunk" is a chunk that is not reachable from any file o Metadata for this "orphaned chunk" is removed as well When a HeartBeat message is sent from master to chunkservers... o Each chunkserver reports a subset of the chunks it has o Master tells the chunkserver the identity of all chunks in the subset that is contains no metadata for. o Chunkserver garbage collects those files. Stale Replica Detection o Each chunk contains a"chunk version number" (CVN) o Master tells each chunkserver the up-to-date CVN o If the CVN is out of date, chunkserver updates chunk

Fault Tolerance and Diagnostics Fast recovery with logfiles and checkpoints o GFS can easily detect out-of-date or bad chunks, and then fix them Chunk replication o Master coordinates data movement If Master fails o Outside system tells it to make a new Master o Shadow  Provide read-only in event of total failure  Updates itself just like a normal master Chunks have checksums Diagnostics done mostly with logs

Questions? Thank You!