The google file system Cs 595 Lecture 9

Consistency Model Why Append Only? Overwriting existing data is not state safe. We cannot read data while it is being modified. A customized ("Atomized") append is implemented by the system that allows for concurrent read/write, write/write, and read/write/write events.

Consistency Model File namespace mutation (update) atomic File Region Consistent if all clients see same data Region – defined after file data mutation (all clients see writes in entirety, no interference from writes) Undefined but Consistent - concurrent successful mutations – all clients see same data, but not reflect what any one mutation has written, fragments of updates Inconsistent – if failed mutation (retries)

Consistency Relaxed consistency can be accommodated – relying on appends instead of overwrites Appending more efficient/resilient to failure than random writes Checkpointing allows restart incrementally and no processing of incomplete successfully written data

Namespace Management and Locking Master ops can take time, e.g. revoking leases allow multiple ops at same time, use locks over regions for serialization GFS does not have per directory data structure listing all files Instead lookup table mapping full pathnames to metadata Each name in tree has R/W lock If accessing: /d1/d2/ ../dn/leaf, R lock on /d1, /d1/d2, etc., W lock on /d1/d2 …/leaf

Locking Allows concurrent mutations in same directory R lock on directory name prevents directory from being deleted, renamed or snapshotted W locks on file names serialize attempts to create file with same name twice R/W objects allocated lazily, delete when not in use Locks acquired in total order (by level in tree) prevents deadlocks

Fault Tolerance Fast Recovery Chunk Replication Master/chunkservers restore state and start in seconds regardless of how terminated Abnormal or normal Chunk Replication

Data Integrity Checksumming to detect corruption of stored data Impractical to compare replicas across chunkservers to detect corruption Divergent replicas may be legal Chunk divided into 64KB blocks, each with 32 bit checksums Checksums stored in memory and persistently with logging

Data Integrity Before read, checksum If problem, return error to requestor and reports to master Requestor reads from replica, master clones chunk from other replica, delete bad replica Most reads span multiple blocks, checksum small part of it Checksum lookups done without I/O Checksum computation optimized for appends If partial corrupted, will detect with next read During idle, chunkservers scan and verify inactive chunks

Garbage Collection Lazy at both file and chunk levels When delete file, file renamed to hidden name including delete timestamp During regular scan of file namespace hidden files removed if existed > 3 days Until then can be undeleted When removed, in-memory metadata erased Orphaned chunks identified and erased With HeartBeat message, chunkserver/master exchange info about files, master tells chunkserver about files it can delete, chunkserver free to delete

Garbage Collection Easy in GFS All chunks in file-to-chunk mappings of master All chunk replicas are Linux files under designated directories on each chunkserver Everything else garbage

Conclusions GFS – qualities essential for large-scale data processing on commodity hardware Component failures the norm rather than exception Optimize for huge files appended to Fault tolerance by constant monitoring, replication, fast/automatic recovery High aggregate throughput Separate file system control Large file size

GFS In the Wild - 2003 Google currently has multiple GFS clusters deployed for different purposes. The largest currently implemented systems have over 1000 storage nodes and over 300 TB of disk storage. These clusters are heavily accessed by hundreds of clients on distinct machines. Has Google made any adjustments?

Read paper on New GFS Google’s Colossus