Google Filesystem Some slides taken from Alan Sussman

Data at Google (~2003) Writes: Large scale scraping of entire web Maintain snapshot of all web content inside local datacenter Update snapshot as a result of large scale web crawling Reads: Generating search results Periodically run search algorithm across snapshot Store resulting inverted search index in memory

Design Considerations Failures are the norm, not the exception Datacenter built using commodity components Relatively high failure rates Very large scale means lots of stuff can go wrong Hardware failures Software failures Files are very large Individual web objects are combined into large files Billions of objects, many TBytes of data Storing many small files is inefficient High relative overheads to access a small file More wasted space due to metadata

Design Considerations Writes are appends, reads are mostly sequential Random writes are “practically non-existent” As we saw in LFS, appends are fast and easy Disks are cheap, so just keep adding them as needed Data accesses are mostly for bulk processing Don’t need low latency, instead focus on bandwidth Very large scale systems should be co-designed Building large scale systems requires thinking about every layer Black box abstractions are inefficient Allows removing unneeded features Google can operate with a relaxed consistency model

Interface Follows POSIX semantics But Google has API extensions Open, read, write, etc... Users see a standard directory tree But Google has API extensions Snapshot Low cost versioning of file/directory contents Record append Appends can appear to be atomic Allows concurrent writes, without complex locking Relaxed Consistency model

Architecture Files are broken into discrete chunks ChunkServers Fixed size units of allocated space (64MB) Identified via unique 64 bit ID Represented as a single file on a local file system ChunkServers Stores a set of chunks on a local disk Handles data operations for its local chunks Operates as a key value store Keys = Chunk IDs, Values = Chunk Data Master server Manages Chunkservers Maintains metadata and namespace information Maps filenames to chunk IDs

Architecture

Distributed Systems (Theory vs Practice) Google runs a single master server Single point of contact for accessing file system Chunk locations are easily kept consistent Means that chunks can be easily managed Makes design and implementation easier Distributed consistency is hard and expensive A single master avoids much of the issue

Master Metadata File and chunk namespaces, mappings from files to chunks, locations of each chunk’s replicas Cached in memory Namespaces and mappings must be persistent Updates added to an operation log stored on master’s disk and replicated on other machines Chunk location information not stored persistently acquired from chunkservers at master startup or when a chunkserver joins the cluster Periodically scans entire state in background for chunk garbage collection, replication for data in failed chunkservers, chunk migration to balance load and disk space use Size of metadata not a problem less than 64 bytes per 64MB chunk, and most chunks are full Operation log contains record of critical metadata changes Stored reliably, and changes not made visible until changes are persistent Log is periodically checkpointed and stored on disk locally and remotely Recovery via replaying log from last checkpoint

Consistency Model File namespace operations are atomic, handled in master with locking operation log defines a total ordering of the mutations Relaxed consistency guarantees designed for their specific workloads Master ensures that a given number of replicas are always available chunkserver failures detected via heartbeats chunk is only lost if all replicas fail Apps must explicitly deal with relaxed consistency Atomic appends instead of random writes Data contains self-validating, self-identifying records

Data Writes Client requests primary replica from master Master replies with all replicas Client caches result Client pushes data out to all replicas directly But data is not written yet Replicas ACK that data has been received

Data Writes Client signals primary to complete write Primary commits write Primary forwards write signal to all secondaries Secondaries ACK write operation to primary Primary returns to client with success or error

Snapshots Creates a backup copy of file/directory good for checkpoints of current state before large changes (to undo) applies copy-on-write techniques to only make a real copy when a chunk changes optimized to only do local copies on chunkservers copied chunks placed on same chunkserver as original

Namespace Management Locks allow concurrent operations on part of the namespace Namespace is logically a lookup table maps full pathnames to metadata (no inodes/directories) Each node in namespace tree has a read-write lock Each master operation acquires locks before it runs For operation on /d1/d2/.../dn/leaf acquire read-write locks on /d1, /d1/d2, ..., /d1/d2/.../dn Acquire read or write lock on /d1/d2/.../dn/leaf Locks acquired in top-down order Prevents deadlock (two- phase locking) Correctly serializes concurrent operations in the same part of the tree Also allows concurrent mutations in the same directory

Replication Chunk replicated across system Replicas created at: ensures spread across multiple racks not just multiple machines at the expense of more network bandwidth between racks Replicas created at: chunk creation, for re-replication for rebalancing Goal is maximize availability balance load across chunkservers, balance available disk space across chunkservers Master adds new replicas when replica count falls below threshold Done lazily in background Lazy rebalancing also used to fill up a new chunkserver gradually

Garbage Collection Storage for deleted files reclaimed lazily via garbage collection for files and chunks Master periodically scans for chunks/files to garbage collect Master logs delete operations, but just renames file to be a hidden name with the deletion timestamp Garbage collection removes hidden files and metadata older than 3 days Allows Undo operation until garbage collection Orphaned chunks (not reachable from any file) erased during scan chunkservers report subset of chunks they own in HeartBeat message, Master replies with identity of all chunks no longer in master’s metadata, Chunkserver can delete its replicas of those chunks Users get some control over replication and reclamation policy (e.g., don’t replicate this temp file)

Master Replication Master state replicated for reliability operation logs and checkpoints replicated on multiple machines mutation to state committed only after log record flushed to disk locally and on all master replicas One master process is in charge of all mutations and background activities that change system internal state monitor process can restart the master from checkpoint and logs, obtained from some “shadow” master While primary master is down, shadow masters provide read- only access to file system