1. Google Filesystem
Some slides taken from Alan Sussman

2. Data at Google (~2003) Writes: Large scale scraping of entire web
Maintain snapshot of all web content inside local datacenter
Update snapshot as a result of large scale web crawling
Reads: Generating search results
Periodically run search algorithm across snapshot
Store resulting inverted search index in memory

3. Design Considerations
Failures are the norm, not the exception
Datacenter built using commodity components
Relatively high failure rates
Very large scale means lots of stuff can go wrong
Hardware failures
Software failures
Files are very large
Individual web objects are combined into large files
Billions of objects, many TBytes of data
Storing many small files is inefficient
High relative overheads to access a small file
More wasted space due to metadata

4. Design Considerations
Writes are appends, reads are mostly sequential
Random writes are “practically non-existent”
As we saw in LFS, appends are fast and easy
Disks are cheap, so just keep adding them as needed
Data accesses are mostly for bulk processing
Don’t need low latency, instead focus on bandwidth
Very large scale systems should be co-designed
Building large scale systems requires thinking about every layer
Black box abstractions are inefficient
Allows removing unneeded features
Google can operate with a relaxed consistency model

5. Interface Follows POSIX semantics But Google has API extensions
Open, read, write, etc...
Users see a standard directory tree
But Google has API extensions
Snapshot
Low cost versioning of file/directory contents
Record append
Appends can appear to be atomic
Allows concurrent writes, without complex locking
Relaxed Consistency model

6. Architecture Files are broken into discrete chunks ChunkServers
Fixed size units of allocated space (64MB)
Identified via unique 64 bit ID
Represented as a single file on a local file system
ChunkServers
Stores a set of chunks on a local disk
Handles data operations for its local chunks
Operates as a key value store
Keys = Chunk IDs, Values = Chunk Data
Master server
Manages Chunkservers
Maintains metadata and namespace information
Maps filenames to chunk IDs

7. Architecture

8. Distributed Systems (Theory vs Practice)
Google runs a single master server
Single point of contact for accessing file system
Chunk locations are easily kept consistent
Means that chunks can be easily managed
Makes design and implementation easier
Distributed consistency is hard and expensive
A single master avoids much of the issue

9. Master Metadata
File and chunk namespaces, mappings from files to chunks, locations of each chunk’s replicas
Cached in memory
Namespaces and mappings must be persistent
Updates added to an operation log stored on master’s disk and replicated on other machines
Chunk location information not stored persistently
acquired from chunkservers at master startup or when a chunkserver joins the cluster
Periodically scans entire state in background
for chunk garbage collection, replication for data in failed chunkservers, chunk migration to balance load and disk space use
Size of metadata not a problem
less than 64 bytes per 64MB chunk, and most chunks are full
Operation log contains record of critical metadata changes
Stored reliably, and changes not made visible until changes are persistent
Log is periodically checkpointed and stored on disk locally and remotely
Recovery via replaying log from last checkpoint

10. Consistency Model File namespace operations are atomic,
handled in master with locking
operation log defines a total ordering of the mutations
Relaxed consistency guarantees designed for their specific workloads
Master ensures that a given number of replicas are always available
chunkserver failures detected via heartbeats
chunk is only lost if all replicas fail
Apps must explicitly deal with relaxed consistency
Atomic appends instead of random writes
Data contains self-validating, self-identifying records

11. Data Writes Client requests primary replica from master
Master replies with all replicas
Client caches result
Client pushes data out to all replicas directly
But data is not written yet
Replicas ACK that data has been received

12. Data Writes Client signals primary to complete write
Primary commits write
Primary forwards write signal to all secondaries
Secondaries ACK write operation to primary
Primary returns to client with success or error

13. Snapshots Creates a backup copy of file/directory
good for checkpoints of current state before large changes (to undo)
applies copy-on-write techniques to only make a real copy when a chunk changes
optimized to only do local copies on chunkservers
copied chunks placed on same chunkserver as original

14. Namespace Management
Locks allow concurrent operations on part of the namespace
Namespace is logically a lookup table
maps full pathnames to metadata (no inodes/directories)
Each node in namespace tree has a read-write lock
Each master operation acquires locks before it runs
For operation on /d1/d2/.../dn/leaf
acquire read-write locks on /d1, /d1/d2, ..., /d1/d2/.../dn
Acquire read or write lock on /d1/d2/.../dn/leaf
Locks acquired in top-down order
Prevents deadlock (two- phase locking)
Correctly serializes concurrent operations in the same part of the tree
Also allows concurrent mutations in the same directory

15. Replication Chunk replicated across system Replicas created at:
ensures spread across multiple racks not just multiple machines
at the expense of more network bandwidth between racks
Replicas created at:
chunk creation,
for re-replication
for rebalancing
Goal is maximize availability
balance load across chunkservers,
balance available disk space across chunkservers
Master adds new replicas when replica count falls below threshold
Done lazily in background
Lazy rebalancing also used to fill up a new chunkserver gradually

16. Garbage Collection
Storage for deleted files reclaimed lazily via garbage collection for files and chunks
Master periodically scans for chunks/files to garbage collect
Master logs delete operations, but just renames file to be a hidden name with the deletion timestamp
Garbage collection removes hidden files and metadata older than 3 days
Allows Undo operation until garbage collection
Orphaned chunks (not reachable from any file) erased during scan
chunkservers report subset of chunks they own in HeartBeat message,
Master replies with identity of all chunks no longer in master’s metadata,
Chunkserver can delete its replicas of those chunks
Users get some control over replication and reclamation policy (e.g., don’t replicate this temp file)

17. Master Replication Master state replicated for reliability
operation logs and checkpoints replicated on multiple machines
mutation to state committed only after log record flushed to disk locally and on all master replicas
One master process is in charge of all mutations and background activities that change system internal state
monitor process can restart the master from checkpoint and logs, obtained from some “shadow” master
While primary master is down, shadow masters provide read- only access to file system