PETAL: DISTRIBUTED VIRTUAL DISKS E. K. Lee C. A. Thekkath DEC SRC

Paper presents a distributed storage management system: –Petal consists of a collection of network- connected servers that cooperatively manage a pool of physical disks –Client see Petal as a highly available block- level storage partitioned into virtual disks Highlights

Introduction Petal is a distributed storage system that –Tolerates single component failures –Can be geographically distributed to tolerate site failures –Transparently reconfigures to expand in performance or capacity –Uniformly balances load and capacity –Provides fast efficient support for backup and recovery

Petal User Interface Petal appears to its clients as a collection of virtual disks: – Block-level interface –Lower-level service than a DFS –Makes system easier to model, design, implement and tune –Can support heterogeneous clients and applications

Client view Scalable Network BSD FFS NTFSEXT2 FSNTFS Virtual disks Petal

Physical view Storage Server Scalable Network BSD FFS NTFSEXT2 FSNTFS

Petal Server Modules Global State Module Recovery Module Data Access Module Liveliness Module Virtual to Physical

Overall design (I) All state information is maintained on servers –Clients maintain only hints Liveness module ensures that all servers will agree on the system operational status –Uses majority consensus and periodic exchanges of “I’m alive”/”You’re alive?” messages

Overall design (II) Information describing –current members of storage system and –currently supported virtual disks is replicated across all servers Global state module keeps this information consistent –Uses Lamport’s Paxos algorithm –Assumes fail-silent failures of servers

Overall design (III) Data access and recovery modules –Control how client data are distributed and stored –Support Simple data striping w/o redundancy Chained declustering –It distributes mirrored data in a way that balances load in the event of a failure

Address translation (I) Must translate virtual addresses into physical addresses Mechanism should be fast and fault-tolerant

Address translation (II) Uses three replicated data structures – Virtual disk directory: translates virtual disk ID into a global map ID – Global map: locates the server responsible for translating the given offset (block number) – Physical map: Locates physical disk and computers physical offset within that disk

Virtual to physical mapping VDir GMap PMap0 Server 0 VDir GMap PMap2 Server 2 VDir GMap PMap1 Server 1 vdiskID offset VDir GMap PMap2 Server 2 diskID and diskOffset on this server

Address translation (III) Three step process: 1.VDir translates virtual disk ID given by client into a GMap ID 2.Specified GMap finds server that can translate given offset 3.PMap of server translates GMap ID and offset to a physical disk and a disk offset Last two steps are almost always performed by same server

Address translation (IV) There is one GMap per virtual disk That GMap specifies –Tuple of servers spanned by the virtual disk –Redundancy scheme used to protect data –GMaps are immutable Cannot be modified Must create a new GMap

Address translation (V) PMaps are similar to page tables –Each PMap entry maps 64 KB of physical disk space –Server that performs the translation will usually perform the disk I/O Keeping GMaps and PMaps separate minimizes amount of global information that must be replicated

Support for backups Petal supports snapshots of virtual disks Snapshots are immutable copies of virtual disks –Created using copy-on-write VDir maps into –Epoch identifies current version of virtual disks and snapshots of past versions

Incremental reconfiguration (I) Used to add/remove new servers and new disks Three simple steps 1.Create new GMap 2.Update VDir entries 3.Redistribute the data Challenge is to perform the reconfiguration concurrently with normal client requests

Incremental reconfiguration (II) To solve the problem –Read requests will Try first new GMap Switch to old GMap if new GMap has no appropriate translation –Write requests will always use new GMap

Incremental reconfiguration (III) Observe that new GMap must be created before any data are moved –Too many read requests will have to consult both GMaps Seriously degrades system performance Do instead incremental changes over a fenced region of a virtual disk

Chained declustering (I) Virtual Disk Server 2 D2 D1 D6 D5 Server 1 D1 D0 D5 D4 Server 0 D0 D3 D4 D7 Server 3 D3 D2 D7 D6

Chained declustering (II) If one server fails, its workload will be almost equally distributed among remaining servers Petal uses a primary/secondary scheme for managing copies – Read requests can go to either primary or secondary copy – Write requests must go first to primary copy

Petal prototype Four servers –Each has fourteen 4.3 GB disks Four clients Links are 155 Mb/s ATM links Petal RPC interface has 24 calls

Latency of a virtual disk

Throughput of a virtual disk Throughput is mostly limited by CPU overhead (233 MHZ CPUs!)

File system performance (Modified Andrew Benchmark)

Conclusion Block-level interface s simpler and more flexible than a FS interface Use of distributed software solutions allows geographic distribution Petal performance is acceptable but for write requests –Must wait for primary and secondary copies to be successfully updated

Paxos: the main idea Proposers propose decision values from an arbitrary input set and try to collect acceptances from a majority of the accepters Learners observe this ratification process and attempt to detect that ratification has occurred Agreement is enforced because only one proposal can get the votes of a majority of accepters

Paxos: the assumptions Algorithm for consensus in a message-passing system Assumes the existence of Failure Detectors that let processes give up on stalled processes after some amount of time Processes can act as proposers, accepters, and learners –A process may combine all three roles

Paxos: the tricky part The tricky part is to avoid deadlocks when – There are more than two proposals – Some of the processes fail Paxos lets –Proposers make new proposals –Accepters release their earlier votes for losing proposals