The SMART Way to Migrate Replicated Stateful Services Jacob R. Lorch, Atul Adya, Bill Bolosky, Ronnie Chaiken, John Douceur, Jon Howell Microsoft Research First EuroSys Conference 19 April 2006

Replicated stateful services Replicated stateful services B Paxos Problem: Machine failure leads to unavailability Solution: Replicate the service for fault tolerance Problem: Replica state can become inconsistent Solution: Use replicated state machine approach The SMART Way to Migrate Replicated Stateful Services

Migrating replicated services B C D E Migration: Changing the configuration – the set of machines running replicas Uses of migration Replace failed machines for long-term fault tolerance Load balancing Increasing or decreasing number of replicas The SMART Way to Migrate Replicated Stateful Services

Limitations of current approaches Limitations of current approaches addressed by SMART Limitations of current approaches Can remove non-failed machines Enables autonomic migration, i.e., migration without human involvement Enables load balancing Can do concurrent request processing Can perform arbitrary migrations, even ones replacing entire configuration Completely described in our paper Cannot remove non-failed machines without creating window of vulnerability Can only remove known-failed machines Cannot use migration for load balancing Cannot process requests in parallel The SMART Way to Migrate Replicated Stateful Services

The SMART Way to Migrate Replicated Stateful Services Outline Introduction Background on Paxos Limitations of existing approaches SMART: Service Migration And Replication Technique Configuration-specific replicas Shared execution modules Implementation and evaluation Conclusions The SMART Way to Migrate Replicated Stateful Services

The SMART Way to Migrate Replicated Stateful Services Background on Paxos The SMART Way to Migrate Replicated Stateful Services

Background: Paxos overview protocol A B C … … … requests: slots: 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 Goal: Every service replica runs the same sequence of requests Deterministic service ensures state changes and replies are consistent Approach: Paxos assigns requests to virtual “slots” No two replicas assign different requests to same slot Each replica executes requests in slot order The SMART Way to Migrate Replicated Stateful Services

Background: Paxos protocol Z A B C PROPOSE DECIDED PROPOSE DECIDED LOGGED LOGGED Req client server replicas One replica is the leader Clients send requests to the leader Leader proposes a request by sending PROPOSE message to all replicas Each replica logs it and sends a LOGGED message to the leader When leader receives LOGGED messages from a majority, it decides it and sends a DECIDED message The SMART Way to Migrate Replicated Stateful Services

Background: Paxos leader change Poll Poll Reply If leader fails, another replica “elects” itself New leader must poll replicas and hear replies from a majority Ensures it learns enough about previous leaders’ actions to avoid conflicting proposals The SMART Way to Migrate Replicated Stateful Services

Background: Paxos migration α Service state A 79 80 81 82 83 84 85 B 79 80 81 82 83 84 85 C 79 80 81 82 83 A, B, C A, B, D D 84 85 Service state includes current configuration Request that changes that part of the state migrates the service Configuration after request n responsible for requests n+α and beyond The SMART Way to Migrate Replicated Stateful Services

The SMART Way to Migrate Replicated Stateful Services Rationale for α Z A B C Req Req PROPOSE PROPOSE LOGGED LOGGED With α=1, slot n can change the configuration responsible for slot n+1 Leader can’t propose slot n+1 until n is decided Doesn’t know who to make proposal to, let alone whether it can make proposal at all Prevents pipelining of requests Request may wait a network round trip and a disk write The SMART Way to Migrate Replicated Stateful Services

Limitations of existing approaches The SMART Way to Migrate Replicated Stateful Services

The SMART Way to Migrate Replicated Stateful Services No request pipelining Leader change is complicated How to ensure that new leader knows the right configuration to poll? How to handle some outstanding proposals being from one configuration and some from another? Other problems To avoid this complexity, current approaches use α=1 But, this prevents request pipelining The SMART Way to Migrate Replicated Stateful Services

Window of vulnerability C D Poll Poll DECIDED PROPOSE DECIDED PROPOSE LOGGED LOGGED Removing a machine creates window of vulnerability Effectively, it induces a failure of the removed replica Consequently, service can become permanently unavailable even if less than half the machines fail Considered acceptable since machines only removed when known to a human to have permanently failed Not suitable for autonomic migration using imperfect failure detectors, or for load balancing The SMART Way to Migrate Replicated Stateful Services

The SMART Way to Migrate Replicated Stateful Services

Configuration-specific replicas Replica 1A Replica 1B Replica 1C Replica 2A Replica 2B Replica 2D A B C D Each configuration has its own set of replicas and its own separate instance of Paxos Simplifies leader change so we can pipeline requests Election always happens in a static configuration No window of vulnerability because a replica can remain alive until next configuration is established The SMART Way to Migrate Replicated Stateful Services

SMART migration protocol Replica 1A FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED JOIN JOIN JOIN Replica 1B FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED Replica 1C FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED Replica 2A READY READY READY Replica 2B PREPARE READY READY READY JOIN Replica 2D A B C JOIN-REQ D After creating new configuration, send JOIN msgs After executing request n+α-1, send FINISHED msgs Tells new replicas where they can get starting state Makes up for possibly lost JOIN messages When a majority of successor configuration have their starting state, replica kills itself If a machine misses this phase, it can still join later The SMART Way to Migrate Replicated Stateful Services

Shared execution modules Agreement 1A Replica 1A Replica 1B Agreement 1B Agreement 1C Replica 1C Execution 1A Execution 1B Execution 1C Agreement 2A Replica 2A Agreement 2B Replica 2B Agreement 2D Replica 2D Execution 2A Execution 2B Execution 2D Execution Execution Execution Execution A B C D Configuration-specific replicas have a downside One copy of service state for each replica Need to copy state to new replicas Solution: Shared execution modules Divide replica into agreement and execution modules One execution module for all replicas on machine The SMART Way to Migrate Replicated Stateful Services

Implementation and evaluation SMART implemented in a replicated state machine library, LibSMART Lets you build a service as if it were single-machine, then turns it into a replicated, migratable service Farsite distributed file system service ported to LibSMART Straightforward because LibSMART uses BFT interface Experimental results using simple key/value service Pipelining reduces average client latency by 14% Migration happens quickly, so clients only see a bit of extra latency, less than 30 ms The SMART Way to Migrate Replicated Stateful Services

The SMART Way to Migrate Replicated Stateful Services Conclusions Migration is useful for replicated services Long-term fault tolerance, load balancing Current approaches to migration have limitations SMART removes these limitations by using configuration-specific replicas Can remove live machines, enabling autonomic migration and load balancing Can overlap processing of concurrent requests SMART is practical Implementation supports large, complex file system service The SMART Way to Migrate Replicated Stateful Services