1. S-Paxos: Eliminating the Leader Bottleneck
Martin Biely, Zarko Milosevic, Nuno Santos, André Schiper
Ecole Polytechnique Fédérale de Lausanne (EPFL)
Switzerland
October 9, 2012

2. Context: State Machine Replication
Nuno Santos
Context and Motivation
Context: State Machine Replication
Consistency among replicas ensured by
Deterministic service
Same initial state
Same sequence of requests
System model
Partially synchronous
Crash stop (max 𝑓 crashes)
Replicated Service
Service
Service
Service
Fault tolerance by replication.
Multiple independent replicas, so that if one fails the others will continue providing the service.
The state of the replicas must remain consistent even in the presence of faults. This is ensured by…
Ordering protocol (Paxos)
Clients

3. Context and Motivation
Nuno Santos
Context and Motivation
The Paxos Protocol
Paxos is a leader-based protocol
A distinguished process (leader) coordinates the others (followers)
Paxos is leader-based
Leader-election Oracle elects a leader.
Once leader is elected, runs Phase 1 where it prepares to order requests. The purpose of this phase is for the leader to get updated with any previous decisions.
Once Phase 1 completes, the leader is ready to order client requests.
Clients send the requests directly to leader.
For that, it executes several Phase 2 instances, each ordering one or more requests. Once the leader has a request that it wants to order, it assigns it a tentative position and sends a Phase 2a message to all, containing the chosen position and the request. In the good case, the other replicas reply with a Phase 2b message accepting the order and once the leader receives a majority of replies it decides the order. The request can then be executed and the reply sent to the client.
Observation: leader receives and sends more messages than the followers
Potential system bottleneck…

4. Paxos Performance The bottleneck in Paxos is typically the leader
Nuno Santos
Context and Motivation
Paxos Performance
Experimental settings
JPaxos – implementation of Paxos in Java (protocol shown previously)
n=3, request size=20 bytes, CPU
The bottleneck in Paxos is typically the leader

5. Paxos is Leader-centric
Nuno Santos
Context and Motivation
Paxos is Leader-centric
Leader-centric protocol
The leader does considerably more work than the followers
Therefore, the leader is prone to being the system bottleneck
Paxos and most leader-based protocols are also leader-centric

6. Leader-based vs Leader-centric
Nuno Santos
Context and Motivation
Leader-based vs Leader-centric
Note that leader-based ≠ leader-centric
Leader-based – algorithmic concept, leader is a distinguished process
Leader-centric – resource usage, leader is a bottleneck
Mention that the problem is unbalanced resource usage
Question: do leader-based protocols like Paxos
must also be leader-centric?

7. S-Paxos Overview Leader-based but not leader-centric Nuno Santos
Say S stands for scalable. Or just omit it.
Leader-based but not leader-centric

8. Why Paxos is Leader-centric
Nuno Santos
S-Paxos Overview
Why Paxos is Leader-centric
Leader does the following
Receives requests from clients
Coordinates protocol to order requests
Replies to clients
Followers do much less
Receive client requests from leader
Acknowledge order proposed by leader
Underlying problem: unbalanced resource utilization
Leader runs out of resources (CPU, network bandwidth)
While followers are lightly loaded
Mention semantics of arrows.

9. S-Paxos: A Balanced Paxos Variant
Nuno Santos
S-Paxos Overview
S-Paxos: A Balanced Paxos Variant
S-Paxos balances workload across replicas
Leader and followers have similar resource usage
The full resources of all replicas become available to the ordering protocol
S-Paxos is leader-based but not leader-centric
Combines several well-known ideas in a novel way
All replicas handle client communication
All replicas disseminate requests
Ordering done on IDs

10. S-Paxos key ideas Distribute client communication
Nuno Santos
S-Paxos Overview
S-Paxos key ideas Distribute client communication
All replicas handle client communication
Commonly used in practice
For instance, ZooKeeper
But by itself, still leader-centric
Leader runs the ordering protocol on requests (Phase 2a messages of Paxos)
Followers have to forward requests to leader
And hence, sends requests to other followers

11. S-Paxos key ideas Distribute request dissemination
Nuno Santos
S-Paxos Overview
S-Paxos key ideas Distribute request dissemination
Note that Phase 2a messages have a dual purpose
Dissemination of requests
Establishing order
All replicas disseminate requests
Ordering performed on IDs
S-Paxos separates dissemination from ordering

12. S-Paxos Architecture and Data Flow
Nuno Santos
S-Paxos Overview
S-Paxos Architecture and Data Flow

13. S-Paxos balances work among replicas
Nuno Santos
S-Paxos Overview
S-Paxos balances work among replicas
Client communication and request dissemination usually the bulk of the load
In S-Paxos this task is performed by all replicas
Leader still has to coordinate ordering protocol
But IDs are small messages
So leader has minimal additional overhead
Batching: a single instance of the ordering protocol can potentially order many client requests, tens or even hundreds.
Two levels of batching to further reduce load on leader
Dissemination layer: batch client requests and use ordering layer to order ids of batches
Ordering layer: usual Paxos batching, in this case batches of batch ids.

14. Benefits in the presence of faults
Nuno Santos
S-Paxos Overview
Benefits in the presence of faults
Faster view change
Since IDs are small, Phase 1 of Paxos completes quickly
Failures affecting the leader have less impact on throughput
Ordering protocol is interrupted, but dissemination protocol continues among working replicas
When a correct leader emerges, it can quickly order the IDs of the requests that were disseminated while there was no leader
Leader crashes or its links start loosing messages,

15. Dissemination Layer Protocol
Nuno Santos
Dissemination Layer Protocol
Ordering layer uses Paxos with only minor modifications, so I will focus instead in the dissemination layer.

16. Dissemination Layer Overview
Nuno Santos
Dissemination Layer Protocol
Dissemination Layer Overview
Dissemination layer tasks
Receive requests from clients
Disseminate requests and IDs to all replicas
Initiate ordering of IDs
Execute requests in the order established for IDs
Challenges
Once an ID is decided, the corresponding request must remain available in the system
Coordinate view change between ordering and dissemination layers to ensure that ids are ordered once-and-only once 3 4 2 2
Contrary to the conventional Paxos, IDs and requests are handled separately which creates a few new challenges 1

17. Overview of the Protocol
Nuno Santos
Dissemination Layer Protocol
Overview of the Protocol
Disseminating requests
Optimistic implementation of reliable broadcast
When a replica receives a request from a client, it broadcasts <request,ID>
Replicas acknowledge reception of forwarded requests by broadcasting <Ack,ID>
Proposing IDs
Leader proposes an ID once the corresponding request is stable
That is, when it receives 𝑓+1 acknowledgements for the ID
Executing requests
Replica must have: request and decision for corresponding ID
If ID decided before request received, poll other replicas for request after a small delay
Request stable, so at least one correct replica has the request
Explain why broadcast of requests is done unreliable. Efficiency. If request does not become stable

18. Performance Evaluation
Nuno Santos
Performance Evaluation

19. Performance Evaluation
Nuno Santos
Experimental Evaluation
Performance Evaluation
S-Paxos implemented on top of JPaxos, a Java implementation of Paxos
Experiments compare
JPaxos (leader-centric)
S-Paxos (non leader-centric)
Testbed: Grid 5000 (helios cluster)
CPU: 2.2Ghz
Network: 1Gbit Ethernet
Experimental parameters
Request size: 20 bytes
Batch size
S-Paxos: dissemination layer 1450 bytes, ordering layer: 50 bytes
JPaxos: 1450 bytes
Null service
Mention the request size and null services. Do not mention hardware details

20. Load Distribution: Average CPU utilization
Nuno Santos
Experimental Evaluation
Load Distribution: Average CPU utilization
JPaxos
S-Paxos

21. Performance with Increasing Number of Clients (n=3)
Nuno Santos
Experimental Evaluation
Performance with Increasing Number of Clients (n=3)
Throughput
Response time

22. Experimental Evaluation
Nuno Santos
Experimental Evaluation
Scalability
Throughput
Resource intensive tasks are distributed.

23. Throughput with crashes
Nuno Santos
Experimental Evaluation
Throughput with crashes
Request size: 1KB, Batch size: 8KB, 𝑛=5
Crash of the leader

24. Experimental Evaluation
Nuno Santos
Experimental Evaluation
False suspicions
Leader is (wrongly) suspected every 10 seconds

25. Conclusion A leader-based protocol does not need to be leader-centric
Nuno Santos
Conclusion
A leader-based protocol does not need to be leader-centric
S-Paxos: balances the workload across replicas
Benefits
Better performance for the same number of replicas
Better scalability with the number of replicas
Better performance in the presence of faults

26. Nuno Santos
Additional slides

27. Dissemination Layer Protocol
Nuno Santos
Dissemination Layer Protocol
Discussion
Broadcast of <request,ID>: best effort, no retransmission
Avoids cost of reliable broadcast on requests
Recovering from partial delivery (message loss/crashes):
Request does not become stable - client timeouts and retransmits
Request becomes stable – after ID is decided, replicas poll other replicas for request
Broadcast of <Ack,ID>: retransmission
Ensures that once a request is stable, it will be proposed
Almost free in practice: acks are small and can be piggybacked on other messages.
Once a request becomes stable, all replicas learn about and so the corresponding ID is proposed.