1. EEC 688/788 Secure and Dependable Computing
Lecture 9
Wenbing Zhao
Department of Electrical and Computer Engineering
Cleveland State University
Maulik no show 10/5/2009

2. Data and Service Replication
Replication resorts to the use of space redundancy to achieve high availability
Instead of running a single copy of the service, multiple copies are used
Usually deployed across a group of physical nodes for fault isolation
Data and service replication
Usually use different approaches
Transactional data replication
Optimistic replication (omitted)
Balance consistency and performance: CAP theorem (omitted)

3. Data and Service Replication
Service replication: State machine replication
Each replica is modeled as a state machine:
state, interface, deterministic state change via interface
Replica consistency issue: coordination needed
Total order of requests to the server replicas
Sequential execution of requests
Data replication:
Direct access on data
Operation on data: read or write
Context: transaction processing => concurrent access to replicated data essential

4. Service Replication State is encapsulated
Clients interact with exported interfaces (APIs)
Replication algorithm used to coordinate replicas (for consistency)
Fault tolerance middleware

5. EEC688/788: Secure & Dependable Computing
Replication Styles
Active replication
Every input (request) is executed by every replica
Every replica generates the outputs (replies)
Voting is needed to cope with non-fail-stop faults
Passive replication
One of the replicas is designated as the primary replica
Only the primary replica executes requests
The state of the primary replica is transferred to the backups periodically or after every request processing
Semi-active replication
One of the replicas is designated as the leader (or primary)
The leader determines the order of execution
Every input is executed by every replica per the leader’s instruction
1/16/2019
EEC688/788: Secure & Dependable Computing

6. EEC688/788: Secure & Dependable Computing
Active Replication
Actively Replicated
Client Object A
Actively Replicated
Server Object B
Duplicate
Invocation
Suppressed
Duplicate
Responses
Suppressed RM RM RM RM RM
1/16/2019
EEC688/788: Secure & Dependable Computing

7. Active Replication with Voting
Question: to cope with f number of faults (non-malicious), how many replicas are needed?
1/16/2019
EEC688/788: Secure & Dependable Computing

8. EEC688/788: Secure & Dependable Computing
Passive Replication
Passively Replicated
Client Object A
Passively Replicated
Server Object B
Primary
Replica
Primary
Replica
Response
Invocation
State Transfer
State RM RM RM RM RM
Question: can passive replication tolerate non-fail-stop faults?
1/16/2019
EEC688/788: Secure & Dependable Computing

9. Semi-Active Replication
Semi-Actively Replicated
Client Object A
Semi-Actively Replicated
Server Object B
Primary
Replica
Primary
Replica
Response
Invocation
Ordering info RM RM RM RM RM
1/16/2019
EEC688/788: Secure & Dependable Computing

10. EEC688/788: Secure & Dependable Computing
Implementation of Service Replication: Ensuring Strong Replica Consistency
For active replication,
use a group communication system or a consensus algorithm that guarantees total ordering of all messages (plus deterministic processing in each replica)
Passive replication with systematic checkpointing
Semi-active replication
Use two-phase commit
1/16/2019
EEC688/788: Secure & Dependable Computing

11. Total Ordering of Messages
What is total ordering of messages?
All replicas receive the same set of messages in the same order
Atomic multicast – If a message is delivered to one replica, it is also delivered to all non-faulty replicas
With replication, we need to ensure total ordering of messages sent by a group of replicas to another group of replicas
FIFO ordering between one sender and a group is not sufficient m1 m2
1/16/2019
EEC688/788: Secure & Dependable Computing

12. Potential Sources of Non-determinisms
Multithreading
The order of accesses of shared data by different threads might not be the same at different replicas
System calls/library calls
A call at one replica might succeed while the same call might fail at another replica. E.g., memory allocation, file access
Host/process specific information
Host name, process id, etc.
Local clocks - gettimeofday()
Interrupts
Delivered and handled asynchronously – big problem
Not required
1/16/2019
EEC688/788: Secure & Dependable Computing

13. Data Replication Transactional data replication
Read/write ops on a set of data items within the scope of a transaction
At the transaction level, executions appear to be sequential (One-copy serializable)
Actual ops on each data item often concurrent
Optimistic data replication
Eventual consistency: eventually, all updates will be propagated to all data items

14. Transactional Data Replication
One-copy serializable
A transactional data replication algorithm should ensure that the replicated data appear to the clients as a single copy
The interleaving of the execution of the transactions be equivalent to a sequential execution of those transactions on a single copy of the data.
Make read ops cheaper than updates: read ops are more prevalent
It is challenging to design sound replication algorithms

15. Wrong Data Replication Algorithms
Write-all
A read op on a data item x can be mapped to any replica of x
Write on x must be applied to all replicas of x
Problem: what if a replica becomes faulty?
Blocking! Any single replica fault => bring down the entire system!

16. Wrong Data Replication Algorithms
Write-all-available
A read op on a data item x can be mapped to any replica of x
Write on x is applied to available replicas of x
Problem: cannot ensure one-copy serializable execution!

17. Attempting to Fix Write-All-Available
Problem caused by accessing the not-fully-recovered replica => how about preventing this?

19. Attempting to Fix Write-All-Available
Still won’t work
Ti does not precedes Tj because Tj reads y before Ti writes to y
Tj does not precedes Ti because Ti reads x before Tj writes to x
Ti: R(x), W(y)
Tj: R(y), W(x)
Hence, Ti and Tj are not serializable!

20. Insight to the Problem
The problem is caused by the fact that conflicting operations are performed at difference replicas
We must prevent this from happening
A solution: use quorum-based consensus
What is a quorum?
Given a system with n processes, a quorum is formed by a subset of the processes in the system
Any two quorums must intersect in at least one process
Read quorum: a quorum formed for read ops
Write quorum: a quorum formed for write ops

21. A Quorum-Based Replication Algorithm
Basic idea:
Write ops apply to a write quorum
Read ops apply to a read quorum
Fault tolerance: given total number replicas N and write quorum size W (>= read quorum size R), can tolerate up to N-W failures
Quorum rule
Each replica assigned a positive weight, e.g., 1
A read quorum has a min total weight RT
A write quorum has a min total weight WT
RT+WT > total weight && 2WT > total weight
What if RT=1? WT would include all replicas => not fault tolerant!

22. A Quorum-Based Replication Algorithm
Since update is applied to a quorum of replicas, we need to track which replica has the latest value => use version numbers
Version number is incremented after each update
Read rule
A read on data x is mapped to a read quorum replicas of x
Each replica returns both the value of x and its version number
The client select the value that has the highest version number

23. A Quorum-Based Replication Algorithm
Write rule
A write op on data x is mapped to a write quorum replicas of x
First, retrieve version numbers from the replicas, set v=vmax+1 for this write op
Write to the replicas (in the write quorum) with new value and version # v. A replica overwrites both the value and version number v

24. Quorum-Based Replication Algorithm: Example

25. Exercise 1: Quorum-based data replication
Consider the following replicas from R1 to R5 with the (Value, Version number) sets respectively as R1 (0,0), R2 (0,0), R3 (0,0), R4 (0,0), R5 (0,0). The following sequence of read/write operations between replicas are performed as follows
Read operation on R1,R2,R3
Write operation on R3, R4, R5 with a value 2
Read operation on R4, R3, R2
Write operation on R4, R5, R1 with a value 5
Write operation on R2,R4,R5 with a value 7
Give the final values from R1 to R5 after all the above operations are performed?

26. Exercise 2: Quorum-based data replication
Data replication. Assume we have a distributed system with 6 replicas.
If the read quorum has size of 3, what is the minimum size of write quorum?
Assume a read quorum consists of 2 replicas. A read operation on data x is mapped to two replicas with one replica has value 2 and version number 1, and the other replica has value 3 and version number 2. Which value will be selected?