Database Replication in WAN Yi Lin McGill University Distributed Information Systems

What? Why? How? …… Montreal TorontoOttawa Toronto Montreal Ottawa Without ReplicationWith Replication Benefits: Performance, Scalability, Failover How?

How? Replication Protocols based on Group Communication Group Communication: Each group member can multicast messages to all members which will receive the messages in specified order such as –FIFO: messages of one sender are received in the order as they were sent. –TOTAL: if two members receive messages m1 and m2, both receive them in the same order. FIFO m1 m2 m1 m2 FIFO, not TOTAL m1 m2 m1 m2 m3 FIFO, TOTAL m1 m2 m1 m2 m3

Existing Replication Protocol for LAN Basic idea: –1. Upon receiving a read request from the client, submit it to execute immediately –2. Upon receiving a update request from the client: multicast the request to all sites in TOTAL order –3. Upon receiving an update request in TOTAL order: put the necessary lock requests in a locktable. –4. submit the transactions for execution once locks are granted

Example: 2 txns accessing X in 2 sites T2 Resp T1T2 T1 Resp X: T1 Locktable X: T1, T2 X:T2 multicast in TOTAL order Site ASite B X: T1 X: T1, T2 X:T2

Replication Protocol Variations T2 Resp T1 T2 T1 exe T2 exe T1 Resp Apply ws1 Apply ws2T2 Resp (a) Symmetric(b) Primary copy(c) Local copy Def Each site executes entire transactions Txn is executed in master site and its writeset is multicast to other sites to be applied Txn is executed in local submitted site and its writeset is multicast to other sites to be applied Pro s Simple, 1 msg/txnCPU lightweight, Load balance, Deterministic Fast local response with large data Con s CPU costly. Cann’t nonDeterministic 2 msg/txn, wait for its own writeset from primary 2 msg/txn, might need to wait for remote writesets T1 T2 T1 exe T2 exeApply ws1 Apply ws2 T1 T2 T1 exe T2 exe T2 Resp

Lesson 1 learned from experiments in Planetlab(WAN) Message delivery time accounts for 70-80% of txn response time whereas execution time only 5-10% Solution: a. Symmetric protocol preferred in WAN b. Propose a fast TOTAL order algorithm (Local Token) Msg delivery time Txn Exe time Waiting in locktable time Resp time

LAN prefers primary copy WAN prefers Symmetric LANWAN Resp Time Primary copy <local copy <symmetric Symmetric<local copy < primary copy Reasons1.Applying write set(e.g 10ms) is faster than executing the whole txn(e.g. 15ms). 2.Msg delivery time in LAN is very fast(e.g. 3-4 ms). Main part of response time is attributed to execution time. 3.Primary copy is good in load balancing 1.Msg delivery time(e.g. 100ms) in WAN is very slow comparing to execution time (e.g. 15ms). 2.Symmetric approach only requires one msg delay within txn boundary 3.Primary copy has up to two msgs delay within txn boundary. 4.If with no conflict to remote txns local copy has only 1 msg delay.

Local Token TOTAL order In each site there is one FIFO queue for one sender. All queues compose a ring (same ring configuration in each site). At any time one queue has a token. Upon receiving msg, append it in the corresponding queue. Upon holding token, deliver first msg (blocked if no msg) and pass the token to the next queue. Comparing Local Token with other TOTAL order algorithm (4 planetlab sites )

Lesson 2 learned from experiments Locking granularity in locktable has significant influence on performance. Table Level locking –In middleware level, since tuples to be accessed by txns are possibly unknown (if primary key is not provided), the reasonable locking mechanism is locking the tables to be accessed. Solution: –Optimistic delivery –Pseudo-tuple level locking

Optimistic Delivery In WAN a msg may be physically received much earlier than when its TOTAL order is determined. We can optimistically start execution (appending txn to locktable) upon receiving the msg instead of upon msg’s TOTAL order is determined. However txn will not be committed until its TOTAL order is determined. If there is no difference between optimistic delivery and TOTAL order delivery, or mismatch happens with non-conflicting txns, execution overlaps with determining the TOTAL order. Otherwise, txn is aborted and rollbacked. Total Order Txn Execution Response Time Time Total Order Optimistic Txn Execution Response Time (a)Regular delivery Optimistic-delivery TOTAL-delivery (b) With Optimistic delivery

Pseudo-Tuple Level locking DBMS serializable isolation level provides: T1 T2 x If two concurrent txns access the same data, the later txn will be blocked by the first txn and aborted upon the first txn’s commit. Table-level locking may lead to unnecessary blocking. We can take advantage of serializable isolation level to detect if two txns conflict at tuple level by optimistically applying both txns and checking if second txn will be blocked by the previous txn. This approach in conjunction with optimistic delivery may be able to detect tuple level conflict before txn TOTAL order is determined. x Executed blocked T1 commits T2 abort time