1. Fine-Grained Replication and Scheduling with Freshness and Correctness Guarantees F.Akal 1, C.Türker 1, H.-J.Schek 1, Y.Breitbart 2, T.Grabs 3, L.Veen 4 1 ETH Zurich, Institute of Information Systems, 8092 Zurich, Switzerland, {akal,tuerker,schek}@inf.ethz.ch 2 Kent State University, Department of Computer Science, Kent OH 44240, USA, yuri@cs.kent.edu 3 One Microsoft Way, Redmond, WA 98052, USA, grabs@acm.org 4 University of Twente, 7500 AE Enschede, The Netherlands, lourens@rainbowdesert.net This work was supported partially by Microsoft 31 st International Conference on Very Large Data Bases, Trondheim, Norway, 30 August – 2 September, 2005.

2. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 2 Overview  Introduction and Motivation  Replication in a Database Cluster  Need for a New Replication Scheme  PowerDB Replication (PDBREP)  Overview of The PDBREP Protocol  Freshness Locking  Experimental Evaluations  Conclusions

3. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 3 Introduction  Replication is an essential technique to improve performance of reads when writes are rare  Different approaches have been studied so far  eager replication -synchronization within the same transaction -conventional protocols have drawbacks regarding performance and scalability Newer protocols reduce these drawbacks by using group communication  lazy replication -decoupled replica maintenance -additional efforts are necessary to guarantee serializable executions -older works focused on performance and correctness, freshness of data was not considered enough  Recently, coordinated replication management proposed within PowerDB project at ETH Zürich addresses freshness issues

4. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 4 The PowerDB Approach  Cluster of databases  Cluster of off-the-shelf PCs  Each PC runs a commercially available RDBMS  Fast Ethernet Connection (100 Mbit/s)  Middleware Architecture  Clients access the cluster over the middleware only  Distinguished cluster into two parts  Lazy replication management -Eager from user‘s perspective  The “Scale-out” vision  Adding new nodes for higher performance  More nodes allow to increase parallelism Coordination Middleware Clients Cluster of DBs OLTP OLAP Update and Read-only Transactions

5. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 5 The Early PowerDB Approach to Replication : FAS (Freshness Aware Scheduling)  Relies on full replication  Read-only transactions execute where they are initiated  Users may specify their freshness needs  How much the accessed data may deviate from up- to-date data  Freshness at the database level -Locks entire database  Read-only sites are maintained by means of decoupled refresh transactions, e.g., on- demand refreshment a,b,c,d Update Sites Update Transactions Read-only Transactions a,b,c,d T2T2 r(b)r(a) T3T3 r(d)r(c) a,b,c,d T1T1 w(a) Read-only Sites PowerDB Middleware „I am fine with 2 minutes old data“ „I want fresh data“ Decoupled Refresh Transactions

6. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 6 Systems Are Evolving… So Is PowerDB…  Customized node groups for certain query types  Requires support for arbitrary physical data design  Read-only transactions may span over many nodes providing query parallelizm  Users may still specify their freshness needs  Freshness at the database object level -Fine-grained locking  Read-only transactions should be served fast even with the higher update rates and freshness requirements  Read-only sites must be kept as up-to-date as possible d Update Sites Update Transactions Read-only Transactions a,b,db,da,c a,b,c,d T1T1 w(a) a,b,c,d T2T2 r(b)r(a) T3T3 r(d)r(b) PowerDB Middleware Continuous Update Propagation to Read-only Sites Read-only Sites

7. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 7 Why There Is Need For A New Replication Protocol?  Distributed executions of read-only transactions might cause non- serializable global schedules  Continuous update propagation must be coordinated with read-only transaction execution  Having arbitrary physical layouts and extending locking to finer granules require more effort to maintain replicas and to execute read-only transactions  Sophisticated replication mechanism is required...

8. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 8 Overview of The PDBREP Protocol a,c c,da,b,dc,d a,b all changes are serialized in a global log Site s 1 Site s 2 Site s 3 Site s 4 Site s 5 T1T1 T3T3 w(d) update transactions r(c) w(a)w(b) r(a) T2T2 w(c) T G3 : SN: 031013 OP: w(d) T G2 : SN: 031012 OP: w(c) T G1 : SN: 031011 OP: w(a) w(b) SN: 031013 v[a]: 6 v[b]: 8 v[c]: 4 v[d]: 2 local propagation queue SN: 031011 v[a]: 6 v[b]: 8 v[c]: 3 v[d]: 1 SN: 031011 v[a]: 6 v[b]: 8 v[c]: 3 v[d]: 1 SN: 031010 v[a]: 5 v[b]: 7 v[c]: 3 v[d]: 1 T L1 Update Sites Read- only Sites Broadcast to Read-only Sites T L2 T L1 T L2 T L1 T L2 Update Counter Vector Global Counter Vector read-only transactions T4T4 r(a) w(a)w(b) T L1 T5T5 w(a)w(b) T L1 T6T6 Propagation transactions Localized log records are applied to the site by using propagation transactions when that site is idle Continuous broadcasting and update propagation keep the site as up-to-date as possible Global log records are continuously being broadcast to read-only sites They are enqueued in the local propagation queues in their serialization order System does not allow propagation transactions to overwrite versions needed by read-only transactions Freshness Locks

9. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 9 Overview of The PDBREP Protocol a,c c,da,b,dc,d a,b all changes are serialized in a global log Site s 1 Site s 2 Site s 3 Site s 4 Site s 5 T1T1 T3T3 w(d) update transactions r(c) w(a)w(b) r(a) T2T2 w(c) T G3 : SN: 031013 OP: w(d) T G2 : SN: 031012 OP: w(c) T G1 : SN: 031011 OP: w(a) w(b) SN: 031013 v[a]: 6 v[b]: 8 v[c]: 4 v[d]: 2 local propagation queue SN: 031011 v[a]: 6 v[b]: 8 v[c]: 3 v[d]: 1 SN: 031011 v[a]: 6 v[b]: 8 v[c]: 3 v[d]: 1 SN: 031010 v[a]: 5 v[b]: 7 v[c]: 3 v[d]: 1 T L2 Update Sites Read- only Sites Broadcast to Read-only Sites T L2 T L1 T L2 T4T4 r(a) read-only transactions Update Counter Vector Global Counter Vector T8T8 r(d) r(a) fresh data required T7T7 r(d) r(c) Freshness is not explicitly specified SN: 031013 v[a]: 6 v[b]: 8 v[c]: 4 v[d]: 2 SN: 031013 v[a]: 6 v[b]: 8 v[c]: 4 v[d]: 2 w(c)w(d) T 10 T L2 T G3 Refresh Transactions w(c)w(d) T9T9 T L2 T G3 To ensure correct executions, each read-only transaction determines the version of the objects it reads at its start

10. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 10 Freshness Locks Propagation While propagation transactions continue, read- only transaction T1 arrives. Then, propagation transactions stop. A refresh transaction is invoked. Propagation stops T1 may continue When T1 commits, it releases its freshness locks. T2 causes another refresh to be invoked. T1 commits Propagation stops T2 may continue T2 commits Propagation stops T3 may continue Data Required Timestamp T1T1 T2T2 T3T3 Propagation Tx Refresh Tx  Freshness locks are placed on the objects to ensure that ongoing replica maintenance transactions do not overwrite versions needed by ongoing read-only transactions  Freshness locks keep the objects accessed by read-only transaction at a certain freshness level during the execution of that transaction

11. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 11 Current freshness freshness lock (request) Scheduling a Read-only Transaction Required Timestamp Data b a T1T1 T1T1 T2T2 T2T2 T3T3 T3T3 1 2 3 4 5 6 7 8 T1 (r1(a), r1(b), TS=1) Sites are older T2 (r2(a), r2(b), TS=3) Both sites younger b´s freshness lock upgraded TS becomes 5 T3 (r3(a), r3(b), TS=7) There are younger site b´s lock upgraded TS becomes 8

12. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 12 Experimental Evaluations  Investigating the influence of continuous update broadcasting and propagations on cluster performance. We considered…  three different … -settings : There are two basic options that we can switch on or off 1.No-Broadcasting and No-Propagation 2.Broadcasting and No-Propagation 3.Broadcasting and Propagation -workloads : 50%, 75% and 100% loaded clusters 50% loaded cluster means that the cluster is busy with evaluating queries for the half of the experiment duration  freshness -Five freshness levels: 0.6, 0.7, 0,8, 0.9, 1.0, e.g., 1.0 means the freshest -Freshness window of 30 seconds, e.g., 0.0 means 30 second old data  Looking at the scalability of PDBREP  Comparing PDBREP to its predecessor (FAS)

13. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 13 Experimental Setup  Cluster of 64 PCs  1GHz Pentium III, 256 MB RAM, 2 SCSI Discs, 100MBit Ethernet  SQL Server 2000 running under Windows 2000 Advanced Server  TPCR-R Database with scale factor 1  ~4.3G together with indexes, 200 updates per second  Node Groups of 4 nodes  Small tables are fully-replicated, the huge ones are partitioned (over order_key) within the NGs.

14. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 14 Average Query Evaluation Times for Different Workloads and Freshness Values Turning on propagation and/or broadcasting always improves the performance. The lower the workload is, the higher the gain in perfomance becomes, e.g., improvement is 82% for 50% loaded cluster.

15. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 15 Average Refresh Transaction Size for Different Workloads and Freshness Values Propagation eliminates the need for refresh transactions except for the maximum freshness requirements and workloads. This results in query execution times practically independent of the overall workload for the given update rate. For fully loaded cluster, there is simply no time for propagations except at the beginning and end of transactions, which results in small performance improvement.

16. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 16 Scalability of PDBREP : Query Throughput for Varying Cluster Sizes PDBREP scales up with the increasing cluster size (Above chart shows the scalability for 50% loaded cluster) The results for freshness 0.9 and below are virtually identical due to local refresh transactions

17. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 17 PDBREP vs. FAS (freshness aware scheduling) : Relative Query Throughput For all three workloads, PDBREP performs significantly better than FAS (30%, 72% and 125%) PDBREP uses partitioning of data while FAS relies on full replication, which results in small refresh transactions PDBREP allows distributed executions and gains from parallelization

18. September 1 st, 2005 Fuat Akal, ETH Zürich, akal@inf.ethz.ch 18 Conclusions  PDBREP respects user demanded freshness requirements and extends the notion of freshness to finer granules of data  PDBREP requires less refresh effort to serve queries due to continuous propagations of updates  PDBREP allows distributed executions of read-only transactions and produces globally correct schedules  PDBREP supports different physical data organization schemes  PDBREP scales even with higher update rates