Exploiting Distributed Version Concurrency in a Transactional Memory Cluster Kaloian Manassiev, Madalin Mihailescu and Cristiana Amza University of Toronto, Canada

Transactional Memory Programming Paradigm Each thread executing a parallel region:  Announces start of a transaction  Executes operations on shared objects  Attempts to commit the transaction If no data race, commit succeeds, operations take effect Otherwise commit fails, operations discarded, transaction restarted  Simpler than locking!

Transactional Memory  Used in multiprocessor platforms  Our work: the first TM implementation on a cluster Supports both SQL and parallel scientific applications (C++)

TM in a Multiprocessor Node  Multiple physical copies of data  High memory overhead A Copy of A T1: Read(A) T2: Write(A) T1: Active T2: Active

TM on a Cluster Key Idea 1. Distributed Versions  Different versions of data arise naturally in a cluster  Create new version on different node, others read own versions writeread

Exploiting Distributed Page Versions mem0 txn0 mem1 txn1 mem2 txn2 memN txnN network... Distributed Transactional Memory (DTM) v3v2v1v0

Key Idea 2: Concurrent “Snapshots” Inside Each Node read v1 v2 Txn0 (v1) Txn1 (v2)

Key Idea 2: Concurrent “Snapshots” Inside Each Node read v1 v2 Txn0 (v1) Txn1 (v2) v1 v2

Key Idea 2: Concurrent “Snapshots” Inside Each Node read v1 v2 Txn0 (v1) Txn1 (v2) v1 v2

Distributed Transactional Memory A novel fine-grained distributed concurrency control algorithm Low memory overhead Exploits distributed versions Supports multithreading within the node Provides 1-copy serializability

Outline  Programming Interface  Design Data access tracking Data replication Conflict resolution  Experiments  Related work and Conclusions

Programming Interface  init_transactions()  begin_transaction()  allocate_dtmemory()  commit_transaction()  Need to declare TM variables explicitly

Data Access Tracking  DTM traps reads and writes to shared memory by either one of:  Virtual memory protection Classic page-level memory protection technique  Operator overloading in C++ Trapping reads: conversion operator Trapping writes: assignment ops (=, +=, …) & increment/decrement(++/--)

Data Replication …… Page 1 Page 2 Page n T1(UPDATE) …… Page 1 Page 2 Page n

Twin Creation …… Page 1 Page 2 Page n T1(UPDATE) …… Page 1 Page 2 Page n Wr p1 P1 Twin

Twin Creation …… Page 1 Page 2 Page n T1(UPDATE) …… Page 1 Page 2 Page n Wr p2 P1 Twin P2 Twin

Diff Creation …… Page 1 Page 2 Page n T1(UPDATE) …… Page 1 Page 2 Page n

Broadcast of the Modifications at Commit …… Page 1 Page 2 Page n T1(UPDATE) …… Page 1 Page 2 Page n Diff broadcast (vers 8) Latest Version = 7 v2v1

Other Nodes Enqueue Diffs …… Page 1 Page 2 Page n T1(UPDATE) …… Page 1 Page 2 Page n Diff broadcast (vers 8) v2v1v8 v1 Latest Version = 7

Update Latest Version …… Page 1 Page 2 Page n T1(UPDATE) …… Page 1 Page 2 Page n v2v1v8 v1 Latest Version = 7Latest Version = 8

Other Nodes Acknowledge Receipt …… Page 1 Page 2 Page n T1(UPDATE) …… Page 1 Page 2 Page n v2v1 v8v1 Ack (vers 8) v8 Latest Version = 7Latest Version = 8

T1 Commits …… Page 1 Page 2 Page n T1(UPDATE) …… Page 1 Page 2 Page n v2v1 v8v1 v8 Latest Version = 8

Lazy Diff Application... Page 1 V0 Page 2 V0 V8V1 Page N V3 V5V4 T2(V2): Rd(…, P1, P2) Latest Version = 8 V2V1V8

Lazy Diff Application... Page 1 Page 2 V0 Page N V3 V5V4 V8 V2 V8V1 T2(V2): Rd(…, P1, P2) Latest Version = 8

Lazy Diff Application... Page 1 V2 V8 Page 2 V1 V8 Page N V3 V5V4 T2(V2): Rd(…, P1, P2) Latest Version = 8

Lazy Diff Application... Page 1 V2 V8 Page 2 V1 V8 Page N V3 V5V4 T3(V8): Rd(PN) T2(V2): Rd(…, P1, P2) Latest Version = 8

Lazy Diff Application... Page 1 V2 V8 Page 2 V1 V8 Page N V5 T3(V8): Rd(PN) T2(V2): Rd(…, P1, P2) Latest Version = 8

Waiting Due to Conflict T3(V8): Rd(PN, P2)... Page 1 V2 V8 Page 2 V1 V8 Page N V5 T2(V2): Rd(…, P1, P2) Wait until T2 commits Latest Version = 8

Transaction Abort Due to Conflict... Page 1 Page 2 V0 Page N V3 V5V4 V8 V2 V8V1 T3(V8): Rd(P2) T2(V2): Rd(…, P1, P2) Latest Version = 8

Transaction Abort Due to Conflict... Page 1 Page 2 V8 Page N V3 V5V4 V8 V2 T3(V8): Rd(P2) CONFLICT! T2(V2): Rd(…, P1, P2) Latest Version = 8

Write-Write Conflict Resolution  Can be done in two ways Executing all updates on a master node, which enforces serialization order OR Aborting the local update transaction upon receiving a conflicting diff flush  More on this in the paper

Experimental Platform  Cluster of Dual AMD Athlon Computers 512 MB RAM 1.5GHz CPUs RedHat Fedora Linux OS

Benchmarks for Experiments  TPC-W e-commerce benchmark Models an on-line book store Industry-standard workload mixes  Browsing (5% updates)  Shopping (20% updates)  Ordering (50% updates) Database size of ~600MB  Hash-table micro-benchmark (in paper)

Application of DTM for E-Commerce

We use a Transactional Memory Cluster as the DB Tier

Cluster Architecture

Implementation Details  We use MySQL’s in-memory HEAP tables RB-Tree main-memory index No transactional properties  Provided by inserting TM calls  Multiple threads running on each node

Baseline for Comparison  State-of-the-art Conflict-aware protocol for scaling e-commerce on clusters Coarse grained (per-table) concurrency control (USITS’03, Middleware’03)

Throughput Scaling

Fraction of Aborted Transactions # of slavesOrderingShoppingBrowsing 11.15%1.44%0.63% 20.35%2.27%1.34% 40.07%1.70%2.37% 60.02%0.41%2.07% 80.00%0.22%1.59%

Comparison (browsing)

Comparison (shopping)

Comparison (ordering)

Related Work  Distributed concurrency control for database applications Postgres-R(SI), Wu and Kemme (ICDE’05) Ganymed, Plattner and Alonso (Middleware’04)  Distributed object stores Argus (’83), QuickStore (’94), OOPSLA’03  Distributed Shared Memory TreadMarks, Keleher et al. (USENIX’94) Tang et al. (IPDPS’04)

Conclusions  New software-only transactional memory scheme on a cluster Both strong consistency and scaling  Fine-grained distributed concurrency control Exploits distributed versions, low memory overheads  Improved throughput scaling for e- commerce web sites

Questions?

Backup slides

Example Program #include typedef struct Point { dtm_int x; dtm_int y; } Point; init_transactions(); for (int i = 0; i < 10; i++) { begin_transaction(); Point * p = allocate_dtmemory(); p->x = rand(); p->y = rand(); commit_transaction(); }

Query weights

Decreasing the fraction of aborts

Micro benchmark experiments

Micro benchmark experiments (with read-only optimization)

Fraction of aborts # of machines % aborts