A Log(n) Multi-Mode Locking Protocol for Distributed Systems Nirmit Desai, Frank Mueller mueller@cs.ncsu.edu Department of Computer Science North Carolina State University

Background Problem Motivation Approach Contribution Large parallel and distributed systems Scalability of synchronization Motivation Increasing need to share resources Inadequate scalability of current protocols Approach Leveraging hierarchical locking in distributed synchronization Contribution Highly scalable protocol Supports hierarchical locking May 6, 2019 NC State University

Large Distributed Environments Cluster Environments Shared Resource 1 Shared Resource 2 Site 1 Site 2 Site 3 Network Site N May 6, 2019 NC State University

Distributed Synchronization (Naimi et al.) Request from A Request from C T is done Single locking mode - Read and Write are not distinguished Single level of locking granularity (either entire table or a tuple) May 6, 2019 NC State University

Hierarchical Locking (a.k.a. Multi-granularity Locking) How is a lock associated with data items ? Data Fine grain Data Coarse grain Data Multi grain - High overhead - High concurreny - Low overhead - No concurrency - Low overhead - High concurrency May 6, 2019 NC State University

Multi-Mode Locking Protocol 1. Intention Read (IR) /Intention Write (IW) on all upper level resources 2. Read (R) locks and Write (W) locks on the resource itself A : Reads tuple 2 t0:Acquire IR on the table t1:Acquire R on tuple #2 t2:Read tuple #2 t3:Release R t4:Release IR Data table B : Writes tuple 3 t0:Acquire IW on the table t1:Acquire W on tuple #3 t2:Write tuple #3 t3:Release W t4:Release IW May 6, 2019 NC State University

Multi-Mode Locking Protocol (cont.) Modes incompatible?  Conflict  Block A: reads entire table t0:Acquire R on the table t1:Read table t2:Release R on the table Data table B : Writes a tuple t0:Request IW on the table t1: wait for lock t2:Acquires IW t3:Acquire W on tuple t4:... X May 6, 2019 NC State University

IR < R < U = IW < W Compatibility Matrix ‘X’ indicates a conflict Upgrade: Read followed by write to same data assures consistent data Lock Mode Strength: Less compatible  Stronger lock mode IR < R < U = IW < W May 6, 2019 NC State University

Our Protocol - Conventions Held Mode: Mh = Mode of held lock Owned Mode: Mo = Strongest held mode in tree (rooted=owner) Pending Mode: Mp = await this mode Node state: (Mo, Mh, Mp) Request arrives  test compatibility  grant/queue/forward based on result Actions: rules & transition tables Requester becomes a child of granter A(0,0,0) C(IR,IR,0) D(IR,IR,0) B(R,R,0) E(U,U,0) May 6, 2019 NC State University

A Granting/Forwarding Example Invariant : “Node cannot grant request for stronger mode than it owns”  has to forward it to parent A(0,0,0) C(0,0,0) D(0,0,IR) B(0,0,0) E(IR,IR,0) A(0,0,R) C(0,0,0) D(IR,IR,0) B(0,0,0) E(IR,IR,0) A(R,R,0) C(0,0,0) D(IR,IR,0) B(0,0,0) E(IR,IR,0) A requests R Token transfer (D,IR) (A,R) Grant (A,R) If Mr compatible with Mo  Mr compatible w/ all modes in subtree May 6, 2019 NC State University

A Release Example Invariant: “Node knows owned mode of its children” C(IR,IR,0) D(IR,IR,0) B(IR,IR,0) E(R,R,0) A(0,0,0) C(IR,IR,0) D(IR,IR,0) B(IR,0,0) E(R,R,0) B(IR,0,0) A(0,0,0) C(0,0,0) D(IR,IR,0) E(R,R,0) B releases IR C releases IR (B,IR) B(0,0,0) (C,IR) Notice: If a child releases a mode but owned mode is unchanged, no need to notify our parent May 6, 2019 NC State University

A Queuing Example Invariants: “Queue if request is conflicting. Freeze modes if it is a) incompatible with queued request and b) grantable otherwise” A(0,0,IR) C(0,0,W) D(R,R,0) B(IR,IR,0) E(R,R,0) IR IR, R (C,W) IR, R, U A(0,0,0) C(0,0,W) D(R,R,0) B(IR,IR,0) E(R,R,0) IR, R, U A(0,0,0) C(0,0,W) D(R,R,0) B(IR,IR,0) E(R,R,0) (C,W) IR, R, U Send freeze A requests IR (C,W) (C,W)(A,IR) Freeze (IR) (IR, R) (C,W) (A,IR) IR IR, R (C,W) (A,IR) Starvation of queued request is avoided, FIFO is ensured May 6, 2019 NC State University

Measurements Metrics of interest: Scalability of Message overhead per lock request Request latency time Effect of changing concurrency level on scalability higher ratio  lower concurency Compare with Naimi’s protocol Coarse: Lock the entire table Fine: Lock each entry Multi-airline reservation system IBM SP Power3 May 6, 2019 NC State University

Comparison with Naimi’s protocols Message overhead per Request Response time per Request Naimi coarse has different functionality May 6, 2019 NC State University

Effect of concurrency level Concurrency level increases as # of nodes increase May 6, 2019 NC State University

Applications State management of replicated data Distributed databases (Oracle, MS-SQL) Distributed transaction processing Middleware concurrency services (CORBA) Coordinated, distributed Server Farms Fault tolerance in clusters State replication for redundant computing Distributed agreement State coherence May 6, 2019 NC State University

Related Work Naimi et. al. (JPDC ‘96) Ramamritham et. Al. (SIGMOD ‘90) Distributed mutual exclusion, single level of granularity Ramamritham et. Al. (SIGMOD ‘90) Concurrency protocols, centralized transaction coordinator Other approaches: complexity > O(log (n)) Hierarchical locking: only in context of distributed synchronization problems in past Scalability: had not been the focus May 6, 2019 NC State University

Conclusions Novel peer-to-peer distributed concurrency protocol Compact formulation: rules and tables  simplicity intriguing Highly scalable: O(log n) msgs, asymptote 3-5 msgs High degree of concurrency common Scalability unaffected by concurrency level Response time: linear increasing w/ concurrency level Best scalable protocol for hierarchical locking known Wide applications in parallel and distributed world May 6, 2019 NC State University

Breakup of Messages Request propagation is major component C May 6, 2019 NC State University

Request Message Concurrency level Request forwarding Request Nodes Tree height Path length Request more request msgs propagation paths longer May 6, 2019 NC State University

Token Message Concurrency level Granting Token transfers Nodes Tree height Granting Token transfers more contention lower chance to grant token May 6, 2019 NC State University

Grant Message Concurrency level Granting Nodes Tree height Granting more contention more child grants May 6, 2019 NC State University

Release Message Concurrency level Granting Release Nodes Tree height Granting Release higher concurrency more trees & release msgs May 6, 2019 NC State University

Freeze Message Concurrency level Conflicts Freeze Nodes Requests Conflicts Freeze higher concurrency more freezes (be fair) May 6, 2019 NC State University

Rule for Granting at Non-root X = cannot grant May 6, 2019 NC State University

Rule for Queuing/Forwarding Queue May 6, 2019 NC State University

Rule for Freezing Modes Modes to freeze May 6, 2019 NC State University