1. © 2016 A. Haeberlen, Z. Ives CIS 455/555: Internet and Web Systems 1 University of Pennsylvania Distributed transactions April 11, 2016

2. © 2016 A. Haeberlen, Z. Ives Announcements Project plan is due today 2-page PDF that describes your project architecture (components, interfaces, interaction between them) Must describe division of labor (who does what) and project milestones (what will be completed when) Second midterm: April 27 (Location TBA) Reading for today: Tanenbaum chapters 8.5-8.6 2 University of Pennsylvania

3. © 2016 A. Haeberlen, Z. Ives Plan for today Transactions ACID properties Serializability Concurrency control: 2PL Log-based recovery: ARIES Distributed transactions: 2PC Distributed concurrency control 3 University of Pennsylvania NEXT

4. © 2016 A. Haeberlen, Z. Ives 4 We need more than synchronization What needs to happen when you… Click on “purchase” on Amazon? Suppose you purchased by credit card? Use online bill-paying services from your bank? Place a bid in an eBay-like auction system? Order music from iTunes? What if your connection drops in the middle of downloading? Is this more than a case of making a simple Web Service (-like) call?

5. © 2016 A. Haeberlen, Z. Ives 5 A problem confronted by eBay eBay wants to sell an item to: The highest bidder, once the auction is over, or The person who’s first to click “Buy It Now!” But: What if the bidder doesn’t have the cash? A solution: Tentatively record the item as sold Validate the PayPal or credit card info with a 3 rd party If not valid, discard this bidder and resume in prior state

6. © 2016 A. Haeberlen, Z. Ives 6 “No Payment” isn’t the only source of failure Suppose we start to transfer the money, but a server goes down… Purchase: sb = Seller.bal bb = Buyer.bal Write Buyer.bal= bb - $100 Write Item.sellTo = Buyer Write Seller.bal= sb + $100 CRASH!

7. © 2016 A. Haeberlen, Z. Ives 7 Transactions There are many (especially, financial) applications where we want to create atomic operations that either commit or roll back... despite hardware/software/application/other failures... whether or not other operations are executed concurrently This is one of the most basic services provided by database management systems, but we want to do it in a broader sense Part of “ACID” semantics…

8. © 2016 A. Haeberlen, Z. Ives 8 ACID Semantics Atomicity: operations are atomic, either committing or aborting as a single entity Consistency: the state of the data is internally consistent Isolation: all operations act as if they were run by themselves Durability: all writes stay persistent! What would a violation of each property look like?

9. © 2016 A. Haeberlen, Z. Ives 9 Providing Atomicity and Consistency Database systems provide transactions with the ability to abort a transaction upon some failure condition Based on transaction logging – record all operations and undo them as necessary Database systems also use the log to perform recovery from crashes Undo all of the steps in a partially-complete transaction Then redo them in their entirety This is part of a protocol called ARIES These can be the basis of persistent storage, and we can use middleware like J2EE to build distributed transactions with the ability to abort database operations if necessary

10. © 2016 A. Haeberlen, Z. Ives Plan for today Transactions ACID properties Serializability Concurrency control: 2PL Log-based recovery: ARIES Distributed transactions: 2PC Distributed concurrency control 10 University of Pennsylvania NEXT

11. © 2016 A. Haeberlen, Z. Ives Recall: Need for isolation What if more than one transaction runs? Desirable - system throughput is higher! Challenging - need to maintain isolation! Earlier example: Accounting system in a bank Maintains the current balance of each customer's account Customers can transfer money to other customers 11 University of Pennsylvania void transferMoney(customer A, customer B, int amount) { showMessage("Transferring "+amount+" to "+B); int balanceA = getBalance(A); int balanceB = getBalance(B); setBalance(B, balanceB + amount); setBalance(A, balanceA - amount); showMessage("Your new balance: "+(balanceA-amount)); }

12. © 2016 A. Haeberlen, Z. Ives Recall: The bank example What can happen if this code runs concurrently? 12 University of Pennsylvania 1) B=Balance(Bob) 2) A=Balance(Alice) 3) SetBalance(Bob,B+100) 4) SetBalance(Alice,A-100) 1) A=Balance(Alice) 2) B=Balance(Bob) 3) SetBalance(Alice,A+500) 4) SetBalance(Bob,B-500) Alice Bob $100 $500 Time Alice's balance: Bob's balance: 1 2 $200 $800 1 2 3 4 4 3 $200 $900 $700 $900 $700 $300 $100 $300

13. © 2016 A. Haeberlen, Z. Ives What is a 'good' execution? We would like result to be the same as if the transactions had executed in some sequential order (serializability) Similar to sequential consistency, but for transactions How can we achieve this? (remember from earlier) What needs to be changed to support transactions? 13 University of Pennsylvania T1 T3 T6 T2 T4 T5 Core #1: Core #2: Time Single core: T1T2 T3 T4 T5 T6 Same start state Same result Actual execution Hypothetical execution

14. © 2016 A. Haeberlen, Z. Ives Plan for the next two lectures Transactions ACID properties Serializability Concurrency control: 2PL Log-based recovery: ARIES Distributed transactions: 2PC Distributed concurrency control 14 University of Pennsylvania NEXT

15. © 2016 A. Haeberlen, Z. Ives 15 Concurrency control A means of ensuring that transactions are serializable There are many methods, of which we’ll see one Lock-based concurrency control (2-phase locking) Optimistic concurrency control (no locks – based on timestamps) Multiversion CC …

16. © 2016 A. Haeberlen, Z. Ives Lock-based concurrency control Strict Two-phase Locking (Strict 2PL) Protocol: Each transaction must obtain: a S (shared) lock on object before reading an X (exclusive) lock on object before writing An owner of an S lock can upgrade it to X if no one else is holding the lock All locks held by a transaction are released when the transaction completes Locks are handled in a “growing” phase, then a “shrinking” phase (Non-strict) 2PL Variant: Release locks anytime, but cannot acquire locks after releasing any lock. #locks held time Strict #locks held time Non-strict

17. © 2016 A. Haeberlen, Z. Ives 17 Benefits of Strict 2PL Strict 2PL allows only serializable schedules. Additionally, it simplifies transaction aborts (Non-strict) 2PL also allows only serializable schedules, but involves more complex abort processing

18. © 2016 A. Haeberlen, Z. Ives Aborting a transaction If a transaction T i is aborted, all its actions have to be undone Not only that, if T j reads an object last written by T i, T j must be aborted as well! Most systems avoid such cascading aborts by releasing a transaction’s locks only at commit time If T i writes an object, T j can read this only after T i commits Actions are undone by consulting the transaction log mentioned earlier

19. © 2016 A. Haeberlen, Z. Ives A danger with locks: Deadlocks Deadlock: Cycle of transactions waiting for locks to be released by each other A waits for a lock that B holds B waits for a lock that A holds Three ways of dealing with deadlocks: Deadlock prevention Deadlock avoidance Deadlock detection

20. © 2016 A. Haeberlen, Z. Ives Deadlock avoidance Need to break symmetry somehow! When we notice that a cyclic dependency may be abort to form (unsafe!), we abort one of the involved transactions. Idea: Assign priorities based on timestamps Let's say we have two transactions: T old and T young Wait-die approach: If T old wants a lock T young holds, T old waits. If T young wants a lock T old holds, T young dies. Wound-wait approach: If T old wants a lock T young holds, T young dies. If T young wants a lock T old holds, T young waits. Older transactions never have to wait for younger transactions! If a transaction re-starts, make sure it keeps its original timestamp (keeps it from always getting aborted!)

21. © 2016 A. Haeberlen, Z. Ives Plan for the next two lectures Transactions ACID properties Serializability Concurrency control: 2PL Log-based recovery: ARIES Distributed transactions: 2PC 21 University of Pennsylvania NEXT

22. © 2016 A. Haeberlen, Z. Ives Challenge: Atomicity and Durability Recall: We need to ensure that... a transaction is either applied completely, or not at all (A) the effects of a committed transaction are persistent (D) How can we do this? Clearly, for D we need some persistent storage (hard disk?) How about the following: Two copies of the database on disk, plus a file that says which copy is 'current' Execute transactions one at a time: Copy current to non- current, apply to non-current, then update the file to swap current and non-current Is this a good solution? 22 University of Pennsylvania

23. © 2016 A. Haeberlen, Z. Ives STEAL and NOFORCE To achieve good performance, we want to have only one copy of the database on disk keep parts of the database in memory When do changes get written to the disk? One approach: Keep all changes in memory until committed (NOSTEAL) Write all changes to disk before committing (FORCE) Minor problem: How to make sure all changes are written atomically? Is this good for performance? No! We would much rather have STEAL: May write changes to disk before committing NOFORCE: May keep changes in memory after committing 23 University of Pennsylvania

24. © 2016 A. Haeberlen, Z. Ives Problem: How to recover? What if the machine with the database fails? Power outage, OS crash, operator error,... Problem: Database is inconsistent Some changes by uncommitted transactions may have made it to disk already (due to STEAL) Some changes by committed transactions may not yet have made it to disk (due to NOFORCE) Idea: Additionally keep a log of all changes Write each change to the log first and only then to the DB After a crash, we can use this log to recover Use strict 2PL (reason will become clear soon) 24 University of Pennsylvania

25. © 2016 A. Haeberlen, Z. Ives The transaction log The log contains entries for: Starting, committing, and aborting transactions Changes to values in the database (old value + new value) 25 University of Pennsylvania T0: read(A) A := A - 50 write(A) read(B) B := B + 50 write (B) T1: read(C) C := C - 100 write(C) Log:

26. © 2016 A. Haeberlen, Z. Ives Recovery: ARIES After a crash, we can do the following: Step 1: Analysis Read the log from beginning to end to find all pending transactions and all unwritten pages Step 2: Redo Reapply all the updates to unwritten pages. This brings the database to the exact state it was in before the crash happened. Step 3: Undo Using the old values in the log entries, undo all the changes made by uncommitted transactions. (why is this allowed?) This algorithm is called ARIES Used in most modern database systems Some additional wrinkles: Handling crashes during recovery, avoiding the need to read the entire log (checkpointing),... 26 University of Pennsylvania

27. © 2016 A. Haeberlen, Z. Ives A simplified example Suppose we see the following after a crash: 27 University of Pennsylvania 32> 20> 60> 7> 30> 40> 2> B: 80 D: 90 C: 2 A: 30 Log E: 89 Data (on disk)

28. © 2016 A. Haeberlen, Z. Ives 28 Recap: Transactions and Concurrency Control The basic goal was to guarantee ACID properties Transactions and logging provide Atomicity and Consistency Locks ensure Isolation The transaction log (plus RAID, backups, etc.) is also used to ensure Durability So far, we’ve been in the realm of databases – how does this extend to the distributed context?

29. © 2016 A. Haeberlen, Z. Ives Plan for the next two lectures Transactions ACID properties Serializability Concurrency control: 2PL Recovery: ARIES Distributed transactions: 2PC Distributed concurrency control 29 University of Pennsylvania NEXT

30. © 2016 A. Haeberlen, Z. Ives 30 Distributed transactions We generally rely on a middleware layer called application servers, aka TP monitors, to provide transactions across systems Tuxedo, iPlanet, WebSphere, etc. For atomicity, two-phase commit protocol For isolation, need distributed concurrency control DB Transact Server Transact Server Workflow Controller Msg Queue Web Server App Server Client

31. © 2016 A. Haeberlen, Z. Ives Why two-phase commit? Terminology: Site at which a transaction originates is the coordinator Other sites at which it executes are subordinates Naïve implementation: One-phase commit Coordinator sends commit or abort to all the subordinates Each subordinate executes the action and acknowledges Is this a good solution? 31 University of Pennsylvania

32. © 2016 A. Haeberlen, Z. Ives Why one-phase commit fails Problem #1: Subordinate can't independently abort the transaction (e.g., because of a deadlock) Problem #2: Subordinate might crash before receiving the message; partial transaction results are lost 32 University of Pennsylvania But I already aborted! COMMIT OK! Coordinator crashes in the middle ???

33. © 2016 A. Haeberlen, Z. Ives Two-Phase Commit (2PC) Two rounds of communication, initiated by the coordinator: Phase #1: Voting Coordinator sends prepare messages, waits for yes or no votes Phase #2: Decision or termination Coordinator sends commit or rollback messages, waits for acks Any site can decide to abort a transaction!

34. © 2016 A. Haeberlen, Z. Ives 34 Steps in 2PC When a transaction wants to commit: Coordinator sends prepare message to each subordinate Subordinate force-writes an abort or prepare log record and then sends a no (abort) or yes (prepare) message to coordinator Coordinator considers votes: If unanimous yes votes, force-writes a commit log record and sends commit message to all subordinates Else, force-writes abort log record, and sends abort message Subordinates force-write abort/commit log records based on message they get, then send ack message to coordinator Coordinator writes end log record after getting all acks

35. © 2016 A. Haeberlen, Z. Ives 35 Illustration of 2PC CoordinatorSubordinate 1Subordinate 2 force-write begin log entry force-write prepared log entry force-write prepared log entry send “prepare” send “yes” force-write commit log entry send “commit” force-write commit log entry force-write commit log entry send “ack” write end log entry

36. © 2016 A. Haeberlen, Z. Ives Comments on 2PC Every message reflects a decision by the sender; to ensure that this decision survives failures, it is first recorded in the local log All log records for a transaction contain its ID and the coordinator’s ID The coordinator’s abort/commit record also includes IDs of all subordinates (why?) Theorem: There exists no distributed commit protocol that can recover without communicating with other processes, in the presence of multiple failures!

37. © 2016 A. Haeberlen, Z. Ives What if a site fails in the middle? Suppose we find a commit or abort log record for transaction T, but not an end record? Need to redo/undo T If this site is the coordinator for T, keep sending commit / abort msgs to subordinates until acks have been received Suppose we find a prepare log record for transaction T, but not commit/abort? This site is a subordinate for T Repeatedly contact the coordinator to find status of T, then write commit/abort log record; redo/undo T; and write end log record Suppose we don’t find even a prepare record for T? Unilaterally abort and undo T This site may be coordinator! If so, subordinates may send messages and need to also be undone

38. © 2016 A. Haeberlen, Z. Ives Coordinator failure What should a subordinate do when the coordinator fails after it has already voted yes? Problem: Cannot decide whether to commit or abort T until coordinator recovers - T is blocked! Consequences? Suppose all the subordinates know each other? Can be implemented (requires extra overhead in prepare msg) But: They are still blocked, unless one of them voted no, or one of them has already received the coordinator's decision

39. © 2016 A. Haeberlen, Z. Ives Link and remote site failures What should a node do when a remote site does not respond during the commit protocol for transaction T (either because the site failed or the link failed)? If the current site is the coordinator for T, should abort T If the current site is a subordinate, and has not yet voted yes, it should abort T If the current site is a subordinate and has voted yes, it is blocked until the coordinator responds!

40. © 2016 A. Haeberlen, Z. Ives Observations on 2PC Ack msgs used to let coordinator know when it’s done with a transaction; until it receives all acks, it must keep T in the transaction- pending table If the coordinator fails after sending prepare msgs but before writing commit/abort log recs, when it comes back up it aborts the transaction More elaborate solutions available (e.g., 3PC) But 2PC blocking is rare in practice

41. © 2016 A. Haeberlen, Z. Ives Plan for the next two lectures Transactions ACID properties Serializability Concurrency control: 2PL Log-based recovery: ARIES Distributed transactions: 2PC Distributed concurrency control 41 University of Pennsylvania NEXT

42. © 2016 A. Haeberlen, Z. Ives 42 Distributed Concurrency Control What we saw were the steps involved in preserving atomicity and consistency in a distributed fashion Let’s briefly look at distributed isolation (locking)…

43. © 2016 A. Haeberlen, Z. Ives Distributed Locking How do we manage locks across many sites? Centralized: One site does all locking Vulnerable to single site failure Primary Copy: All locking for an object done at the primary copy site for this object Reading requires access to locking site as well as site where the object is stored We’ll see how this is used in PNUTS Fully Distributed: Locking for a copy done at site where the copy is stored Locks at all sites holding the object being written

44. © 2016 A. Haeberlen, Z. Ives Distributed deadlock detection Each site maintains a local waits-for graph A global deadlock might exist even if the local graphs contain no cycles: T1 T2 SITE ASITE BGLOBAL Three solutions:  Centralized (send all local graphs to one site)  Hierarchical (organize sites into a hierarchy and send local graphs to parent in the hierarchy)  Timeout (abort transaction if it waits too long)

45. © 2016 A. Haeberlen, Z. Ives 45 Recap: Transactions and concurrency There are many (especially monetary) transfers that need atomicity and isolation Transactions and concurrency control provide these features Two-phase locking ensures isolation Two-phase commit is a voting scheme for doing distributed commit