CPSC-608 Database Systems Fall 2017 Instructor: Jianer Chen Office: HRBB 315C Phone: 845-4259 Email: chen@cse.tamu.edu Notes #24

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist.

Coping with System Failures Therefore, the logging strategy must be carefully designed to ensure Atomicity and Durability. If you are going to be in the logging business, one of the things that you have to do is to learn about heavy equipment. Robert VanNatta, Logging History of Columbia County We will study three logging strategies: Undo-logging Redo-logging Undo/Redo-logging

Some Drawbacks of Undo-logging <T, commit> reaches disk very late All “real works” of T may have been done much earlier. If a crash occurs before <T, commit> reaches disk, all these done works are wasted. The memory log may need to be flushed often (in order to satisfy U1) This takes disk I/O’s. <T,V1, a1> … <T, commit> New V1 <T, start> <T,V3, a3> <T,V2, a2> <T,V4, a4> New V3 New V4 New V2 The order in which the values reach disk 4

Coping with System Failures Therefore, the logging strategy must be carefully designed to ensure Atomicity and Durability. If you are going to be in the logging business, one of the things that you have to do is to learn about heavy equipment. Robert VanNatta, Logging History of Columbia County We will study three logging strategies: Undo-logging Redo-logging Undo/Redo-logging

Redo-Logging The format of the Redo-log: <T, start> <T, A, anew> …… <T, B, bnew> <T, commit> (or <T, abort>) transaction T starts transaction T wrote on A/B, and A/B had an new value anew/bnew transaction T commits/abort

Undo-logging uses OLD values Redo-Logging The format of the Redo-log: <T, start> <T, A, anew> …… <T, B, bnew> <T, commit> (or <T, abort>) transaction T starts transaction T wrote on A/B, and A/B had an new value anew/bnew transaction T commits/abort Undo-logging uses OLD values

Redo-Logging The format of the Redo-log: Main idea of Redo-logging: <T, start> <T, A, anew> …… <T, B, bnew> <T, commit> (or <T, abort>) transaction T starts transaction T wrote on A/B, and A/B had an new value anew/bnew transaction T commits/abort Main idea of Redo-logging: If the Redo-log record <T, commit> has not reached disk, then no changes made by T have reached disk.

Redo-Logging The format of the Redo-log: Main idea of Redo-logging: <T, start> <T, A, anew> …… <T, B, bnew> <T, commit> (or <T, abort>) transaction T starts Compare with Undo: If <T, commit> reaches disk, then all changes by T have reached disk transaction T wrote on A/B, and A/B had an new value anew/bnew transaction T commits/abort Main idea of Redo-logging: If the Redo-log record <T, commit> has not reached disk, then no changes made by T have reached disk.

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. … Redo-logging rule (Rule R1): The entire Redo-log for T (including <T, commit>) reaches disk before any change by T reaches disk <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. … Redo-logging rule (Rule R1): The entire Redo-log for T (including <T, commit>) reaches disk before any change by T reaches disk <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … Remarks: <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. … Redo-logging rule (Rule R1): The entire Redo-log for T (including <T, commit>) reaches disk before any change by T reaches disk <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … Remarks: 1. Log-record of all results of T reaches disk before T’s costly disk I/O’s; <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. … Redo-logging rule (Rule R1): The entire Redo-log for T (including <T, commit>) reaches disk before any change by T reaches disk <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … Remarks: 1. Log-record of all results of T reaches disk before T’s costly disk I/O’s; 2. One flush-log is sufficient for the entire T-log; <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. … Redo-logging rule (Rule R1): The entire Redo-log for T (including <T, commit>) reaches disk before any change by T reaches disk <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … Remarks: 1. Log-record of all results of T reaches disk before T’s costly disk I/O’s; 2. One flush-log is sufficient for the entire T-log; 3. Drawback: needs main memory space to hold intermediate results <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. Thus, when the system crashes, we can look at disk-log: … <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. Thus, when the system crashes, we can look at disk-log: If we do not see <T, commit>, by R1, we know that none of the changes made by T has reached disk, yet. Thus, we can ignore T; … <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. Thus, when the system crashes, we can look at disk-log: If we do not see <T, commit>, by R1, we know that none of the changes made by T has reached disk, yet. Thus, we can ignore T; If we see <T, commit>, then all records for T have reached disk, although we do not know which also have their corresponding changes reach disk. To ensure atomicity, we must Redo T; … <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. Thus, when the system crashes, we can look at disk-log: If we do not see <T, commit>, by R1, we know that none of the changes made by T has reached disk, yet. Thus, we can ignore T; If we see <T, commit>, then all records for T have reached disk, although we do not know which also have their corresponding changes reach disk. To ensure atomicity, we must Redo T; When Redoing T, for each <T, V, anew> we see, we are unsure if the new value anew of V has reached disk. However, Redoing the action (i.e., assigning the new value anew to V) is always safe. … <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. Thus, when the system crashes, we can look at disk-log: If we do not see <T, commit>, by R1, we know that none of the changes made by T has reached disk, yet. Thus, we can ignore T; If we see <T, commit>, then all records for T have reached disk, although we do not know which also have their corresponding changes reach disk. To ensure atomicity, we must Redo T; When Redoing T, for each <T, V, anew> we see, we are unsure if the new value anew of V has reached disk. However, Redoing the action (i.e., assigning the new value anew to V) is always safe. The Redo should follow the forward order recorded in the log. … <T, start> <T,V1, a1> … <T,V2, a2> <T,V3, a3> … <T,V4, a4> The order in which the values reach disk <T, commit> New V4 … New V1 New V2 … New V3 …

Redo-Logging Algorithms for Redo-logging Log recording Main idea of Redo-logging: If <T, commit> is not in disk, then no changes made by T have reached disk. Algorithms for Redo-logging Log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, anew> in memory-log (no output(A) is allowed yet); 3. Add <T, commit> to memory-log; 4. Flush memory-log to disk; 5. Perform output(A) for disk elements A that were written by T. Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A;

Redo-logging: Example Log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, anew> in memory-log (no output(A) is allowed yet); 3. Add <T, commit> to memory-log; 4. Flush memory-log to disk; 5. Perform output(A) for disk elements A that were written by T. Redo-logging: Example T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU A=10 B=10 C=30 memory disk 21

Redo-logging: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, anew> in memory-log (no output(A) is allowed yet); 3. Add <T, commit> to memory-log; 4. Flush memory-log to disk; 5. Perform output(A) for disk elements A that were written by T. Redo-logging: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=10 B=10 C=30 memory disk 22

Redo-logging: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, anew> in memory-log (no output(A) is allowed yet); 3. Add <T, commit> to memory-log; 4. Flush memory-log to disk; 5. Perform output(A) for disk elements A that were written by T. Redo-logging: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU log A=10 B=10 C=30 A=30 <T, start> <T, A, 30> <T, B, 30> B=30 memory disk 23

Redo-logging: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, anew> in memory-log (no output(A) is allowed yet); 3. Add <T, commit> to memory-log; 4. Flush memory-log to disk; 5. Perform output(A) for disk elements A that were written by T. Redo-logging: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU log A=10 B=10 C=30 A=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> B=30 memory disk 24

Redo-logging: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, anew> in memory-log (no output(A) is allowed yet); 3. Add <T, commit> to memory-log; 4. Flush memory-log to disk; 5. Perform output(A) for disk elements A that were written by T. Redo-logging: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU log A=10 B=10 C=30 A=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> <T, start> <T, A, 30> <T, B, 30> <T, commit> B=30 <T, start> <T, A, 10> <T, B, 10> memory disk 25

Redo-logging: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, anew> in memory-log (no output(A) is allowed yet); 3. Add <T, commit> to memory-log; 4. Flush memory-log to disk; 5. Perform output(A) for disk elements A that were written by T. Redo-logging: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU log A=10 B=10 C=30 A=30 A=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> B=30 B=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> <T, start> <T, A, 10> <T, B, 10> <T, commit> memory disk 26

Redo-logging: Failure Recovery Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery CPU A=10 B=10 C=30 memory disk 27

Redo-logging: Failure Recovery Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=10 B=10 C=30 memory disk 28

crash occurs after this point, Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=30 B=30 C=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> crash occurs after this point, memory disk 29

crash occurs after this point, Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=30 B=30 C=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> crash occurs after this point, memory disk 30

crash occurs at this point, Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=30 B=10 C=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> crash occurs at this point, memory disk 31

crash occurs at this point, Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=30 B=30 C=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> crash occurs at this point, memory disk 32

crash occurs at this point, Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=10 B=10 C=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> crash occurs at this point, memory disk 33

crash occurs at this point, Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=30 B=30 C=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> crash occurs at this point, memory disk 34

crash occurs before <T, commit> reaches disk Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=10 B=10 C=30 crash occurs before <T, commit> reaches disk memory disk 35

crash occurs before <T, commit> reaches disk Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU log A=10 B=10 C=30 <T, start> <T, A, 30> <T, B, 30> <T, commit> <T, start> <T, A, 30> <T, B, 30> crash occurs before <T, commit> reaches disk memory disk 36

crash occurs before <T, commit> reaches disk Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU log A=10 B=10 C=30 <T, start> <T, A, 30> <T, B, 30> <T, start> <T, A, 30> crash occurs before <T, commit> reaches disk memory disk 37

crash occurs before <T, commit> reaches disk Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=10 B=10 C=30 <T, start> <T, A, 30> crash occurs before <T, commit> reaches disk memory disk 38

crash occurs before <T, commit> reaches disk Recovery with Redo-log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. If <T, commit> is not in log Then Ignore T; \\but write <T, abort> to disk-log 4. For T with <T, commit> in log For each <T, A, anew> in log Redo by assigning anew to A; Redo-logging: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 30> add-log: <T, commit> flush-log; output (A); output (B); CPU A=10 B=10 C=30 <T, start> <T, A, 30> <T, abort> crash occurs before <T, commit> reaches disk memory disk 39

Checkpoint for Redo-logging …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> 40

Checkpoint for Redo-logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> 41

Copy changes by committed Xacts to disk Checkpoint for Redo-logging log Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; Copy all elements in main memory that were written by committed transactions to disk; …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> Copy changes by committed Xacts to disk … 42

Checkpoint for Redo-logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; Copy all elements in main memory that were written by committed transactions to disk; \\ at the meanwhile, the system can continuously \\ accept new transactions (non-quiescent) …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> Copy changes by committed Xacts to disk … <T5,start> Copy changes by committed Xacts to disk 43

Checkpoint for Redo-logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; Copy all elements in main memory that were written by committed transactions to disk; \\ at the meanwhile, the system can continuously \\ accept new transactions (non-quiescent) When step 2 is done, place a log record <End CKPT>. …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> Copy changes by committed Xacts to disk … <T5,start> Copy changes by committed Xacts to disk <End CKPT> 44

Checkpoint for Redo-logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; Copy all elements in main memory that were written by committed transactions to disk; \\ at the meanwhile, the system can continuously \\ accept new transactions (non-quiescent) When step 2 is done, place a log record <End CKPT>. \\ at this point, all log records before the earliest \\ <Ti, start> among (T1, …, Tk) can be ignored <End CKPT> …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> Copy changes by committed Xacts to disk … <T5,start> Copy changes by committed Xacts to disk <End CKPT> 45

Checkpoint for Redo-logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; Copy all elements in main memory that were written by committed transactions to disk; \\ at the meanwhile, the system can continuously \\ accept new transactions (non-quiescent) When step 2 is done, place a log record <End CKPT>. \\ at this point, all log records before the earliest \\ <Ti, start> among (T1, …, Tk) can be ignored \\ if crash occurs between <Start CKPT(T1, …, Tk)> \\ and <End CKPT>, then look for a previous \\ <End CKPT> <End CKPT> …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> Copy changes by committed Xacts to disk … <T5,start> Copy changes by committed Xacts to disk <End CKPT> 46

Coping with System Failures Therefore, the logging strategy must be carefully designed to ensure Atomicity and Durability. If you are going to be in the logging business, one of the things that you have to do is to learn about heavy equipment. Robert VanNatta, Logging History of Columbia County We will study three logging strategies: Undo-logging Redo-logging Undo/Redo-logging

Undo/Redo Logging The format of the Undo/Redo log: <T, start> <T, A,aold,anew> …… <T, B,bold,bnew> <T, commit> (or <T, abort>) transaction T starts transaction T wrote on A/B, both old and new values are recorded transaction T commits/abort

Undo/Redo Logging The format of the Undo/Redo log: <T, start> <T, A,aold,anew> …… <T, B,bold,bnew> <T, commit> (or <T, abort>) transaction T starts transaction T wrote on A/B, both old and new values are recorded transaction T commits/abort Main idea of Undo/Redo logging: Increase flexibility and compromise undo logging and redo logging (at the expense of using more space).

Undo/Redo Logging The format of the Undo/Redo log: <T, start> <T, A,aold,anew> …… <T, B,bold,bnew> <T, commit> (or <T, abort>) transaction T starts Undo: helps guarantee Atomicity, but transactions commit later, more flush of log files Redo: helps guarantee Durability, but requires more main memory space transaction T wrote on A/B, both old and new values are recorded transaction T commits/abort Main idea of Undo/Redo logging: Increase flexibility and compromise undo logging and redo logging (at the expense of using more space).

Undo/Redo Logging Main idea of Undo/Redo logging: Compromising undo logging and redo logging, increasing flexibility. <T,V1, a1, b1> … <T, commit> New V1 = b1 <T, start> <T,V3, a3, b3> <T,V2, a2, b2> <T,V4, a4, b4> New V3 = b3 New V4 = b4 New V2 = b2 The order in which the values reach disk Undo/Redo logging rule (Rule UR): An undo/redo log record must reach disk before the corresponding change reaches disk.

Undo/Redo Logging Undo/Redo logging rule (UR): Log records reach disk before the corresponding changes reach disk. Thus, when the system crashes, we can look at disk-log: <T,V1, a1, b1> … <T, commit> New V1 = b1 <T, start> <T,V3, a3, b3> <T,V2, a2, b2> <T,V4, a4, b4> New V3 = b3 New V4 = b4 New V2 = b2 The order in which the values reach disk

Undo/Redo Logging Undo/Redo logging rule (UR): Log records reach disk before the corresponding changes reach disk. Thus, when the system crashes, we can look at disk-log: If we see <T, commit>, then all log records for T have reaches disk (although we are not sure if the corresponding changes have also reached disk), so we have all information to Redo T; <T,V1, a1, b1> … <T, commit> New V1 = b1 <T, start> <T,V3, a3, b3> <T,V2, a2, b2> <T,V4, a4, b4> New V3 = b3 New V4 = b4 New V2 = b2 The order in which the values reach disk

Undo/Redo Logging Undo/Redo logging rule (UR): Log records reach disk before the corresponding changes reach disk. Thus, when the system crashes, we can look at disk-log: If we see <T, commit>, then all log records for T have reaches disk (although we are not sure if the corresponding changes have also reached disk), so we have all information to Redo T; If we do not see <T, commit>, then we do not know if the log record for T is complete. However: <T,V1, a1, b1> … <T, commit> New V1 = b1 <T, start> <T,V3, a3, b3> <T,V2, a2, b2> <T,V4, a4, b4> New V3 = b3 New V4 = b4 New V2 = b2 The order in which the values reach disk

Undo/Redo Logging Undo/Redo logging rule (UR): Log records reach disk before the corresponding changes reach disk. Thus, when the system crashes, we can look at disk-log: If we see <T, commit>, then all log records for T have reaches disk (although we are not sure if the corresponding changes have also reached disk), so we have all information to Redo T; If we do not see <T, commit>, then we do not know if the log record for T is complete. However: * If record <T, V, aold, anew> is not in disk, then the corresponding change has not reached disk, yet, so we can ignore it; <T,V1, a1, b1> … <T, commit> New V1 = b1 <T, start> <T,V3, a3, b3> <T,V2, a2, b2> <T,V4, a4, b4> New V3 = b3 New V4 = b4 New V2 = b2 The order in which the values reach disk

Undo/Redo Logging Undo/Redo logging rule (UR): Log records reach disk before the corresponding changes reach disk. Thus, when the system crashes, we can look at disk-log: If we see <T, commit>, then all log records for T have reaches disk (although we are not sure if the corresponding changes have also reached disk), so we have all information to Redo T; If we do not see <T, commit>, then we do not know if the log record for T is complete. However: * If record <T, V, aold, anew> is not in disk, then the corresponding change has not reached disk, yet, so we can ignore it; * If record <T, V, aold, anew> is in disk, then, though we do not know if the corresponding change has reached disk, Undoing this action is always safe. <T,V1, a1, b1> … <T, commit> New V1 = b1 <T, start> <T,V3, a3, b3> <T,V2, a2, b2> <T,V4, a4, b4> New V3 = b3 New V4 = b4 New V2 = b2 The order in which the values reach disk

Undo/Redo Logging Undo/Redo logging rule (UR): Log records reach disk before the corresponding changes reach disk. Thus, when the system crashes, we can look at disk-log: If we see <T, commit>, then all log records for T have reaches disk (although we are not sure if the corresponding changes have also reached disk), so we have all information to Redo T; If we do not see <T, commit>, then we do not know if the log record for T is complete. However: * If record <T, V, aold, anew> is not in disk, then the corresponding change has not reached disk, yet, so we can ignore it; * If record <T, V, aold, anew> is in disk, then, though we do not know if the corresponding change has reached disk, Undoing this action is always safe. Redo should go forwards while Undo should go backwards. <T,V1, a1, b1> … <T, commit> New V1 = b1 <T, start> <T,V3, a3, b3> <T,V2, a2, b2> <T,V4, a4, b4> New V3 = b3 New V4 = b4 New V2 = b2 The order in which the values reach disk

Undo/Redo Logging Algorithms for Undo/Redo logging Undo/Redo logging rule (UR): Log records reach disk before the corresponding changes reach disk. Algorithms for Undo/Redo logging Undo/Redo log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, aold, anew> in memory-log; \\ output(A) can be performed now, \\ either before or after step 3 3. Add <T, commit> to memory-log. Recovery with Undo/Redo log 1. Bring disk-log to memory; 2. Scan the log forwards; 3. For T with <T,commit> in log For <T, A, aold, anew> in log Redo by assigning anew to A; 4. Scan the log backwards; 5. For T without <T,commit> in log Undo by assigning aold to A.

Undo/Redo-logging: Example Log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, aold, anew> in memory-log; \\ output(A) can be performed now, \\ either before or after step 3 3. Add <T, commit> to memory-log. Undo/Redo-logging: Example T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU A=10 B=10 C=30 memory disk 59

Undo/Redo: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, aold, anew> in memory-log; \\ output(A) can be performed now, \\ either before or after step 3 3. Add <T, commit> to memory-log. Undo/Redo: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU A=10 B=10 C=30 memory disk 60

Undo/Redo: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, aold, anew> in memory-log; \\ output(A) can be performed now, \\ either before or after step 3 3. Add <T, commit> to memory-log. Undo/Redo: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU log A=10 B=10 C=30 A=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> B=30 memory disk 61

Undo/Redo: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, aold, anew> in memory-log; \\ output(A) can be performed now, \\ either before or after step 3 3. Add <T, commit> to memory-log. Undo/Redo: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU log A=10 B=10 C=30 A=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> <T, start> <T, A, 10, 30> <T, B, 10, 30> B=30 memory disk 62

Undo/Redo: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, aold, anew> in memory-log; \\ output(A) can be performed now, \\ either before or after step 3 3. Add <T, commit> to memory-log. Undo/Redo: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU log A=10 B=10 C=30 A=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> B=30 B=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> memory disk 63

Undo/Redo: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, aold, anew> in memory-log; \\ output(A) can be performed now, \\ either before or after step 3 3. Add <T, commit> to memory-log. Undo/Redo: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU log A=10 B=30 C=30 A=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> <T, commit> B=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> memory disk 64

Undo/Redo: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, aold, anew> in memory-log; \\ output(A) can be performed now, \\ either before or after step 3 3. Add <T, commit> to memory-log. Undo/Redo: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU log A=10 B=30 C=30 A=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> <T, commit> <T, start> <T, A, 10, 30> <T, B, 10, 30> <T, commit> B=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> memory disk 65

Undo/Redo: log recording 1. Add <T, start> to memory-log; 2. When T writes on A, add <T, A, aold, anew> in memory-log; \\ output(A) can be performed now, \\ either before or after step 3 3. Add <T, commit> to memory-log. Undo/Redo: log recording T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU log A=10 B=30 C=30 A=30 A=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> <T, commit> B=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> <T, commit> memory disk 66

Undo/Redo: Failure Recovery Recovery with Undo/Redo 1. Bring disk-log to memory; 2. Scan the log forwards; 3. For T with <T,commit> in log For <T, A, aold, anew> in log Redo by assigning anew to A; 4. Scan the log backwards; 5. For T without <T,commit> in log Undo by assigning aold to A. Undo/Redo: Failure Recovery CPU A=10 B=10 C=30 memory disk 67

Undo/Redo: Failure Recovery Recovery with Undo/Redo 1. Bring disk-log to memory; 2. Scan the log forwards; 3. For T with <T,commit> in log For <T, A, aold, anew> in log Redo by assigning anew to A; 4. Scan the log backwards; 5. For T without <T,commit> in log Undo by assigning aold to A. Undo/Redo: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU A=10 B=10 C=30 memory disk 68

crash occurs after <T, commit> is flushed to disk Recovery with Undo/Redo 1. Bring disk-log to memory; 2. Scan the log forwards; 3. For T with <T,commit> in log For <T, A, aold, anew> in log Redo by assigning anew to A; 4. Scan the log backwards; 5. For T without <T,commit> in log Undo by assigning aold to A. Undo/Redo: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU A=10 B=30 C=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> <T, commit> crash occurs after <T, commit> is flushed to disk memory disk 69

crash occurs after <T, commit> is flushed to disk Recovery with Undo/Redo 1. Bring disk-log to memory; 2. Scan the log forwards; 3. For T with <T,commit> in log For <T, A, aold, anew> in log Redo by assigning anew to A; 4. Scan the log backwards; 5. For T without <T,commit> in log Undo by assigning aold to A. Undo/Redo: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU A=30 B=30 C=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> <T, commit> crash occurs after <T, commit> is flushed to disk memory disk 70

crash occurs before <T, commit> is flushed to disk Recovery with Undo/Redo 1. Bring disk-log to memory; 2. Scan the log forwards; 3. For T with <T,commit> in log For <T, A, aold, anew> in log Redo by assigning anew to A; 4. Scan the log backwards; 5. For T without <T,commit> in log Undo by assigning aold to A. Undo/Redo: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU A=10 B=30 C=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> crash occurs before <T, commit> is flushed to disk memory disk 71

crash occurs before <T, commit> is flushed to disk Recovery with Undo/Redo 1. Bring disk-log to memory; 2. Scan the log forwards; 3. For T with <T,commit> in log For <T, A, aold, anew> in log Redo by assigning anew to A; 4. Scan the log backwards; 5. For T without <T,commit> in log Undo by assigning aold to A. Undo/Redo: Failure Recovery T: A  A  3 B  B  3 add-log: <T, start> read (A,t); t  t3; write (A,t); add-log: <T, A, 10, 30> read (B,t); t  t3; write (B,t); add-log: <T, B, 10, 30> flush-log; output (B); add-log: <T, commit> output (A); CPU A=10 B=10 C=30 <T, start> <T, A, 10, 30> <T, B, 10, 30> crash occurs before <T, commit> is flushed to disk memory disk 72

Checkpoint for Undo/Redo logging …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> 73

Checkpoint for Undo/Redo logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> 74

Checkpoint for Undo/Redo logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; Copy all “dirty data” in main memory to disk, this can be due to either committed or incomplete transactions; \\ at the meanwhile, the system can continuously \\ accept new transactions (non-quiescent) …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> copy dirty data to disk … <T5,start> copy dirty data to disk 75

Checkpoint for Undo/Redo logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; Copy all “dirty data” in main memory to disk, this can be due to either committed or incomplete transactions; \\ at the meanwhile, the system can continuously \\ accept new transactions (non-quiescent) When step 2 is done, place a log record <End CKPT>. …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> copy dirty data to disk … <T5,start> copy dirty data to disk <End CKPT> 76

Checkpoint for Undo/Redo logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; Copy all “dirty data” in main memory to disk, this can be due to either committed or incomplete transactions; \\ at the meanwhile, the system can continuously \\ accept new transactions (non-quiescent) When step 2 is done, place a log record <End CKPT>. \\ at this point, nothing beyond <Start CKPT(…)> \\ needs Redone, but Undo still has to trace back to \\ the earliest <Ti, start> among (T1, …, Tk). …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> copy dirty data to disk … <T5,start> copy dirty data to disk <End CKPT> 77

Checkpoint for Undo/Redo logging Place a log record <Start CKPT(T1, …, Tk)>, where T1, …, Tk are the currently active transactions; Copy all “dirty data” in main memory to disk, this can be due to either committed or incomplete transactions; \\ at the meanwhile, the system can continuously \\ accept new transactions (non-quiescent) When step 2 is done, place a log record <End CKPT>. \\ at this point, nothing beyond <Start CKPT(…)> \\ needs Redone, but Undo still has to trace back to \\ the earliest <Ti, start> among (T1, …, Tk). \\ if crash occurs between <Start CKPT(T1, …, Tk)> \\ and <End CKPT>, then look for a previous \\ <End CKPT> <End CKPT> …… <T1,start> … <T2,commit> <T3,start> … <T4,commit> <Start CKPT (T1,T3)> copy dirty data to disk … <T5,start> copy dirty data to disk <End CKPT> 78

Tranactions What can affect transaction ACID? Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist. What can affect transaction ACID? Accident system failures may destroy Atomicity and Durability, thus Consistency; Concurrent executions of transactions (which is highly desired) may affect Isolation, thus Consistency.