1. CS4432: Database Systems II
Lecture #24
Database Consistency and Transactions
Professor Elke A. Rundensteiner
cs4432
recovery

2. Transactions, etc. Crash recovery Ch.8 [17] Concurrency control
Transaction processing
Ch.10 [19]
cs4432
recovery

3. Integrity or correctness of data ?
We would like data in our database to be “accurate” ( “correct” ) at all times.
EMP
How DBMS decides if data is consistent?
Name
White
Green
Gray
Age 52
3421 1
cs4432
recovery

4. Integrity or consistency constraints
Utilize predicates data must satisfy
Examples:
- x is key of relation R
- x  y holds in R
- Domain(x) = {Red, Blue, Green}
- a is valid index for attribute x of R
- no employee should make more than twice the average salary
cs4432
recovery

5. Definitions: Consistent state: satisfies all constraints
Consistent DB: DB in consistent state
cs4432
recovery

6. Such constraints may not capture “full correctness”
Example 1 : Transaction constraints
When salary is updated,
new salary > old salary
When account record is deleted,
balance = 0
cs4432
recovery

7. Constraints (as we use here) may not capture “full correctness”
Example Database should reflect real world
Reality DB
cs4432
recovery

8. in any case, continue with constraints ...
Observation: DB cannot be consistent always
Example:
Constraint : a1 + a2 +…. an = TOT
Action:
Deposit $100 in a2: a2  a
TOT  TOT + 100
cs4432
recovery

9. Example: a1 + a2 +…. an = TOT (constraint) TOT  TOT + 100
Deposit $100 in a2: a2  a
TOT  TOT + 100 a2
TOT . . . 50
150
150 . . .
1000
1000
1100
cs4432
recovery

10. Transaction: a collection of actions that preserve consistency
Consistent DB
Consistent DB’ T
cs4432
recovery

11. Big assumption: If T starts with consistent state AND
T executes in isolation
 T leaves consistent state
cs4432
recovery

12. Correctness (informally)
If we stop running transaction(s), DB left consistent
Each transaction sees a consistent DB
cs4432
recovery

13. How can constraints be violated?
Transaction bug
DBMS bug
Hardware failure
e.g., disk crash alters balance of account
Data sharing
e.g.: T1: give 10% raise to programmers T2: change programmers  systems analysts
cs4432
recovery

14. Will not consider: How to write correct transactions
How to write correct DBMS system
Constraint checking & repair
cs4432
recovery

15. How can we prevent/fix violations?
Chapter 8[17]: due to failures only
Chapter 9[18]: due to data sharing only
Chapter 10[19]: due to failures and sharing
cs4432
recovery

16. Chapter 8 [17]: Recovery
First : Failure Model
cs4432
recovery

17. Events Desired
Undesired Expected
Unexpected
cs4432
recovery

18. Our failure model
processor
memory disk
CPU D M
cs4432
recovery

19. Desired events: see product manuals…. Undesired expected events:
System crash
- memory lost
- cpu halts, resets
Undesired Unexpected: Everything else!
that’s it!!
cs4432
recovery

20. Undesired Unexpected: Everything else!
Examples:
Disk data is lost
Memory lost without CPU halt
CPU implodes wiping out universe….
You name it …
cs4432
recovery

21. Operations re Storage Hierarchy:
Memory Disk x
Operations re Storage Hierarchy:
Input (x): block containing x  memory
Output (x): block containing x  disk
Read (x,t): do input(x) if necessary t  value of x in block
Write (x,t): do input(x) if necessary value of x in block  t
cs4432
recovery

22. Key problem : Unfinished transaction
Example: Constraint: A=B
T1: A  A  2
B  B  2
cs4432
recovery

23. T1: Read (A,t); t  t2 Write (A,t); Read (B,t); t  t2 Write (B,t);
Output-to-Disk (A);
Output-to-Disk (B); 16
failure!
A: 8
B: 8
A: 8
B: 8 16
memory
disk
cs4432
recovery

24. Need atomicity: execute all actions of a transaction or none at all
cs4432
recovery

25. One solution: undo logging (immediate
modification)
A la Hansel and Gretel recording their navigation through forest via bread crumbs …
Must have durable undo logging !!!
cs4432
recovery

26. Undo logging (Immediate modification)
T1: Read (A,t); t  t A=B
Write (A,t);
Read (B,t); t  t2
Write (B,t);
Output-to-disk (A);
Output-to-disk (B);
<T1, start>
<T1, A, 8>
A:8
B:8
A:8
B:8 16 16
<T1, B, 8> 16
<T1, commit>
disk
memory
log
cs4432
recovery

27. One “complication” : First disk, then log
Log is first written in memory
Not written to disk on every action
memory DB
Log
A: 8
B: 8 16
BAD STATE
# 1
A: 8 16
B: 8 16
Log:
<T1,start>
<T1, A, 8>
<T1, B, 8>
cs4432
recovery

28. One “complication” : first log, then disk.
Log is first written in memory
Not written to disk on every action
memory DB
Log
A: 8
B: 8 16
BAD STATE
# 2
A: 8 16
B: 8 16
Log:
<T1,start>
<T1, A, 8>
<T1, B, 8>
<T1, commit>
...
<T1, B, 8>
<T1, commit>
cs4432
recovery

29. Undo logging rules
For every action generate undo log record (containing old value)
(2) Before x is modified on disk,
log records pertaining to x must be
on disk (write ahead logging)
(3) Before commit is written to log on disk,
all writes of transaction must be
reflected on disk
cs4432
recovery

30. Recovery rules: Undo logging
For every Ti with <Ti, start> in log: - If <Ti,commit> or <Ti,abort> in log, do nothing - Else For all <Ti, X, v> in log:
write (X, v)
output (X )
Write <Ti, abort> to log
IS THIS CORRECT??
cs4432
recovery

31. Recovery rules: Undo logging
(1) Let S = set of transactions with <Ti, start> in log, but no
<Ti, commit> (or <Ti, abort>) record in log
(2) For each undo record <Ti, X, v> in log,
in reverse order (latest  earliest) do:
- write old value from log back to disk:
- if Ti  S then - write (X, v)
- output (X)
(3) For each Ti  S do
- write <Ti, abort> to log
cs4432
recovery

32. What if failure during recovery? No problem!  Undo idempotent
cs4432
recovery

33. To discuss: Redo logging Undo/redo logging, why both?
Real world actions
Checkpoints
Media failures
cs4432
CS 245
Notes 08
recovery 34

34. Simple Redo logging (deferred modification)
T1: Read(A,t); t t2; write (A,t);
Read(B,t); t t2; write (B,t);
Output(A); Output(B) 16
<T1, start>
<T1, A, 16>
<T1, B, 16>
<T1, commit>
A: 8
B: 8
output 16
A: 8
B: 8
memory DB
LOG
cs4432
recovery

35. Redo logging rules
(1) For every action, generate redo log record (containing new value)
(2) Before X is modified on disk (DB), all log records for transaction that modified X (including commit) must be on disk
(3) Flush log at commit
cs4432
recovery

36. Recovery rules: Redo logging
For every Ti with <Ti, commit> in log:
For all <Ti, X, v> in log:
Write(X, v)
Output(X)
cs4432
CS 245
Notes 08
recovery 37

37. Recovery is very, very SLOW !
Redo log:
First T1 wrote A,B Last
Record Committed a year ago Record
(1 year ago) --> STILL, Need to redo after crash!!
...
...
...
Crash
cs4432
recovery

38. Solution: Checkpoint Periodically: (1) Do not accept new transactions
simple checkpoint
Periodically:
(1) Do not accept new transactions
(2) Wait until all transactions finish
(3) Flush all log records to disk (log)
(4) Flush all buffers to disk (DB) (do not discard buffers)
(5) Write “checkpoint” record on disk (log)
(6) Resume transaction processing
CS 245
cs4432
Notes 08
recovery 39

39. Example: what to do at recovery?
Undo or Redo log (disk):
<T1,A,16>
<T1,commit>
Checkpoint
<T2,B,17>
<T2,commit>
<T3,C,21>
Crash
...
...
...
...
...
...
CS 245
cs4432
Notes 08
recovery 40

40. AGAIN : Redo logging (deferred modification)
T1: Read(A,t); t t2; write (A,t);
Read(B,t); t t2; write (B,t);
Output(A); Output(B)
A: 8
B: 8
A: 8
B: 8
memory DB
LOG
cs4432
CS 245
recovery
Notes 08 41

41. Redo logging (deferred modification)
T1: Read(A,t); t t2; write (A,t);
Read(B,t); t t2; write (B,t);
Output(A); Output(B) 16
<T1, start>
<T1, A, 16>
<T1, B, 16>
<T1, commit>
A: 8
B: 8
A: 8
B: 8
memory DB
LOG
CS 245
cs4432
recovery
Notes 08 42

42. Redo logging (deferred modification)
T1: Read(A,t); t t2; write (A,t);
Read(B,t); t t2; write (B,t);
Output(A); Output(B) 16
<T1, start>
<T1, A, 16>
<T1, B, 16>
<T1, commit>
A: 8
B: 8
output 16
A: 8
B: 8
memory DB
LOG
CS 245
cs4432
recovery
Notes 08 43

43. Redo logging (deferred modification)
T1: Read(A,t); t t2; write (A,t);
Read(B,t); t t2; write (B,t);
Output(A); Output(B) 16
<T1, start>
<T1, A, 16>
<T1, B, 16>
<T1, commit>
A: 8
B: 8
output 16
A: 8
B: 8
<T1, end>
memory DB
LOG
CS 245
cs4432
Notes 08
recovery 44

44. Redo logging rules (1) For every action, generate redo log
record (containing new value)
(2) Before X is modified on disk (DB), all log records for transaction that modified X (including commit) must be first on disk
(3) Flush log at commit to disk
(4) Write END record after DB updates flushed to disk
cs4432
CS 245
Notes 08
recovery 45

45. Recovery rules: Redo logging
For every Ti with <Ti, commit> in log:
For all <Ti, X, v> in log:
Write(X, v)
Output(X)
cs4432
CS 245
Notes 08
recovery 46

46. Recovery rules: Redo logging
For every Ti with <Ti, commit> in log:
For all <Ti, X, v> in log:
Write(X, v)
Output(X)
IS THIS CORRECT??
CS 245
cs4432
Notes 08
recovery 47

47. Recovery rules: Redo logging
(1) Let S = set of transactions with <Ti, commit> (and no <Ti, end>) in log
(2) For each <Ti, X, v> in log, in forward order (earliest  latest) do:
- if Ti  S then Write(X, v)
Output(X)
(3) For each Ti  S, write <Ti, end>
cs4432
CS 245
Notes 08
recovery 48

48. Combining <Ti, end> Records
Want to delay DB flushes for hot objects
Actions:
write X
output X
Say X is branch balance:
T1: ... update X...
T2: ... update X...
T3: ... update X...
T4: ... update X...
CS 245
cs4432
Notes 08
recovery 49

49. Combining <Ti, end> Records
Want to delay DB flushes for hot objects
Actions:
write X
output X
Say X is branch balance:
T1: ... update X...
T2: ... update X...
T3: ... update X...
T4: ... update X...
combined <end> (checkpoint)
cs4432
CS 245
recovery
Notes 08 50

50. Two methods: Each with drawbacks
Undo logging:
cannot bring backup DB copies up to date
Waste undo things that were already “done”
Redo logging:
need to keep all modified blocks in memory until commit
cs4432
CS 245
Notes 08
recovery 51

51. Solution: get best of both methods?
Apply both: undo/redo logging!
Create Undo/Redo Log Entry
Update  <Ti, Xid, New X val, Old X val>
page X
cs4432
CS 245
Notes 08
recovery 52

52. Rules Page X can be flushed before or after Ti commit
Log record flushed before corresponding updated page (WAL)
Flush at commit (log only)
cs4432
CS 245
Notes 08
recovery 53

53. Rules Log record flushed before corresponding updated page (WAL)
Both
Page X can be flushed before Ti commit (or after Ti commit)
Undo: Yes, all page X writes must be flushed FIRST before writing commit to log
Redo: No, all log records including commit must be flushed FIRST before DB changes
Flush log at commit
Undo: No, all page X writes must be flushed FIRST
Redo: Yes, flush log commit FIRST
cs4432
CS 245
Notes 08
recovery 54

54. Example: Undo/Redo logging?
log (disk):
<checkpoint>
<T1, A, 10>
<T1, B, 20>
<T1, commit>
<T2, C, 30>
<T2, D, 40>
Crash
...
...
...
...
...
...
CS 245
cs4432
Notes 08
recovery 55

55. Non-quiesce checkpointL O G
for
undo for redo:
dirty buffer
pool pages
flushed
Start-ckpt
active TR:
T1,T2,...
end
ckpt
...
...
...
...
CS 245
cs4432
Notes 08
recovery 56

56. Examples what to do at recovery time?
no T1 commit L O G
...
T1,- a
...
Ckpt-
start T1
...
Ckpt
end
...
T1- b
CS 245
cs4432
Notes 08
recovery 57

57. Example what to do at recovery time?
no T1 commit L O G
...
T1,- a
...
Ckpt T1
...
Ckpt
end
...
T1- b
 Undo T1 (undo a,b)
cs4432
CS 245
Notes 08
recovery 58

58. Example what to do at recovery time?
T1 commit L O G
... T1 a
...
ckpt-s T1
... T1 b
...
ckpt-
end
... T1 c
... T1
cmt
...
CS 245
cs4432
Notes 08
recovery 59

59. Example L O G  Redo T1: (redo b,c) ... T1 a ... T1 ... T1 b ... ckpt-
ckpt-s T1
... T1 b
...
ckpt-
end
... T1 c
... T1
cmt
...
 Redo T1: (redo b,c)
cs4432
CS 245
Notes 08
recovery 60

60. Recover From Valid Checkpoint:G
...
ckpt
start
...
ckpt
end
... T1 b
...
ckpt-
start
... T1 c
...
start
of latest
valid
checkpoint
CS 245
cs4432
recovery
Notes 08 61

61. Recovery process:
Backwards pass (end of log  latest valid checkpoint start)
construct set S of committed transactions
undo actions of transactions not in S
Undo pending transactions
follow undo chains for transactions in (checkpoint active list) - S
Forward pass (latest checkpoint start  end of log)
redo actions of S transactions
backward pass
start
check-
point
forward pass
cs4432
CS 245
recovery
Notes 08 62

62. Real world actions E.g., dispense cash at ATM
Ti = a1 a2 …... aj …... an $
CS 245
cs4432
Notes 08
recovery 63

63. Solution (1) execute real-world actions after commit
(2) try to make idempotent
cs4432
CS 245
Notes 08
recovery 64

64. $ ATM Give$$ (amt, Tid, time) lastTid: time: give(amt) cs4432 CS 245
recovery
Notes 08 65

65. Media failure (loss of non-volatile storage)
cs4432
CS 245
Notes 08
recovery 66

66. Media failure (loss of non-volatile storage)
Solution: Make copies of data!
CS 245
cs4432
Notes 08
recovery 67

67. Example 1 Triple modular redundancy
Keep 3 copies on separate disks
Output(X) --> three outputs
Input(X) --> three inputs + vote X3 X1 X2
cs4432
CS 245
recovery
Notes 08 68

68. Example #2 Redundant writes, Single reads
Keep N copies on separate disks
Output(X) --> N outputs
Input(X) --> Input one copy if ok, done
- else try another one
 Assumes bad data can be detected
cs4432
CS 245
Notes 08
recovery 69

69. Example #3: DB Dump + Log backup active database database log
If active database is lost,
restore active database from backup
bring up-to-date using redo entries in log
cs4432
CS 245
recovery
Notes 08 70

70. When can log be discarded?db
dump
last
needed
undo
check-
point
log
time
not needed for
media recovery
not needed for undo
after system failure
not needed for
redo after system failure
CS 245
cs4432
recovery
Notes 08 71

71. Summary Consistency of data & Atomicity
One source of problems: memory failure
- Logging / Recovery
One source of problems: disk failure
- Redundancy
cs4432
CS 245
Notes 08
recovery 72