CPSC-608 Database Systems Fall 2018 Instructor: Jianer Chen Office: HRBB 315C Phone: 845-4259 Email: chen@cse.tamu.edu Notes #32

Graduate Database DBMS lock table DDL language DDL complier file administrator DDL complier lock table DDL language file manager logging & recovery concurrency control transaction manager database programmer index/file manager buffer manager query execution engine DML complier main memory buffers DML (query) language secondary storage (disks) DBMS Graduate Database

Graduate Database DBMS lock table DDL language DDL complier file administrator DDL complier lock table DDL language file manager logging & recovery concurrency control transaction manager database programmer index/file manager buffer manager query execution engine DML complier main memory buffers DML (query) language secondary storage (disks) DBMS Graduate Database

Graduate Database DBMS lock table DDL language DDL complier file administrator DDL complier lock table DDL language file manager logging & recovery concurrency control transaction manager database programmer index/file manager buffer manager query execution engine DML complier main memory buffers DML (query) language secondary storage (disks) DBMS Graduate Database

Graduate Database DBMS lock table DDL language DDL complier file administrator DDL complier lock table DDL language file manager logging & recovery concurrency control transaction manager database programmer index/file manager buffer manager query execution engine DML complier main memory buffers DML (query) language secondary storage (disks) DBMS Graduate Database

Graduate Database DBMS administrator DDL complier lock table DDL language file manager logging & recovery concurrency control transaction manager database programmer index/file manager buffer manager query execution engine DML complier main memory buffers DML (query) language System concerns on processes concurrently running in the system, not on individual process. secondary storage (disks) DBMS Graduate Database

Graduate Database DBMS lock table DDL language DDL complier file administrator DDL complier lock table DDL language file manager logging & recovery concurrency control transaction manager database programmer index/file manager buffer manager query execution engine DML complier main memory buffers DML (query) language secondary storage (disks) DBMS Graduate Database

Graduate Database …… DBMS User-1 User-2 User-3 User-n lock table DDL complier lock table file manager logging & recovery concurrency control User-1 DDL User-2 transaction manager index/file manager buffer manager User-3 query execution engine …… DML complier DML main memory buffers User-n secondary storage (disks) DBMS Graduate Database

Graduate Database …… DBMS User-1 User-2 User-3 User-n lock table DDL complier lock table file manager logging & recovery concurrency control User-1 DDL User-2 transaction manager index/file manager buffer manager ……,T2,T1 User-3 query execution engine …… DML complier DML …… Q2,Q1 main memory buffers User-n secondary storage (disks) DBMS Graduate Database

Graduate Database …… DBMS User-1 User-2 User-3 User-n lock table DDL complier lock table file manager logging & recovery concurrency control User-1 DDL User-2 transaction manager index/file manager buffer manager ……,T2,T1 User-3 query execution engine …… DML complier DML …… Q2,Q1 main memory buffers User-n secondary storage (disks) DBMS Graduate Database

Graduate Database …… DBMS User-1 User-2 User-3 User-n lock table DDL complier lock table file manager logging & recovery concurrency control User-1 DDL User-2 transaction manager index/file manager buffer manager ……,T2,T1 User-3 query execution engine …… DML complier DML …… Q2,Q1 main memory buffers User-n secondary storage (disks) DBMS Graduate Database

Graduate Database …… DBMS User-1 User-2 User-3 User-n lock table DDL complier lock table file manager logging & recovery concurrency control User-1 DDL User-2 transaction manager index/file manager buffer manager ……,T2,T1 User-3 query execution engine …… DML complier DML …… Q2,Q1 main memory buffers User-n secondary storage (disks) DBMS Graduate Database

Coping with System Failures

Coping with System Failures System may fail because of Program or data (logic) errors Disk crashes Computer room fires World crashes … System failures (power off, code execution errors …)

Coping with System Failures System may fail because of Program or data (logic) errors Disk crashes Computer room fires World crashes … System failures (power off, code execution errors …) Users’ errors, out of system’s control

Coping with System Failures System may fail because of Program or data (logic) errors Disk crashes Computer room fires World crashes … System failures (power off, code execution errors …) } Data backup

Coping with System Failures System may fail because of Program or data (logic) errors Disk crashes Computer room fires World crashes … System failures (power off, code execution errors …) We can do nothing with it … Backup in Mars?

Coping with System Failures System may fail because of Program or data (logic) errors Disk crashes Computer room fires World crashes … System failures (power off, code execution errors …) Recovery programs and transition scheduling may help

Integrity/Correctness of Data

Integrity/Correctness of Data Data be “accurate/correct” at all times

Integrity/Correctness of Data Data be “accurate/correct” at all times Data satisfy all pre-given (logic) predicates

Integrity/Correctness of Data Data be “accurate/correct” at all times Data satisfy all pre-given (logic) predicates Examples: A is key of relation R A  B holds for all attributes B of R Domain(B) = {Red, Blue, Green} a is a valid index for attribute C of R

Integrity/Correctness of Data Data be “accurate/correct” at all times Data satisfy all pre-given (logic) predicates Examples: A is key of relation R A  B holds for all attributes B of R Domain(B) = {Red, Blue, Green} a is a valid index for attribute C of R Data reflect the real world DB Reality

Integrity/Correctness of Data Data in consistent state: Data that satisfy all (logic/realistic) constraints Consistent Database: Database that contains data in consistent state

Integrity/Correctness of Data Observation: A database cannot always keep consistent!

Integrity/Correctness of Data Observation: A database cannot always keep consistent! Example: Logic constraint: a1 + a2 +…. an = sum Now suppose that we wanted to deposit $100 in a2. We perform: a2  a2 + 100; sum  sum + 100 50 1000 150 1100 a2 sum Inconsistent!

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist.

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist. Disk is non-volatile:

So the results of a transaction should reach disk Tranactions What is a transaction? A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist. Disk is non-volatile: So the results of a transaction should reach disk

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist.

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist. System cannot do much with this: Errors can be introduced by transaction writers

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist. System cannot do much with this: Errors can be introduced by transaction writers We adopt the following (big) assumption: All transactions we deal with are consistent. Consistent DB Consistent DB’ T

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist.

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist. What other things can affect transaction ACID?

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist. What other things can affect transaction ACID? Accident system failures may destroy Atomicity and Durability, thus Consistency;

Tranactions What is a transaction? What is a logic step A sequence of actions that we want to be done as a single “logic step ” What is a logic step (what do we want a transaction to be)? ACID: Atomicity: either all steps of a transaction happen, or none happen Consistency: a transaction transforms a consistent DB into a consistent DB Isolation: execution of a transaction is isolated from that of other transactions Durability: if a transaction commits, its effects persist. What other things can affect transaction ACID? Accident system failures may destroy Atomicity and Durability, thus Consistency; Concurrent executions of transactions (which is highly desired) may affect Isolation, thus Consistency.

We will be focused on: How to prevent/fix constraint violations due to system failures due to concurrent executed transactions due to both

We will be focused on: How to prevent/fix constraint violations due to system failures due to concurrent executed transactions due to both

We will be focused on: How to prevent/fix constraint violations due to system failures How do we recover from system failures? due to concurrent executed transactions due to both

System Failure Recovery What we want to recover? Undesired changes made by transactions How can a transaction make undesired changes? Partial changes made by an incomplete transaction whose execution is interrupted because of system failure.

Read from disk to memory Computational Model CPU A=10 A=10 B=10 C=30 memory Input(A): Read from disk to memory disk 41

Write back from memory to disk Computational Model CPU A=10 B=10 C=30 A=30 A=30 memory Output(A): Write back from memory to disk Value gets changed in memory disk 42

Computational Model CPU When memory shuts off, B=10 C=30 memory When memory shuts off, the changes in disk remain. disk 43

Computational Model CPU memory disk input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model CPU A=10 B=10 C=30 memory disk 44

Computational Model CPU memory disk T1: A  A  3 B  B  3 input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 CPU A=10 B=10 C=30 memory disk 45

Computational Model CPU memory disk T1: A  A  3 B  B  3 input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU A=10 B=10 C=30 memory disk 46

Computational Model CPU memory disk T1: A  A  3 B  B  3 input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU A=10 A=10 B=10 C=30 memory disk 47

Computational Model CPU memory disk T1: A  A  3 B  B  3 input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU A=10 B=10 C=30 A=30 memory disk 48

Computational Model CPU memory disk T1: A  A  3 B  B  3 input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU A=10 B=10 B=10 C=30 A=30 B=30 memory disk 49

Computational Model CPU memory disk T1: A  A  3 B  B  3 input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU A=10 B=10 C=30 A=30 A=30 B=30 memory disk 50

Computational Model CPU memory disk T1: A  A  3 B  B  3 input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU A=30 B=10 C=30 A=30 A=30 B=30 B=30 memory disk 51

Computational Model CPU memory disk T1: A  A  3 B  B  3 input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU A=30 B=30 C=30 memory disk 52

Computational Model CPU memory disk T1: A  A  3 B  B  3 input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU What if the system crashes incidentally? A=10 B=10 C=30 memory disk 53

If the crash occurs after this point, input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU If the crash occurs after this point, we are good. A=30 B=30 C=30 A=30 B=30 memory disk 54

Computational Model CPU If the crash occurs before this point, input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU If the crash occurs before this point, we are probably fine: just redo T1. A=10 B=10 C=30 A=30 B=30 memory disk 55

Computational Model CPU If the crash occurs at this point, input (x): disk Bx  memory Bx output (x): memory Bx  disk Bx read (x,t): \\ do input(x) if necessary t  value of x in memory Write (x,t): \\ do input(x) if necessary Bx in memory  value of t Computational Model T1: A  A  3 B  B  3 read (A,t); t  t3; write (A,t); read (B,t); t  t3; write (B,t); output (A); output (B); CPU If the crash occurs at this point, we are left with an undesired result. A=30 B=10 C=30 A=30 B=30 memory disk 56

Computational Model CPU Thus, we must find a way to record the changes made by the transactions: memory disk 57

Computational Model CPU Thus, we must find a way to record the changes made by the transactions: 1. This can only be done in main memory; memory disk 58

Computational Model CPU Thus, we must find a way to record the changes made by the transactions: 1. This can only be done in main memory; 2. The record must be copied to disk memory disk 59