1. Behind The Scenes: Updating A Record
Gus Björklund, Progress Software
DBA Day, 7 December 2010

2. Preface
We examine what happens “under the covers” as we execute a simple four line 4GL program.
What happens in the 4GL runtime ?
What happens in the database ?
cheated a little to keep it short
Please ask questions as we go 2 2

3. DO TRANSACTION: FIND customer WHERE cust.cust-num = 20.
A little 4GL program
DO TRANSACTION:
FIND customer WHERE
cust.cust-num = 20.
cust.city = “Nashua”.
END. 3 3

4. Operations Performed By Our Simple Program
In the 4GL runtime, the following will happen
Step 0: Compile program and run it
Step 1: Start transaction
Step 2: Find row
Step 3: Upgrade lock
Step 4: Assign new value
Step 5: End block
Step 6: Commit

5. Operations Performed By Our Simple Program
In the database, the following will happen
Step 2a: Start a transaction.
Step 2b: Find index entry for customer 20
Step 2c: Lock customer row S
Step 2d: Fetch customer row
Step 2e: Send customer row to client
Step 3: Upgrade row lock to EX
Step 4a: Update customer row
Step 4b: Delete old index entry
Step 4c: Insert new index entry
Step 6: Commit.
! stuff on database side has this light blue background ! 8

6. DO TRANSACTION: FIND customer WHERE cust.cust-num = 20.
The 7 Steps
DO TRANSACTION:
FIND customer WHERE
cust.cust-num = 20.
cust.city = “Nashua”.
END. 1 2 3 4 5 6 3 3

7. DO TRANSACTION: FIND customer WHERE cust.cust-num = 20.
Step 0: Compile and Run
DO TRANSACTION:
FIND customer WHERE
cust.cust-num = 20.
cust.city = “Nashua”.
END. 3 3

8. Step 0: Compile and Run Compile (oversimplified)
Get schema share lock on every connected db
Analyse source code
Fetch whatever is needed from schema tables
Choose index(es) that will be used
Bind buffers to databases
Generate r-code
Release schema lock

9. Step 0: Compile and Run Binding The Buffer
FIND customer …
FIND foo.customer …
SMCI
for customer
SMDB dbname = “foo” dbtype = “PROGRESS
for database "foo"

10. Step 0: Compile and Run Why Doesn’t This Work?
CREATE ALIAS bar FOR DATABASE foo.
FIND bar.CUSTOMER WHERE …

11. Digression: Client side schema cache
FILDES structures
One per _File entry
_file
CONDB
_field
DBDES “demo”
customer
order
order-line
item
Built when connecting to db
Etc.

12. Step 0: Compile and Run Run program
Match SMDB structures in r-code to CONDB’s of connected databases
Get share schema locks if needed
Make runtime CI's From r-code SMCI's
Look up index numbers from index names
Index cust-num is index #113
Do other interesting stuff ….
Start executing in action segment
When program ends, clean up mess

13. Step 0: Compile and Run Make CI from SMCI
CONDB
Etc.
order
customer
DBDES
FDTBL for databse foo
CI for table customer

14. Step 1: Start Transaction
DO TRANSACTION:
FIND customer WHERE
cust.cust-num = 20.
cust.city = “Nashua”.
END. 1 3 3

15. Step 1: Start Transaction
CONDB
Etc.
order
customer
DBDES
transaction start
is delayed
FDTBL for database foo
CI for table customer

16. DO TRANSACTION: FIND customer WHERE cust.cust-num = 20.
Step 2: Find Row
DO TRANSACTION:
FIND customer WHERE
cust.cust-num = 20.
cust.city = “Nashua”.
END. 1 2 3 3

17. Step 2: FIND Row Send delayed start transaction message to server
Server starts transaction
Send ‘find” message to server with
equality bracket from WHERE clause,
index number 113,
table number 3
Server finds index entry
Lock row
Find row
Send client a reply with row

18. Step 2a: Find row Server starts a transaction
Receives start transaction message
Allocates a transaction table entry and transaction id
Label transaction table entry "allocated"
Transaction begin note NOT written

19. Step 2b: Find row What is in an index?
cust-num rowid
an ordered list of key value/rowid pairs

20. Step 2b: Find Row Index entriesKS
Key Value IS
Info
Two parts: 0) key value, 1) key info
Key value: byte string, up to ~3000 bytes
Key info: byte string up to 255 bytes – rowids go here
Index Entries ordered by
Numeric value, or
Collation value of character data, or
A combination of values (i.e. multi-component keys)
Compared left to right, byte by byte

21. Step 2b: Find Row Server FINDs a Row
Index # 113 will be searched for key value 20
Look up _storageobject entry for table # 3
get table's area number
Look up _storageobject entry for index # 113
get index area number and root block location
Get index # 113 root block in area 7
Search down b-tree for key value 20
Extract rowid 9006 from index entry
Fetch customer 20’s row at rowid 9006 in area 8

22. ?
Wrong! What did I forget

23. Step 2b: Find Row Server FINDs a Row
Index “cust-num” is used for the find
The compiler chose it and found out it is index # 113
Index # 113 will be searched for key value 20
Look up _storageobject entry for table # 3
get table's area number
Look up _storageobject entry for index # 113
get index area number and root block location
Get index # 113 root block in area 7
Search down b-tree for key value 20
Extract rowid 9006 from index entry
Get a SHARE LOCK on table 3, rowid 9006
Fetch customer 20’s row at rowid 9006 area 8

24. Start digression Program execution suspended you are swapped out

25. Digression: Locking Rows can be locked as
share (other transactions can read, no one can update)
exclusive (no other transaction can read or update).
When you read a row, you get a share lock (default).
Share locks can be acquired outside a transaction.
Exclusive locks can only be acquired within a transaction.

26. Digression: More Locking
Share locks are automatically upgraded to exclusive when you update a row.
You can read without any locking (NO-LOCK) too
You will see uncommitted and inconsistent data
You may see partially updated rows
You may see partially updated indexes
You may see data that is not really there.

27. Digression: Two-phase Locking
The 2 Phases:
Growing
Shrinking
Within a transaction (growing)
Only acquire or upgrade locks
NEVER release
At transaction commit (shrinking)
Release or downgrade to share

28. What if you could update without any locking?
What if you could update without any locking

29. End digression resume program execution

30. Step 2b: Find row Where is index root block?
We have area number 7 from index _storageobject
Index root block is block 8
Database block size is 8192
Block 8 begins bytes after extent header

31. Step 2b: Find row Fetch index root block
blocks on disk in storage area with 1 extent
dbread (file, byteoffset, blocksize, buffer)
blocks in shared memory in primary buffer pool
blocks in shared memory in secondary buffer pool

32. Step 2b: Find row Index blocks have subset of index entries
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
…….
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . . 26 30

33. Step 2b: Find row Index b-tree search down to bottom level
root
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
internal
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
leaf
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Top
Reserved
Free Space
o o o
. . . Compressed Index Entries . . .
Bot
Index No.
Num Entries
Bytes Used
Dummy Entry . . .
rows

34. Step 2c: Find row Insert share lock into lock table
Free List
Active & Queued (Waiting) Lock Entries X SQ SQ SQ SQ
Row-id
Table-id
Owner
Lock State S
Next Pointer S X
Lock
Table
Entry
Hash
Table

35. Step 2d: Find row Where is the row stored?
We have area number 8 from table's _storageobject
We have rowid 9006 from index entry
Database block size is 8192
Maximum rows per block is 64
9006 / 64 yields 140, remainder 46
Row is 47th in block 139 of first extent of area 8
Block 139 begins bytes after extent header

36. Step 2d: Find row Fetch data block that contains the row
blocks on disk in storage area with 2 extents
dbread (file, byteoffset, numbytes, buffer)
blocks in shared memory in primary buffer pool
blocks in shared memory in secondary buffer pool

37. Step 2d: Find row Fetch row from data block
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Free Space
Free
Dirs.
Rec 0 Offset
Rec 1 Offset
Rec 2 Offset
Rec n Offset
Num
Used Data Space
Record 0
Record 2
Record 1
Share
locked RM
block
in buffer
pool
Server’s buffer 27 31

38. Step 2e: Find row Send row to client
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Free Space
Free
Dirs.
Rec 0 Offset
Rec 1 Offset
Rec 2 Offset
Rec n Offset
Num
Used Data Space
Record 0
Record 2
Record 1
Share
locked RM
block
in buffer
pool
row buffer
Server’s buffer
TCP/IP
Server’s network message buffer
Client’s network message buffer 27 31

39. Step 2: Find row Where does the row go now?
CONDB
Etc.
order
customer
DBDES
FDTBL for database foo
CI for customer

40. Step 2: Find row Where does the row go?
CONDB
Etc.
order
customer
DBDES
ICB for table customer
FDTBL for database foo
CI for customer
row buffer (cust 20)

41. Start digression Program execution suspended you are swapped out

42. Digression what if we have 2 buffers for customer table ?
DEFINE BUFFER bcust FOR TABLE customer
FIND FIRST customer.
FIND LAST bcust.

43. Digression what if we have 2 buffers for customer table ?
ICB for table customer
CI for buffer bcust
row buffer (last)
ICB for table customer
CI for buffer customer
row buffer (first)

44. Digression what if we have 2 buffers for customer table ?
What if we fetch the same row?
DEFINE BUFFER bcust FOR TABLE customer
FIND FIRST customer.
FIND FIRST bcust.

45. Digression customer and bcust buffers, same row
CI for buffer bcust
ICB for table customer
CI for buffer customer
row buffer (first)

46. End digression resume program execution

47. DO TRANSACTION: FIND customer WHERE cust.cust-num = 20.
Step 3: Upgrade lock
DO TRANSACTION:
FIND customer WHERE
cust.cust-num = 20.
cust.city = “Nashua”.
END. 1 2 3 3 3

48. Step 3: Upgrade lock please upgrade to EX client server Lock table okSQ S
client
server
Lock table ok

49. Step 3: Upgrade lock Change lock table entry from S to X
Free List
Active & Queued (Waiting) Lock Entries X SQ SQ SQ SQ
Row-id
Table-id
Owner
Lock State X
Next Pointer X X
Lock
Table
Entry
Hash
Table

50. Step 4: Assgn new value to city
DO TRANSACTION:
FIND customer WHERE
cust.cust-num = 20.
cust.city = “Nashua”.
END. 1 2 3 4 3 3

51. Step 4: Assign new value to city
“Nashua”
ICB for table customer
stack
field n
Boston
CI for customer
row buffer

52. Step 4: Assign new value to city
we also keep a copy of old key value
“Nashua”
ICB for customer
stack
field n
Nashua
CI for customer
row buffer

53. Step 4: Assign new value to city After assign statement completed
Because a key field was changed:
Send updated row to server
Delete existing index entry using saved copy
Create new index entry with new value

54. Step 4a: Assign new value to city Server updates data block with new row
Block’s DBKEY
Type
Chain
Backup Ctr
Next DBKEY in Chain
Block Update Counter
Free Space
Free
Dirs.
Rec 0 Offset
Rec 1 Offset
Rec 2 Offset
Rec n Offset
Num
Used Data Space
row 0
row 2
row 1
Exclusive
locked RM
block
in buffer
pool
record buffer
Server’s buffer
TCP/IP
Server’s network message buffer
Client’s network message buffer 27 31

55. Step 4a: Assign new value to city Server updates data block with new row
Locate RM data block from rowid
Exclusive lock data block buffer
Compute difference of old and new row.
Allocate memory for BI log note
Lock MTX latch
Spool a “transaction begin” note
We did not write it in step 2a before
Change transaction state from "allocated" to “active”
Spool a “row change” note to BI log.
Unlock MTX latch
Then… 31 31

56. Step 4a: Assign new value to city Server updates data block with new row
Adjust block free space if needed
Copy new row over old
Mark database block modified
Increment block's version number
Release buffer lock
Done 32

57. Step 4b: Assign new value to city Server delete old index entry
Locate index entry in index b-tree
Exclusive lock index block buffer
Lock MTX latch
Spool "index delete" before-image log note
Unlock MTX latch
Remove index entry from index block
Mark buffer modified
Increment block's version number
Release buffer lock

58. Step 4c: Assign new value to city Server inserts new index entry
Locate spot in b-tree for new index entry
Exclusive lock index block buffer
Lock MTX latch
Spool "index insert" before-image log note
Unlock MTX latch
Insert index entry into block
Mark buffer modified
Increment block's version number
Release buffer lock

59. DO TRANSACTION: FIND customer WHERE cust.cust-num = 20.
Step 5: End the block
DO TRANSACTION:
FIND customer WHERE
cust.cust-num = 20.
cust.city = “Nashua”.
END. 1 2 3 4 5 3 3

60. Step 5: End the block
Buffer goes out of scope
Disconnect ICB

61. Step 5: End the block Disconnect ICB
CONDB
Etc.
order
customer
DBDES
ICB for customer
FDTBL for database foo
CI for customer
Row buffer

62. Step 5: End the block Disconnect ICB
CONDB
Etc.
order
customer
DBDES
FDTBL for demo
CI for customer

63. Step 6: Commit transaction
DO TRANSACTION:
FIND customer WHERE
cust.cust-num = 20.
cust.city = “Nashua”.
END. 1 2 3 4 5 6 3 3

64. Step 6: Commit transaction
If not distributed transaction
only need phase 2.
Send commit message to server
Wait for response
client
server
commit ok 40 40

65. Step 6: Commit transaction Server side
Find transaction table entry
Change transaction state
from “active” to “committing”.
Spool “transaction commit” note to Before-Image Log.
from "committing" to “committed”
Release transaction table entry.
from "committed" to "dead"
Release (or downgrade) row locks.
Wait for commit note to be flushed to disk
Skipped if Lazy Commit option (-Mf) is used. 40 40

66. ?
Are we done

67. QUIZ! What could go wrong?
QUIZ! What could go wrong

68. A few possible complications
Lock conflict on SHARE lock
Row larger than network message bufffer
Row deleted by uncommitted transaction
Lock conflict on upgrade to EXCLUSIVE lock
Need space for larger row
Need new allocation cluster for table
Need more disk space
Need new BI Log cluster
Updated field of unique index key
After Image Journalling enabled
Auditing enabled
Transparent Data Encryption enabled 43

69. ?
Questions