1. Birth, Death, Infinity Gus Björklund. Progress. Dan Foreman. BravePoint. PUG Challenge Americas, 9-12 June 2013

2. © 2013 Progress Software Corporation. All rights reserved. 2 Birth, Death, Infinity abstract In this talk we follow the life of a database record from its initial conception, creation, reading, & updates, to eventual deletion. The discussion will include the environment that record CRUD activities take place in and the different resources that are impacted: disk, memory, latches, semaphores, record locks, blocks, etc. We will also touch on the aspects of a record's life that directly and indirectly impact performance. This may sound like a mega-geek presentation, and it is, however there are many practical aspects to this information that have a direct impact on DBAs as well as developers.

3. © 2013 Progress Software Corporation. All rights reserved. 3 Birth, Death, Infinity who are we ?

5. © 2013 Progress Software Corporation. All rights reserved. 5 Birth, Death, Infinity Conception

6. © 2013 Progress Software Corporation. All rights reserved. 6 Birth, Death, Infinity A Table is defined  Use the Data Dictionary Tool Type in all the information  Load a "df" file run prodict/load_df.p (os-getenv ("SCHEMADIR") + "/" "customer.df").  Execute a SQL DDL statement CREATE TABLE pub.customer (cust-num integer,... ) AREA "a51"; Methods for defining a new table:

7. Where does the definition go?

8. © 2013 Progress Software Corporation. All rights reserved. 8 Birth, Death, Infinity Table definitions _file _field One _field record (row) for each field (column) of a table The _* tables live in the Schema Area. Put yours elsewhere.

9. © 2013 Progress Software Corporation. All rights reserved. 9 Birth, Death, Infinity List Your Tables and Their Fields output to tables.txt. for each _file where (0 < _file-num): put _file-name skip. for each _field of _file: put “ “ _field-name skip. end. put “” skip. end. output close.

10. © 2013 Progress Software Corporation. All rights reserved. 10 Birth, Death, Infinity Index definitions _file _index _index-field One _index record for each index of a table One _index-field record for each key component of an index _index

11. © 2013 Progress Software Corporation. All rights reserved. 11 Birth, Death, Infinity List indexes and key components by table output to index.txt. for each _file where (0 < _file-num): put _file-name skip. for each _index of _file: put " " _index-name skip. for each _index-field of _index: find _field where recid (_field) = _field-recid. put " " _field-name skip. end. put "" skip. end. output close.

12. Where do the default values go?

13. © 2013 Progress Software Corporation. All rights reserved. 13 Birth, Death, Infinity the template record  Each table has a template record  Column values in the template become the default values  Template records NOT stored with schema metadata go in table's home area or schema area (v11) have to read from database to get template  TODAY and NOW defaults replaced with current values

14. © 2013 Progress Software Corporation. All rights reserved. 14 Birth, Death, Infinity Birth

15. © 2013 Progress Software Corporation. All rights reserved. 15 Birth, Death, Infinity CREATE a new row in a 4GL program create customer.

16. © 2013 Progress Software Corporation. All rights reserved. 16 Birth, Death, Infinity CREATE a new row in a 4GL program create customer. what happens? we read template record from database (if we don't have it already) client buffer is filled with a copy of the template record current values inserted for TODAY, NOW no write to database – db knows nothing about the create at this point.

17. © 2013 Progress Software Corporation. All rights reserved. 17 Birth, Death, Infinity Where does customer row go? CONDB customer order Etc. DBDES CI for customer FDTBL for database foo ICB for table customer row buffer

18. © 2013 Progress Software Corporation. All rights reserved. 18 Birth, Death, Infinity assign column values Address Address2 City Contact Curr-bal Cust-num Discount Max-credit Mnth-sales Name Phone Sales-region Sales-rep St Tax-no Terms Ytd-sls Zip

19. when do we send it to the database ?

20. © 2013 Progress Software Corporation. All rights reserved. 20 Birth, Death, Infinity database accelerator

21. © 2013 Progress Software Corporation. All rights reserved. 21 Birth, Death, Infinity send row to database client's row buffer server's network message buffer server's row buffer client's network message buffer TCP/IP

22. where do we put it ?

23. © 2013 Progress Software Corporation. All rights reserved. 23 Birth, Death, Infinity Database Pages (Blocks)  Database divided into fixed-size blocks or “pages”  Adjacent pages form clusters  Different kinds of pages store different kinds of data  Disk i/o done in page size units  Each page has unique identifier – its “dbkey” see these slides for space allocation algorithm details Space, the final frontier http://communities.progress.com/pcom/docs/DOC-107729

24. © 2013 Progress Software Corporation. All rights reserved. 24 Birth, Death, Infinity Find data block, read into database buffer pool Hash Table LRU Chain “oldest” “newest” Page Writer Queue MMM MM MMMM Checkpoint Queue

25. © 2013 Progress Software Corporation. All rights reserved. 25 Birth, Death, Infinity Update data block  acquire exclusive lock on buffer  generate "create" bi note  spool bi note  copy from row buffer to data block  update block free space if needed  mark buffer modified  release buffer lock note: simplified. some details omitted to protect the innocent

26. © 2013 Progress Software Corporation. All rights reserved. 26 Birth, Death, Infinity Record In A Row Data Block (aka RM Block) block header RM block header dir entry 0 free space record 0 Newly created row is stored in some row data block and assigned an available directory entry. The chosen block and directory entry determine the ROWID record 1 record 2 record 3 dir entry 1 dir entry 2 dir entry 3

27. © 2013 Progress Software Corporation. All rights reserved. 27 Birth, Death, Infinity Birth Alternatives

28. © 2013 Progress Software Corporation. All rights reserved. 28 Birth, Death, Infinity Other ways records are born  sql: INSERT INTO customer (cust-no, name street, city, state) VALUES (642, 'gus', '14 oak park', 'bedford', 'ma');  sql: INSERT INTO customer (cust-no, name street, city, state) SELECT... FROM... WHERE... ;  binary load: proutil foo -C load bar.bd

29. © 2013 Progress Software Corporation. All rights reserved. 29 Birth, Death, Infinity Possible Problems

30. © 2013 Progress Software Corporation. All rights reserved. 30 Birth, Death, Infinity Things to consider  When record is sent to database before all the values are assigned, one database operation create  becomes several create update etc.  The record expands as more values are assigned fragmentation will occur

31. © 2013 Progress Software Corporation. All rights reserved. 31 Birth, Death, Infinity Life

32. © 2013 Progress Software Corporation. All rights reserved. 32 Birth, Death, Infinity READ find customer where cust-num = 3447.

33. © 2013 Progress Software Corporation. All rights reserved. 33 Birth, Death, Infinity READ  Compiler determines index to use (perhaps cust-num) R-code has index information Query used to form equality or range bracket  At runtime, load schema cache into memory  Look up the table number (# 3)  Find _storage-object records for table #3 (for tenant #t)  Look up the index number (# 113)  Find _storage-object records for index #113 (for tenant #t)  Load into “om cache” so we can use again (set -omsize)  Get location of index root block from _storage-object  Traverse index b-tree to leaf, perhaps cust-num = 3447  Get row's rowid 9006 from index leaf block  Get share lock on row 9006  Read data block(s) containing row fragments  Copy row fragments to 4GL buffer or network buffer

34. © 2013 Progress Software Corporation. All rights reserved. 34 Birth, Death, Infinity Find row in database Block’s DBKEYTypeChainBackup Ctr Next DBKEY in ChainBlock Update Counter Free Space Free Dirs. Rec 0 OffsetRec 1 Offset Rec 2 OffsetRec n Offset Num Dirs. Free Space Used Data Space Record 0 Record 2 Record 1 server’s row buffer Share locked RM block in buffer pool client’s network message buffer client's row buffer server’s network message buffer TCP/IP

35. © 2013 Progress Software Corporation. All rights reserved. 35 Birth, Death, Infinity row buffer connected CONDB customer order Etc. DBDES CI for customer FDTBL for database foo ICB for table customer row buffer (cust 3447)

36. © 2013 Progress Software Corporation. All rights reserved. 36 Birth, Death, Infinity UPDATES do transaction: find customer where cust-num = 3447 exclusive-lock: customer.city = "Westford". end. end.

37. © 2013 Progress Software Corporation. All rights reserved. 37 Birth, Death, Infinity Assign new value to city “Westford” 4GL runtime stack row buffer (cust 3447) Boston ICB for table customer CI for customer field n

38. © 2013 Progress Software Corporation. All rights reserved. 38 Birth, Death, Infinity Assign new value to city “Westford” 4GL runtime stack row buffer Westford ICB for table customer CI for customer field n

39. when do we send it to the database ?

40. © 2013 Progress Software Corporation. All rights reserved. 40 Birth, Death, Infinity send row to database client's row buffer server's network message buffer server's row buffer client's network message buffer TCP/IP

41. © 2013 Progress Software Corporation. All rights reserved. 41 Birth, Death, Infinity Update data block  Acquire exclusive lock block buffer  read data block into buffer pool  generate "record difference" bi note  spool bi note  modify data block  update free space  mark block modified  release buffer lock

42. © 2013 Progress Software Corporation. All rights reserved. 42 Birth, Death, Infinity Update other data blocks  Indexes have to be updated too (city is a key field)  delete existing index entry for "Boston"  insert new index entry for "Westford"

43. © 2013 Progress Software Corporation. All rights reserved. 43 Birth, Death, Infinity Fragment chain block header RM block header dir entry 0 free space record 0 record 1 record 2 record 3 (part 1 of row) dir entry 1 dir entry 2 dir entry 3 block header RM block header dir entry 0 free space record 0 record 1 (part 2 of row) dir entry 1

44. © 2013 Progress Software Corporation. All rights reserved. 44 Birth, Death, Infinity Death

45. © 2013 Progress Software Corporation. All rights reserved. 45 Birth, Death, Infinity Deleting do transaction: for each customer where city = "Boston": delete customer. end. end.

46. © 2013 Progress Software Corporation. All rights reserved. 46 Birth, Death, Infinity Deleting (sql) delete from customer where city = "Boston";

47. © 2013 Progress Software Corporation. All rights reserved. 47 Birth, Death, Infinity Update data block  Acquire exclusive lock block buffer  read data block into buffer pool  generate "record delete" bi note  spool bi note  modify data block  update free space  mark block modified  release buffer lock

48. © 2013 Progress Software Corporation. All rights reserved. 48 Birth, Death, Infinity Update other data blocks  Indexes have to be updated too  After each record delete delete existing index entry for "Boston" delete all the other index entries too  For unique index entries, replace with a reservation (placeholder) the key value cannot be used until deleting transaction commits

49. © 2013 Progress Software Corporation. All rights reserved. 49 Birth, Death, Infinity deleted ROWID Reservation block header RM block header dir entry 0 free space record 0 record 1 record 2 dir entry 1 dir entry 2 dir entry 3 ROWID cannot be recycled until after the deleting transaction commits. Why ?

50. © 2013 Progress Software Corporation. All rights reserved. 50 Birth, Death, Infinity deleted ROWID Reservation block header RM block header dir entry 0 free space record 0 record 1 record 2 dir entry 1 dir entry 2 dir entry 3 ROWID cannot be recycled until after the deleting transaction commits. Why ? What ELSE can go wrong during UNDO ?

51. What could go wrong ?

52. © 2013 Progress Software Corporation. All rights reserved. 52 Birth, Death, Infinity deleted ROWID Reservation block header RM block header dir entry 0 free space record 0 record 1 record 2 dir entry 1 dir entry 2 dir entry 3 ROWID cannot be recycled until after the deleting transaction commits. Why ? What ELSE can go wrong during UNDO ?

53. © 2013 Progress Software Corporation. All rights reserved. 53 Birth, Death, Infinity deleted ROWID Reservation block header RM block header dir entry 0 free space record 0 record 1 record 2 dir entry 1 dir entry 2 dir entry 3 ROWID cannot be recycled until after the deleting transaction commits. Why ? What can go wrong during UNDO ? Not enough space. another transaction has taken it.

54. Infinity

55. How can a row live after death ?

56. © 2013 Progress Software Corporation. All rights reserved. 56 Birth, Death, Infinity Life after death  Your records live on forever in: Backup archives Archived after-image extents Audit data Archived audit data Binary dump files Reports Dropbox etc.  If you have any of these, where? Can you find a specific record?  Are the data encrypted? Where are the keys ?  Can we subpeona your records ? We may.

57. © 2013 Progress Software Corporation. All rights reserved. 57 Birth, Death, Infinity That’s all we have time for today, except

58. © 2013 Progress Software Corporation. All rights reserved. 58 Birth, Death, Infinity Answers email: gus@progress.com danf@prodb.com