Presentation is loading. Please wait.

Presentation is loading. Please wait.

Storing Data: Disks, Buffers and Files

Similar presentations


Presentation on theme: "Storing Data: Disks, Buffers and Files"— Presentation transcript:

1 Storing Data: Disks, Buffers and Files
Talk starts with wrap up of previous lecture

2 Query Parsing & Optimization
Architecture of a DBMS SQL Client Organized in layers Each layer abstracts the layer below Manage complexity Perf. Assumptions Example of good systems design Database Disk Space Management Buffer Management Files and Index Management Relational Operators Query Parsing & Optimization

3 Disk Space & Buffer Management
Frame Page 4 Page 2 Page 1 Read Page Write Page Database operates on in memory pages. Disk Space Mngmt. Page 1 Page 2 Page 3 Page 4 Page 5 Page 6

4 Overview Table Record File Byte Rep. Record Slotted Page Bob M 32
94703 Harmon Name Sex Age Zip Addr Alice F 33 Mabel Jose 31 94110 Chavez Jane 30 Table Record Bob M 32 94703 Harmon Varchar Char Int File Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Byte Rep. Record Header M 32 94703 Bob Harmon Slotted Page Page Header

5 Overview: Files of Pages of Records
Tables stored as a logical files consisting of pages each containing a collection of records Pages are managed in memory by the buffer manager: higher levels of database only operate in memory on disk by the disk space manager: reads and writes pages to physical disk/files Big Ideas in this Lecture Block/Pages granularity reasoning Exploit access patterns in memory management Efficient binary representation of data

6 Query Parsing & Optimization
Architecture of a DBMS SQL Client Database Disk Space Management Buffer Management Files and Index Management Relational Operators Query Parsing & Optimization Illusion of operating in memory RAM Frame Frame Frame Page 1 Page 4 Page 2 Disk Space Management

7 Mapping Pages into Memory
Disk Space Mngmt. Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Main Memory (Buffer Pool) Frame Frame Frame Page 1 Page 4 Database operates on in memory pages. Load Page Write Page

8 Challenges for Buffer Manager
What happens if a page is modified? Writing “dirty” pages back to disk No free frames  evict which page? Replacement Policy What if page is being used? Pinning Concurrent page operations? Solve this in lock manager …

9 Mapping Pages into Memory
Files and Index Management Disk Space Mngmt. Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Noticed pages changed from earlier example Buffer Pool Pointer * Pinned Pinned Frame Frame Frame Page 5 Page 1 Page 6 Load Page Request Page 6 Write Page

10 Mapping Pages into Memory
Disk Space Mngmt. Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Noticed pages changed from earlier example Buffer Pool 1 2 1 Frame Frame Frame Page 5 Page 2 Page 3 Load Page Finished Pointer * Page 2 Write Page

11 Page Replacement Policy
Page is chosen for replacement by a replacement policy: Least-recently-used (LRU), Clock Most-recently-used (MRU) Policy can have big impact on #I/O’s; Depends on the access pattern. Once a hot area of research but LRU/Clock has become pretty popular

12 LRU Replacement Policy
Least Recently Used (LRU) Pinned Frame: not available to replace track time each frame last unpinned (end of use) replace the frame which least recently unpinned Very common policy: intuitive and simple Works well for repeated accesses to popular pages (temporal locality) Can be costly. Why? Need to maintain heap data-structure Solution?

13 Clock Replacement Policy
Empty Frame Approx. to LRU Arrange frames in logical cycle Introduce “Ref. Bit” Clock arm advances until request satisfied

14 Clock Replacement Policy
Pinned Empty Frame A Want to insert page: Current frame has pin-count > 0  Skip Pinned G F B E C D

15 Clock Replacement Policy
Pinned Empty Frame A Want to insert page: Not Pinned and Ref. bit is set  Clear Ref. Bit Skip Pinned G F B E C D

16 Clock Replacement Policy
Pinned Empty Frame A Want to insert page: Not pinned and Ref. bit unset: Evict Copy Page Set pinned Set ref. bit Advanced clock Pinned G F B E C Pinned D

17 Clock Replacement Policy
Pinned Empty Frame A Request for Page C: Cache Hit!: Pin (inc.) Set ref bit. Pinned F B Pinned E C Pinned G

18 Is LRU/Clock Always Best?
Very common policy: intuitive and simple Works well for repeated accesses to popular pages temporal locality LRU can be costly  Clock policy is cheap When might it perform poorly What about repeated scans of big files?

19 Repeated Scan of Big File (LRU)
Kind of big File Empty Frame Buffer A B C D Scan Cache Hits: Attempts:

20 Repeated Scan of Big File (LRU)
Empty Frame Buffer A B C D Scan A Scan Cache Hits: Attempts: 1

21 Repeated Scan of Big File (LRU)
Empty Frame Buffer A B C D A B Scan Scan Cache Hits: Attempts: 2

22 Repeated Scan of Big File (LRU)
Empty Frame Buffer A B C D A B C Scan Scan Cache Hits: Attempts: 3

23 Repeated Scan of Big File (LRU)
Empty Frame Buffer A B C D A B C LRU Scan Cache Hits: Attempts: 3 4 D Scan

24 Repeated Scan of Big File (LRU)
Empty Frame Buffer A B C D A Scan D B C LRU Scan Cache Hits: Attempts: 5 4

25 Repeated Scan of Big File (LRU)
Empty Frame Buffer A B C D D A C B Scan LRU Scan Cache Hits: Attempts: 5 6

26 Repeated Scan of Big File (LRU)
Empty Frame Buffer A B C D Sequential Flooding D A B Scan LRU Scan No Cache Hits! Cache Hits: Attempts: 6

27 Repeated Scan of Big File (MRU)
Most Recently Used File Empty Frame Buffer A B C D Scan Scan Cache Hits: Attempts:

28 Repeated Scan of Big File (MRU)
Empty Frame Buffer A B C D Scan A Scan Cache Hits: Attempts: 1

29 Repeated Scan of Big File (MRU)
Empty Frame Buffer A B C D A B Scan Scan Cache Hits: Attempts: 2

30 Repeated Scan of Big File (MRU)
Empty Frame Buffer A B C D A B C Scan Scan Cache Hits: Attempts: 3

31 Repeated Scan of Big File (MRU)
Empty Frame Buffer A B C D A B C MRU Scan Cache Hits: Attempts: 4 3 D Scan

32 Repeated Scan of Big File (MRU)
Empty Frame Buffer A B C D Hit! Scan A B D MRU Scan 1 Cache Hits: Attempts: 5

33 Repeated Scan of Big File (MRU)
Empty Frame Buffer A B C D Hit! A B D Scan MRU Scan 2 Cache Hits: Attempts: 6

34 Repeated Scan of Big File (MRU)
Empty Frame Buffer A B C D A B D MRU Scan C 2 Scan Cache Hits: Attempts: 6 7

35 Repeated Scan of Big File (MRU)
Empty Frame Buffer A B C D Hit! A C D MRU Scan 3 Improved Cache Hit Rate! Cache Hits: Attempts: 8 Scan

36 Repeated Scan of Big File (MRU)
Empty Frame Buffer A B C D Hit! What else might we do for sequential scans? A B D MRU Scan 3 Improved Cache Hit Rate! Cache Hits: Attempts: 8 Scan

37 Background Prefetching
File Empty Frame Buffer A B C D Prefetched Prefetched Scan A B C Scan Load (prefetch) “next” pages in a background thread Why does this help? Disk Scheduling & Parallel IO Interleave IO and compute

38 DBMS vs. OS Buffer Cache OS has page/buffer management. Why not let OS manage these tasks? Buffer management in DBMS requires ability to: pin page in buffer pool, force page to disk, order writes important for implementing CC & recovery adjust replacement policy, and pre-fetch pages based on access patterns in typical DB operations.

39 Query Parsing & Optimization
Architecture of a DBMS SQL Client Database Disk Space Management Buffer Management Files and Index Management Relational Operators Query Parsing & Optimization Illusion of operating in memory RAM Frame Frame Frame Page 1 Page 4 Page 2 Disk Space Management

40 Query Parsing & Optimization
Architecture of a DBMS SQL Client Database Disk Space Management Buffer Management Files and Index Management Relational Operators Query Parsing & Optimization Organizing tables and records as groups of pages in a logical file. Bob M 32 94703 Harmon Name Sex Age Zip Addr Alice F 33 Mabel Jose 31 94110 Chavez Jane 30 Page Header

41 Files Table Record File Byte Rep. Record Slotted Page Bob M 32 94703
Harmon Name Sex Age Zip Addr Alice F 33 Mabel Jose 31 94110 Chavez Jane 30 Table Record Bob M 32 94703 Harmon Varchar Char Int File Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Byte Rep. Record Header M 32 94703 Bob Harmon Slotted Page Page Header

42 Files of Pages of Records
Higher levels of DBMS operate on pages of records and files of pages. FILE: A collection of pages, each containing a collection of records. Must support: insert/delete/modify record fetch a particular record by record id ... Think: pointer encoding Page_ID and location on page. scan all records (possibly with some conditions on the records to be retrieved) Could span multiple OS files and even machines Or “raw” disk devices

43 Many Kinds of Database Files
Unordered Heap Files Records placed arbitrarily across pages Clustered Heap File and Hash Files Records and pages are grouped Sorted Files Page and records are in sorted order Index Files B+ Trees, Hash Tables, … May contain records or point to records in other files

44 Basic Unordered Heap Files
Collection of records in no particular order Not to be confused with ”heap data-structure” As file shrinks/grows, pages (de)allocated To support record level operations, we must: keep track of the pages in a file keep track of free space on pages keep track of the records on a page There are many alternatives for keeping track of this, we’ll consider two in this class

45 Heap File Implemented as a List
Data Page Data Page Data Page Full Pages Header Page Data Page Data Page Data Page Pages with Free Space Header page ID and Heap file name stored elsewhere Database “catalog” Each page contains 2 “pointers” plus free-space and data. What is wrong with this? How do I find a page with enough space for a 20 byte record?

46 Better: Use a Page Directory
Data Page 1 Page 2 Page N Header Page DIRECTORY Directory entries include #free bytes on the page. Header pages accessed often  likely in cache What eviction policy is best here? Finding a page to fit a record required far fewer page loads than linked list (Why?) One header page load reveals free space of many pages.

47 Indexes (sneak preview)
A Heap file allows us to retrieve records: by specifying the record id (page id + slot) by scanning all records sequentially Would like to fetch records by value, e.g., Find all students in the “CS” department Find all students with a gpa > 3 AND blue hair Indexes: file structures for efficient value-based queries

48 Table encoded as files which are collections of pages.
Overview Bob M 32 94703 Harmon Name Sex Age Zip Addr Alice F 33 Mabel Jose 31 94110 Chavez Jane 30 Table Record Bob M 32 94703 Harmon Varchar Char Int Summary Heap File Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Table encoded as files which are collections of pages. Byte Rep. Record Header M 32 94703 Bob Harmon Slotted Page Page Header

49 How do we store records on a page?
Overview Bob M 32 94703 Harmon Name Sex Age Zip Addr Alice F 33 Mabel Jose 31 94110 Chavez Jane 30 Table Record Bob M 32 94703 Harmon Varchar Char Int How do we store records on a page? Heap File Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Byte Rep. Record Header M 32 94703 Bob Harmon Slotted Page Page Header

50 Page Basics: The Header
Blank Page 128KB

51 Page Basics: The Header
Page Header Header may contain: Number of records Free space Maybe next/last pointer Slot Table … more soon

52 Things to Consider Record length? Fixed or Variable
Page Header Record length? Fixed or Variable Find records by record id? Offset… How do we add and delete records? Bitmaps & Slot Tables

53 Fixed Length Records: Packed
Page Header Record Record Id: (Page 2, Record 4) Record Record Record Record Record Record Record Free Space Pack records densely Record id: record number in page Easy to add: just append Delete?

54 Fixed Length Records: Packed
Page Header Record Record Id: (Page 2, Record 4) Record Record Record Wrong Record Free Space Pack records densely Record id: record number in page Easy to add: just append Delete? Re-arrange …

55 Fixed Length Records: Unpacked
Page Header Bitmap Record Id: (Page 2, Record 4) Record Record Record Record Record Record Record Record Record Free Space Bitmap denotes “slots” with records Record id: record number in page Insert: find first empty slot Delete?

56 Fixed Length Records: Unpacked
Page Header Bitmap Record Id: (Page 2, Record 4) Record Record Record Record Record Record Record Record Free Space Bitmap denotes “slots” with records Record id: record number in page Insert: find first empty slot Delete: Clear bit

57 Variable Length Records
Header Record Record Record Record Record Record Record Record Free Space How do we know where each record begins? What happens when we add and delete records?

58 Slotted Page Introduce slot directory in header
Record Id: (Page 2, Record 4) 16 Header 24 32 12 18 Slot Directory Record Record Record Record Record Record Record Record Free Space Introduce slot directory in header Length + Pointer to beginning of record Pointer to free space Record ID = location in slot table Delete? (e.g., 3rd record on the page)

59 Slotted Page Delete: Set pointer to null.
Record Id: (Page 2, Record 4) 16 Header 24 32 12 18 Slot Directory Record Record Record Record Record Record Record Record Free Space Delete: Set pointer to null. Doesn’t affect pointers to other records However, need to make sure we remove any references to record_id in indexes

60 Slotted Page Insert? Header Record Record Record Record Record Record
16 Header 24 12 18 Slot Directory Record Record Record Record Record Record Free Space Insert? Record

61 Slotted Page Insert: Place record in free space on page Header Record
16 Header 24 12 18 Slot Directory Record Record Record Record Record Record Free Space Insert: Place record in free space on page Record

62 Slotted Page Insert: Place record in free space on page
16 Header 24 18 12 18 Slot Directory Record Record Record Record Record Record Record Free Space Insert: Place record in free space on page Create pointer in next open slot in slot directory Fragmentation?

63 Slotted Page Insert: Place record in free space on page
Header 16 24 18 12 18 Slot Directory Record Record Record Record Record Record Record Free Space Insert: Place record in free space on page Create pointer in next open slot in slot directory Reorganize data on page. Is this safe?

64 What if we need more slots?
Slotted Page Header 16 18 24 12 18 Slot Directory Record Record Record Record Is this safe? Record Record Free Space When? What if we need more slots? Insert: Place record in free space on page Create pointer in next open slot in slot directory Reorganize data on page.

65 Slotted Page: Growing Slots
16 Header 6 18 24 12 18 Slot Directory Record Record Record Record Record Record Free Space Track number of slots in slot directory

66 Slotted Page: Growing Slots
Header 6 16 24 18 12 18 Slot Directory Free Space Record Record Record Record Record Record Track number of slots in slot directory Grow records from other end of page Why?

67 Slotted Page: Growing Slots
Header 6 7 6 16 24 18 12 18 Slot Directory Free Space Record Record Record Record Record Record Record Track number of slots in slot directory Grow records from other end of page Extend slot directory on insert Add record in free space & update counter

68 Slotted Page: Growing Slots
Header 7 6 16 24 18 12 18 6 Slot Directory Header 12 Free Space Record Record Record Record Record Record Record Record Record Track number of slots in slot directory Grow records from other end of page Extend slot directory on insert Add record in free space & update counter

69 Slotted Page Summary Typically use slotted Page
Header 7 6 16 24 18 12 18 6 Slot Directory Header 12 Free Space Record Record Record Record Record Record Record Record Record Typically use slotted Page Good for variable and fixed length records Good for fixed length records too. Why? Re-arrange (e.g., sort) and squash null fields

70 Store records on slotted page
Overview Bob M 32 94703 Harmon Name Sex Age Zip Addr Alice F 33 Mabel Jose 31 94110 Chavez Jane 30 Table Record Bob M 32 94703 Harmon Varchar Char Int Store records on slotted page Heap File Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Byte Rep. Record Header M 32 94703 Bob Harmon Slotted Page Page Header

71 Overview Table Record Heap File Byte Rep. Record Slotted Page Bob M 32
94703 Harmon Name Sex Age Zip Addr Alice F 33 Mabel Jose 31 94110 Chavez Jane 30 Table Record Bob M 32 94703 Harmon Varchar Char Int Heap File Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Byte Rep. Record Header M 32 94703 Bob Harmon Slotted Page Page Header

72 Record Formats Goals: Easy Case: Fixed Length Fields
Relational Model  Each record in table has same fixed type Assume System Catalog with Schema No need to store type information (save space!) This will be another table … (bootstraping) Goals: Compact in memory & disk format Fast access to fields (why?) Easy Case: Fixed Length Fields Interesting Case: Variable Length Fields

73 Record Formats: Fixed Length
3 T HELLO_W INTEGER DOUBLE BOOLEAN CHARACTER(7) 3.142 4 8 1 7 24 Field types same for all records in a file. Type info stored separately in system catalog On disk byte representation same as in memory Finding i’th field? done via arithmetic (fast) Compact? (Nulls?) Compact? We could have repeated fields or allocate large integers when small integers would suffice for most entries. Nulls? How do we encode if a particular field is missing. Need a missing value. Could use a bitset. May opt for variable length record format next …

74 Record Formats: Variable Length
What happens if fields are variable length? Could store with padding? (Fixed Length) Record Bob M 32 94703 Big, St. VARCHAR CHAR INT Wasted Space Bob M 32 94703 Big, St. CHAR(20) CHAR(18) CHAR INT Alice M 32 94703 CHAR(20) CHAR(18) CHAR INT Boulevard of the Allies Field Not Big Enough

75 Record Formats: Variable Length
What happens if fields are variable length? Could use delimiters (i.e., CSV): Issues? Record Bob M 32 94703 Big, St. VARCHAR CHAR INT Comma Separated Values (CSV) Bob VARCHAR Big, St. M CHAR 32 INT 94703 ,

76 Record Formats: Variable Length
What happens if fields are variable length? Could use delimiters (i.e., CSV): Requires scan to access field What if text contains commas? Record Bob M 32 94703 Big, St. VARCHAR CHAR INT Comma Separated Values (CSV) Bob VARCHAR Big, St. M CHAR 32 INT 94703 , ,

77 Record Formats: Variable Length
What happens if fields are variable length? Store length information before fields: Requires scan to access field What if text contains commas? Record Bob M 32 94703 Big, St. VARCHAR CHAR INT Variable Length Fields with Offsets Bob VARCHAR Big, St. M CHAR 32 INT 94703 3 8 Move all variable length fields to end enable fast access

78 Record Formats: Variable Length
What happens if fields are variable length? Introduce a record header: Requires scan to access field. Why? What if text contains commas? Record Bob M 32 94703 Big, St. VARCHAR CHAR INT Bob VARCHAR Big, St. M CHAR 32 INT 94703 Header

79 Record Formats: Variable Length
What happens if fields are variable length? Introduce a record header: Direct access & no “escaping”, other adv.? Handle null fields  useful for fixed length Record Bob M 32 94703 Big, St. VARCHAR CHAR INT Bob VARCHAR Big, St. M CHAR 32 INT 94703 Header

80 How is all this organized?
Overview Bob M 32 94703 Harmon Name Sex Age Zip Addr Alice F 33 Mabel Jose 31 94110 Chavez Jane 30 Table Record Bob M 32 94703 Harmon Varchar Char Int Heap File Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Byte Rep. Record Header M 32 94703 Bob Harmon How is all this organized? Slotted Page Page Header

81 System Catalogs For each relation: For each index: For each view:
name, file location, file structure (e.g., Heap file) attribute name and type, for each attribute index name, for each index integrity constraints For each index: structure (e.g., B+ tree) and search key fields For each view: view name and definition Plus statistics, authorization, buffer pool size, etc. Catalogs are themselves stored as relations!

82 pg_attribute

83 Summary Disk manager loads and stores pages
Block level reasoning Abstracts device and file system; provides fast next Buffer manager brings pages into RAM page pinned while reading/writing dirty pages written to disk good replacement policy essential for performance DBMS “File” tracks collection of pages, records within each. Heap-files: unordered records organized with directories

84 Summary (Contd.) Slotted page format Variable length record format
Variable length records and intra-page reorg Variable length record format Direct access to i’th field and null values. Catalog relations store information about relations, indexes and views.

85 Query Parsing & Optimization
Architecture of a DBMS SQL Client Organized in layers Each layer abstracts the layer below Manage complexity Perf. Assumptions Example of good systems design Database Disk Space Management Buffer Management Files and Index Management Relational Operators Query Parsing & Optimization


Download ppt "Storing Data: Disks, Buffers and Files"

Similar presentations


Ads by Google