1 PART III STORAGE AND INDEXING CH. 8: OVERVIEW OF STORAGE AND INDEXING (introduction) CH. 9: STORING DATA: DISKS AND FILES (hardware) CH. 10: TREE-STRUCTURED INDEXING CH. 11: HASH-BASED INDEXING (two important types of indexing)

2 APPROACH NO FORMAL UNIFIED THEORY. PRESENTATION COVERS KEY IDEAS IN AD HOC FASHION. BASIC METHODOLOGY: SOFTWARE ENGINEERING

3 Engineering as Optimization or Rather as Nonlinear Programming The goal of engineering is to create a design for an object or a system so as to optimize a specified Criterion Function subject to certain Constraints.

4 FORMAL STATEMENT OF THE GOAL OF ENGINEERING Max (CF( x 1, x 2,..., x n, p 1, p 2,..., p m )) (x 1, x 2, …, x n ) S 1 ( x 1, x 2,..., x n, p 1, p 2,..., p m )  k 1 S 2 ( x 1, x 2,..., x n, p 1, p 2,..., p m )  k 2  (constraint  equations)  S q ( x 1, x 2,..., x n, p 1, p 2,..., p m )  k q  where p i : parameters x j : variables If all the equations are linear: LINEAR PROGRAMMING PROBLEM If any equation is nonlinear: NONLINEAR PROGRAMMING PROBLEM

5 ILLUSTRATION: 1-variable SQ1 (X>k1) SQ2 (X<k2) CF X Figure 1. CF vs X CF max X CFmax

6 PROBLEMS CF generally unknown (subjective decision) Parameters partly known Variables partly known (subjective decision) No algorithm for the nonlinear programming problem CONCLUSION: The formal approach is to be used solely as a guideline. Actual Software Engineering: Seek algorithms and techniques minimizing - memory space requirements - computing time

7 CONSTRAINTS Database size requirements + Cost of High Speed Memory  Use of secondary storage for Database. Data processing to be done in HSM  constant stream of I/O operations High cost (in time) of I/O  minimize the number of I/Os Unavoidable defects in magnetic disk layer  guaranteed error detection and correction

8 TERMINOLOGY  Attribute a value  Tuple 1:    Attribute k value  Instance of Relation A    Conceptual Schema:   Tuple n   Instance of Relation Z  Field a value  Record # 1:    Field k value  File for Relation A    Physical Schema:   Record # n   File for Relation Z

9 BLOCKS AND PAGES Confusing descriptions: “A disk block is a contiguous sequence of bytes (residing on a disk) and is the unit in which data is written to a disk and read from a disk” (text p. 306). “The unit of information read from or written to disk is a page” (text p.274). Disc concept seems more fundamental: block sizes are set by the OS at initialization and cannot be changed dynamically. Pages describe the subdivision of the buffer space set aside in the HSM to accommodate disc blocks during I/O transfers. Their size is set by the DBMS. Page size is normally set equal to block size The words “page” and “block” are often used interchangeably; No harm as long as their size is the same.

10 FILE ORGANIZATION AND INDEXING Two (fairly independent) software tools for improving performance: File organization: a method of arranging the records in a file when the file is stored on disks. Five such methods are described in the text. Indexing; a technique (used in conjunction with certain file organizations) which can improve record searches by adding index files to the database (quite similar to book indexes). Several indexing methods are mentioned in the text, the main emphasis being on Hashing and B trees-based methods.

11 MAJOR CONCEPTS AND NOTATION Search Key: attribute value(s) of sought record(s), denoted k. Data Record(s): actual records as stored on disk. Record Id: disk address of a record permitting DBMS to locate it, denoted rid. Data Entry: record stored in an index file permitting to locate the sought record having search key value k; denoted k *.

12 SOME PROPERTIES OF INDEXES Clustering An index is clustered when the ordering of data records is the same as or close to the ordering of the data entries. An index that is not clustered is said to be unclustered, Clustered indexes are quite beneficial as they generally minimize search times. Primary and Secondary indexes A primary index is an index on a set of fields that includes the primary key. Other indexes are called secondary indexes

13 CLUSTERD vs UNCLUSTERED INDEXES

14 ALTERNATIVE METHODS FOR STORING DATA ENTRIES Alternative (1): A data entry k* is an actual data record with search key value k (see left side of figure 8.2). Alternative (2): A data entry is a pair, where rid is the record id of a data record with search key value k (see right side of figure 8.2). Alternative (3): A data entry is a pair, where rid-list is a list of record ids of data records with search key value k (this alternative is not shown on figure 8.2. It corresponds precisely to the entries in a book index).

15 FIGURE 8.2 Alternative 1 Alternative 2

16 HASH-BASED INDEXING The Hash Function: h(k) = bucket address - Bucket: Primary page + optional chain linked overflow pages’. - A ‘good’ hashing function: even distribution of keys into buckets. - A poor hashing function: too many records per bucket producing long linked chains of overflow pages requiring multiple I/Os. - The search key can be any sequence of one or more fields. - There are several hash based indexing techniques (see chapter 11), with various ways to solve OF problem.

17 TREE-BASED INDEXING

18 TREE-BASED INDEXING - Data entries located in leaf levels, arranged in sorted order by search key values. - Non-leaf pages contain node pointers separated by search key values. - Hierarchical search data structure directs searches to correct page of data entries. - Each node is a data page; retrieving a node = 1 disk I/O. - Number of I/O needed for search = path length (root-to-leaf) + no. of pages with qualifying data entries. - B+ tree: balanced B tree: same length for every path (root-to-leaf). height = path length. fan-out = average no. of children in non-leaf nodes. Detail study in chapter 1.0 +

19 COMPARISON OF FILEORGANIZATIONS Files to be compared: Heap file Sorted files Clustered B+ tree files Heap file with unclustered B+ tree index Heap file with unclustered hash index Operations considered: Scan Search with Equality Selection Search with Range Selection Insert a Record Delete a Record

20 COST MODEL - ASSUMPTIONS B = # of (packed) data pages R = # of records /page D = avg time for I/O (≈15 ms) C = avg time to process a record (≈100 ns) H = avg time to apply hash function (≈ 100 ns)

21 COST MODEL - RESULTS FILE TYPE ScanEquality SearchRange SearchInsertDelete Heap B(D + RC) ≅ BD 0.5B(D+RC) ≅ 0.5 BD B(D+RC) ≅ BD 2D + C ≅ 2D Search+C+D ≅ Search+D SortedB(D+RC) ≅ BD Dlog 2 B+Clog 2 R ≅ Dlog 2 B Dlog 2 B+# matching pages Search + BD Search + BD Clustered1.5 B(D+RC) ≅ 1.5 BD Dlog F 1.5B+Clog 2 R ≅ Dlog F 1.5B DlogF1.5B +# matching pages Search + D Search + D Unclustered tree index ≅ BD (R )D(1+log F 0.15B) D(1+log F 0.15B+# matching records) D(3+ log F 0.15B) Search + 2D Unclustered hash index BD(R )2DBD4DSearch + 2D