1. Bitmap Indexes

2. Bitmap Indices
Bitmap indices are a special type of index designed for efficient querying on multiple keys
Very effective on attributes that take on a relatively small number of distinct values
E.g. gender, country, state, …
E.g. income-level (income broken up into a small number of levels such as , , , infinity)
A bitmap is simply an array of bits
For each gender, we associate a bitmap, where each bit represents whether or not the corresponding record has that gender.

3. Bitmap Indices (Cont.)
In its simplest form a bitmap index on an attribute has a bitmap for each value of the attribute
Bitmap has as many bits as records
In a bitmap for value v, the bit for a record is 1 if the record has the value v for the attribute, and is 0 otherwise

4. Bitmap Indices (Cont.)
Bitmap indices are useful for queries on multiple attributes
not particularly useful for single attribute queries
Queries are answered using bitmap operations
Intersection (and)
Union (or)
Complementation (not)
Each operation takes two bitmaps of the same size and applies the operation on corresponding bits to get the result bitmap
E.g AND =
OR = NOT =
Males with income level L1:
And’ing of Males bitmap with Income Level L1 bitmap
10010 AND = 10000
Can then retrieve required tuples.
Counting number of matching tuples is even faster

5. Bitmap Indices (Cont.)
Bitmap indices generally very small compared with relation size
E.g. if record is 100 bytes, space for a single bitmap is 1/800 of space used by relation.
If number of distinct attribute values is 8, bitmap is only 1% of relation size
Deletion needs to be handled properly
Existence bitmap to note if there is a valid record at a record location
Needed for complementation
not(A=v): (NOT bitmap-A-v) AND ExistenceBitmap
Should keep bitmaps for all values, even null value
To correctly handle SQL null semantics for NOT(A=v):
intersect above result with (NOT bitmap-A-Null)

6. Index Definition in SQL
Create a B-tree index (default in most databases)
create index <index-name> on <relation-name> (<attribute-list>)
-- create index b-index on branch(branch_name)
-- create index ba-index on branch(branch_name, account) -- concatenated index
-- create index fa-index on branch(func(balance, amount)) – function index
Use create unique index to indirectly specify and enforce the condition that the search key is a candidate key.
Hash indexes: not supported by every database (but implicitly in joins,…)
PostgresSQL has it but discourages due to performance
Create a bitmap index
create bitmap index <index-name> on <relation-name> (<attribute-list>)
For attributes with few distinct values
Mainly for decision-support(query) and not OLTP (do not support updates efficiently)
To drop any index
drop index <index-name>

7. Query Processing

8. General Overview Relational model - SQL Functional Dependencies
Formal & commercial query languages
Functional Dependencies
Normalization
Physical Design
Indexing
Query Processing and Optimization

9. Review Data Retrieval at the physical level:
Indices: data structures to help with some query evaluation:
SELECTION queries (ssn = 123)
RANGE queries (100 <= ssn <=200)
Index choices: Primary vs secondary, dense vs sparse, ISAM vs B+- tree vs Extendible Hashing vs Linear Hashing
But what about join queries? Or other queries not directly supported by the indices? How do we evaluate these queries?
Sometimes, indexes not useful, even for SELECTION queries. When? What decides when to use them?
A: Query Processing (one of the most complex
components of a database system)

10. QP & O
SQL Query
Query
Processor
Data: result of the query

11. QP & O SQL Query Query Processor Parser Query Optimizer Evaluator
Algebraic
Expression
Execution plan
Evaluator
Data: result of the query

12. QP & O Query Optimizer Algebraic Query Rewriter Representation
Algebraic Representation
Plan Generator
Data Stats
Query Execution Plan

13. Query Processing and Optimization
Parser / translator (1st step)
Input: SQL Query (or OQL, …)
Output: Algebraic representation of query (relational algebra expression)
Eg SELECT balance
FROM account
WHERE balance < 2500
balance(balance2500(account)) or
balance
balance2500
account

14. QP & O Plan Evaluator (last step) Input: Query Execution Plan
Output: Data (Query results)
Query execution plan
Algorithms of operators that read from disk:
Sequential scan
Index scan
Merge-sort join
Nested loop join
…..

15. QP & O Query Rewriting Input: Algebraic representation of query
Output: Algebraic representation of query
Idea: Apply heuristics to generate equivalent expression that is likely to lead to a better plan
e.g.: amount > 2500 (borrower loan)
borrower (amount > 2500(loan))
Why is 2nd better than 1st?

16. QP & O Plan Generator Input: Algebraic representation of query
Output: Query execution plan
Idea:
generate alternative plans on evaluating a query
amount > 2500
Estimate cost for each plan
Choose the plan with the lowest cost
Sequential scan
Index scan

17. QP & O Goal: generate plan with minimum cost (i.e., fast as possible)
Cost factors:
CPU time (trivial compared to disk time)
Disk access time
main cost in most DBs
Network latency
Main concern in distributed DBs
Our metric: count disk accesses

18. Cost Model How do we predict the cost of a plan? Ans: Cost model
For each plan operator and each algorithm we have a cost formula
Inputs to formulas depend on relations, attributes
Database maintains statistics about relations for this (Metadata)

19. Metadata
Given a relation r, DBMS likely maintains the following metadata:
Size (# of tuples) nr
Size (# of blocks) br
Block size (#tuples) fr
(typically br =  nr / fr  )
Tuple size (in bytes) sr
Attribute Variance (for each attribute r, # of different values) V(att, r)
Selection Cardinality (for each attribute in r, expected size of a selection: att = K (r ) ) SC(att, r)

20. Example naccount = 6 saccount = 33 bytes faccount = 4K/33
V(balance, account) = 3
V(acct_no, account) = 6
S(balance, account) = ( nr / V(att, r))

21. Some typical plans and their costs
Query: att = K (r )
A1 (linear search). Scan each file block and test all records to see whether they satisfy the selection condition.
Cost estimate (number of disk blocks scanned) = br
br denotes number of blocks containing records from relation r
If selection is on a key attribute, cost = (br /2)
stop on finding record (on the average in the middle of the file)
Linear search can be applied regardless of
selection condition or
ordering of records in the file, or
availability of indices

22. Selection Operation (Cont.)
Query: att = K (r )
A2 (binary search). Applicable if selection is an equality comparison on the attribute on which file is ordered.
Requires that the blocks of a relation are stored contiguously
Cost estimate:
log2(br) — cost of locating the first tuple by a binary search on the blocks
Plus number of blocks containing records that satisfy selection condition
EA2 = log2(br) + sc(att, r) / fr -1
What is the cost if att is a key?
EA2 = log2(br)

23. Example V(bname, account) = 50
Query: bname =“Perry” ( account )
V(bname, account) = 50
naccount = 10K faccount = 20 tuples/block
Primary index on bname
Key: acct_no
Cost Estimates:
A1: EA1 = naccount / faccount  = 500 I/O’s
A2: EA2 = log2(br) + sc(att, r) / fr -1 = = 18 I/O’s

24. More Plans for selection
What if there is an index on att?
We need metadata on size of index (i). DBMS keeps that of:
Index height: HTi
Index “Fan Out”: fi
Average # of children per node (not same as order..)
Index leaf nodes: LBi
Note: HTi ~ logfi(LBi) + 1

25. More Plans for selection
Query: att = K (r )
A3: Index scan, Primary Index
What: Follow primary index, searching for key K
Prereq: Primary index on att, i
Cost:
EA3 = HTi + 1, if att is a candidate key
EA3 = HTi + SC(att, r) / fr, if not

26. A5: Index scan, Secondary Index
What: Follow according index, searching for key K
Prereq: Secondary index on att, i
Cost:
if att not a key:
EA4 = HTi SC(att, r)
Else, if att is a key: EA4 = HTi + 1
bucket read
Index block
reads
File block reads
(in worst case, each
tuple on different block)

27. Cardinalities
Cardinality: the size (number of tuples) in the query result
Why do we care?
Ans: Cost of every plan depends on nr
e.g.
Linear scan: br +  nr / fr
Primary Index: HTi +1 ~ logfi(LBi) +2 ≤ logfi(nr / fr )+2
But, what if r is the result of another query?
Must now the size of query results as well as cost
Size of att = K (r ) ? SC(att, r)

28. Selections Involving Comparisons
Query: Att  K (r )
A6 (primary index, comparison). (Relation is sorted on Att)
For Att  V(r) use index to find first tuple  v and scan relation sequentially from there
For AttV (r) just scan relation sequentially till first tuple > v; do not use index
Cost: EA5 =HTi + c / fr (where c is the cardinality of result)
HTi k
... k

29. Query: Att  K (r ) Cardinality: More metadata on r are needed:
min (att, r) : minimum value of att in r
max(att, r): maximum value of att in r
Then the selectivity of Att = K (r ) is estimated as:
(or nr /2 if min, max unknown)
Intuition: assume uniform distribution of values between min and max
min(attr, r) K
max(attr, r)

30. Plan generation: Range Queries
Att K (r )
A6: (secondary index, comparison).
Cost:
EA6 = HTi -1+ #of leaf nodes to read + # of file blocks to read
= HTi -1+ LBi * (c / nr) + c, if att is a candidate key
HTi
...
k, k+1
k+m
...
k+1
k+m k

31. Plan generation: Range Queries
A6: (secondary index, range query). If att is NOT a candidate key
HTi
...
k, k+1
k+m
... k
k+1
k+m k
...
...

32. Cost: EA6 = HTi -1+ #of leaf nodes to read + #of file blocks to read +#buckets to read
= HTi -1+ LBi * (c / nr) + c + x

33. Join Operation Metadata: ncustomer = 10,000 ndepositor = 5000
Size and plans for join operation
Running example: depositor customer
Metadata:
ncustomer = 10, ndepositor = 5000
fcustomer = fdepositor = 50
bcustomer= bdepositor= 100
V(cname, depositor) = 2500 (each customer has on average 2 accts)
cname in depositor a foreign key for customer
depositor(cname, acct_no)
customer(cname, cstreet, ccity)

34. Cardinality of Join Queries
What is the cardinality (number of tuples) of the join?
E1: Cartesian product: ncustomer * ndepositor = 50,000,000
E2: Attribute cname common in both relations, 2500
different cnames in depositor
Size: ncustomer * (avg# of tuples in depositor with same cname)
= ncustomer * (ndepositor / V(cname, depositor))
= 10,000 * (5000 / 2500)
= 20,000

35. Cardinality of Join Queries
E3: cname is a foreign key for depositor on customer
Size: ndepositor * (avg # of tuples in customer with same cname)
= ndepositor * 1
= 5000
Note: If cname is a key for customer but NOT a foreign key for depositor,
then 5000 an UPPER BOUND
Some customer names may not match w/ any customers in customer

36. Cardinality of Joins in general
Assume join: R S
If R, S have no common attributes: nr * ns
If R,S have attribute A in common:
(take min)
If R, S have attribute A in common and:
A is a candidate key for R: ≤ ns
A is candidate key in R and candidate key in S : ≤ min(nr, ns)
A is a key for R, foreign key for S: = ns

37. Nested-Loop Join Algorithm 1: Nested Loop Join Idea: Query: R S t1 u1
Blocks
of... t2 u2 t3 u3 R S
results
Compare: (t1, u1), (t1, u2), (t1, u3) .....
Then: GET NEXT BLOCK OF S
Repeat: for EVERY tuple of R

38. Nested-Loop Join Algorithm 1: Nested Loop Join
Query: R S
Algorithm 1: Nested Loop Join
for each tuple tr in R do for each tuple us in S do test pair (tr,us) to see if they satisfy the join condition if they do (a “match”), add tr • us to the result.
R is called the outer relation and S the inner relation of the join.

39. Nested-Loop Join (Cont.)
Cost:
Worst case, if buffer size is 3 blocks br + nr  bs disk accesses.
Best case: buffer big enough for entire INNER relation + 2
br + bs DAs.
Assuming worst case memory availability cost estimate is
5000  = 2,000,100 disk accesses with depositor as outer relation, and
10000  = 1,000,400 disk accesses with customer as the outer relation.
If smaller relation (depositor) fits entirely in memory, the cost estimate will be 500 disk accesses. (actually we need 2 more blocks)

40. Join Algorithms Algorithm 2: Block Nested Loop Join Idea: Query: R S
Blocks
of... t2 u2 t3 u3 R S
results
Compare: (t1, u1), (t1, u2), (t1, u3)
(t2, u1), (t2, u2), (t2, u3)
(t3, u1), (t3, u2), (t3, u3)
Then: GET NEXT BLOCK OF S
Repeat: for EVERY BLOCK of R

41. Block Nested-Loop Join
for each block BR of R do for each block BS of S do for each tuple tr in BR do for each tuple us in Bs do begin Check if (tr,us) satisfy the join condition if they do (“match”), add tr • us to the result.

42. Block Nested-Loop Join (Cont.)
Cost:
Worst case estimate: br  bs + br block accesses.
Best case: br + bs block accesses. Same as nested loop.
Improvements to nested loop and block nested loop algorithms for a buffer with M blocks:
In block nested-loop, use M — 2 disk blocks as blocking unit for outer relations, where M = memory size in blocks; use remaining two blocks to buffer inner relation and output
Cost = br / (M-2)  bs + br
If equi-join attribute forms a key or inner relation, stop inner loop on first match
Scan inner loop forward and backward alternately, to make use of the blocks remaining in buffer (with LRU replacement)