CS222: Principles of Data Management Notes #13 Set operations, Aggregation Instructor: Chen Li

Relational Set Operations Intersection and cross-product special cases of join. Union (distinct) and Except similar; we’ll do union. Sorting-based approach to union: Sort both relations (on combination of all attributes). Scan sorted relations and merge them. Alternative: Merge runs from Pass 0+ for both relations (!). Hash-based approach to union (from Grace): Partition both R and S using hash function h1. For each S-partition, build in-memory hash table (using h2), then scan corresponding R-partition, adding truly new S tuples to hash table while discarding duplicates. 19

Sets versus Bags UNION, INTERSECT, and DIFFERENCE (EXCEPT or MINUS) use the set semantics. UNION ALL just combine two relations without considering any correlation. 19

Sets versus Bags Example: R = {1, 1, 1, 2, 2, 2, 2, 3} S = {1, 2, 2} R UNION S = {1, 2, 3} R UNION ALL S = {1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3} R INTERSECT S = {1, 2} R EXCEPT S = {3} Notice that some DBs such as MySQL don't support some of these operations. 19

Aggregate Operations (AVG, MIN, ...) Without grouping: In general, requires scanning the full relation. Given an index whose search key includes all attributes in the SELECT or WHERE clauses, can do an index-only scan. With grouping: Sort on the group-by attributes, then scan sorted result and compute the aggregate for each group. (Can improve on this by combining sorting and aggregate computations.) Or, similar approach via hashing on the group-by attributes. Given tree index whose search key includes all attributes in SELECT, WHERE and GROUP BY clauses, can do an index-only scan; if group-by attributes form prefix of search key, can retrieve data entries (tuples) in the group-by order. 20

Aggregation State Handling State: init( ), next(value), finalize( )  agg. value Consider the following query SELECT E.dname, avg(E.sal) FROM Emp E GROUP BY E.dname (dname count sum) Emp Toy 2 35K ename sal dname h(dname) mod 4 Joe 10K Toy Shoe 2 33K h Sue 20K Shoe Mike 13K Shoe Chris 25K Toy Aggregate state (per unfinished group): Count: # values Min: min value Max: max value Sum: sum of values Avg: count, sum of values Zoe 50K Book … … … 3

Operator Buffering Considerations If several operations are executing concurrently, estimating the “right” number of buffer pages is tough. (Intra- vs. inter-query considerations, too!) Repeated access patterns can interact with buffer pool’s page replacement policy as well. e.g., Inner relation scanned repeatedly in Simple Nested Loop Join. With enough buffer pages to hold the entire inner, replacement policy does not matter. Otherwise MRU best, LRU worst (sequential flooding)! Memory management for multi-user DBMSs is still a challenging problem (too many “knobs” today). 21