Yue (Jenny) Cui and William Perrizo North Dakota State University Aggregate Function Computation and Iceberg Querying in Vertical Databases Yue (Jenny) Cui and William Perrizo North Dakota State University

Outline Introduction Review of Aggregate Functions Review of Iceberg Queries Algorithms of Aggregate Function Computation Using P-trees SUM, COUNT, and AVERAGE. MAX, MIN, MEDIAN, RANK, and TOP-K. Iceberg Query Operation Using P-trees An Iceberg Query Example Performance Analysis Conclusion

Introduction Commonly used aggregation functions include COUNT, SUM, AVERAGE, MIN, MAX, MEDIAN, RANK, and TOP-K. Iceberg queries perform aggregate functions across attributes and then eliminate aggregate values that are below some specified threshold. We use an example to review iceberg queries. SELECT Location, Product Type, Sum (# Product) FROM Relation Sales GROUPBY Location, Product Type HAVING Sum (# Product) >= T

Introduction (Cont.) We illustrate the procedure of calculating by three steps. Step one: Generate Location-list. SELECT Location, Sum (# Product) FROM Relation Sales GROUPBY Location HAVING Sum (# Product) >= T Step Two: Generate Product Type-list. SELECT Type, Sum (# Product) FROM Relation Sales GROUPBY Product Type HAVING Sum (# Product) >= T Step Three: Generate location & Product Type pair groups. From the Location-list and the Type-list we generated in first two steps, we can eliminate many of the location & Product Type pair groups

Algorithms of Aggregate Function Computation Using P-trees The dataset we used in our example. We use the data in relation Sales to illustrate algorithms of aggregate function. Id Mon Loc Type On line # Product 1 Jan New York Notebook Y 10 2 Minneapolis Desktop N 5 3 Feb Printer 6 4 Mar 7 11 Chicago 9 Apr Fax Table 1. Relation Sales.

Algorithms of Aggregate Function Computation Using P-trees (Cont.) Table 2 shows the binary representation of data in relation Sales. Id Mon Loc Type On line # Product P0,3 P0,2 P0,1 P0,0 P1,4 P1,3 P1,2 P1,1 P1,0 P2,2 P2,1 P2,0 P3,0 P4,3 P4,2 P4,1 P4,0 1 0001 00001 001 1010 2 00101 010 0101 3 0010 100 0110 4 0011 0111 5 1011 6 00110 1001 7 0100 101 Table 2. Binary Form of Sales.

Algorithm of Aggregate Function COUNT COUNT function: It is not necessary to write special function for COUNT because P-tree RootCount function has already provided the mechanism to implement it. Given a P-tree Pi, RootCount(Pi) returns the number of 1s in Pi. Id Mon Loc Type On line # Product P0,3 P0,2 P0,1 P0,0 P1,4 P1,3 P1,2 P1,1 P1,0 P2,2 P2,1 P2,0 P3,0 P4,3 P4,2 P4,1 P4,0 1 0001 00001 001 1010 2 00101 010 0101 3 0010 100 0110 4 0011 0111 5 1011 6 00110 1001 7 0100 101 Table 1. Relation Sales.

Algorithm of Aggregate Function SUM SUM function: Sum function can total a field of numerical values. Algorithm 4.1 Evaluating sum () with P-tree. total = 0.00; For i = 0 to n { total = total + 2i * RootCount (Pi); } Return total Algorithm 4. 1. Sum Aggregate

Algorithm of Aggregate Function SUM P4,3 P4,2 P4,1 P4,0 10 5 6 7 11 9 3 1 1 1 1 For example, if we want to know the total number of products which were sold out in relation Sales, the procedure is showed on left {3} {3} {5} {5} 23 * + 22 * + 21 * + 20 * = 51

Algorithm of Aggregate Function AVERAGE Average function: Average function will show the average value in a field. It can be calculated from function COUNT and SUM. Average () = Sum ()/Count ().

Algorithm of Aggregate Function MAX Max function: Max function returns the largest value in a field. Algorithm 4.2 Evaluating max () with P-tree. max = 0.00; c = 0; Pc is set all 1s For i = n to 0 { c = RootCount (Pc AND Pi); If (c >= 1) Pc = Pc AND Pi; max = max + 2i; } Return max; Algorithm 4. 2. Max Aggregate.

Algorithm of Aggregate Function MAX Steps IF Pos Bits P4,3 P4,2 P4,1 P4,0 1. Pc = P4,3 RootCount (Pc) = 3 >= 1 10 5 6 7 11 9 3 1 1 1 1 {1} 2. RootCount (Pc AND P4,2) = 0 < 1 Pc = Pc AND P’4,2 {0} 3. RootCount (Pc AND P4,1 ) = 2 >= 1 Pc = Pc AND P4,1 {1} 4. RootCount (Pc AND P4,0 ) = 1 >= 1 {1} 23 * + 22 * + 21 * + 20 * = {1} {0} {1} {1} 11

Algorithm of Aggregate Function MIN Min function: Min function returns the smallest value in a field. Algorithm 4.3. Evaluating Min () with P-tree. min = 0.00; c = 0; Pc is set all 1s For i = n to 0 { c = RootCount (Pc AND NOT (Pi)); If (c >= 1) Pc = Pc AND NOT (Pi); Else min = min + 2i; } Return min; Algorithm 4. 2. Max Aggregate.

Algorithm of Aggregate Function MIN Steps IF Pos Bits P4,3 P4,2 P4,1 P4,0 1. Pc = P’4,3 RootCount (Pc) = 4 >= 1 10 5 6 7 11 9 3 1 1 1 1 {0} 2. RootCount (Pc AND P’4,2) = 1 >= 1 Pc = Pc AND P’4,2 {0} 3. RootCount (Pc AND P’4,1 ) = 0 < 1 Pc = Pc AND P4,1 {1} 4. RootCount (Pc AND P’4,0 ) = 0 < 1 {1} 23 * + 22 * + 21 * + 20 * = {0} {0} {1} {1} 3

Algorithms of Aggregate Function MEDIAN and RANK Algorithm 4.4. Evaluating Median () with P-tree median = 0.00; pos = N/2; for rank pos = K; c = 0; Pc is set all 1s for single attribute For i = n to 0 { c = RootCount (Pc AND Pi); If (c >= pos) median = median + 2i; Pc = Pc AND Pi; Else pos = pos - c; Pc = Pc AND NOT (Pi); } Return median; Median function returns the median value in a field. Rank (K) function returns the value that is the kth largest value in a field. Algorithm 4. 2. Median Aggregate.

Algorithm of Aggregate Function MEDIAN Steps IF Pos Bits P4,3 P4,2 P4,1 P4,0 1. Pc = P4,3 RootCount (Pc) = 3 < 4 Pc = P’4,3 pos = 4 – 3 = 1 10 5 6 7 11 9 3 1 1 1 1 {0} 2. RootCount (Pc AND P4,2) = 3 >= 1 Pc = Pc AND P4,2 {1} 3. RootCount (Pc AND P4,1 ) = 2 >= 1 Pc = Pc AND P4,1 {1} 4. RootCount (Pc AND P4,0 ) = 1 >= 1 {1} 23 * + 22 * + 21 * + 20 * = {0} {1} {1} {1} 7

P3 P2 P1 P0 Rank = 3 1. Count = 3 3 >= 3 B3 = 1 2. Count = 1 1 < 3 B2 = 0 r = r-c = 3-1 = 2 3. Count = 2 2 >= 2 B1 = 1 4. Count = 1 1 < 2 B0 = 0 Rank = 4 1. Count = 3 3 < 4 B3 = 0 r = r – c = 1 2. Count = 4 4 > 1 B2 = 1 3. Count = 2 2 >= 1 B1 = 1 4. Count = 1 1 >= 1 B0 = 1 7 3 11 1 5 14 10 6 4 1 3 4 5 6 7 10 11 14 P3 R=4 C=3 C<R P3 R=3 C=3 C>=R root P3’P2 R=1 C=4 C>=R P3P2 R=3 C=1 C<R P3 P2 P1 P0 1 0 1 0 P3 P2 P1 P0 0 1 1 1 P3 P3’ P3’P2P1 R=1 C=2 C>=R P3P2’P1 R=2 C=2 C>=R P3P2 P3P2’ P3’P2 P3’P2’ P3’P2P1P0 R=1 C=1 C>=R P3P2’P1P0 R=2 C=1 C<R P3P2P1 P3P2P1’ P3P2’P1 P3P2’P1’ P3’P2P1 P3’P2P1’ P3’P2’P1 P3’P2’P1’ 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

Algorithm of Aggregate Function TOP-K Top-k function: In order to get the largest k values in a field, first, we will find rank k value Vk using function Rank (K). Second, we will find all the tuples whose values are greater than or equal to Vk. Using ENRING technology of P-tree

Performance Analysis Figure 15. Iceberg Query with multi-attributes aggregation Performance Time Comparison

Performance Analysis Our experiments are implemented in the C++ language on a 1GHz Pentium PC machine with 1GB main memory running on Red Hat Linux. In figure 15, we compare the running time of P-tree method and bitmap method on calculating multi-attribute iceberg query. In this case P-trees are proved to be substantially faster.

Conclusion we believe our study confirms that the P-tree approach is superior to the bitmap approach for aggregation of all types and multi-attribute iceberg queries. It also proves that the advantages of basic P-tree representations of files are: First, there is no need for redundant, auxiliary structures. Second basic P-trees are good at calculating multi-attribute aggregations, numeric value, and fair to all attributes.