Spatial Queries Nearest Neighbor and Join Queries Most slides are based on slides provided By Prof. Christos Faloutsos (CMU) and Prof. Dimitris Papadias (UST-HK)

Spatial Queries Given a collection of geometric objects (points, lines, polygons,...) organize them on disk, to answer efficiently point queries range queries k-nn queries spatial joins ( ‘ all pairs ’ queries)

Spatial Queries Given a collection of geometric objects (points, lines, polygons,...) organize them on disk, to answer point queries range queries k-nn queries spatial joins ( ‘ all pairs ’ queries)

Spatial Queries Given a collection of geometric objects (points, lines, polygons,...) organize them on disk, to answer point queries range queries k-nn queries spatial joins ( ‘ all pairs ’ queries)

Spatial Queries Given a collection of geometric objects (points, lines, polygons,...) organize them on disk, to answer point queries range queries k-nn queries spatial joins ( ‘ all pairs ’ queries)

Spatial Queries Given a collection of geometric objects (points, lines, polygons,...) organize them on disk, to answer point queries range queries k-nn queries spatial joins ( ‘ all pairs ’ queries)

R-tree …

R-trees - Range search pseudocode: check the root for each branch, if its MBR intersects the query rectangle apply range-search (or print out, if this is a leaf)

R-trees - NN search A B C D E F G H I J P1 P2 P3 P4 q

R-trees - NN search Q: How? (find near neighbor; refine...) A B C D E F G H I J P1 P2 P3 P4 q

R-trees - NN search A1: depth-first search; then range query A B C D E F G H I J P1 P2 P3 P4 q

R-trees - NN search A1: depth-first search; then range query A B C D E F G H I J P1 P2 P3 P4 q

R-trees - NN search A1: depth-first search; then range query A B C D E F G H I J P1 P2 P3 P4 q

R-trees - NN search: Branch and Bound A2: [Roussopoulos+, sigmod95]: At each node, priority queue, with promising MBRs, and their best and worst-case distance main idea: Every side (face) of any MBR contains at least one point of an actual spatial object!

MBR face property MBR is a d-dimensional rectangle, which is the minimal rectangle that fully encloses (bounds) an object (or a set of objects) MBR f.p.: Every face of the MBR contains at least one point of some object in the database

Search improvement Visit an MBR (node) only when necessary How to do pruning? Using MINDIST and MINMAXDIST

MINDIST MINDIST(P, R) is the minimum distance between a point P and a rectangle R If the point is inside R, then MINDIST=0 If P is outside of R, MINDIST is the distance of P to the closest point of R (one point of the perimeter)

MINDIST computation MINDIST(p,R) is the minimum distance between p and R with corner points l and u the closest point in R is at least this distance away r i = l i if p i < l i = u i if p i > u i = p i otherwise p p p R l u MINDIST = 0 l=(l 1, l 2, …, l d ) u=(u 1, u 2, …, u d )

MINMAXDIST MINMAXDIST(P,R): for each dimension, find the closest face, compute the distance to the furthest point on this face and take the minimum of all these (d) distances MINMAXDIST(P,R) is the smallest possible upper bound of distances from P to R MINMAXDIST guarantees that there is at least one object in R with a distance to P smaller or equal to it.

MINDIST and MINMAXDIST MINDIST(P, R) <= NN(P) <=MINMAXDIST(P,R) R1 R2 R3 R4 MINDIST MINMAXDIST MINDIST MINMAXDIST MINDIST MINMAXDIST

Pruning in NN search Downward pruning: An MBR R is discarded if there exists another R ’ s.t. MINDIST(P,R)>MINMAXDIST(P,R ’ ) Downward pruning: An object O is discarded if there exists an R s.t. the Actual-Dist(P,O) > MINMAXDIST(P,R) Upward pruning: An MBR R is discarded if an object O is found s.t. the MINDIST(P,R) > Actual-Dist(P,O)

Pruning 1 example Downward pruning: An MBR R is discarded if there exists another R ’ s.t. MINDIST(P,R)>MINMAXDIST(P,R ’ ) MINDIST MINMAXDIST R R’R’

Pruning 2 example Downward pruning: An object O is discarded if there exists an R s.t. the Actual-Dist(P,O) > MINMAXDIST(P,R) Actual-Dist MINMAXDIST O R

Pruning 3 example Upward pruning: An MBR R is discarded if an object O is found s.t. the MINDIST(P,R) > Actual-Dist(P,O) MINDIST Actual-Dist R O

Ordering Distance MINDIST is an optimistic distance where MINMAXDIST is a pessimistic one. We use MINDIST! P MINDIST MINMAXDIST

NN-search Algorithm 1. Initialize the nearest distance as infinite distance 2. Traverse the tree depth-first starting from the root. At each Index node, sort all MBRs using an ordering metric and put them in an Active Branch List (ABL). 3. Apply pruning rules 1 and 2 to ABL 4. Visit the MBRs from the ABL following the order until it is empty 5. If Leaf node, compute actual distances, compare with the best NN so far, update if necessary. 6. At the return from the recursion, use pruning rule 3 7. When the ABL is empty, the NN search returns.

K-NN search Keep the sorted buffer of at most k current nearest neighbors Pruning is done using the k-th distance

Another NN search: Best-First Global order [HS99] Maintain distance to all entries in a common Priority Queue Use only MINDIST Repeat Inspect the next MBR in the list Add the children to the list and reorder Until all remaining MBRs can be pruned

HS algorithm Initialize PQ (priority queue) InesrtQueue(PQ, Root) While not IsEmpty(PQ) R= Dequeue(PQ) If R is an object Report R and exit (done!) If R is a leaf page node For each O in R, compute the Actual-Dists, InsertQueue(PQ, O) If R is an index node For each MBR C, compute MINDIST, insert into PQ

30 Depth-First (DF) NN Algorithm Roussoulos et al., SIGMOD, x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E Note: distances not actually stored inside nodes. Only for illustration 5

31 DF Search – Visit E x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E 7 2 5

32 DF Search – Find Candidate NN a First Candidate NN: a with distance a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E 7 5

33 DF Search – Backtrack to Root and Visit E 2 First Candidate NN: a with distance a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E 7 5

34 DF Search – Find Actual NN i First Candidate NN: a with distance a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E 7 Actual NN: i with distance 2 5

35 Optimality x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E 7 Question: Which is the minimal set of nodes that must be visited by any NN algorithm? Answer: The set of nodes whose MINDIST is smaller than or equal to the distance between q and its NN (e.g., E 1, E 2, E 6 ).

36 BF Search – Visit root E 1 1 E 2 2 Visit Root ActionHeap x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E

37 BF Search – Visit E 1 E 1 1 E 2 2 Visit Root follow E 1 E 2 2 E 5 3 E 5 5 E 9 4 ActionHeap x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E

38 BF Search – Visit E 2 E 1 1 E 2 2 Visit Root follow E 1 E 2 2 E 5 3 E 5 5 E 9 4 ActionHeap follow E 2 E 2 6 E 5 3 E 5 5 E 9 4 E x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E

39 BF Search – Visit E 6 E 1 1 E 2 2 Visit Root follow E 1 E 2 2 E 5 3 E 5 5 E 9 4 ActionHeap follow E 2 E 2 6 E 5 3 E 5 5 E 9 4 E 13 7 follow E 6 j 10 i 2 E 5 3 E 5 5 E 9 4 E 13 7 k x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E

40 BF Search – Find Actual NN i E 1 1 E 2 2 Visit Root follow E 1 E 2 2 E 5 3 E 5 5 E 9 4 ActionHeap follow E 2 E 2 6 E 5 3 E 5 5 E 9 4 E 13 7 follow E 6 Report i and terminate j 10 i 2 E 5 3 E 5 5 E 9 4 E 13 7 k x axis y axis b c a E 3 d e f g h i j k l m query E 4 E 5 E 1 E 2 E 6 E a b c d e E 1 E 2 E 3 E 4 E 5 Root E 1 E 2 E 3 E 4 f g h E 5 l m E 7 i j k E 6 E 6 E

Best-First vs Branch and Bound Best-First is the “ optimal ” algorithm in the sense that it visits all the necessary nodes and nothing more! BF is incremental! After you find the top-10 you can continue to find the 11 th etc… But needs to store a large Priority Queue in main memory. If PQ becomes large, we have thrashing… BB uses small Lists for each node. Also uses MINMAXDIST to prune some entries

Spatial Queries Given a collection of geometric objects (points, lines, polygons,...) organize them on disk, to answer point queries range queries k-nn queries spatial joins ( ‘ all pairs ’ queries)

Spatial Join Find all parks in each city in MA Find all trails that go through a forest in MA Basic operation find all pairs of objects that overlap Single-scan queries nearest neighbor queries, range queries Multiple-scan queries spatial join

Algorithms No existing index structures Transform data into 1-d space [O89] z-transform; sensitive to size of pixel Partition-based spatial-merge join [PW96] partition into tiles that can fit into memory plane sweep algorithm on tiles Spatial hash joins [LR96, KS97] Sort data using recursive partitioning [BBKK01] With index structures [BKS93, HJR97] k-d trees and grid files R-trees

R-tree based Join [BKS93] R S

Join1(R,S) Tree synchronized traversal algorithm Join1(R,S) Repeat Find a pair of intersecting entries E in R and F in S If R and S are leaf pages then add (E,F) to result-set Else Join1(E,F) Until all pairs are examined CPU and I/O bottleneck R S

CPU – Time Tuning Two ways to improve CPU – time Restricting the search space Spatial sorting and plane sweep

Reducing CPU bottleneck R S

Join2(R,S,IntersectedVol) Join2(R,S,IV) Repeat Find a pair of intersecting entries E in R and F in S that overlap with IV If R and S are leaf pages then add (E,F) to result-set Else Join2(E,F,CommonEF) Until all pairs are examined In general, number of comparisons equals size(R) + size(S) + relevant(R)*relevant(S) Reduce the product term

Restricting the search space Now: 3 of R * 2 of S Plus Scanning: 7 of R + 7 of S Join1: 7 of R * 7 of S 1 = 49 comparisons =6 comp = 14 comp

Using Plane Sweep R S Consider the extents along x-axis Start with the first entry r1 sweep a vertical line r1 r2 r3 s1 s2

Using Plane Sweep R S r1 r2 r3 s1 s2 Check if (r1,s1) intersect along y-dimension Add (r1,s1) to result set

Using Plane Sweep R S r1 r2 r3 s1 s2 Check if (r1,s2) intersect along y-dimension Add (r1,s2) to result set

Using Plane Sweep R S r1 r2 r3 s1 s2 Reached the end of r1 Start with next entry r2

Using Plane Sweep R S r1 r2 r3 s1 s2 Reposition sweep line

Using Plane Sweep R S r1 r2 r3 s1 s2 Check if r2 and s1 intersect along y Do not add (r2,s1) to result

Using Plane Sweep R S r1 r2 r3 s1 s2 Reached the end of r2 Start with next entry s1

Using Plane Sweep R S r1 r2 r3 s1 s2 Total of 2(r1) + 1(r2) + 0 (s1)+ 1(s2)+ 0(r3) = 4 comparisons

I/O Tunning Compute a read schedule of the pages to minimize the number of disk accesses Local optimization policy based on spatial locality Three methods Local plane sweep Local plane sweep with pinning Local z-order

Reducing I/O Plane sweep again: Read schedule r1, s1, s2, r3 Every subtree examined only once Consider a slightly different layout

Reducing I/O R S r1 r2 r3 s1 s2 Read schedule is r1, s2, r2, s1, s2, r3 Subtree s2 is examined twice

Pinning of nodes After examining a pair (E,F), compute the degree of intersection of each entry degree(E) is the number of intersections between E and unprocessed rectangles of the other dataset If the degrees are non-zero, pin the pages of the entry with maximum degree Perform spatial joins for this page Continue with plane sweep

Reducing I/O R S r1 r2 r3 s1 s2 After computing join(r1,s2), degree(r1) = 0 degree(s2) = 1 So, examine s2 next Read schedule = r1, s2, r3, r2, s1 Subtree s2 examined only once

Local Z-Order Idea : 1. Compute the intersections between each rectangle of the one node and all rectangles of the other node 2. Sort the rectangles according to the Z-ordering of their centers 3. Use this ordering to fetch pages

Local Z-ordering s1 r1 r2 s2 r3 r4 IV II I III IV Read schedule: II I III

Number of Disk Access Size of LRU Buffer > <

67 Skyline Queries Question: Which are the best hotels Answer: The ones that are not dominated (a, i, k). These points constitute the skyline A lot of algorithms for conventional DBs, data streams etc. Here we focus on NN-based methods. Assume that the data points correspond to hotel records with 2 attributes: price, distance from the beach

68 NN algorithm, Kossman et al., VLDB 2002 NN uses the results of nearest neighbor search to partition the data universe recursively.

69 NN (cont) NN uses the results of nearest neighbor search to partition the data universe recursively.

70 Disadvantages of NN Handling empty queries consumes most of the time. Large main memory requirements – in the worst case it might be the order of the dataset! number e of empty queries e=(s+r)  (d-1)+1, where s is # of skyline points r is # of redundant queries e.g., for d=2, e=s+1

71 Disadvantages of NN For dimensionality d, each skyline point leads to d more queries. First skyline point 1st query [0,n x ) [0,  ) [0,  ) 2nd query [0,  ) [0,n y ) [0,  ) 3rd query [0,  ) [0,  ) [0,n z )

72 Disadvantages of NN Need for duplicate elimination, if dimensionality d > 2. First skyline point 1st query [0,n x ) [0,  ) [0,  ) 2nd query [0,  ) [0,n y ) [0,  ) 3rd query [0,  ) [0,  ) [0,n z )

73 Disadvantages of NN Need for duplicate elimination, if dimensionality d > 2. First skyline point 1st query [0,n x ) [0,  ) [0,  ) 2nd query [0,  ) [0,n y ) [0,  ) 3rd query [0,  ) [0,  ) [0,n z )

74 mindist(MBR) = the L 1 distance between its lower-left corner and the origin. Each heap entry keeps the mindist of the MBR. Branch and Bound Skyline (BBS) Papadias et al., SIGMOD 2003

75 Example of BBS Process entries in ascending order of their mindists.

76 Example of BBS

77 Example of BBS

78 Example of BBS

79 Example of BBS

80 Example of BBS