4/14/2017 10:52 PM DBI405 Troubleshooting SQL Server Spatial Query Performance: A Deep Dive into Spatial Indexing Michael Rys Principal Program Manager Microsoft Corporation @SQLServerMike © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Q: Why is my Query so Slow? A: Usually because the index isn’t being used. Q: How do I tell? A: SELECT * FROM T WHERE g.STIntersects(@x) = 1 NO INDEX INDEX!

Hinting the Index Spatial indexes can be forced if needed. SELECT * FROM T WHERE g.STIntersects(@x) = 1 Use SQL Server 2008 SP1 or later! WITH(INDEX(T_g_idx))

But Why Isn't My Index Used? Plan choice is cost-based QO uses various information, including cardinality When can we estimate cardinality? Variables: never Literals: not for spatial since they are not literals under the covers Parameters: yes, but cached, so first call matters EXEC sp_executesql N'SELECT * FROM T WHERE T.g.STIntersects(@x) = 1', N'@x geometry', N'POINT (0 0)' DECLARE @x geometry = 'POINT (0 0)' SELECT * FROM T WHERE T.g.STIntersects(@x) = 1 SELECT * FROM T WHERE T.g.STIntersects('POINT (0 0)') = 1

Spatial Indexing Basics Primary Filter (Index lookup) Secondary Filter (Original predicate) E In general, split predicates in two Primary filter finds all candidates, possibly with false positives (but never false negatives) Secondary filter removes false positives The index provides our primary filter Original predicate is our secondary filter Some tweaks to this scheme Sometimes possible to skip secondary filter

Using B+-Trees for Spatial Index SQL Server has B+-Trees Spatial indexing is usually done through other structures Quad tree, R-Tree Challenge: How do we repurpose the B+-Tree to handle spatial queries? Add a level of indirection!

Mapping to the B+-Tree B+-Trees handle linearly ordered sets well We need to somehow linearly order 2D space Either the plane or the globe We want a locality-preserving mapping from the original space to the line i.e., close objects should be close in the index Can’t be done, but we can approximate it

SQL Server Spatial Indexing Story Planar Index Geographic Index Requires bounding box Only one grid No bounding box Two top-level projection grids Indexing Phase Secondary Filter Primary Filter 1 2 15 16 4 3 14 13 5 8 9 12 6 7 10 11 1 2 15 16 4 3 14 13 5 8 9 12 6 7 10 11 1 2 15 16 4 3 14 13 5 8 9 12 6 7 10 11 1. 3. 2. 5. Apply actual CLR method on candidates to find matches 4. Intersecting grids identifies candidates 1. Overlay a grid on the spatial object 2. Identify grids for spatial object to store in index 3. Identify grids for query object(s)

SQL Server Spatial Indexing Story Multi-Level Grid Much more flexible than a simple grid Hilbert numbering Modified adaptable QuadTree Grid index features 4 levels Customizable grid subdivisions Customizable maximum number of cells per object (default 16) NEW IN SQL Server 2012: New Default tessellation with 8 levels of cell nesting

Multi-Level Grid /4/2/3/1 / (“cell 0”) Deepest-cell Optimization: Only keep the lowest level cell in index Covering Optimization: Only record higher level cells when all lower cells are completely covered by the object Cell-per-object Optimization: User restricts max number of cells per object

Implementation of the Index Persist a table-valued function Internally rewrite queries to use the table 0 – cell at least touches the object (but not 1 or 2) 1 – guarantee that object partially covers cell 2 – object covers cell Spatial Reference ID Have to be the same to produce match Varbinary(5) encoding of grid cell id 15 columns and 895 byte limitation Prim_key geography 1 g1 2 g2 3 g3 Prim_key cell_id srid cell_attr 1 0x00007 42 3 0x0000A 2 0x0000B 0x0000C 0x0000D 0x00014 Base Table T CREATE SPATIAL INDEX sixd ON T(geography) Internal Table for sixd

Auto Grid Spatial Index New spatial index Tessellations: geometry_auto_grid geography_auto_grid Uses 8 Grid levels instead of the previous 4 No GRIDS parameter needed (or available) Fixed at HLLLLLLL default number of cells per object: 8 for geometry 12 for geography More stable performance for windows of different size for data with different spatial density For default values: Up to 2x faster for longer queries > 500 ms More efficient primary filter Fewer rows returned 10ms slower for very fast queries < 50 ms Increased tessellation time which is constant

Spatial Index Performance New grid gives much stable performance for query windows of different size Better grid coverage gives fewer high peaks

Index Creation and Maintenance Create index example GEOMETRY: CREATE SPATIAL INDEX sixd ON spatial_table(geom_column) WITH (BOUNDING_BOX = (0, 0, 500, 500), GRIDS = (LOW, LOW, MEDIUM, HIGH), CELLS_PER_OBJECT = 20) Create index example GEOGRAPHY: CREATE SPATIAL INDEX sixd ON spatial_table(geogr_column) USING GEOGRAPHY_GRID WITH (GRIDS = (LOW, LOW, MEDIUM, HIGH), NEW IN SQL Server 2012 (equivalent to default creation): USING GEOGRAPHY_AUTO_GRID WITH (CELLS_PER_OBJECT = 20) Use ALTER and DROP INDEX for maintenance.

Indexing and Performance 4/14/2017 10:52 PM demo Indexing and Performance © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Spatial queries supported by index in SQL Server Geometry: STIntersects() = 1 STOverlaps() = 1 STEquals()= 1 STTouches() = 1 STWithin() = 1 STContains() = 1 STDistance() < val STDistance() <= val Nearest Neighbor Filter() = 1 Geography STIntersects() = 1 STOverlaps() = 1 STEquals()= 1 STWithin() = 1 STContains() = 1 STDistance() < val STDistance() <= val Nearest Neighbor Filter() = 1 New in SQL Server 2012

How Costing is Done The stats on the index contain a trie constructed on the string form of the packed binary(5) typed Cell ID. When a window query is compiled with a sniffable window object, the tessellation function on the window object is run at compile time. The results are used to construct a trie for use during compilation. May lead to wrong compilation for later objects No costing on: Local variables, constants, results of expressions Use different indices and different stored procs to account for different query characteristics

Understanding the Index Query Plan 4/14/2017 Understanding the Index Query Plan © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Seeking into a Spatial Index Minimize I/O and random I/O Intuition: small windows should touch small portions of the index A cell 7.2.4 matches Itself Ancestors Descendants 7 7.2 7.2.4 Spatial Index S

Understanding the Index Query Plan 4/14/2017 Understanding the Index Query Plan T(@g) Optional Sort Remove dup ranges Ranges Spatial Index Seek © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Spatial index tessellation Better and more continuous coverage Fully contained cells Partially contained cells 64 cells 128 cells 256 cells

Query window number of cells Optimal value (theoretical) is somewhere between two extremes Typical spatial query performance Time needed to process false positives Default values: 512 - Geometry AUTO grid 768 - Geography AUTO grid 1024 - MANUAL grids SELECT * FROM table t WITH (SPATIAL_WINDOW_MAX_CELLS=256) WHERE t.geom.STIntersects(@window)=1;

Query Window Hinting (SQL Server 2012) SELECT * FROM table t with(SPATIAL_WINDOW_MAX_CELLS=1024) WHERE t.geom.STIntersects(@window)=1 Used if an index is chosen (does not force an index) Overwrites the default (512 for geometry, 768 for geography) Rule of thumb: Higher value makes primary filter phase longer but reduces work in secondary filter phase Set higher for dense spatial data Set lower for sparse spatial data

Index Hinting FROM T WITH (INDEX (<Spatial_idxname>)) Spatial index is treated the same way a non-clustered index is the order of the hint is reflected in the order of the indexes in the plan multiple index hints are concatenated no duplicates are allowed The following restrictions exist: The spatial index must be either first in the first index hint or last in the last index hint for a given table. Only one spatial index can be specified in any index hint for a given table.

demo Spatial Query Hinting 4/14/2017 10:52 PM © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Additional Query Processing Support Index intersection Enables efficient mixing of spatial and non-spatial predicates Matching New in SQL Server 2012: Nearest Neighbor query Distance queries: convert to STIntersects Commutativity: a.STIntersects(b) = b.STIntersects(a) Dual: a.STContains(b) = b.STWithin(a) Multiple spatial indexes on the same column Various bounding boxes, granularities Outer references as window objects Enables spatial join to use one index

Other Spatial Performance Improvements in SQL Server 2012 Spatial index build time for point data can be as much as four to five times faster Optimized spatial query plan for STDistance and STIntersects like queries Faster point data queries Optimized STBuffer, lower memory footprint

Spatial Nearest Neighbor Main scenario Give me the closest 5 Italian restaurants Execution plan SQL Server 2008/2008 R2: table scan SQL Server 2012: uses spatial index Specific query pattern required SELECT TOP(5) * FROM Restaurants r WHERE r.type = ‘Italian’ AND r.pos.STDistance(@me) IS NOT NULL ORDER BY r.pos.STDistance(@me)

Spatial Performance in SQL Server 2012 4/14/2017 10:52 PM demo Spatial Performance in SQL Server 2012 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Nearest Neighbor Performance Find the closest 50 business points to a specific location (out of 22 million in total) NN query vs best current workaround (sort all points in 10km radius) *Average time for NN query is ~236ms

Limitations of Spatial Plan Selection Off whenever window object is not a parameter: Spatial join (window is an outer reference) Local variable, string constant, or complex expression Has the classic SQL Server parameter-sensitivity problem SQL compiles once for one parameter value and reuses the plan for all parameter values Different plans for different sizes of window require application logic to bucketize the windows

Error 8635: Cannot find a plan Error: The query processor could not produce a query plan for a query with a spatial index hint.  Reason: %S_MSG.  Try removing the index hints or removing SET FORCEPLAN. Possible Reasons (%S_MSG): The spatial index is disabled or offline The spatial object is not defined in the scope of the predicate Spatial indexes do not support the comparand supplied in the predicate Spatial indexes do not support the comparator supplied in the predicate Spatial indexes do not support the method name supplied in the predicate The comparand references a column that is defined below the predicate The comparand in the comparison predicate is not deterministic The spatial parameter references a column that is defined below the predicate Could not find required binary spatial method in a condition Could not find required comparison predicate

Index Support Can be built in parallel Can be hinted File groups/Partitioning Aligned to base table or Separate file group Full rebuild only New catalog views, DDL Events DBCC Checks Supportability stored procedures New in SQL Server 2012: Index Page and Row Compression Ca. 50% smaller indices, 0-15% slower queries Not supported Online rebuild Database Tuning advisor

SET Options Spatial indexes requires: ANSI_NULLS: ON ANSI_PADDING: ON ANSI_WARNINGS: ON CONCAT_NULL_YIELDS_NULL: ON NUMERIC_ROUNDABORT: OFF QUOTED_IDENTIFIER: ON

Spatial Indices and Partitions and Filegroups Default partitioned to the same filegroups as the base table. Overwrite with: [ ON { filegroup_name | "default" } ] If filegroup_name is specified, the index will be placed on the specified filegroup regardless of the table’s partitioning scheme. If “default” is specified, the base table’s default filegroup/partitioning scheme is applied. Altering the base table’s partition scheme is not allowed unless the spatial index was created with the “ON filegroup” option (and is hence not aligned with the partitioning anyway). The index has to be dropped and then the base table repartitioned.

Spatial Catalog Views sys.spatial_indexes catalog view sys.spatial_index_tessellations catalog view Entries in sys.indexes for a spatial index: A clustered index on the internal table of the spatial index A spatial index (type = 4) for spatial index An entry in sys.internal_tables An entry to sys.index_columns

New Spatial Histogram Helpers sp_spatial_help_geometry_histogram sp_spatial_help_geography_histogram Used for spatial data and index analysis Histogram of 22 million business points over US Left: SSMS view of a histogram Right: Custom drawing on top of Bing Maps

Indexing Support Procedures sys.sp_help_spatial_geometry_index sys.sp_help_spatial_geometry_index_xml sys.sp_help_spatial_geography_index sys.sp_help_spatial_geography_index_xml Provide information about index: 64 properties 10 of which are considered core

sys.sp_help_spatial_geometry_index Arguments Results in property name/value pair table of the format: Parameter Type Description @tabname nvarchar(776) the name of the table for which the index has been specified @indexname sysname the index name to be investigated @verboseoutput tinyint 0 core set of properties is reported 1 all properties are being reported @query_sample geometry A representative query sample that will be used to test the usefulness of the index. It may be a representative object or a query window. PropName: nvarchar(256) PropValue: sql_variant

sys.sp_help_spatial_geography_index_xml Arguments Parameter Type Description @tabname nvarchar(776) the name of the table for which the index has been specified @indexname sysname the index name to be investigated @verboseoutput tinyint 0 core set of properties is reported 1 all properties are being reported @query_sample geography A representative query sample that will be used to test the usefulness of the index. It may be a representative object or a query window. @xml_output xml This is an output parameter that contains the returned properties in an XML fragment

Some of the returned Properties Property Type Description Base_Table_Rows Bigint All Number of rows in the base table Index properties - index properties: bounding box, grid densities, cell per object Total_Primary_Index_R ows Number of rows in the index Total_Primary_Index_P ages Number of pages in the index Total_Number_Of_Obje ctCells_In_Level0_For_ QuerySample Core Indicates whether the representative query sample falls  outside of the bounding box of the geometry index and into the root cell (level 0 cell). This is either 0 (not in level 0 cell) or 1. If it is in the level 0 cell, then the investigated index is not an appropriate index for the query sample. Total_Number_Of_Obje ctCells_In_Level0_In_I ndex Number of cell instances of indexed objects that are tessellated in level 0. For geometry indexes, this will happen if the bounding box of the index is smaller than the data domain. A high number of objects in level 0 may require a costly application of secondary filters if the query window falls partially outside the bounding box. If the query window falls inside the bounding box, having a high number of objects in level 0 may actually improve the performance.

Some of the returned Properties Property Type Description Number_Of_Rows_Selected_By_Primary _Filter bigint Core P = Number of rows selected by the primary filter. Number_Of_Rows_Selected_By_Internal _Filter S = Number of rows selected by the internal filter. For these rows, the secondary filter is not called. Number_Of_Times_Secondary_Filter_Is_ Called Number of times the secondary filter is called. Percentage_Of_Rows_NotSelected_By_Pr imary_Filter float Suppose there are N rows in the base table, suppose P are selected by the primary filter. This is (N-P)/N as percentage. Percentage_Of_Primary_Filter_Rows_Sel ected_By_Internal_Filter This is S/P as a percentage. The higher the percentage, the better is the index in avoiding the more expensive secondary filter. Number_Of_Rows_Output O=Number of rows output by the query. Internal_Filter_Efficiency This is S/O as a percentage. Primary_Filter_Efficiency This is O/P as a percentage. The higher the efficiency is, the less false positives have to be processed by the secondary filter.

Spatial Tips on index settings Some best practice recommendations (YMMV): Start out with new default tesselation Point data: always use HIGH for all 4 level. CELL_PER_OBJECT are not relevant in the case. Simple, relatively consistent polygons: set all levels to LOW or MEDIUM, MEDIUM, LOW, LOW Very complex LineString or Polygon instances: High number of CELL_PER_OBJECT (often 8192 is best) Setting  all 4 levels to HIGH may be beneficial Polygons or line strings which have highly variable sizes: experimentation is needed.  Rule of thumb for GEOGRAPHY: if MMMM is not working, try HHMM

What to do if my Spatial Query is slow? Make sure you are running SQL Server 2008 SP1, 2008 R2 or 2012 Check query plan for use of index Make sure it is a supported operation Hint the index (and/or a different join type) Do not use a spatial index when there is a highly selective non-spatial predicate Run above index support procedure: Assess effectiveness of primary filter (Primary_Filter_Efficiency) Assess effectiveness of internal filter (Internal_Filter_Efficiency) Redefine or define a new index with better characteristics More appropriate bounding box for GEOMETRY Better grid densities

Summary: Spatial Index Improvements in SQL Server 2012 Auto Grid Spatial Index Spatial Index Hint More supported Operations Spatial Index Compression Improved “Create Spatial Index” Time For Point Data

Related Content Breakout Sessions DBI324 - Taking SQL Server into the World of Spatial Data Management Some Spatial Presentations (with further links) and whitepapers: http://www.slideshare.net/MichaelRys/sqlbits-x-sql-server-2012-spatial http://www.slideshare.net/MichaelRys/sqlbits-x-sql-server-2012-spatial-indexing http://social.technet.microsoft.com/wiki/contents/articles/9694.tuning-spatial-point-data-queries-in-sql-server-2012.aspx Forum: http://social.msdn.microsoft.com/Forums/en-US/sqlspatial/threads Blogs: http://blogs.msdn.com/b/edkatibah/ Find Me Later At… @SQLServerMike mrys@microsoft.com

How to contact me Michael Rys http://sqlblog.com/blogs/michael_rys/ mrys@microsoft.com http://sqlblog.com/blogs/michael_rys/ @SQLServerMike

mva Track Resources SQL Server 2012 Eval Copy Hands-On Labs @sqlserver @TechEd_europe #msTechEd mva Microsoft Virtual Academy Get Certified!

Resources Learning TechNet http://europe.msteched.com Connect. Share. Discuss. http://europe.msteched.com Microsoft Certification & Training Resources www.microsoft.com/learning TechNet Resources for IT Professionals http://microsoft.com/technet Resources for Developers http://microsoft.com/msdn

Submit your evals online 4/14/2017 10:52 PM Evaluations Submit your evals online http://europe.msteched.com/sessions © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4/14/2017 10:52 PM © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.