1. Resumable Online Index Rebuild (ROIR) in SQL 2017 & SQL DB
PNWSQL Presentation- February 2017
Mirek Sztajno, Senior Program Manager SQL Server
Resumable Index Preview

2. Agenda Problem statement and proposed solution
Current and Resumable Online Index Architecture
Syntax and Remarks
Permissions, resources and limitations
Metadata extensions
Performance and troubleshooting
Release plan
Roadmap
Demo
Q&A

3. Problem Statement
Failure during long running index rebuild operation ( i.e. failover, crash…etc) causes the whole operation to be abandoned and restart from scratch.
In Azure frequent failover (due to deployments, incidents, etc.) may often hit such failures
Index Build is executed in a single transaction requiring significant amount of log space
Potential issue for Azure SQL DB affecting large index rebuild
Log truncation issue
As long as index operation is running transaction logs cannot be truncated
On-premise customers have always asked for the ability to pause and resume index build operations during their maintenance windows
Supports ability to resume the operations in another maintenance window

4. Proposed Solution
Allows to resume an index rebuild operation after an unexpected failure has occurred
No need to restart from the beginning potentially saving significant amount of work when rebuilding indexes for large tables.
Ability to pause and resume an index build operation at any time, for example, to temporarily free up systems resources in order to execute a high priority task. 
Instead of terminating the index rebuild process the DBA can pause this task and resume it later without losing any prior execution progress. 
The second scenario is critical in those situations when a single maintenance window is too short to complete the index rebuild operation for a large table.
Finally, the operation supports rebuilding large indexes online without requiring significant log space, allowing to perform log truncation while the resumable rebuild operation is running.
Especially for Azure SQL Database the resumable index build capability allows the creation of indexes on even the largest databases supported by our service offering.

7. T-SQL Syntax ( 1 of 2)
Execute an index operation as “resumable” the following syntax is used
ALTER INDEX { index_name | ALL } ON <object> { REBUILD [ WITH ( <rebuild_index_option> [ ,...n ] ) ]  } [ ; ]
rebuild_index_option > ::= { …. | ONLINE = { ON [ ( <low_priority_lock_wait> )] | OFF } | RESUMABLE = { ON | OFF } | MAX_DURATION = <time> [MINUTES] Where RESUMABLE = ON – index operation is resumable RESUMABLE = OFF – default – operation cannot be resumed MAX_DURATION –  indicates time (an integer in minutes) that a resumable online index operation is executed before being paused

8. T-SQL Syntax ( 2 of 2)
To resume/abort an index rebuild the following syntax is used
ALTER INDEX { index_name | ALL } ON <object> { REBUILD [ WITH ( <rebuild_index_option> [ ,...n ] ) ] | DISABLE     | REORGANIZE [ PARTITION = partition_number ] [ WITH ( <reorganize_option>) ] | SET ( <set_index_option> [ ,...n ] ) | RESUME [WITH (<resumable_index_options>,[…n])] | PAUSE | ABORT } [;] <resumable_index_options>::= { MAXDOP = max_degree_of_parallelism | MAX_DURATION = <time> [MINUTES] | <low_priority_lock_wait> } <low_priority_lock_wait>::=  {  WAIT_AT_LOW_PRIORITY ( MAX_DURATION = <time> [ MINUTES ] ,   ABORT_AFTER_WAIT = { NONE | SELF | BLOCKERS } )  } 
Where
RESUME – Resume an index operation paused manually or due to a failure
MAXDOP –Maximum degree of parallelism (legacy option)
PAUSE Pause a resumable online index operation
ABORT – Abort an index operation. Affects running or paused index rebuild operations executed as resumable
<low_priority_lock_wait>::=  { 
    WAIT_AT_LOW_PRIORITY ( MAX_DURATION = <time> [ MINUTES ] ,  
                          ABORT_AFTER_WAIT = { NONE | SELF | BLOCKERS } )  } 

9. Remarks (1 of 2) /see also Appendix/
Currently only ONLINE INDEX OPERATIONS can be specified as resumable using RESUMABLE= ON option
Executing again the original ALTER INDEX REBUILD statement for resumable index, resumes automatically the index rebuild operation
Having a long pause for an index operation may impact the DML performance on a specific table as well as the database disk
To pause immediately the index operation user can
Stop (Ctrl-C) the ongoing command,
Execute the ALTER INDEX PAUSE command or
Execute KILL <session_id> command.
Once the command is paused it can be resumed using ALTER INDEX command described below

10. Remarks (2 of 2)
Executing again the original ALTER INDEX REBUILD statement for resumable index, resumes automatically the index rebuild operation in case the index rebuild was paused.
The PAUSE command allows to pause resumable online index rebuild operation using T-SQL command
Once the command is paused it can be resumed using ALTER INDEX command described
The ABORT command kills the session that hosted the original index rebuild and aborts the index operation
Only index operations that are paused are resumable.
During PAUSE state the affected table remains in a state that prevents schema modifications.
During PAUSE state the new index will be updated with the new DML records
The new MAXDOP setting can be change to a new setting when resuming a rebuild index operation

11. Permissions and Resources
No new permissions have been added to support this feature 
To mark an index operation “resumable” the same permissions apply for  ALTER INDEX
To execute ALTER INDEX, at a minimum, ALTER permission on the table or view is required. 
Resumable Index Rebuild should use the same resources as regular online Index Rebuild, with the exception that the new index needs to be updated while the operation is in paused state and no schema modifications can be performed at this time.
Ghost cleanup on the target and mapping indexes will be stopped while the operation is running, but is will be restarted when the index operation pauses 

12. Limitations
The following functionality is disabled for resumable index rebuild operation
ALTER INDEX REBUILD ALL command
ALTER TABLE using index rebuild
Rebuilding an index that is disabled is not supported with RESUMABLE=ON
DDL command with “RESUMABLE = ON” cannot be executed inside an explicit transaction (cannot be part of begin tran … commit block)
Rebuild an index that has computed or TIMESTAMP column(s) as key columns
The SORT_IN_TEMPDB=ON option is not supported for resumable index

13. Metadata Introducing new system view - sys.index_resumable_operations
object_id
int
ID of the object to which this index belongs ( not nullable )
index_id
ID of the index ( not nullable )
name
sysname
Name of the index ( not nullable )
sql_text
nvarchar(max)
DDL T-SQL statement text
last_max_dop_used
smallint
Last Max DOP used (default is 0)
partition_number
Partition number within the owning index or heap. For non-partitioned tables and indexes or in case all partitions are being rebuild the value of this column is NULL.
state
tinyint
Operational state for resumable index
0= Running
1-=Pause
state_desc
nvarchar(60)
Description of the operational state for resumable index (Running or Pause)
start_time
datetime
Index operation start time ( not nullable )
last_pause_time
Index operation last pause time ( nullable)
NULL if operation is running and never paused
total_execution_time
Total execution time from start time in minutes ( not nullable )
percent_complete
real
Index operation progress completion in % ( not nullable )
page_count
bigint
Total number of index pages allocated by the index build operation for the new and mapping indexes ( not nullable )
This system view shows only indexes executed as resumable (RESUMABLE=ON).
Aborted and inactive resumable index is not displayed because such operation doesn’t exist anywhere
The view data is persisted

14. Performance and Troubleshooting
No performance difference between resumable and non-resumable Online Index Rebuild
Extra DML updating the new index during index rebuild pause
Expected average <10% TPCC throughput degradation when rebuilding most indexes
Troubleshooting
sys.index_resumable_operations – details about resumable index
KILL – kill session command
sys.dm_db_index_physical_stats - details (i.e. size & fragmentation info) for specified table or view
sys.dm_exec_sessions – session information
sys.dm_db_index_usage_stats – details about different types of index operations
sys.dm_db_index_operational_stats - additional index information

15. Release Plan for ROIR Box / On-prem Azure SQL DB
Private Preview ( on-prem)
Available since CTP1.1
Latest version (almost Pubclic Preview) is available in CTP1.3 SQL release mid February 2017
Public preview
Scheduled for mid March (CTP1.4)
GA scheduled with SQL 2017 release
Azure SQL DB
Private preview - Available for all SKUs under feature switch
Please contact Resumable Index Preview
Public preview- end of March (T29)
GA – July 2017
Current user documentation is available

16. Roadmap Long term vision Short term vision
Make all scheme maintenance operations for index and table resumable
Online operations, next
Offline operations
Short term vision
Deliver this solution for Online Index Rebuild /H1 2017/
Next delivery – support resumable index capability for create/drop online index /H2 2017

17. DEMO

18. Demo Details
Create table with 4 partitioned using DOB datetime as partition column
Create clustered index on the DOB column
Populate partitions with date
Rebuild index online as non-resumable on one partition
Truncate logs
Rebuild index online as resumable on one partition
Kill the rebuild operation ( CTR-C) / Kill the server
Repeat the original command
Check the index status
Continue populating the table with date (update/delete)
Resume the index rebuild operation changing MAXDOP
Run CHECKDB

19. Q&A Resumable Index Preview <ResumableIDXPreview@microsoft.com>

20. Appendix

21. Remarks (1 of 3)
Currently only ONLINE INDEX OPERATIONS can be specified as resumable using RESUMABLE= ON option
The RESUMABLE option is not persisted in the metadata for a given index and applies only to the duration of a current DDL statement. Therefore, the RESUMABLE=ON clause must be specified explicitly to enable resumability
MAX_DURATION option is currently only supported for RESUMABLE=ON option. The <time> in minutes for MAX_DURATION must be greater than 0 minutes and less or equal one week (7*24*60= minutes). If MAX_DURATION option is omitted, the index operation will continue until its completion or until a failure occurs.
Having a long pause for an index operation may impact the DML performance on a specific table as well as the database disk capacity since both indexes the original one and the newly created one require disk space and need to be updated during DML operations. 
To pause immediately the index operation user can stop (Ctrl-C) the ongoing command, execute the ALTER INDEX PAUSE command, or execute KILL <session_id> command. Once the command is paused it can be resumed using ALTER INDEX command described below

22. Remarks (2 of 3)
Executing again the original ALTER INDEX REBUILD statement for resumable index, resumes automatically the index rebuild operation in case the index rebuild was paused.
The PAUSE command allows to pause resumable online index rebuild operation using T-SQL command
Once the command is paused it can be resumed using ALTER INDEX command described
The ABORT command kills the session that hosted the original index rebuild and aborts the index operation
Only index operations that are paused are resumable.
During PAUSE state the affected table remains in a state that prevents schema modifications.
During PAUSE state the new index will be updated with the new DML records
The new MAXDOP setting can be change to a new setting when resuming a rebuild index operation

23. Remarks (3 of 3)
The <low_priority_lock_wait> was added to RESUMABLE=ON operation. For more information about this option, see us/library/ms aspx
Using <low_priority_lock_wait> customer can decide how the index operation can proceed when blocked on the SCH-M lock.
Please note two different MAX_DURATION options. One is related to low_priority_lock_wait and the other is related to RESUMABLE=ON option
The DDL command is running until it completes, pauses or fails.
In case the command pauses, an error will be issued indicating that the operation was paused and the index creation did not complete. In case of a failure an error will be issued as well.
More information about the current index status can be obtained from sys.index_resumable_operations.