Les Conférences Groupe des Utilisateurs SQL Server Juin 2013 – SQL Server in-memory Alexandre Chemla – Masao Frédéric Pichaut – Microsoft

Memory-optimized Table Filegroup Data Filegroup SQL Server.exe Hekaton Engine: Memory_optimized Tables & Indexes TDS Handler and Session Management Natively Compiled SPs and Schema Buffer Pool for Tables & Indexes Proc/Plan cache for ad-hoc T- SQL and SPs Client App Transaction Log Query Interop Non-durable Table T1 T4 T3 T2 T1 T4 T3 T2 T1 T4 T3 T2 T1 T4 T3 T2 Tables Indexes Interpreter for TSQL, query plans, expressions T1 T4 T3 T2 T1 T4 T3 T2 Checkpoint & Recovery Access Methods Parser, Catalog, Algebrizer, Optimizer Hekaton Compiler Hekaton Component Key Existing SQL Component Generated.dll

Memory-optimized Table Filegroup Data Filegroup SQL Server.exe Hekaton Engine for Memory_optimized Tables & Indexes TDS Handler and Session Management Natively Compiled SPs and Schema Buffer Pool for Tables & Indexes Proc/Plan cache for ad- hoc T-SQL and SPs Client App Transaction Log Query Interop Interpreter for TSQL, query plans, expressions Access Methods Parser, Catalog, Algebrizer, Optimizer Hekaton Compiler 10-30x more efficient Reduced log bandwidth & contention. Log latency remains Checkpoints are background sequential IO No improvements in communication stack, parameter passing, result set generation Hekaton Component Key Existing SQL Component Generated.dll

OperationFactor faster (slower) than regular SQL Comments InteropNative Select count(*) 1 (2.5)= No clustered index scan in Hekaton Hash Join 1 (1.3)N/A Uses index scan Nested-loop Join 1 4.0N/A Probes into hash index Single-row selects SP doing selects in loop Single-row selects Native compiled SP calls SQL’s rand() Single-row updates 1 N/A10 SP doing update in loop Bwin Session State6 Version M4 Expectation for OLTP workloads Advantage of pushing work to SPs Interop targets app migration, not perf (1) 1 million rows accessed in single query or SP

CREATE TABLE DDLTable code generatedCompiler invokedTable DLL producedTable DLL loaded

90,150 Susan Bogota 50, ∞ JanePrague 100, 200 JohnParis 200, ∞ JohnBeijing TimestampsNameChain ptrsCity Hash index on City B P Hash index on Name J S Garbage Collection Removes Unused Rows

Hekaton Memory Transaction log 237 | 001 | George | SEA 235 | 002 | Fred | CHI 237 | 001 | George | SEA 234 | 001 | George | LAX 235 | 002 | Fred | CHI 237 | 001 | George | SEA Checkpoint File Delta File 234 | 001 | 237 XID RowID Name(PK) Airport Create Delete XID RowID XID Del Add 235 | 002 | Fred | CHI 234 | 001 | George | LAX

T-SQL Compiled to Machine Code T-SQL compiled to machine code via C code generator and VC Invoking a procedure is just a DLL entry-point Aggressive compile-time Stalling CPU clock rate Hardware trends Efficient, business- logic processing Customer Benefits Hekaton Tech Pillars Drivers Native Compiled Stored Procedures Non-Native Compilation Performance High. Significantly less instructions to go through No different than T-SQL calls in SQL Server today Migration StrategyApplication changes – development overhead Easier app migration as can still access Memory Optimized (MO) tables Access to objectsCan only interact with Memory Optimized tables All Objects. Access for transactions across MO and b-tree tables Support for T-SQL Constructs Limited.T-SQL surface area (limit on MO interaction) Optimization/Stats and Query Plan Statistics utilized at CREATE -> Compile time Statistics updates can be utilized to modify plan at runtime FlexibilityLimited (e.g., no ALTER procedure, compile-time isolation level) Ad-hoc query patterns

TimeTransaction T1 (SNAPSHOT)Transaction T2 (SNAPSHOT) 1BEGIN 2 3UPDATE t SET c1=‘bla’ WHERE c2=123 4UPDATE t SET c1=‘bla’ WHERE c2=123 (write conflict) First writer wins

CREATE TABLE dbo.Orders( OrderID int NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT= ), CustomerID int NOT NULL INDEX IX_CustomerID HASH WITH (BUCKET_COUNT= ), OrderDate datetime NOT NULL, OrderDescription nvarchar(1000) ) WITH (MEMORY_OPTIMIZED=ON) Assume the Orders table has 1M rows, and the average length of OrderDescription 78 characters. Index size: The bucket_count This is rounded up to the nearest power of 2: * * = bytes Table data size [row size] * [row count] = [row size] * 8379 [row size] = [row header size] + [actual row body size] [row header size] = * [number of indices] = * 2 = 40 bytes [actual row body size] SUM([size of shallow types]) = 4 [int] + 4 [int] + 8 [datetime] = * [number of deep type columns] = * 1 = 4 NULL array = 1 + NULL array padding = 1 Size so far is = 22. Padding to Nearest multiple of 8 is [actual row body size] = *78 = 180 bytes. So [row size] = = 220 bytes [table size] = * = bytes ~= 230Mb

Memory Optimized Data Filegroup Range Range Range Range Range 500- offline checkpoint Thread A transaction with a commit timestamp of 600 inserts one new row and deletes rows inserted by transactions with a commit timestamp of 150, 250 and 420

Memory Optimized Data Filegroup Key Range Range Range Range Range Range Range Range Data file with rows generated in timestamp range a-b Delta file with IDs of Deleted Rows Merge Range Range Deleted Files

sys.dm_db_xtp_checkpointReturns database that has one or more IM objects sys.dm_db_xtp_checkpoint_filesDisplays information about checkpoint files sys.dm_db_xtp_hash_index_statsUseful for understanding and tuning the bucket counts and duplicates for index key sys.dm_db_xtp_index_statsReports statistics about scans on an index sys.dm_db_xtp_memory_consumersReports the database-level memory consumers in the IM database engine. sys.dm_db_xtp_object_statsReports statistics about operations on a memory optimized object. sys.dm_db_xtp_table_memory_statsReturns memory usage statistics for each IM table (user and system) in the current database. sys.dm_db_xtp_transactionsReports the active transactions in the IM database engine. sys.dm_xtp_consumer_memory_usageReports memory usage for all memory consumers level level. sys.dm_xtp_gc_statsReports information about the current behavior of the IM garbage-collection process. sys.dm_xtp_system_memory_consumersReports information about memory usage. sys.dm_xtp_threadsReports the threads that the IM database engine has started internally. sys.dm_xtp_transaction_statsReports statistics about transactions that have run since the server started.

New or updated property, system view, stored procedures, or DMV Change OBJECTPROPERTYEXNew property: TableIsMemoryOptimized. SERVERPROPERTYNew property: IsXTPSupported. sys.data_spacesThe following columns display additional values: type and type_desc sys.indexesThe following columns display additional values: type and type_desc. sys.parametersNew column: is_nullable. sys.all_sql_modulesNew column: uses_native_compilation. sys.sql_modulesNew column: uses_native_compilation. sys.table_typesNew column: is_memory_optimized. sys.tablesNew columns: durability, durability_desc, and is_memory_optimized. sys.hash_indexesNew: Shows the current hash indexes and the hash index properties sp_xtp_merge_checkpoint_filesNew stored procedure: Merges all data and delta files in the transaction range specified.

Column Store (CS) Deleted Bitmap Row Store (RS) Partition

CREATE TABLE ( … ) CREATE CLUSTERED COLUMNSTORE INDEX on Converts entire table to CS format Take care of memory needed and parallelism (MAXDOP 1) BULK INSERT, SELECT INTO on Creates new CS row groups INSERT/UPDATE Store in Row Store Tuple Mover When RS reaches 1M rows, convert to a CS row group Runs every 5 minutes by default Started explicitly by ALTER INDEX ON REORGANIZE Partitioning works on clustered columnstores Just like any other table The motivation is manageability more than performance

Les Conférences Groupe des Utilisateurs SQL Server GUSS.fr