Vedran Kesegić

About me  M.Sc., FER, Zagreb  HRPro d.o.o. Before: Vipnet, FER  13+ years with SQL Server (since SQL 2000)  Microsoft Certified Master: SQL 2008  Author: XDetails – addin for SQL Studio  Hobby: Photography

What you will hear  What is new in SQL2014 ?  What is In-Memory OLTP ?  New DB engine and RAM structures  RAM - do we have Durability? How?  Memory-optimized tables  Memory-optimized types (and table variables)  New Indexes: hash index, range index  Natively compiled stored procedures  Demo

SQL Server 2014 (Hekaton) Released date: (RTM)  In-memory tables and table types (table variables)  Compiled procedures (eXtreme Transaction Processing)  Buffer Pool Extension (to SSD)  Delayed Durability (WRTELOG wait eliminator)  Rebuild Partition ONLINE  Partition-level statistics  Updatable Clustered Columnstore Index (INS/UPD/DEL)  Resource governor can now limit IO (IOPS)  AlwaysOn enhanced (8 replicas, better algorithms)  Standard Edition max. RAM raised from 64 to 128GB

What is In-Memory OLTP ?  Today servers: 1TB RAM, 32 CPU < $  It is cheaper to buy RAM than pay CPU licences  NOT cached tables!  New database engine in SQL 2014  Built for speed, not comfort (lots of limitations)  Extreme concurrency (no locking – no blocking)  XTP (eXtreme Transaction Processing) – machine code compiled SP’s

How ?  Tables in RAM  Faster and smaller transaction log  MVCC (MultiVersion Concurrency Control)  No locks, no latches, no spinlocks  New indexes: HASH index, RANGE index  Natively compiled SP (machine code)

Overview

Memory-optimized Tables  Disk-based tables: 8KB pages, with rows in them  Memory-optimized tables: linked ROW structures in RAM, no pages!  ROW limited to 8060 bytes (no row overflow, no LOB)  DLL per table generated for table access and traverse rows In-memory ROW structure

Memory-optimized Tables No locks, latches, spinlocks - really? Really!  What about writing EndTs? Lock must exist!?  hardware CAS instructions (compare and switch)

Indexes  The only thing that connects rows into table (table MUST have min. ONE index!)  Max 8 indexes per table  Created together with the table (cannot be created later)  Not durably stored, built on db start  Two types: HASH indexes RANGE indexes

HASH Index  Array of pointers  Key value is hashed = location  BUCKET_COUNT = #distinct key values

HASH Index  Two indexes = Two pointers  Max 8 indexes per table (8 pointers)

RANGE Index  When no idea about bucket count  When you need a range scan (not just equality)  Bw-tree (new data struture)  Lock-free, latch-free variation of B-tree  Envisioned by Microsoft Research 2011  Variable-length pages, immutable  Pointers are not direct memory address: PID + Page Mapping Table = address

RANGE Index

Durability  Checkpoint – stored as FILESTREAM Data stream (INSERTS) Delta stream (DELETES – int filter)  Transaction log Written into regular transaction log Only REDO info (UNDO not needed) Uncommitted transactions are not logged Index operations are not logged Grouped into one big IO = very efficient  All indexes rebuilt on recovery, during table data loading into RAM

Durability SCHEMA_AND_DATA (default) SCHEMA_ONLY Schema is durable Rows are durable  Is in checkpoint files  Is logged (Tran. Log)  Is in backup  Survives AG failover  Blazing fast

Natively Compiled SP  TSQL (CREATE TABLE, SELECT,...) converted to C source code  C compiled to DLL (by Visual C compiler)  One machine-code DLL per SP  DLL is in RAM, not durably stored. Only metadata (TSQL) needed for DLL creation  DLL is generated (compiled) on every start of a database or instance, from metadata

Natively Compiled SP -- name = "...\xtp_t_ _.dll" SELECT m.name, m.description FROM sys.dm_os_loaded_modules m WHERE m.description='XTP Native DLL'  Stored in file system for debug purposes  C source code also

Natively Compiled SP  Marked „WITH NATIVE_COMPILATION”  SCHEMABINDING is required  EXECUTE AS is required (CALLER not supported. Only OWNER, SELF, user)  One ATOMIC block required: TRANSACTION ISOLATION LEVEL (SNAPSHOT, REPEATABLEREAD, SERIALIZABLE) LANGUAGE (sys.syslanguages)  Cannot partially succeed (rollbacked to savepoint at the beginning)

Natively Compiled SP Limitations:  „two.part” names must be used  Not all TSQL can be used  Restrictions are documented, will change us/library/dn246937(v=sql.120).aspx us/library/dn246937(v=sql.120).aspx

Natively Compiled SP CREATE PROCEDURE INT) WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER AS BEGIN ATOMIC WITH ( TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = 'English' ) SELECT Name, Address, Phone FROM dbo.Customers WHERE Id END GO

Interop TSQL  Interpreted TSQL accessing memory-optimized tables  Interop can access both memory-optimized and disk-based tables  Natively compiled can access only memory- optimized tables  NOT as fast as native (but still VERY fast!)  Useful as a step in migration to native

Out of presentation scope  Transaction isolation  Garbage collection  Checkpoint files automerging  New DMV’s (%XTP%) and other changes  Special Transaction log records (XTP) inspection

Summary  In-Memory OLTP is a major advance  100x faster on the same hardware is achievable  Speed has tradeoff in restrictions (less restrictions in the future)  Migrate only performance critical tables and procedures

Literature  Kalen Delaney: „SQL Server In-Memory OLTP Internals Overview for CTP2”, SQL 2014 CTP2 Whitepaper, October 2013  Microsoft (multiple authors): „ Hekaton: SQL Server’s Memory-Optimized OLTP Engine”