Sofia Event Center 21-22 November 2013 Margarita Naumova SQL Master Academy.

Slides:



Advertisements
Similar presentations
Module 13: Performance Tuning. Overview Performance tuning methodologies Instance level Database level Application level Overview of tools and techniques.
Advertisements

new database engine component fully integrated into SQL Server 2014 optimized for OLTP workloads accessing memory resident data achive improvements.
Big Data Working with Terabytes in SQL Server Andrew Novick
6 SQL Server Integration Same manageability, administration & development experience Integrated queries & transactions Integrated HA and backup/restore.
Project “Hekaton” adds in-memory technology to boost performance of OLTP workloads in SQL Server.
Dandy Weyn Sr. Technical Product Mkt.
Planning on attending PASS Summit 2014? The world’s largest gathering of SQL Server & BI professionals Take your SQL Server skills to the.
Dos and don’ts of Columnstore indexes The basis of xVelocity in-memory technology What’s it all about The compression methods (RLE / Dictionary encoding)
Meanwhile RAM cost continues to drop Moore’s Law on total CPU processing power holds but in parallel processing… CPU clock rate stalled… Because.
IIS Server ETL IIS Server This is OPERATIONAL ANALYTICS.
Midterm Review Lecture 14b. 14 Lectures So Far 1.Introduction 2.The Relational Model 3.Disks and Files 4.Relational Algebra 5.File Org, Indexes 6.Relational.
1 External Sorting for Query Processing Yanlei Diao UMass Amherst Feb 27, 2007 Slides Courtesy of R. Ramakrishnan and J. Gehrke.
Evolving SQL Server for Modern Hardware Paul Larson, Eric N. Hanson, Mike Zwilling Microsoft plus to the many members of the Apollo and Hekaton teams Paul.
Architecting a Large-Scale Data Warehouse with SQL Server 2005 Mark Morton Senior Technical Consultant IT Training Solutions DAT313.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1 Preview of Oracle Database 12 c In-Memory Option Thomas Kyte
External Sorting Chapter 13.. Why Sort? A classic problem in computer science! Data requested in sorted order  e.g., find students in increasing gpa.
Module 7 Reading SQL Server® 2008 R2 Execution Plans.
SQL Server 2014: In In-memory OLTP for Database Developers.
Applications hitting a wall today with SQL Server Locking/Latching Scale-up Throughput or latency SLA Applications which do not use SQL Server.
SQL Server 2014 adds in-memory technology to boost performance of OLTP workloads.
Sofia Event Center November 2013 Margarita Naumova
Parallel Execution Plans Joe Chang
IN-MEMORY OLTP By Manohar Punna SQL Server Geeks – Regional Mentor, Hyderabad Blogger, Speaker.
Meet Kevin Liu Principal Lead Program Manager Kevin Liu has been with Microsoft and the SQL Server engine team for 7 years, working on key projects like.
Ἑ κατόν by Niko Neugebauer. Niko Neugebauer PASS EvangelistPASS Evangelist SQL Server MVPSQL Server MVP SQLPort ( founder & leaderSQLPort.
Moore’s Law means more transistors and therefore cores, but… CPU clock rate stalled… Meanwhile RAM cost continues to drop.
Buffer Pool Memory Optimized Tables Available Memory Buffer Pool Memory Optimized Tables Buffer Pool Memory Optimized Tables Buffer Pool Memory.
Sofia Event Center ноември 2013 г. Маги Наумова/ Боряна Петрова.
Chapter 5 Index and Clustering
INTRODUCING SQL SERVER 2012 COLUMNSTORE INDEXES Exploring and Managing SQL Server 2012 Database Engine Improvements.
Query Execution. Where are we? File organizations: sorted, hashed, heaps. Indexes: hash index, B+-tree Indexes can be clustered or not. Data can be stored.
Lock Tuning. Overview Data definition language (DDL) statements are considered harmful DDL is the language used to access and manipulate catalog or metadata.
5 Trends in the Data Warehousing Space Source: TDWI Report – Next Generation DW.
October 15-18, 2013 Charlotte, NC Accelerating Database Performance Using Compression Joseph D’Antoni, Solutions Architect Anexinet.
How to kill SQL Server Performance Håkan Winther.
In-Memory OLTP The faster is now simpler in SQL Server 2016.
PASS Virtual Chapter presentation March 27, 2014.
Vedran Kesegić. About me  M.Sc., FER, Zagreb  HRPro d.o.o. Before: Vipnet, FER  13+ years with SQL Server (since SQL 2000)  Microsoft Certified.
Mladen Prajdić SQL Server MVP Hekaton The New SQL Server In-Memory OLTP Engine.
Oracle Announced New In- Memory Database G1 Emre Eftelioglu, Fen Liu [09/27/13] 1 [1]
Doing fast! Optimizing Query performance with ColumnStore Indexes in SQL Server 2012 Margarita Naumova | SQL Master Academy.
SQL Server 2016: Real-time operational analytics
Best Practices for Columnstore Indexes Warner Chaves SQL MCM / MVP SQLTurbo.com Pythian.com.
Introducing Hekaton The next step in SQL Server OLTP performance Mladen Prajdić
Memory-Optimized Tables Querying at the speed of light.
In-Memory Capabilities
In-Memory OLTP for DBA Management
SQL Server In-Memory OLTP: What Every SQL Professional Should Know
UFC #1433 In-Memory tables 2014 vs 2016
TechEd /6/2018 7:34 PM © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks.
SQL Server 2014 In-Memory OLTP
The Vocabulary of Performance Tuning
PREMIER SPONSOR GOLD SPONSORS SILVER SPONSORS BRONZE SPONSORS SUPPORTERS.
ColumnStore Index Primer
Real world In-Memory OLTP
SQL 2014 In-Memory OLTP What, Why, and How
11/29/2018 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks.
TechEd /2/2018 7:32 AM © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks.
Selected Topics: External Sorting, Join Algorithms, …
1/3/2019 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks.
In Memory OLTP Not Just for OLTP.
Statistics for beginners – In-Memory OLTP
In-Memory OLTP for Database Developers
Sunil Agarwal | Principal Program Manager
The Vocabulary of Performance Tuning
Clustered Columnstore Indexes (SQL Server 2014)
In Memory OLTP Not Just for OLTP.
The Vocabulary of Performance Tuning
SQL Server Columnar Storage
Sunil Agarwal | Principal Program Manager
Presentation transcript:

Sofia Event Center November 2013 Margarita Naumova SQL Master Academy

Administrator

Memory-optimized Table: Row Format Row headerPayload (table columns) Begin Ts End Ts StmtId IdxLinkCount 8 bytes 4 bytes 2 bytes 8 bytes * (IdxLinkCount – 1)

Key lookup: B-tree vs. Memory-optimized Table Matching index record Hash index on Name R1 R2 R3

Memory Optimized Tables and Indexes 50, ∞ JanePrague TimestampsNameChain ptrsCity Hash index on City Hash index on Name 90, ∞ Susan Bogota f(Jane) f(Susan) f(Prague) f(Bogota)

Memory Optimized Tables and Indexes 50, ∞ JanePrague TimestampsNameChain ptrsCity Hash index on City Hash index on Name T100: INSERT (John, Prague) 100, ∞ JohnPrague 90, ∞ Susan Bogota f(John) f(Prague)

90, ∞ Susan Bogota Memory Optimized Tables and Indexes 50, ∞ JanePrague TimestampsNameChain ptrsCity Hash index on City Hash index on Name T150: DELETE (Susan, Bogota) 100, ∞ JohnPrague 90, 150

Memory Optimized Tables and Indexes 90, 150 Susan Bogota 50, ∞ JanePrague TimestampsNameChain ptrsCity Hash index on City Hash index on Name T200: UPDATE (John, Prague) to (John, Beijing) 100, ∞ JohnPrague 200, ∞ JohnBeijing 100, 200 f(Beijing) f(John)

Memory Optimized Tables and Indexes 90, 150 Susan Bogota 50, ∞ JanePrague TimestampsNameChain ptrsCity Hash index on City Hash index on Name T250: Garbage collection 100, 200 JohnPrague 200, ∞ JohnBeijing f(John) f(Jane) f(Beijing) f(Prague)

Storage: Data and Delta Files Data File Delta File 0100 TS (ins)RowIdTableId TS (ins)RowIdTableId TS (ins)RowIdTableId TS (ins)RowIdTS (del) TS (ins)RowIdTS (del) TS (ins)RowIdTS (del) Checkpoint File Pair Row pay load Data files ~128MB in size, write 256KB chunks at a time Stores only the inserted rows (i.e. table content) Chronologically organized streams of row versions Delta files File size is not constant, write 4KB chunks at a time. Stores IDs of deleted rows

Populating Data/Delta files Offline Checkpoint Thread Memory-optimized Table Filegroup Range Range Range Range Range 500- New Inserts Delete 450 TS Delete 250 TS Delete 150 TS Data file with rows generated in timestamp range IDs of Deleted Rows (height indicates % deleted) Del Tran2 (TS 450) Del Tran3 (TS 250) Del Tran1(TS150 ) Insert into Hekaton T1 Log in disk Table Del Tran1 (row TS150) Del Tran2 (row TS 450) Del Tran3 (row TS 250) Insert into T1 SQL Transaction log

Merge Operation Memory-optimized data Filegroup Files as of Time 600 Range Range Range Range Data file with rows generated in timestamp range IDs of Deleted Rows (height indicates % deleted) Merge Deleted Files Files Under Merge Files as of Time 500 Memory-optimized data Filegroup Range Range Range Range Range Range Range Range

15 What’s being delivered High-performance, memory-optimized OLTP engine integrated into SQL Server and architected for modern hardware trends Main benefits Optimized for in-memory data up to 20–30 times throughput Indexes (hash and range) exist only in memory; no buffer pool, B-trees T-SQL compiled to machine code via C code generator and Visual C compiler Core engine uses lock-free algorithms; no lock manager, latches, or spinlocks Multiversion optimistic concurrency control with full ACID support On-ramp existing applications Integrated experience with same manageability, administration, and development experience In-Memory OLTP summary

In-Memory DW

Columnstore Storage Model Data stored column- wise Each page stores data from a single column Highly compressed Each column can be accessed independently

In-Memory DW Index Processing an Example 18

Horizontally Partition Row Groups 19

Vertical Partition Segments 20

Compress Each Segment 21 Some Compress More than Others

Fetch Only Needed Columns Segment Elimination 22

Fetch Only Needed Segments Segment Elimination 23

Updatable Columnstore Index Table consists of column store and row store DML (update, delete, insert) operations leverage delta store SELECT Unifies data from Column and Row stores - internal UNION operation. “Tuple mover” converts data into columnar format once segment is full (1M of rows) REORGANIZE statement forces tuple mover to start. C1 C2 C3 C5C6C4 Column Store C1 C2 C3 C5C6C4 Delta (row) store tuple mover

Structure of In-Memory DW How It Works 25 CREATE CLUSTERED COLUMNSTORE ‐ Organizes and compresses data into columnstore BULK INSERT ‐ Creates new columnstore row groups INSERT ‐ Rows are placed in the row store (heap) ‐ When row store is big enough, a new columnstore row group is created ColumnStore Deleted Bitmap Row Store Partition

Structure of In-Memory DW How It Works (cont'd) DELETE ‐ Rows are marked in the deleted bitmap UPDATE ‐ Delete plus insert Most data is in columnstore format ColumnStore Deleted Bitmap Row Store Partition

Columnstore usage recommendations Use NCSI If you need constraints and forcing uniqueness Use CCI in other cases Use for big tables JOINs on string columns is still slow! Use integer keys instead Tune batch sizes for BULK INSERTS You don’t need any special maintenance if your data loading is tuned and you don’t have many deletes

Resource Governor

Resource Governor components

Resource Pools CREATE RESOURCE POOL pool_name [ WITH ( [ MIN_CPU_PERCENT = value ] [ [, ] MAX_CPU_PERCENT = value ] [ [, ] CAP_CPU_PERCENT = value ] [ [, ] AFFINITY {SCHEDULER = AUTO | (Scheduler_range_spec) | NUMANODE = (NUMA_node_range_spec)} ] [ [, ] MIN_MEMORY_PERCENT = value ] [ [, ] MAX_MEMORY_PERCENT = value ] [ [, ] MIN_IOPS_PER_VOLUME = value ] [ [, ] MAX_IOPS_PER_VOLUME = value ]) ]

Resource Pools

Create workload groups Create function to classify requests into workload group Register the classification function in the previous step with the Resource Governor Enable Resource Governor Monitor resource consumption for each workload group Use monitor to establish pools Assign workload group to pool Steps to implement Resource Governor

Scenario 1: I just got a new version of SQL Server and would like to make use of resource governor. How can I use it in my environment? Scenario 2 (based on Scenario 1): Based on monitoring results I would like to see an event any time a query in the ad-hoc group (groupAdhoc) runs longer than 30 seconds. Scenario 3 (based on Scenario 2): I want to further restrict the ad- hoc group so it does not exceed 50 percent of CPU usage when all requests are cumulated. Resource Governor scenarios

Monitoring Resource Governor

Complete and consistent data platform

Споделете вашата обратна връзка за тази сесия и за цялостната организация на конференцията и участвайте в томболата за HTC 8S и други награди!