SQL Server Internals 101 AYMAN SENIOR MICROSOFT.

Slides:



Advertisements
Similar presentations
Overview of Mass Storage Structure
Advertisements

Storing Data: Disks and Files: Chapter 9
SEMINAR ON FILE SLACK AND DISK SLACK
IELM 230: File Storage and Indexes Agenda: - Physical storage of data in Relational DB’s - Indexes and other means to speed Data access - Defining indexes.
1 Storing Data: Disks and Files Yanlei Diao UMass Amherst Feb 15, 2007 Slides Courtesy of R. Ramakrishnan and J. Gehrke.
Silberschatz, Galvin and Gagne ©2009 Operating System Concepts – 8 th Edition Mass-Storage Systems Revised Tao Yang.
CPSC 231 Secondary storage (D.H.)1 Learning Objectives Understanding disk organization. Sectors, clusters and extents. Fragmentation. Disk access time.
Introduction to Database Systems 1 The Storage Hierarchy and Magnetic Disks Storage Technology: Topic 1.
12.1 Silberschatz, Galvin and Gagne ©2009 Operating System Concepts with Java – 8 th Edition Chapter 12: Mass-Storage Systems.
Secondary Storage Unit 013: Systems Architecture Workbook: Secondary Storage 1G.
Fast Track, Microsoft SQL Server 2008 Parallel Data Warehouse and Traditional Data Warehouse Design BI Best Practices and Tuning for Scaling SQL Server.
Layers of a DBMS Query optimization Execution engine Files and access methods Buffer management Disk space management Query Processor Query execution plan.
Operating Systems CMPSC 473 I/O Management (2) December Lecture 24 Instructor: Bhuvan Urgaonkar.
1 Lecture 7: Data structures for databases I Jose M. Peña
CS4432: Database Systems II Data Storage (Better Block Organization) 1.
1 Recitation 8 Disk & File System. 2 Disk Scheduling Disks are at least four orders of magnitude slower than main memory –The performance of disk I/O.
Lecture 11: DMBS Internals
Silberschatz, Galvin and Gagne ©2013 Operating System Concepts – 9 th Edition Chapter 10: Mass-Storage Systems.
Disk Access. DISK STRUCTURE Sector: Smallest unit of data transfer from/to disk; 512B 2/4/8 adjacent sectors transferred together: Blocks Read/write heads.
The Computer Systems By : Prabir Nandi Computer Instructor KV Lumding.
Introduction to Database Systems 1 Storing Data: Disks and Files Chapter 3 “Yea, from the table of my memory I’ll wipe away all trivial fond records.”
IT 344: Operating Systems Winter 2010 Module 13 Secondary Storage Chia-Chi Teng CTB 265.
Architecture Rajesh. Components of Database Engine.
GCSE Information Technology Storing data Data storage devices can be divided into 2 main categories: Backing storage is used to store programs and data.
Chapter 12: Mass-Storage Systems Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Chapter 12: Mass-Storage.
The disk surface is divided into tracks. into tracks. 1.
ICS 321 Fall 2011 Overview of Storage & Indexing (i) Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa 11/9/20111Lipyeow.
OSes: 11. FS Impl. 1 Operating Systems v Objectives –discuss file storage and access on secondary storage (a hard disk) Certificate Program in Software.
External Storage Primary Storage : Main Memory (RAM). Secondary Storage: Peripheral Devices –Disk Drives –Tape Drives Secondary storage is CHEAP. Secondary.
Chapter 8 External Storage. Primary vs. Secondary Storage Primary storage: Main memory (RAM) Secondary Storage: Peripheral devices  Disk drives  Tape.
DMBS Internals I. What Should a DBMS Do? Store large amounts of data Process queries efficiently Allow multiple users to access the database concurrently.
Disk Basics CS Introduction to Operating Systems.
HARD DISK Prabhavati M. (985450) Kamal Gidwani (962841) Sharvari Joshi (962281) Poonam Rode (985500) Rajesh S. (962206) )
DMBS Internals I February 24 th, What Should a DBMS Do? Store large amounts of data Process queries efficiently Allow multiple users to access the.
DMBS Internals I. What Should a DBMS Do? Store large amounts of data Process queries efficiently Allow multiple users to access the database concurrently.
CPSC 231 Secondary storage (D.H.)1 Learning Objectives Understanding disk organization. Sectors, clusters and extents. Fragmentation. Disk access time.
Programmer’s View of Files Logical view of files: –An a array of bytes. –A file pointer marks the current position. Three fundamental operations: –Read.
Computer Performance. Hard Drive - HDD Stores your files, programs, and information. If it gets full, you can’t save any more. Measured in bytes (KB,
Optimizing SQL Server and Databases for large Fact Tables =tg= Thomas Grohser, NTT Data SQL Server MVP SQL Server Performance Engineering SQL Saturday.
Lecture 3 Secondary Storage and System Software I
Storage Tuning for Relational Databases Philippe Bonnet – Spring 2015.
Select Operation Strategies And Indexing (Chapter 8)
1 Components of the Virtual Memory System  Arrows indicate what happens on a lw virtual address data physical address TLB page table memory cache disk.
Ayman El-Ghazali Senior Microsoft.
Indexing strategies and good physical designs for performance tuning Kenneth Ureña /SpanishPASSVC.
SQL Server Storage Inside. About Hemantgiri S. Goswami Hemantgiri S. Goswami is a Lead Database Consultant for Pythian, a company head quartered in Ottawa,
Operating System Concepts with Java – 7 th Edition, Nov 15, 2006 Silberschatz, Galvin and Gagne ©2007 Chapter 11: File System Implementation.
Chapter 10: Mass-Storage Systems
Operating System (013022) Dr. H. Iwidat
A Day in the Life of a Row Eddie Wuerch, mcm
What Is a Latch? …and Why Do I Care? Eddie Wuerch, mcm
Lecture 16: Data Storage Wednesday, November 6, 2006.
Database Management Systems (CS 564)
Computer Science 210 Computer Organization
Chapter 12: Mass-Storage Structure
I/O Resource Management: Software
Database Management Systems (CS 564)
CPSC-608 Database Systems
Oracle SQL*Loader
Mass-Storage Structure
Memory chips Memory chips have two main properties that determine their application, storage capacity (size) and access time(speed). A memory chip contains.
Hustle and Bustle of SQL Pages
File Processing : Storage Media
Lecture 11: DMBS Internals
What Is a Latch? …and Why Do I Care? Eddie Wuerch, mcm
File Processing : Storage Media
CSE 451: Operating Systems Autumn 2004 Secondary Storage
Mass-Storage Systems (Disk Scheduling)
Hybrid Buffer Pool The Good, the Bad and the Ugly
Presentation transcript:

SQL Server Internals 101 AYMAN SENIOR MICROSOFT

Agenda How Data is Stored: Pages & Extents How Data is Retrieved: The Query Lifecycle How the Infrastructure Works: Storage (HDD,SSD,RAID) & Memory Speed Assessment

How Data is Stored and Organized: Pages, Extents, Allocation Bitmap Pages

Pages 8KB in Size or 8192 Bytes Header is 96 Bytes Contains: PageID Pointers to Previous and Next Pages Free space Total # of rows on the page 128 Pages = 1 Megabyte | 128,000 = 1 Gigabyte Smallest Unit of allocation; there are no half pages

Pages Page Header (96 Bytes) Data Row 1 (100 Bytes) Data Row 3 (80 Bytes) Data Row 2 (200 Bytes) 321 Warning: Actual colors may vary! Row offset: Distance from the first byte in the row to the start of the page For this example: Row Offset 1 would have the value 196 Row Offset 2 would have the value 396 Row Offset 3 would have the value 476

Extent A set of 8 Pages 1 Page = 8KB | 8 Pages = 64 KB 16 Extents per MB | 16,000 Per GB Basic unit of space management for SQL Server data and index pages Two Types Mixed: contains pages owned by multiple objects (up to 8) Uniform: all pages contained within come from one object and the entire extent can only be used by that object

Allocation Bitmap Pages The internal “directory” of information on pages GAM Which extents are available for allocation Covers a 4GB interval of data SGAM Which mix extents have at least one page to allocate Covers a 4GB interval of data PFS Used to track how much free space is on pages Tracks other attributes as well Covers a 64MB interval of data

GAM Extent 0 Extent 250 Tracks 64,000 Extents (4GB) 1 Represents if that extent is free GAM Page

Any Free Storage Space Available?

Index Allocation Map (IAM) Special internal page on a data file that tracks all extent allocations for tables, indexes, and partitions Basically, lets SQL Server know what extent belongs to what specific entity Covers a 4GB interval of data

Uniform ExtentsMixed Extent Extent 1 (Table1) Extent 2 (Table2) Extent 3 (Table1) IAM1 Allocation Bitmaps Table3 SGAM GAM PFS IAM2

Does it sound Greek? PFS, GAM and SGAM help SQL Server determine where free space is and how much so that it can allocate it appropriately without having to scan all the pages IAM pages and indexes help SQL Server fulfil queries The take away is that these few pages help keep an internal “directory” for SQL Server to be able to find what it needs quickly and pull back as few pages as necessary It’s all about reduction of cost to the system

The Query Lifecycle Client Query Engine Storage Engine Storage (RAM or HD) Query Engine

FSB CPU North Bridge (Memory Control) South Bridge (I/O Control) VIA DMA SATA/SAS PCI-E RAM

Refer to the book: Professional SQL Server 2008 Internals and Troubleshooting

How the Infrastructure Works: Storage (HDD,SDD,RAID) & Memory

Magnetic Storage 1 Platter(s) 5 Drive Head 6 Track(s) 7 Sector(s) 8 Cluster(s) * Sector sizes in many modern disks are now 4K in size (previously 512 bytes)

Magnetic Storage Depending on how you format your partitions (cluster size), disk performance can be affected Microsoft recommends formatting the clusters to 64K allocation units Why? Extents are 64K in size and this can help with Read operations since tables/indexes are organizes in Extents by SQL Server Think of clusters as data “shopping bags” You could use several huge bags to carry all your groceries You can use a larger quantity of smaller bags to carry all your groceries Each bag costs you 5 cents from the grocery store, and all bags are equal in strength

Allocation Units 64KB 1 Physical read per Extent Track Sectors Drive Head

Allocation Units 4KB 2 Physical Reads per Page Track Sectors Drive Head

SSD – NAND Flash 64KB Allocation Units 4KB Allocation Units

Partition1 RAID Basics RAID O (Zero) Reads are divided among hard drives (multiple read heads) content/uploads/2012/02/Hard_Disk1.png

LUN1 LUN2 LUN3 LUN4 SAN with RAID content/uploads/2012/02/Hard_Disk1.png

RAM Entire pages are loaded into RAM (not rows!) Therefore if page is 50% full, 8KB of space is consumed from RAM Imagine if the example below was in GB not KB 128 KB of Data in pages that are full takes up 128KB (Fill Factor 100%) 64 KB of Data in pages that are half full takes up 128KB (Fill Factor 50%)

What about compression? Affects CPU usage (negatively but not very noticeable) Pages are then decompressed and take same space in RAM So why do it? Saves I/O which is the real bottleneck! Just think of the math. It will take X% less time to push compressed data from disk to RAM. X% is compression rate. Pages stay compressed in memory until used (*not the entire page, but only the data values of interest; from BOL) 64KB pages compressed down to 32KB (50%) 32KB pages decompressed to 64KB (if all pages had to be decompressed) Transfer 32KB Decompress

ZYNZ1GD3ZYNZ1GD4ZYNZ1GD5ZYNZ1GD6 ZYNZ1GD7ZYNZ1GD8ZYNZ1GD9ZYNZ1GD10 How pages are stored in RAM Using Hash Buckets Page InfoBuffer Cache Address 1:1:200XYNPUG19 3:7:98AZX444ST1 5:1:199ZYNZ1GD8 2:2:17XYAPUG10 1:1:21TRW56F7F Request for Page #199 on File #1 from DatabaseID 5 Stored in reverse DatabaseID:FileNumber:PageNumber 5:1:199 5:1: 199 Hash Bucket Data Cache (RAM)

Speed Assessment

Speed Assessment (Price Estimates from Feb 2015) MetricDD4 RAMDDR3 RAMSSD (Solid State Drive)HDD (Hard Disk Drive) Random Access Speed or Latency (RAM) Slightly higher than DDR3 10ns (0.01 Microsecond) 100 Microseconds (0.1 ms) 100,000 Nanoseconds (ns) 4-9 milliseconds (ms) Microseconds Cost per GB$18+$5-$40$0.5-$3.00 (Enterprise PCI-E $2+)$0.05-$0.15 Power ConsumptionLower than DDR32.5-5W0.6W-3W (PCI-E ~20W)5-20 Watts Data Transfer Rate (Peak) SATA III (600 MB/s)N/A MB/sLess than 15K RPM PCI-E rev 3 x16 (~16 GB/s)N/A MB/sN/A DIMM25,000+ MB/s ,000 MB/sN/A *DDR4 Memory expected in 2014 (Doubling speed and Density; lower power consumption) *Drive price and specs for Single Drives not SAN Storage *SATA III and SAS are very close in performance *Fibre Channel connection on a SAN Speeds are between MB/s for transfer rates *SSD sizes max out around 1TB for mainstream drives; HDD sizes max around 4TB

Picture worth 1000 MB? Pages per second on peak transfer HDD 10,000 Pages SSD DDR3 RAM SSD PCI-E 50,000 Pages 250,000 Pages

Access Speed or Latency time Every nanosecond you waste, a cookie is burned in someone’s oven RAM SSD HDD 1 Cookie 10,000 Cookies 1,000,000 Cookies

Why does all this matter? We are Geeks, we should be excited by this stuff, the world depends on us! Truth is, the CPU is a busy worker on the system that pulls data from RAM Data from disk goes to RAM, and it is the main bottle neck for performance The slower the data request goes from CPU – Disk – Back to CPU the slower work is performed

FSB CPU North Bridge (Memory Control) South Bridge (I/O Control) VIA DMA SATA/SAS PCI-E RAM CPU does all the “Thinking” Requests data from disk Disk returns data to Memory via Direct Memory Access Memory pushes data to CPU via Front Side Bus For SQL Server, when the pages are kept in memory, they can be brought back from memory without having to go to disk.

Thank you! AYMAN