Thursday April 17, 2008 ScaleDB Technical Presentation Scaling MySQL to New Heights

ScaleDB ScaleDB for MySQL Database Storage Engine InnoDB, MyISAM, Cluster, Falcon, BDB, Merge, etc.

ScaleDB What Makes ScaleDB Better? ScaleDB Advantages: Performance: New indexing delivers dramatic performance improvement Scalability: Designed for clustering with Plug- and-Cluster™ Architecture

Improving Performance

ScaleDB ScaleDB Indexing Conventional Indexing (B-tree) General Purpose Indexing HashBitmapAggregate Etc. Special-purpose Index Add-ons* *Only supported by high-end commercial databases ScaleDB Index: A general purpose index that also delivers much of the functionality and performance of special-purpose index add-ons

ScaleDB ScaleDB: Multi-Table Indexing B-tree: Only indexes the data in tables Index #1 #1#2 Index #2Index #3Index #4Index #5 #3#4#5 ScaleDB Index #1 #2 #3 #4 #5 ScaleDB: Indexes the data and relationships Advantages: Faster Smaller Referential integrity More functionality

ScaleDB Describing Our Demo Scenario: Select information that is spread across 3 tables: Colleges, Students and Enrollment Relationships: Students are enrolled in courses within departments of colleges DDL Definitions

ScaleDB The Query SELECT c1.CollName, s.StudName, c2.CourseName, e.Grade FROM College AS c1 STRAIGHT_JOIN Student AS s STRAIGHT_JOIN Enrollment AS e STRAIGHT_JOIN Course AS c2 ON ( c1.CollNo = s.CollNo AND s.CollNo = e.CollNo AND s.StudentNo = e.StudentNo AND e.CollNo = c2.CollNo AND e.DeptNo = c2.DeptNo AND e.CourseNum = c2.CourseNum ) WHERE c1.CollNo = X AND s.StudentNo = Y ;

ScaleDB A Sample Scenario Scenario: I need information that is spread across 3 tables: Colleges, Students and Enrollment Col_ID# Col_Name Col_Budget Col_Description Colleges 0001Amhearst$1,234,567Nice place to visit 0002Berkeley$5,432,567Sports not so good 0003Harvard$9,999,666Cool logo 0004Holy Cross$3,234,567Ugh Worcester 0005MIT$8,238,568Serious work 0006Cornell$7,237,767Jumpy students 0007Stanford$9,898,777Pretty campus 0008TCU$5,987,004In Texas Students 0001Amhearst$1,234,567Nice place to visit 0002Berkeley$5,432,567Sports not so good 0003Harvard$9,999,666Cool logo 0004Holy Cross$3,234,567Ugh Worcester 0005MIT$8,238,568Serious work 0006Cornell$7,237,767Jumpy students 0007Stanford$9,898,777Pretty campus 0008TCU$5,987,004In Texas Enrollment 0001Amhearst$1,234,567Nice place to visit 0002Berkeley$5,432,567Sports not so good 0003Harvard$9,999,666Cool logo 0004Holy Cross$3,234,567Ugh Worcester 0005MIT$8,238,568Serious work 0006Cornell$7,237,767Jumpy students 0007Stanford$9,898,777Pretty campus 0008TCU$5,987,004In Texas Coll_ID#Coll_NameColl_BudgetColl_Description Dept_ID#Dept_NameColl_ID#Dept_BudgetCourse_ID#Course_NameColl_ID#Dept_ID# Options: #1: Conventional Joins #2: Materialized View #3: ScaleDB

ScaleDB The Query SELECT c1.CollName, s.StudName, c2.CourseName, e.Grade FROM College AS c1 STRAIGHT_JOIN Student AS s STRAIGHT_JOIN Enrollment AS e STRAIGHT_JOIN Course AS c2 ON ( c1.CollNo = s.CollNo AND s.CollNo = e.CollNo AND s.StudentNo = e.StudentNo AND e.CollNo = c2.CollNo AND e.DeptNo = c2.DeptNo AND e.CourseNum = c2.CourseNum ) WHERE c1.CollNo = X AND s.StudentNo = Y ; |table | type | key | key_len | rows | filtered | | c1 | const | PRIMARY | 4 | 1 | | | s | const | PRIMARY | 14 | 1 | | | e | ref | idx_EnrollStud | 4 | 3 | | | c2 | eq_ref | PRIMARY | 17 | 1 | |

ScaleDB Option #1: Conventional Joins Col_ID# Col_Name Col_Budget Col_Description Colleges 001Agriculture$1,234,567Nice place to visit 002Arts$5,432,567Sports not so good 003Business$9,999,666Cool logo 004Education$3,234,567Ugh Worcester 005Engineering$8,238,568Serious work 006Law$7,237,767Jumpy students 007Liberal Arts$9,898,777Pretty campus 008Medicine$5,987,004In Texas Students Mike HoganCaucasian Moshe SmithCaucasian Sally ShadmonNative American Billy FleegleAfrican American Saul GoodeAfrican American Tim CollinsPolynesian Sam GeeAsian Rod PaulinoAsian Enrollment B C B A B C F D Coll_ID#Coll_NameColl_BudgetColl_Description Student_ID# College_ID# Student_Name Student_Desc College_ID# Dept_ID#Student_ID# Grade Colleges Index(s)Students Index(s)Enrollment Index(s) Query Result: 008 Medicine $5,987,004 In Texas | Saul Goode African American | 4455 B+ | Join

ScaleDB Col_ID# Col_Name Col_Budget Col_Description Materialized View Coll_ID#Coll_NameColl_BudgetColl_Description Student_ID# Student_Name Student_Desc Dept_ID# Grade Col_ID# Col_Name Col_Budget Col_Description ………… 001Agriculture$1,234,567Nice place to visit Mike Hogan Caucasian 3345A 001Agriculture$1,234,567Nice place to visit Mike Hogan Caucasian 3235B+ 001Agriculture$1,234,567Nice place to visit Mike Hogan Caucasian 3245A- 001Agriculture$1,234,567Nice place to visit Mike Hogan Caucasian 3245B 001Agriculture$1,234,567Nice place to visit Mike Hogan Caucasian 3235A+ 001Agriculture$1,234,567Nice place to visit Paul Martyn Caucasian 3239A- 001Agriculture$1,234,567Nice place to visit Paul Martyn Caucasian 3239B 001Agriculture$1,234,567Nice place to visit Paul Martyn Caucasian 3240A+ 008Medicine$5,987,004In Texas Saul Goode African American 4455A 008 Medicine $ 5,987,004 In Texas Saul Goode African American 4455A 008 Medicine $ 5,987,004 In Texas Saul Goode African American 4455B+ 008 Medicine $ 5,987,004 In Texas Saul Goode African American 4455A- 008 Medicine $ 5,987,004 In Texas Saul Goode African American 4455B 008 Medicine $ 5,987,004 In Texas Paul Martyn Caucasian 4454A- 008 Medicine $ 5,987,004 In Texas Paul Martyn Caucasian 4454B 008 Medicine $ 5,987,004 In Texas Paul Martyn Caucasian 4454A+ Enrollment B C B A B C F D Col_ID# Col_Name Col_Budget Col_Description Colleges 001Agriculture$1,234,567Nice place to visit 002Arts$5,432,567Sports not so good 003Business$9,999,666Cool logo 004Education$3,234,567Ugh Worcester 005Engineering$8,238,568Serious work 006Law$7,237,767Jumpy students 007Liberal Arts$9,898,777Pretty campus 008Medicine$5,987,004In Texas Students Mike HoganCaucasian Moshe SmithCaucasian Sally ShadmonNative American Billy FleegleAfrican American Saul GoodeAfrican American Tim CollinsPolynesian Sam GeeAsian Rod PaulinoAsian Coll_ID#Coll_NameColl_BudgetColl_Description Dept_ID#Dept_NameColl_ID#Dept_Budget Course_ID#Course_NameColl_ID#Dept_ID# Copies (and synchronizes) the data from individual tables into one massive view Option #2: Materialized View Query Result: 008 Medicine $5,987,004 In Texas | Saul Goode African American | 4455 B+ | Materialized View Indexes

ScaleDB Col_ID# Col_Name Col_Budget Col_Description Colleges 001Agriculture$1,234,567Nice place to visit 002Arts$5,432,567Sports not so good 003Business$9,999,666Cool logo 004Education$3,234,567Ugh Worcester 005Engineering$8,238,568Serious work 006Law$7,237,767Jumpy students 007Liberal Arts$9,898,777Pretty campus 008Medicine$5,987,004In Texas Students Mike HoganCaucasian Moshe SmithCaucasian Sally ShadmonNative American Billy FleegleAfrican American Saul GoodeAfrican American Tim CollinsPolynesian Sam GeeAsian Rod PaulinoAsian Enrollment B C B A B C F D Coll_ID#Coll_NameColl_BudgetColl_Description Student_ID# College_ID# Student_Name Student_Desc College_ID# Dept_ID#Student_ID# Grade Option #3: ScaleDB Query Result: 008 Medicine $5,987,004 In Texas | Saul Goode African American | 4455 B+ | College Students Enrollment ScaleDB’s multi-table index is relationship-aware A Single Index Lookup Departments Courses ScaleDB Index Enrollment

ScaleDB Building Relationships in ScaleDB College Create College Departments Create Department - foreign key: College Courses Create Course - foreign key: Department Students Create Students - foreign key: College Enrollment Create Enrollment - foreign key: Students Relationship creation is automated

ScaleDB Pros & Cons of Each Method Ease of Implementation Real-Time Data PerformanceTuning Conventional Joins Materialized Views ScaleDB

ScaleDB Performance Variables Early performance benchmarks Used a vanilla scenario Our performance advantage increases with: Query/Schema Complexity Referential Integrity Checks Key Size Data Size/Number of Keys Performance Advantage: 2X – 20X+

ScaleDB MySQL Integration ScaleDB leverages its index to assemble data across tables without step-wise joins MySQL query optimizer sees multiple tables, so it calls for step-wise joins ScaleDB tells the query optimizer about joined tables, they are virtualized (built on the fly) When MySQL’s query optimizer recognizes ScaleDB, phantom tables will be removed

Improving Scalability

ScaleDB The Challenges of Scaling How do I partition data? Predict usage patterns, application evolution, data growth patterns…all are moving targets Avoid data skew: bottlenecks caused by frequently accessed data on just a few nodes Data shipping between nodes (2-phase commit) Searches outside the partition column require participation by all nodes Scaling becomes an exercise in fire fighting

ScaleDB ScaleDB’s Plug-and-Cluster™ Cluster-ready solution, just plug in a server No need to partition the data Based on shared-everything architecture Found in the highest-end commercial databases Eliminates all of the data partitioning problems

ScaleDB Local Lock Manager Shared Storage ScaleDB Cluster Local Lock Manager

ScaleDB ScaleDB Cluster Shared Storage Global Lock Manager

ScaleDB Demo

ScaleDB In a nutshell… MySQL MySQL + ScaleDB

ScaleDB Summary Revolutionary indexing solution delivers a quantum leap in performance & scalability Results: Performance improvements of 2X and up 7X smaller index size (average) Stop jumping through hoops to avoid joins…FREE JOINS! Enables more complex applications, fresh data, lower TCO, superior scalability & performance We’re looking for appropriate beta testers