M.Kersten 2008 1 MonetDB, a Column-Store in Midflight Martin Kersten CWI Amsterdam.

M.Kersten 2008 1 MonetDB, a Column-Store in Midflight Martin Kersten CWI Amsterdam

M.Kersten 2008 2 Database view of the world 25 years ago… Design and Analysis of a Relational Join Operation for VLSI The "Group By" Operation in Relational Algebra Implementing Unknown and Imprecise Values in Databases Finding an Optimal Search Sequence of Files. Modelling Events a Data Base Application Design Deferring Updates in a Relational Data Base System

M.Kersten 2008 3 Where is the field heading to Mike Stonebraker, VLDB 2007: One size fits all: A concept whose time has come and gone Martin Kersten, ICDE 2008: Mike is wrong….. Every size fits him always… Even a Euro solution.

M.Kersten 2008 4 If you want to play with a generic column-store, we recommend you download the academic version of Vertica, the commercialization of the C-Store project, or download MonetDB, an open-source column-store.

M.Kersten 2008 5 Paste Present Potency Cracking Columns Chaos

M.Kersten 2008 6 Paste Present Potency PAX stores N-ary stores Column stores

M.Kersten 2008 7 John32Houston OK Early 80s: tuple storage structures for PCs were simple Mary31Houston OK Easy to access at the cost of wasted space Try to keep things simple

M.Kersten 2008 8 Slotted pages Logical pages equated physical pages 32John Houston 31 Mary Houston Try to keep things simple

M.Kersten 2008 9 Slotted pages Logical pages equated multiple physical pages 32John Houston 31 Mary Houston Try to keep things simple

M.Kersten 2008 10 Not all attributes are equally important Avoid things you don’t always need

M.Kersten 2008 11 A column orientation is as simple and acts like an array Attributes of a tuple are correlated by offset Avoid moving too much around

M.Kersten 2008 12 MonetDB Binary Association Tables Try to keep things simple

M.Kersten 2008 13 Physical data organization Binary Association Tables Bat Unit fixed size Dense sequence Memory mapped files Try to avoid doing things twice

M.Kersten 2008 14 Binary Association Tables accelerators Hash-based access Try to avoid doing things twice Column properties: key-ness non-null dense ordered

M.Kersten 2008 15 Binary Association Tables storage control A BAT can be used as an encoding table A VID datatype can be used to represent dense enumerations Type remappings are used to squeeze space 100 Try to avoid doing things twice

M.Kersten 2008 16 Column orientation benefits datawarehousing Brings a much tighter packaging and improves transport through the memory hierarchy Each column can be more easily optimized for storage using compression schemes Each column can be replicated for read-only access Mantra: Try to keep things simple

M.Kersten 2008 17 Try to maximize performance Paste Present Potency Materialize All Model Vectorized model Volcano model

M.Kersten 2008 18 Volcano Refresher Query SELECT name, salary*.19 AS tax FROM employee WHERE age > 25 Try to maximize performance

M.Kersten 2008 19 Volcano Refresher Operators Iterator interface -open() -next(): tuple -close() Try to maximize performance

M.Kersten 2008 20 The Volcano model is based on a simple pull- based iterator model for programming relational operators. The Volcano model minimizes the amount of intermediate store The Volcano model is CPU intensive and inefficient Try to maximize performance Volcano paradigm

M.Kersten 2008 21 MonetDB paradigm The MonetDB kernel is a programmable relational algebra machine Relational operators operate on ‘array’-like structures Based on experiences in database machines 25 years ago, RAP, CASSM,…ICL RAP… PRISMA… IDIOMS (J. Kerridge)… Try to use simple a software pattern

M.Kersten 2008 22 SQL MonetDB Server MonetDB Kernel XQuery MAL function user.s3_1():void; X1:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",0); X6:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",1); X9:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",2); X13:bat[:oid,:oid] := sql.bind_dbat("sys","photoobjall",1); X8 := algebra.kunion(X1,X6); X11 := algebra.kdifference(X8,X9); X12 := algebra.kunion(X11,X9); X14 := bat.reverse(X13); X15 := algebra.kdifference(X12,X14); X16 := calc.oid(0@0); X18 := algebra.markT(X15,X16); X19 := bat.reverse(X18); X20 := aggr.count(X19); sql.exportValue(1,"sys.","count_","int",32,0,6,X20,""); end s3_1; select count(*) from photoobjall; Try to use simple a software pattern

M.Kersten 2008 23 Operator implementation All algebraic operators materialize their result Local optimization decisions Heavy use of code expansion to reduce cost 55 selection routines 149 unary operations 335 join/group operations 134 multi-join operations 72 aggregate operations Try to use simple a software pattern

M.Kersten 2008 24 DBtapestry tables The first column is an ordered sequence All other columns are permutations Organize them as a column store A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 0 0 0 0 2 0 4 1 0 0 1 7 4 4 8 3 8 4 3 3 2 5 9 8 0 6 3 7 7 4 3 4 1 3 3 9 2 2 9 1 4 8 8 5 9 4 6 3 8 6 5 6 5 6 5 7 0 9 1 2 6 1 2 9 4 1 9 5 5 8 7 9 6 7 7 5 5 6 2 9 8 2 7 2 6 2 1 8 6 5 9 3 3 1 1 8 7 0 4 7

M.Kersten 2008 25 Micro-benchmark MonetDB/SQL0.34 N44 MySQL25.1 N238 PostgreSQL10.6 N1230 Commercial 139.0 N800 Commercial 217 N150 In milliseconds/10K Fixed cost in ms Keeping the query result in a new table is often too expensive select * into tmp from tapestry where attr1>=0 and attr1 <=@range create table tmp( attr0 int, attr1 int); insert into tmp select * from tapestry where attr1>=0 and attr1 <=@range ;

M.Kersten 2008 26 Multi-column tapestry Experiments ran on Athlon 1.4, Linux commercial MonetDB/SQL #joins ms

M.Kersten 2008 27 A column store should be designed from scratch to benefit from its characteristics Simulation of a column store on top of an n- ary system using the Volcano model does not work

M.Kersten 2008 28 Try to maximize performance Paste Present Potency Execution Paradigm Database Structures Query optimizer

M.Kersten 2008 29 Applications have different characteristics Platforms have different characteristics The actual state of computation is crucial A generic all-encompassing optimizer cost- model does not work Try to avoid the search space trap

M.Kersten 2008 30 SQL MonetDB Server MonetDB Kernel XQuery MAL Operational optimizer: – Exploit everything you know at runtime – Re-organize if necessary Try to disambiguate decisions

M.Kersten 2008 31 SQL MonetDB Server MonetDB Kernel XQuery MAL Strategic optimizer: – Exploit the semantics of the language – Rely on heuristics Operational optimizer: – Exploit everything you know at runtime – Re-organize if necessary Try to disambiguate decisions

M.Kersten 2008 32 SQL MonetDB Server Tactical Optimizer MonetDB Kernel XQuery MAL y1:bat[:oid,:dbl]:= bpm.take("sys_photoobjall_ra"); y2 := bpm.new(:oid,:oid); barrier rs:= bpm.newIterator(y1,A0,A1); t1:= algebra.uselect(rs,A0,A1); bpm.addSegment(y2,t1); redo rs:= bpm.hasMoreElements(y1,A0,A1); exit rs; x1:bat[:oid,:dbl]:= sql.bind("sys","photoobjall","ra",0); x14:= algebra.uselect(x1,A0,A1); Tactical MAL optimizer: – No changes in front-ends and no direct human guidance – Minimal changes in the engine Try to disambiguate decisions

M.Kersten 2008 33 Code Inliner. Constant Expression Evaluator. Accumulator Evaluations. Strength Reduction. Common Term Optimizer. Join Path Optimizer. Ranges Propagation. Operator Cost Reduction. Foreign Key handling. Aggregate Groups. Code Parallizer. Replication Manager. Result Recycler. MAL Compiler. Dynamic Query Scheduler. Memo-based Execution. Vector Execution. Alias Removal. Dead Code Removal. Garbage Collector. Try to disambiguate decisions

M.Kersten 2008 34 Try to maximize performance Paste Present Potency Execution Paradigm Database Structures Query optimizer

M.Kersten 2008 35 Execution paradigms The MonetDB kernel is set up to accommodate different execution engines The MonetDB assembler program is Interpreted in the order presented Interpreted in a dataflow driven manner Compiled into a C program Vectorised processing X100 project No data from persistent store to the memory trash

M.Kersten 2008 36 MonetDB/x100 Combine Volcano model with vector processing. All vectors together should fit the CPU cache Vectors are compressed Optimizer should tune this, given the query characteristics. ColumnBM (buffer manager) X100 query engine CPU cache networked ColumnBM-s RAM

M.Kersten 2008 37 Varying the vector size on TPC-H query 1 mysql, oracle, db2 X100 MonetDB low IPC, overhead RAM bandwidth bound No data from persistent store to the memory trash

M.Kersten 2008 38 Vectorized-Volcano processing can be used for both multi-core and distributed processing The architecture and the parameters are influenced heavily by Hardware characteristics Data distribution to compress columns No data from persistent store to the memory trash

M.Kersten 2008 39 Does MonetDB stand a ‘real’ test? Is the main memory orientation a bottleneck? Is it functionally complete? The proof of the pudding is in the eating

M.Kersten 2008 40 TPC-H ATHLON X2 3800+ (2000mhz) 2 disks in raid 0, 2G main memory TPC-H 60K rows line_item table Comfortably fit in memory Performance in milliseconds

M.Kersten 2008 41 TPC-H ATHLON X2 3800+ (2000mhz) 2 disks in raid 0, 2G main memory Scale-factor 1 6M row line-item table Out of the box performance Queries produce empty or erroneous results

M.Kersten 2008 42 TPC-H ATHLON X2 3800+ (2000mhz) 2 disks in raid 0, 2G main memory

M.Kersten 2008 43 TPC-H ATHLON X2 3800+ (2000mhz) 2 disks in raid 0, 2G main memory

M.Kersten 2008 44 Code base for MonetDB/SQL is 1.2M lines of C Nightly regression testing on 17 platforms

M.Kersten 2008 45 Try to maximize performance Paste Present Potency Cracking B-tree, Hash Indices Materialized Views

M.Kersten 2008 46 Indices in database systems focus on: All tuples are equally important for fast retrieval There are ample resources to maintain indices MonetDB cracks the database into pieces based on actual query load Find a trusted fortune teller

M.Kersten 2008 Cracking algorithms Physical reorganization happens per column based on selection predicates. Split a piece of a column in two new pieces A<10 A>=10 A<10

M.Kersten 2008 Cracking algorithms Physical reorganization happens per column Split a piece of a column in two new pieces Split a piece of a column in three new pieces A<10 A>=10 A<10 5<A<10 A>=10 5<A<10 A<5

M.Kersten 2008 Cracking example 3 8 6 2 12 13 4 17 15 select A>5 and A<10

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10 <=5

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <=5

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <=5 >5 and <10

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 62 15 13 4 17 12 <=5 >5 and <10

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <=5 >5 and <10

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >5 and <10

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 13 4 17 12 <= 5 >= 10 > 5 15

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5

M.Kersten 2008 racking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 >3 and <14 <=3

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 >3 and <14 <=3

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 <=3

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 10 > 5 <=3

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 14 > 5 <=3 >=10

M.Kersten 2008 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 >3 >= 14 > 5 <=3 >=10 The more we crack the more we learn

M.Kersten 2008 Design The first time a range query is posed on an attribute A, a cracking DBMS makes a copy of column A, called the cracker column of A A cracker column is continuously physically reorganized based on queries that need to touch attribute such as the result is in a contiguous space For each cracker column, there is a cracker index Cracker Index Cracker Column

M.Kersten 2008 A simple range query Try to avoid useless investments

M.Kersten 2008 TPC-H query 6 Try to avoid useless investments

M.Kersten 2008 91 Cracking is easy in a column store and is part of the critical execution path Cracking works under high volume updates Try to avoid useless investments

M.Kersten 2008 Updates Base columns are updated as normally We need to update the cracker column and the cracker index Efficiently Maintain the self-organization properties Two issues: When How

M.Kersten 2008 When to propagate updates in cracking Follow the workload to maintain self-organization Updates become part of query processing When an update arrives, it is not applied For each cracker column there is a pending insertions column and a pending deletions column Pending updates are applied only when a query needs the specific values

M.Kersten 2008 Updates aware select We extended the cracker select operator to apply the needed updates before cracking The select operator: 1. Search the pending insertions column 2. Search the pending deletions column 3. If Steps 1 or 2 find tuples run an update algorithm 4. Search the cracker index 5. Physically reorganize the cracker column 6. Update the cracker index 7. Return a slice of the cracker column

M.Kersten 2008 Merging 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >12 Start position: 1 values: >1 Insert a new tuple with value 9 The new tuple belongs to the blue piece 9

M.Kersten 2008 Merging 7 2 10 29 25 31 57 42 53 Start position: 8 values: >35 Start position: 5 values: >12 Start position: 1 values: >1 Insert a new tuple with value 9 The new tuple belongs to the blue piece 9 Pieces in the cracker column are ordered Tuples inside a piece are not ordered Shifting is not a viable solution

M.Kersten 2008 Merging by Hopping 7 2 10 29 25 31 42 53 Start position: 8 values: >35 Start position: 4 values: >12 Start position: 1 values: >1 57 9 Insert a new tuple with value 9 We need to make enough room to fit the new tuples

M.Kersten 2008 Merge Gradually A query merges only the qualifying values, i.e., only the values that it needs for a correct and complete result Average cost increases significantly We avoid the large peaks but... Merge CompletelyMerge Gradually

M.Kersten 2008 The Ripple Touch only the pieces that are relevant for the current query

M.Kersten 2008 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 Touch only the pieces that are relevant for the current query

M.Kersten 2008 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 Select 7<= A< 15 Touch only the pieces that are relevant for the current query

M.Kersten 2008 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 Select 7<= A< 15 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query

M.Kersten 2008 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Select 7<= A< 15

M.Kersten 2008 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Avoid shifting down non interesting pieces Select 7<= A< 15

M.Kersten 2008 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Immediately make room for the new tuples Avoid shifting down non interesting pieces Select 7<= A< 15

M.Kersten 2008 The Ripple 7 2 10 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions 29 Touch only the pieces that are relevant for the current query Immediately make room for the new tuples Avoid shifting down non interesting pieces Select 7<= A< 15

M.Kersten 2008 The Ripple 7 2 10 25 31 57 42 53 Start position: 7 values: >35 Start position: 5 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions 29 Touch only the pieces that are relevant for the current query Immediately make room for the new tuples Avoid shifting down non interesting pieces Select 7<= A< 15

M.Kersten 2008 The Ripple Maintain high performance through the whole query sequence in a self-organizing way

M.Kersten 2008 The Ripple Maintain high performance through the whole query sequence in a self-organizing way Merge GraduallyMerge Completely Merge Ripple

M.Kersten 2008 116 MonetDB Research & Development line XQuery and Information Retrieval Astronomy databases (SkyServer) Event Stream Engine Relational Cache Effectiveness Multi-core Parallelism DataStorage Rings RDF Engine …. Array Databases become en vogue

M.Kersten 2008 117 Summary MonetDB is a mature column store Cracking based on physical re-organization in the critical path can be made to work There is work for another decade

M.Kersten 2008 118 Cstore SQLserver DB2 PostgreSQL MySQL Whoa MonetDB ! Speed lines !

M.Kersten 2008 119 Martin Kersten Peter Boncz Niels Nes Stefan Manegold Fabian Groffen Sjoerd Mullender Steffen Goeldner Arjen de Vries Menzo Windhouwer Tim Ruhl Romulo Goncalves Jan Rittinger Wouter Alink Jennie Zhang Stratos Idreos Erietta Liarou Lefteris Sidirourgos Florian Waas Albrecht Schmidt Jonas Karlsson Martin van Dinther Peter Bosch Carel van den Berg Wilco Quak Acknowledgements Alex van Ballegooij Johan List Georgina Ramirez Marcin Zukowski Roberto Cornacchia Sandor Heman Torsten Grust Jens Teubner Maurice van Keulen Jan Flokstra Milena Ivanova MonetDB/{SQL,XQuery} open-source platform MonetDB/SQL Version 5 experimentation platform MonetDB/X100 aimed at cpu & IO squeezing MonetDB/RAM aimed at IR and science DB MonetDB/Armada aimed at evolving databases MonetDB/SkyServer portal for Astronomy

M.Kersten 2008 1 MonetDB, a Column-Store in Midflight Martin Kersten CWI Amsterdam.

Similar presentations

Presentation on theme: "M.Kersten 2008 1 MonetDB, a Column-Store in Midflight Martin Kersten CWI Amsterdam."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

M.Kersten 2008 1 MonetDB, a Column-Store in Midflight Martin Kersten CWI Amsterdam.

Similar presentations

Presentation on theme: "M.Kersten 2008 1 MonetDB, a Column-Store in Midflight Martin Kersten CWI Amsterdam."— Presentation transcript:

Similar presentations

About project

Feedback