1. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. MySQL Conference & Expo 2011 Michael “Monty” Widenius monty@askmonty.org Oleksandr “Sanja” Byelkin sanja@askmonty.org MariaDB Dynamic Columns

2. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. RDBMS doesn't solve all common problems The (web) store problem: All items need: ID, Type, Price, Country, Manufacturer) A T-Shirt has the following additional properties: Size, color... A computer has the following additional properties: CPU, MHz, memory, Watt... There is no easy way to store many different types into a relational database!

3. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. RDBMS doesn't solve all common problems One common solutions to this is: ● Store all the 'extra' columns in a BLOB in some format (HTML?) ● You need a lot of extra work to manipulate the blob ● Hard to access column data (usually done in client) ● Overhead in storage (especially when you use HTML) ● All values are 'text'

4. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. RDBMS doesn't solve all common problems Another common solution: ● Create one table for all the 'extra' columns: CREATE TABLE extra (id int auto_increment, extra_column_id char(10), value varchar(255)); INSERT INTO items set type=“t-shirt”, price=10; INSERT INTO extra (NULL, LAST_INSERT_ID(), “color”, “Blue”),(NULL, LAST_INSERT_ID(), “Size”, “M”); The problems with this approach is: ● Every access to an extra column requires a key/row lookup ● Slow performance (if database is not optimized for this) ● Big overhead in storage (especially in index) ● Risk for errors as data is not typed

5. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns Dynamic columns is a bridge between relational databases and non relational databases ● With dynamic columns all extra columns are stored in a packed blob, maintained by the database. ● You can add more columns, remove or query them for a row. ● You can access columns in the server or retrieve the full blob to the client and manipulate it there. ● You can use virtual columns to create indexes on some values. ● True indexes for dynamic columns is planned for later. ● Implemented through functions for use by ODBC, & etc. ● First implementation uses integer to access columns.

6. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: supported types ● unsigned int ● int ● char [character set ] ● double ● decimal [(, )] ● time ● date ● datetime

7. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: syntax Creating a table with a dynamic column for the store: CREATE TABLE item ( ID int auto_increment primary key, Type_id int, Price decimal(7,2), Country_id int, Manufacturer_id int, extra blob); Where column 'extra' is dedicated to store dynamic columns. It could be any column able carry text.

8. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: syntax Creating/initializing a dynamic_column: COLUMN_CREATE(column_nr, value [as type], [column_nr, value [as type]],...) INSERT into item values (NULL, 1 /* T-shirt */, 10, 1 /* Germany */, 1 /* Nike */, COLUMN_CREATE(1 /* color */, "Blue", 2 /* Size */, "M")); INSERT into item values (NULL, 2 /* computer */, 1000, 1 /* Germany */, 2 /* intel */, COLUMN_CREATE(3 /* cpu */, "T9400", 5 /* MHz */, 800 as unsigned int)); The /*..*/ is just there to make the example clearer.

9. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: syntax Updating a dynamic column: COLUMN_ADD(string, column_nr, value [as type], column_nr, value [as type]]...) UPDATE item SET extra= COLUMN_ADD(extra, 6 /* Memory */, 2048) WHERE id=2; If the column already exists, it will be overwritten.

10. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: syntax Deleting a dynamic column (if it exists): COLUMN_DELETE(string, column_nr1, column_nr2,...); UPDATE item SET extra= COLUMN_DELETE(extra, 6) WHERE id=2;

11. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: syntax Querying a dynamic column: COLUMN_EXISTS(string, column_nr); SELECT ID, Type_id, Price, Country_id, Manufacturer_id from item where COLUMN_EXISTS(extra, 3); Querying which columns exist: COLUMN_LIST(string); SELECT COLUMN_LIST(extra) FROM item WHERE id=1; → “1,2”

12. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: syntax Retrieving a dynamic column: COLUMN_GET(column_nr, string as type); SELECT id, COLUMN_GET( 1 /* color*/, extra as char) from item; → 1 Blue → 2 NULL You can of course also do things like: SELECT id, COLUMN_GET(1, extra as char) FROM item where Type_id=1 order by COLUMN_GET(1, extra as char);

13. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: C library C Library allows the same manipulations with dynamic columns on the client side. A description can be found in the worklog http://askmonty.org/worklog/Server-Sprint/?tid=34 and in the source. http://askmonty.org/worklog/Server-Sprint/?tid=34

14. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: encoding How is the dynamic column encoded? Header: Sorted index: Each column is stored as: Where 'offset' is offset from beginning of data part, 'type' is 3 bits in offset. Length of the data could be calculated by offsets of 2 neighbor fields.

15. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: data encoding ● Unsigned integer is just a variable integer field. ● Signed integer coded to make the variable size efficient: ● 0→0 ● -1→1 ● 1→2 ● -2→3 ● 2→4 ●... ● Double, date, time, and date time are fixed-size fields ● String stores collation number and the string ● Decimal stores sizes of parts before and after decimal point and decimal in MySQL format. ● NULL – means removing the field.

16. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: avaliability When will dynamic columns be available? First version is already pushed in separate tree for testing. lp:~maria-captains/maria/5.3-mwl34 Should be available in main MariaDB 5.3 within several weeks.

17. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Dynamic columns: plans ● Adding name directory ● Adding functional indices ● Supporting popular NoSQL data exchange formats (for example, JSON or XML) ● Adding engine support for NoSQL databases as HBase

18. Notice: MySQL is a registered trademark of Sun Microsystems, Inc. Thanks Q & A