Instant Add Columns in MySQL Vin Chen Tencent Game

Who I am Tencent Game DBA Team Profiles Contributor to Oracle MySQL Our Team has been serving game business for a few years. The mission of the DBA Team is to provide stable and efficient online storage services for Tencent Games. As the storage requirements from game business evolve, we keep extending features of our database and improving its performance. We spared no effort to provide reliable, scalable and performant database solutions. Profiles Team leader of Tencent DBA Developer Team Senior engineer of Tencent More than 7 years MySQL development Focus on open source storage solutions(MySQL, MongoDB, Redis, etc.) Contributor to Oracle MySQL Author of instant add columns

Background Why we need instant add columns High Frequency of Adding columns to BIG Table in Production Env Expensive operation with Adding columns to BIG table Causing downtime in the Game business “Instant add columns” in Tencent Game First released in July 2012, TenDB Based on mysql 5.5.24, and 5.6.24, 5.7.20 Contribution to Oracle MySQL in 2017 Merged and RELEASED in 8.0.12 dated July 2018

Implementation Row Format Dictionary DDL & DML Redo

Non-NULL Variable-Length Array Row Format Record Format Characteristic (Compact/Dynamic) Storage order is the same as definition order (except primary key) NULL field not stored. A bitmap is used to indicate which fields are NULL. Field number of a record is index->n_fields Non-NULL Variable-Length Array Nullable Bitmap Record Extra Bytes Record Header Primary Key Transaction ID Roll Ptr Other not null Fields Record Ptr n_owned+Info Bits (4Bits + 4Bits, 1 Byte) Status + Heap No (3Bits + 13Bits, 2 Bytes) Next Ptr (2 Bytes)

Row Format Record Format Changes(Compact/Dynamic) Field Count: the field count of the record(≤ index.n_fields) Forward compatibility Info Bits(higher 4 bits, 2 reserved bits) REC_INFO_MIN_REC_FLAG 0x10 REC_INFO_DELETED_FLAG 0x20 REC_INFO_INSTANT_FLAG 0x80 Only leaf pages of cluster index are affected Record Header Primary Key Transaction ID Roll Ptr Other not null Fields /* The 0x40UL can also be used in the future */ /* The instant ADD COLUMN flag. When it is set to 1, it means this record was inserted/updated after an instant ADD COLUMN. */ Record Ptr Non-NULL Variable-Length Array Nullable Bitmap Record Extra Bytes Non-NULL Variable-Length Array Nullable Bitmap Field Count Record Extra Bytes n_owned+Info Bits (4Bits + 4Bits, 1 Byte) Status + Heap No (3Bits + 13Bits, 2 Bytes) Next Ptr (2 Bytes)

Row Format Record Format Changes(Compact/Dynamic) If REC_INFO_INSTANT_FLAG is set, store the “Field count” using a variable-length encoding of 2 bytes REC_INFO_INSTANT_FLAG would be set for insert/update after instant adding columns. If REC_INFO_INSTANT_FLAG is zero, there are two possibilities: Instant adding columns have never happened in this table. The record is inserted or updated before the first time of instant adding columns. n_core_fields: The field count before the first time instant adding columns happened.

Dictionary New metadata Store n_core_fields Store default value When the first time instant adding columns happens, we need to store the original field count of a table Update sys_tables set (mix_len = mix_len | n_core_fields << 16) where table = :table_id; Store default value A new system table sys_columns_added Dictionary loading When loading a table from dictionary n_core_fields = sys_table.mix_len >> 16 cluster_index.n_core_fields = n_core_fields > 0 ? n_core_fields : cluster_index.n_fields. dict_load_sys_columns_added() after dict_load_sys_columns()

Dictionary Examples MySQL 8 TenDB Use test; create table t1(id int primary key, c1 int); insert into t1 values(1,1); alter table t1 add column d1 varchar(20) not null default 'abc‘, algorithm=‘instant’; alter table t1 alter column d1 set default 'bcd'; Examples MySQL 8 TenDB The number of columns when the table undergoes first instant ADD COLUMN and all default value of newly added columns are stored in the data dictionary. These two pieces of information are stored in se_private_data column of the data dictionary tables.

DDL Key Logic Create table: no behavior change Alter table: Instant adding column No need to copy data Insert or change the related metadata Other: no behavior change Drop/Truncate table: Delete the related metadata Update sys_tables set (mix_len = mix_len | n_core_fields << 16) where table = :table_id; (First time) Insert into sys_columns values(…); Insert into sys_columns_added values(:table_id, :pos, <default value>);

DML Key Logic insert: delete: update: Before the first time Instant adding columns happened, no behavior change Otherwise, REC_INFO_INSTANT_FLAG would be set. delete: No behavior change, just mark the delete flag; update: For updating old version record, if the added columns changed or the length of column changed, it will use non-inplace update(delete + insert) Otherwise, no behavior change.

DML Key Logic select:

Redo Redo a mix of physical and logical operations n_core_fields need to be stored in dict object of redo log default values is useless for recovery

Summary Advantage Limitations Change metadata only, very fast No need to copy data No need to take twice the space Forward compatibility with MySQL Limitations Only support adding columns in one statement, that is if there are other non-INSTANT operations in the same statement, it can’t be done instantly Only support adding columns at last, not in the middle of existing columns Not support COMPRESSED row format, which is seldom used Not support a table which already has any fulltext index Not support any table residing in DD tablespace Not support temporary table(it goes with COPY)

Mass production environment deployment MySQL Usage In Tencent Games Load Test Functional Test Grayscale deployment Mass production environment deployment 400+ Games 10,000+ Machines 20,000+ Instances 20+ FailOver Per Day 17 Table DDL Per week All instances use TenDB, which base on Oracle MySQL 5.5/5.6/5.7 Our Plan of Deploying MySQL 8.0

Thank you for your patience! vinchen@tencent.com