MDEV-14992 Cross-Engine SQL Backup Hot Backup for InnoDB, MyRocks, … & cold for others Marko Mäkelä Lead Developer InnoDB February 2018

Cross-engine BACKUP with Restore by Just Copying Files Eliminate external backup tools

Replacing Mariabackup with SQL BACKUP Mariabackup is an InnoDB-only tool that supports the {differential,snapshot}×{backup,restore}×{of}×{some,all} tables. With the workflow in the previous slide, we can almost cover the snapshot backup and restore of some or all InnoDB tables. BACKUP 〈tablenames〉 [FROM 〈time〉] FROM 〈time〉 is for differential backup: apply changes to a backup that was not modified since the last BACKUP that was completed at time. A result will be returned to the client connection. (filename TEXT NOT NULL, offset BIGINT UNSIGNED NOT NULL, data BLOB) A simple mariacopy client will apply the result to local files. No need for separate “restore” tool. Simply start a server on the files.

Short comparison to Mariabackup Cross-engine operation is possible InnoDB, Aria, TokuDB, MyRocks, ColumnStore in a single BACKUP command All data files, as determined by each storage engine No need to FLUSH TABLES WITH READ LOCK except for MyISAM No recovery log (redo log, WAL) is copied for InnoDB No --prepare or --apply-log step is needed A server can quickly be started up inside the backup target directory Backup produces a copy of all files that the specified tables depend on Including any global persistent files or tables, most notably: The state of all persistent transactions mysql.transaction_registry for system-versioned (temporal) tables (“as of”)

Implementing BACKUP without FROM 〈time〉 BACKUP 〈tablenames〉 Makes a consistent copy of all the files of the tables Protected by MDL that prevents DDL, but allows concurrent DML For InnoDB, uses the InnoDB buffer pool for accessing the *.ib* files This will automatically perform a change buffer merge. Pages that happen to be x-latched must be (initially) skipped. (See below.) Pages marked as free can be skipped. (See also FROM 〈time〉 on a later slide.) At the end, a log checkpoint will be triggered. Any pages that enter the flush_list with a LSN not after the checkpoint LSN will be copied again. BEGIN; BACKUP t1; BACKUP t2; COMMIT; will not deliver 1 snapshot! All tables (or *.*) must be specified in a single BACKUP statement.

Cross-Engine Backup Algorithm Based on Details contributed to MDEV-14992 by Yuan Zhang / Alibaba Cloud Prepare For InnoDB, switch to a mode where we remember future page changes. For TokuDB and MyRocks, disable log checkpoints Copying (could be done on multiple engines in parallel) InnoDB: Copy via the buffer pool all non-freed data pages of persistent data files. TokuDB, MyRocks: Copy the log files Lock tables & cold backup from other storage engines Finish: InnoDB: Re-copy the pages that were dirtied since the copying; reset the mode Unlock tables TokuDB, MyRocks: Copy data files, re-enable log checkpoints

Implementing BACKUP with FROM 〈time〉 BACKUP 〈tablenames〉 FROM 〈time〉 Must know which pages changed since the specified logical time Could fall back to ignoring time (snapshot BACKUP) if the information is missing. Skips pages marked as free, and also pages that were not changed since time. time should be something global, to allow cross-engine BACKUP. Cannot use the InnoDB LSN, if other storage engines are to support time. Use the binlog offset? or some identifier from sys.transactions? Not unique if InnoDB did purge or other background tasks. Might be good enough. What if we had a global recovery log for all crash-safe engines? time would become the global recovery log position. This would also give us cross-engine COMMIT or XA with a single fsync().

Partial Backup & Restore Partial backup is like full backup, but excluding the files for some tables Partial restore would ignore global state. Importing InnoDB tables into an existing server: Any pending transactions for the tables must be rolled back and purged. TO BE DEFINED: What to do WITH SYSTEM VERSIONING (AS OF)? An intermediate server can be used to manipulate the global state: Individually XA COMMIT some XA PREPARE transactions Choose to discard WITH SYSTEM VERSIONING for some tables? Maybe run some ALTER TABLE to drop partitions etc? Export the tables to the final target server, losing GTID and XID

EOF ^Z :wq :q! ␄