Importing and Exporting Data with MySQL Presented by: Sheeri K. Cabral At ODTUG Kaleidoscope 2010

Who I Am MySQL DBA MySQL User Group First Oracle ACE Director for MySQL Lots of community stuff (videos, blog, podcast on hiatus)

How Universal import/export Storage engines Batched INSERTs LOAD DATA INFILE

Storage Engines Different ways of storing data Different load speeds How much does the technique matter? What else are you doing? If you have a storage engine that is just plain faster to load your data, it doesn't matter if you use a special technique or not. Also keep in mind: are you only bulk loading? Are you reading the table at the same time, or later? Are there updates? How frequently? Are you doing any transformations? Consider a “logging” table – if it's insert insert insert, there are tables that can optimize that. Is there a slave?

CSV storage engine Just what it sounds like No indexes No NULLs Drop-in table data – sort of CREATE TABLE music ( Name varchar(75) NOT NULL, Artist varchar(75) NOT NULL, Time_Sec SMALLINT UNSIGNED NOT NULL, Yr YEAR(4) NOT NULL, Date_Modified TIMESTAMP, Bit_Rate SMALLINT UNSIGNED NOT NULL, Sample_Rate INT UNSIGNED NOT NULL default 44100) ENGINE=CSV; INSERT INTO music (Name, Artist, Time_Sec, Yr, Date_Modified, Bit_Rate, Sample_Rate) VALUES ('Piano Man','Billy Joel',338, 1973,"2010_03_22 09:57:05",160,44100); mysql -u root -e "SHOW GLOBAL VARIABLES LIKE 'datadir';" exit dir music* mysql -u root test truncate music select * from music; copy "C:\Documents and Settings\cabral\Desktop\music.csv" . -- reload the table flush table music; CSV example: mysql -u root use test; CSV does not support indexes CREATE TABLE music ( Name varchar(75) NOT NULL, Artist varchar(75) NOT NULL, Time_Sec SMALLINT UNSIGNED default NULL, Yr YEAR(4) NOT NULL, Date_Modified TIMESTAMP, Bit_Rate SMALLINT UNSIGNED DEFAULT NULL, Sample_Rate INT UNSIGNED NOT NULL default 44100, INDEX (Artist,Name)) ENGINE=CSV; CSV does not support nulls Sample_Rate INT UNSIGNED NOT NULL default 44100)

MyISAM concurrent_insert=2 Key cache only, no data cache MyISAM – set concurrent_insert=2 for non-blocking data loading with INSERT 1 = concurrent inserts if there's no gaps (fragmentation), 2 = concurrent inserts even if there are gaps. MyISAM can delay index writes for no tables, tables created with the DELAY_KEY_WRITE option, or all tables. http://dev.mysql.com/doc/refman/5.1/en/create-table.html

InnoDB Consider the buffer pool Loads in memory first, then flushed to disk But data is then already in memory But MyISAM has an fs cache InnoDB worst to load because of the buffer pool But once it loads, it's already in memory, so better if you have lots of updates happening at the same time

Replication Topologies Different table type on the slave Loading directly to the slave Slave and master have different data! I won't go into these, but note that they exist

Regular INSERT statement mysql> INSERT INTO tbl (fld1, fld2,...) VALUES (val1,val2,...); Import a file $ mysql < file.sql with a mysql client command mysql> source file.sql

Batched INSERT statement INSERT INTO tbl (fld1, fld2...) VALUES (val1a, val2a...), (val1b, val2b), .... Faster than single inserts “Extended insert” Any storage engine

Other tricks for any storage engine Turn off foreign key checking Lock the table Build the indexes only at the end Other than extended insert.....

Turn Off Foreign Key Checking Only if you are 100% sure your data is good! SET SESSION FOREIGN_KEY_CHECKS=OFF; [load data] SET SESSION FOREIGN_KEY_CHECKS=ON; Does not EVER test loaded data! This can be dangerous

Lock the Table Other writes/reads may interfere with your load LOCK TABLES tbl WRITE; [load data] UNLOCK TABLES; Only do this if you really want to stop everything on this table though!

Build the Indexes at the End Disable keys, load, enable keys ALTER TABLE tbl DISABLE KEYS; [load data] ALTER TABLE tbl ENABLE KEYS; Wait as index builds all at once Extended insert is one trick to help things load faster no matter what the storage engine.

Exporting with mysqldump mysqldump commandline export tool by default: Extended insert (--skip-extended-insert) Disables keys (--skip-disable-keys) Locks the table for writes (--skip-add-locks) Drops the table and re-adds (--no-create-info -- skip-add-drop-table) Sets foreign key checking off Assumptive statements (--complete-insert) --opt is: --add-drop-table, --add-locks, --extended-insert, --disable-keys, --lock-tables, --create-options, --quick, --set-charset mysqldump --skip-opt -u root test music > "c:\Documents and Settings\cabral\music.sql" Edit "c:\Documents and Settings\cabral\music.sql" --skip-extended-insert –skip-disable-keys –skip-add-locks –skip-add-drop-table skip-add-locks Or --skip-opt –create-options –quick –set-charset –no-create-info –complete-insert mysqldump --skip-opt --no-create-info --complete-insert -u root test music > "c:\Documents and Settings\cabral\music.sql"

INSERT DELAYED Online batches with MyISAM, MEMORY, ARCHIVE, BLACKHOLE Delay queue Ignored on slaves http://dev.mysql.com/doc/refman/5.1/en/insert-delayed.html

INSERT DELAYED delayed_insert_limit 100 by default Handlers Terminates after waiting delayed_insert_limit 300 seconds by default delayed_queue_size 1000 by default This is a per-table limit on the number of rows to queue when handling INSERT DELAYED statements. If the queue becomes full, any client that issues an INSERT DELAYED statement waits until there is room in the queue again. http://dev.mysql.com/doc/refman/5.1/en/insert-delayed.html

Delayed Key Write MyISAM only Index buffers not flushed after each write Flushed (keys written) only on table close FLUSH TABLES closes and re-opens Only for speeding up writes In general, not a good idea

DELAY_KEY_WRITE delay_key_write system variable CREATE TABLE … () ENGINE=MYISAM DELAY_KEY_WRITE=1; delay_key_write system variable ON by default, honors table definition OFF is never delay key writes ALL is always delay key writes

LOAD DATA INFILE Table must already exist Table name does not need to relate to file name: mysql> LOAD DATA INFILE '/tmp/importme.sql' INTO TABLE rental; Complement of SELECT INTO OUTFILE More details on future slides...

LOAD DATA INFILE On an empty MyISAM table, non-unique indexes are created in a separate batch Possibly made even faster by ALTER TABLE tbl DISABLE KEYS LOAD DATA INFILE... ALTER TABLE tbl ENABLE KEYS

LOAD DATA INFILE LOAD DATA [LOW_PRIORITY | CONCURRENT] less/no blocking [LOCAL] from client machine INFILE 'file_name' can be full path

LOAD DATA INFILE LOAD DATA INFILE 'file_name' [REPLACE | IGNORE] What to do if there are duplicates Default is error INTO TABLE tbl_name

LOAD DATA INFILE LOAD DATA INFILE 'file_name' INTO TABLE tbl_name [CHARACTER SET charset_name] CREATE TABLE music ( Name varchar(75) NOT NULL, Artist varchar(75) NOT NULL, Time_Sec SMALLINT UNSIGNED NOT NULL, Yr YEAR(4) NOT NULL, Date_Modified TIMESTAMP, Bit_Rate SMALLINT UNSIGNED NOT NULL, Sample_Rate INT UNSIGNED NOT NULL default 44100) ENGINE=MyISAM; LOAD DATA INFILE 'C:\\Documents and Settings\\cabral\\Desktop\\music.csv' INTO TABLE music FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '”';

LOAD DATA INFILE LOAD DATA INFILE 'file_name' INTO TABLE tbl_name [FIELDS [TERMINATED BY 'string'] (comma) [[OPTIONALLY] ENCLOSED BY 'char'] [ESCAPED BY 'char']] [LINES [STARTING BY 'string'] [TERMINATED BY 'string']] (newline) CREATE TABLE music ( Name varchar(75) NOT NULL, Artist varchar(75) NOT NULL, Time_Sec SMALLINT UNSIGNED NOT NULL, Yr YEAR(4) NOT NULL, Date_Modified TIMESTAMP, Bit_Rate SMALLINT UNSIGNED NOT NULL, Sample_Rate INT UNSIGNED NOT NULL default 44100) ENGINE=MyISAM; LOAD DATA INFILE 'C:\\Documents and Settings\\cabral\\Desktop\\music.csv' INTO TABLE music FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '”';

SELECT INTO OUTFILE SELECT * INTO OUTFILE 'file_name' [CHARACTER SET charset_name] [FIELDS [TERMINATED BY 'string'] [[OPTIONALLY] ENCLOSED BY 'char'] [ESCAPED BY 'char']] [LINES [STARTING BY 'string'] [TERMINATED BY 'string']]

SELECT INTO DUMPFILE SELECT * INTO DUMPFILE 'file_name'; No other options Good for BLOBs

Faster inserts/imports Batched inserts Lock tables Turn off foreign key checking Build index at end of import DELAY_KEY_WRITE INSERT DELAYED

Import/export Import LOAD DATA INFILE Different storage engines (ie, CSV) Export mysqldump SELECT … INTO OUTFILE SELECT … INTO DUMPFILE