1. Importing and Exporting Data with MySQL
Presented by:
Sheeri K. Cabral
At ODTUG Kaleidoscope 2010

2. Who I Am MySQL DBA MySQL User Group
First Oracle ACE Director for MySQL
Lots of community stuff (videos, blog, podcast on hiatus)

3. How Universal import/export Storage engines Batched INSERTs
LOAD DATA INFILE

4. Storage Engines Different ways of storing data Different load speeds
How much does the technique matter?
What else are you doing?
If you have a storage engine that is just plain faster to load your data, it doesn't matter if you use a special technique or not.
Also keep in mind:
are you only bulk loading?
Are you reading the table at the same time, or later?
Are there updates? How frequently?
Are you doing any transformations?
Consider a “logging” table – if it's insert insert insert, there are tables that can optimize that.
Is there a slave?

5. CSV storage engine Just what it sounds like No indexes No NULLs
Drop-in table data – sort of
CREATE TABLE music (
Name varchar(75) NOT NULL,
Artist varchar(75) NOT NULL,
Time_Sec SMALLINT UNSIGNED NOT NULL,
Yr YEAR(4) NOT NULL,
Date_Modified TIMESTAMP,
Bit_Rate SMALLINT UNSIGNED NOT NULL,
Sample_Rate INT UNSIGNED NOT NULL default 44100)
ENGINE=CSV;
INSERT INTO music (Name, Artist, Time_Sec, Yr, Date_Modified, Bit_Rate, Sample_Rate)
VALUES ('Piano Man','Billy Joel',338, 1973,"2010_03_22 09:57:05",160,44100);
mysql -u root -e "SHOW GLOBAL VARIABLES LIKE 'datadir';"
exit
dir music*
mysql -u root test
truncate music
select * from music;
copy "C:\Documents and Settings\cabral\Desktop\music.csv" .
-- reload the table
flush table music;
CSV example:
mysql -u root
use test;
CSV does not support indexes
CREATE TABLE music (
Name varchar(75) NOT NULL,
Artist varchar(75) NOT NULL,
Time_Sec SMALLINT UNSIGNED default NULL,
Yr YEAR(4) NOT NULL,
Date_Modified TIMESTAMP,
Bit_Rate SMALLINT UNSIGNED DEFAULT NULL,
Sample_Rate INT UNSIGNED NOT NULL default 44100,
INDEX (Artist,Name))
ENGINE=CSV;
CSV does not support nulls
Sample_Rate INT UNSIGNED NOT NULL default 44100)

6. MyISAM concurrent_insert=2 Key cache only, no data cache
MyISAM – set concurrent_insert=2 for non-blocking data loading with INSERT
1 = concurrent inserts if there's no gaps (fragmentation), 2 = concurrent inserts even if there are gaps.
MyISAM can delay index writes for no tables, tables created with the DELAY_KEY_WRITE option, or all tables.

7. InnoDB Consider the buffer pool
Loads in memory first, then flushed to disk
But data is then already in memory
But MyISAM has an fs cache
InnoDB worst to load because of the buffer pool
But once it loads, it's already in memory, so better if you have lots of updates happening at the same time

8. Replication Topologies
Different table type on the slave
Loading directly to the slave
Slave and master have different data!
I won't go into these, but note that they exist

9. Regular INSERT statement
mysql> INSERT INTO tbl (fld1, fld2,...)
VALUES (val1,val2,...);
Import a file
$ mysql < file.sql
with a mysql client command
mysql> source file.sql

10. Batched INSERT statement
INSERT INTO tbl (fld1, fld2...)
VALUES (val1a, val2a...),
(val1b, val2b), ....
Faster than single inserts
“Extended insert”
Any storage engine

11. Other tricks for any storage engine
Turn off foreign key checking
Lock the table
Build the indexes only at the end
Other than extended insert.....

12. Turn Off Foreign Key Checking
Only if you are 100% sure your data is good!
SET SESSION FOREIGN_KEY_CHECKS=OFF;
[load data]
SET SESSION FOREIGN_KEY_CHECKS=ON;
Does not EVER test loaded data!
This can be dangerous

13. Lock the Table Other writes/reads may interfere with your load
LOCK TABLES tbl WRITE;
[load data]
UNLOCK TABLES;
Only do this if you really want to stop everything on this table though!

14. Build the Indexes at the End
Disable keys, load, enable keys
ALTER TABLE tbl DISABLE KEYS;
[load data]
ALTER TABLE tbl ENABLE KEYS;
Wait as index builds all at once
Extended insert is one trick to help things load faster no matter what the storage engine.

15. Exporting with mysqldump
mysqldump commandline export tool by default:
Extended insert (--skip-extended-insert)
Disables keys (--skip-disable-keys)
Locks the table for writes (--skip-add-locks)
Drops the table and re-adds (--no-create-info -- skip-add-drop-table)
Sets foreign key checking off
Assumptive statements (--complete-insert)
--opt is:
--add-drop-table, --add-locks, --extended-insert,
--disable-keys, --lock-tables,
--create-options, --quick, --set-charset
mysqldump --skip-opt -u root test music > "c:\Documents and Settings\cabral\music.sql"
Edit "c:\Documents and Settings\cabral\music.sql"
--skip-extended-insert –skip-disable-keys –skip-add-locks –skip-add-drop-table skip-add-locks Or
--skip-opt –create-options –quick –set-charset –no-create-info –complete-insert
mysqldump --skip-opt --no-create-info --complete-insert -u root test music > "c:\Documents and Settings\cabral\music.sql"

16. INSERT DELAYED Online batches with MyISAM, MEMORY, ARCHIVE, BLACKHOLE
Delay queue
Ignored on slaves

17. INSERT DELAYED delayed_insert_limit 100 by default Handlers
Terminates after waiting delayed_insert_limit
300 seconds by default
delayed_queue_size
1000 by default
This is a per-table limit on the number of rows to queue when handling INSERT DELAYED statements. If the queue becomes full, any client that issues an INSERT DELAYED statement waits until there is room in the queue again.

18. Delayed Key Write MyISAM only
Index buffers not flushed after each write
Flushed (keys written) only on table close
FLUSH TABLES closes and re-opens
Only for speeding up writes
In general, not a good idea

19. DELAY_KEY_WRITE delay_key_write system variable
CREATE TABLE … () ENGINE=MYISAM
DELAY_KEY_WRITE=1;
delay_key_write system variable
ON by default, honors table definition
OFF is never delay key writes
ALL is always delay key writes

20. LOAD DATA INFILE Table must already exist
Table name does not need to relate to file name:
mysql> LOAD DATA INFILE '/tmp/importme.sql' INTO TABLE rental;
Complement of SELECT INTO OUTFILE
More details on future slides...

21. LOAD DATA INFILE
On an empty MyISAM table, non-unique indexes are created in a separate batch
Possibly made even faster by
ALTER TABLE tbl DISABLE KEYS
LOAD DATA INFILE...
ALTER TABLE tbl ENABLE KEYS

22. LOAD DATA INFILE LOAD DATA [LOW_PRIORITY | CONCURRENT]
less/no blocking
[LOCAL]
from client machine
INFILE 'file_name'
can be full path

23. LOAD DATA INFILE LOAD DATA INFILE 'file_name' [REPLACE | IGNORE]
What to do if there are duplicates
Default is error
INTO TABLE tbl_name

24. LOAD DATA INFILE LOAD DATA INFILE 'file_name' INTO TABLE tbl_name
[CHARACTER SET charset_name]
CREATE TABLE music (
Name varchar(75) NOT NULL,
Artist varchar(75) NOT NULL,
Time_Sec SMALLINT UNSIGNED NOT NULL,
Yr YEAR(4) NOT NULL,
Date_Modified TIMESTAMP,
Bit_Rate SMALLINT UNSIGNED NOT NULL,
Sample_Rate INT UNSIGNED NOT NULL default 44100)
ENGINE=MyISAM;
LOAD DATA INFILE 'C:\\Documents and Settings\\cabral\\Desktop\\music.csv' INTO TABLE music FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '”';

25. LOAD DATA INFILE LOAD DATA INFILE 'file_name' INTO TABLE tbl_name
[FIELDS
[TERMINATED BY 'string'] (comma)
[[OPTIONALLY] ENCLOSED BY 'char']
[ESCAPED BY 'char']]
[LINES
[STARTING BY 'string']
[TERMINATED BY 'string']] (newline)
CREATE TABLE music (
Name varchar(75) NOT NULL,
Artist varchar(75) NOT NULL,
Time_Sec SMALLINT UNSIGNED NOT NULL,
Yr YEAR(4) NOT NULL,
Date_Modified TIMESTAMP,
Bit_Rate SMALLINT UNSIGNED NOT NULL,
Sample_Rate INT UNSIGNED NOT NULL default 44100)
ENGINE=MyISAM;
LOAD DATA INFILE 'C:\\Documents and Settings\\cabral\\Desktop\\music.csv' INTO TABLE music FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '”';

26. SELECT INTO OUTFILE SELECT * INTO OUTFILE 'file_name'
[CHARACTER SET charset_name]
[FIELDS
[TERMINATED BY 'string']
[[OPTIONALLY] ENCLOSED BY 'char']
[ESCAPED BY 'char']]
[LINES
[STARTING BY 'string']
[TERMINATED BY 'string']]

27. SELECT INTO DUMPFILE SELECT * INTO DUMPFILE 'file_name';
No other options
Good for BLOBs

28. Faster inserts/imports
Batched inserts
Lock tables
Turn off foreign key checking
Build index at end of import
DELAY_KEY_WRITE
INSERT DELAYED

29. Import/export Import LOAD DATA INFILE
Different storage engines (ie, CSV)
Export
mysqldump
SELECT … INTO OUTFILE
SELECT … INTO DUMPFILE