1. School of Computing and Management Sciences © Sheffield Hallam University The need to move data around is not new, and will be with us while ever there are different DBMS –Internally, between systems: Disparate information systems to one EIS Transactional to Warehouse Upgrading legacy systems –Externally, between organisations –Delivering data to clients with browsers Moving Data

2. Other data stores Electronic Data Interchange W3C Standard compliant Common RDBMS Single database Create Table B AS Select * from Table A Processing Degree of task complexity Import/Export.dmp.xml XML file conversion.edi EDI message conversion.csv.txt Flat files of either variable or fixed length Line-by-line processing

3. School of Computing and Management Sciences © Sheffield Hallam University Between applications using same DBMS: –internal DBMS processes can include: Import/Export - caution, this means different things to different manufacturers! Bulk/Batch Copy/Move Backup and Restore Create …..as select * from….. Oracle 10g onwards has DataPump –OS level is possible for some (cruder?) DBMS Moving Data

4. School of Computing and Management Sciences © Sheffield Hallam University Between different environments: –Enable the internal database functionality of target by making two systems logically the same: Linking DBMS by internal drivers Linking DBMS through external drivers (eg ODBC) –Dump from source into readable file and then import to target Moving Data

5. School of Computing and Management Sciences © Sheffield Hallam University When dealing with actual data it is all too easy to forget that each row is potentially as important as the others. Our task is not only to move the data, but to move it accurately. Indeed, in some circumstances the data we are provided with is itself unclean. We therefore need to ensure we understand the data we are dealing with and that when it completes its journey it is a valid and exact representation of the information being stored. Moving Data

6. School of Computing and Management Sciences © Sheffield Hallam University Variable length files –Need to identify where fields and records terminate –Probably the most common is the CSV Comma Separated Variable most dbms can at least dump to csv file –EG: External File formats Arnold,Band Music,Dallas Wind Symphony,Junkin,2 Arnold,Film Music,Moscow SO,Stromberg,1 Arnold,Symphonies 5 and 6,Ireland NSO,Penny,1 Arnold,Symphonies 7 and 8,Ireland NSO,Penny,1

7. School of Computing and Management Sciences © Sheffield Hallam University Watch out for the CSV Gotchas –Gotcha No.1 - the dreaded apostrophe! If we had chosen to enclose a string with single quotes, this record would probably fall over when we tried to insert it as the record would read: –'Bartok','Duke Bluebeard's Castle','NYP','Kubelik',1 –Gotcha No.2 - the non-separating comma. Antheil,"Jazz Symphony, Ballet Mechanique, etc.",Orch,Peress,1 External File formats

8. School of Computing and Management Sciences © Sheffield Hallam University Fixed length files –Each data field occurs at a set location in a record, or at a precise offset from position 1 Watch out for record terminators! External File formats

9. School of Computing and Management Sciences © Sheffield Hallam University Three key command line utilities: –SQLLOADER –IMPort –EXPort These are installed in the Oracle_Home\Bin directory on the server. –This causes a problem for using these tools when logging on from a client with restricted privileges, as you do on shu10g. –Solution : Ora10g Utilities from the start button Oracle tools for external data

10. School of Computing and Management Sciences © Sheffield Hallam University SQLLDR.exe Pass it a series of parameters: –Username/password@instance –Where to find a control file –Optionally: What to do with discarded rows What to do with invalid rows S:\Oracle\Client10g\BIN\SQLLDR userid=peter/fleur@shu10g control="f:\advdb\movingdata\composer.con" Oracle SQLLOADER

11. School of Computing and Management Sciences © Sheffield Hallam University The key item is the control file. It allows Oracle to know: –How to map file data to columns and tables in your schema –What you want to do with the data (insert, update) –How to react to errors –Optionally control the incoming data by setting validation tests –Optionally manipulate the incoming data Sequences decode Oracle SQLLOADER

12. School of Computing and Management Sciences © Sheffield Hallam University Sample Control file OPTIONS (ERRORS=999) LOAD DATA INFILE “f:\advdb\movingdata\CDList.csv" BADFILE “f:\advdb\movingdata\CDList.bad" INSERT INTO TABLE COMPOSERS FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ( COMPOSERID SEQUENCE(MAX,1), COMPOSER) Oracle SQLLOADER

13. School of Computing and Management Sciences © Sheffield Hallam University Generates logical records from the infile You can then use either –Conventional load (default) Actually generates an INSERT sql statement for each logical row Will therefore process as normal things like consistency checks and triggers –Direct Load Allow you to insert directly to the segment Much faster, but more DANGEROUS! Oracle SQLLOADER

14. School of Computing and Management Sciences © Sheffield Hallam University Exp.exe –Outputs to a dump file, traditionally given extension of.DMP –Can output database, schema or individual tables –Is a command line utility –Can be run by just typing EXP and then answering the questions –Or you can pass parameters, similar to sqlldr: S:\Oracle\Client10g\BIN\EXP peter/fleur@shu10g PARFILE="f:\advdb\movingdata\expparams.dat“ –The parfile (parameter file) is like a control file Oracle EXPort

15. School of Computing and Management Sciences © Sheffield Hallam University Sample Parfile contents: FILE="f:\advdb\movingdata\composers.dmp" TABLES=(COMPOSERS) GRANTS=y INDEXES=y ROWS=y TRIGGERS=y Oracle EXPort

16. School of Computing and Management Sciences © Sheffield Hallam University Imp.exe –Data must have been generated by an Oracle Export (.dmp) as uses proprietary compression –Is a command line utility –Can be run by just typing IMP and then answering the questions –Or you can pass parameters, similar to sqlldr: S:\Oracle\Client10g\BIN\IMP peter/fleur@shu10g PARFILE="f:\advdb\movingdata\expparams.dat“ –Can be selective on what you import, for example using FROMUSER or TOUSER parameters Oracle IMPort