1. Supporting High- Performance Data Processing on Flat-Files Xuan Zhang Gagan Agrawal Ohio State University

2. Motivation Challenges of bioinformatics integration –Data volume: overwhelming DNA sequence: 100 gigabases (August, 2005) –Data growth: exponential Figure provided by PDB

3. Existing Solutions –(Relational) Databases Support for indexing and high-level queries Not suitable for biological data –Flat Files with Scripts Compact, Perl Scripts available Lack indexing and high-level query processing –Web-services Significant overhead

4. Enhance information integration systems on –Functionality On-the-fly data incorporation Flat file data process –Usability Declarative interface Low programming requirement –Performance Incorporate indexing support Our Approach

5. Approach Summary Metadata –Declarative description of data –Data mining algorithms for semi-automatic writing –Reusable by different requests on same data Code generation –Request analysis and execution separated –General modules with plug-in data module

6. System Overview Understand DataProcess Data Data File User Request Answer Metadata Description Layout Descriptor --------------------------------------------------- Schema Descriptor Layout Descriptor --------------------------------------------------- Schema Descriptor Layout Descriptor --------------------------------------------------- Schema Descriptor Code Generation Request Processor Layout Miner Schema Miner Information Integration System

7. Advantages Simple interface –At metadata level, declarative General data model –Semi-structured data –Flat file data Low human involvement –Semi-automatic data incorporation –Low maintenance cost OK Performance –Linear scale guaranteed –Can improve by using indexing

8. System Components Understand data –Layout mining –Schema mining Process data –Wrapper generation –Query Process –Query Process with indices

9. Data Process Overview Automatic code generation approach Input –Metadata about datasets involved –Optional: Implicit data transformation task Request by users Indexing functions Output –Executable programs General modules Task-specific data module

10. Metadata Description Two aspects of data in flat files –Logical view of the data –Physical data organization Two components of every data descriptor –Schema description –Layout description Design goals –Powerful –Easy for writing and interpretation

11. Schema Descriptors Follow XML DTD standard for semi-structured data Simple attribute list for relational data [FASTA]//Schema Name ID = string//Data type definitions DESCRIPTION = string SEQ = string

12. Layout Descriptors Overall structure (FASTA example) DATASET “FASTAData” {//Dataset name DATATYPE {FASTA} //Schema name DATASPACE LINESIZE=80 { // ---- File layout details goes here ---- } DATA {osu/fasta}//File location }

13. Wrapper Generation System Overview DataReaderDataWriter Synchronizer Source Dataset Target Dataset WRAPINFO Wrapper generation system wrapper Mapping File Mapping Parser Schema Mapping Mapping Generator Schema Descriptors Layout Parser Layout Descriptor Data Entry Representation Application Analyzer

14. Query With Indices Motivation Goal –Improve the performance of query-proc program Index –Maintain the advantages Flat file based Low requirement on programming

15. Challenges & Approaches Various indexing algorithms for various biological data –User defined indexing functions –Standard function interfaces Flat file data –Values parsed implicitly and ready to be indexed –Byte offset as pointer Metadata about indices –Layout descriptor

16. System Revisited query Query parser Metadata collection Dataset descriptors Descriptor parser Application analyzer QUERYINFOR DataReaderDataWriter Synchronizer Source data files Target data file Source/target names Schema & Layout information mappings Query analysis Query execution Index file Index functions

17. Language Enhancement Describe indices –Indexing is a property of dataset –Extend layout descriptors –Maintain query format DATASET “name”{ … INDEX {attribute:index_file_loc:index_gen_fun:index_retr_fun:fun_loc [, attribute:index_file_loc:index_gen_fun:index_retr_fun:fun_loc]} } AUTOWRAP GNAMES FROM CHIPDATA, YEASTGENOME BY CHIPDATA.GENE = YEASTGENOME.ID WHERE … New meaning of “=“: If index available, use index retrieving function Else, compare values directly

18. System Enhancement Metadata Descriptor Parser + parse index information Application Analyzer + index information: index look-up table + test condition: compare_field_indexing

19. Microarray Gene Information Look-up Goal: gather information about genes (120) Query: microarray output join genome database Index: gene names in genome

20. BLAST-ENHANCE Query Goal: Add extra information to BLAST output Query: BLAST output join Swiss- Prot database Index: protein ID in Swiss-Prot

21. OMIM-PLUS Query Goal: add Swiss- Prot link to OMIM Query: OMIM join Swiss-Prot Index: protein ID in Swiss-Prot

22. Homology Search Query Goal: find similar sequences Query: query sequence list * sequence database Indexing algorithm –Sequence-based –Transformation of sub-string composition –Indexing n-D numerical values

23. Homology Search (1) Index (Singh’s algorithm) –Data: yeast genome –wavelet coefficients –minimum bounding rectangles

24. Homology Search (2) Index (Ferhatosmanoglu’s algorithm) –Data: GenBank –Wavelet coefficients –Scalar quantization –R-tree

25. Conclusions A frame work and a set of tools for on-the-fly flat file data integration –New data source understood semi-automatically by data mining tools –New data processed automatically by generated programs –Support for indexing incorporated flexibly