1. Supporting High-Performance Data Processing on Flat-Files
Xuan Zhang
Gagan Agrawal
Ohio State University

2. Motivation Challenges of bioinformatics integration
Data volume: overwhelming
DNA sequence: 100 gigabases (August, 2005)
Data growth:
exponential
Figure provided by PDB

3. Existing Solutions (Relational) Databases Flat Files with Scripts
Support for indexing and high-level queries
Not suitable for biological data
Flat Files with Scripts
Compact, Perl Scripts available
Lack indexing and high-level query processing
Web-services
Significant overhead

4. Our Approach Enhance information integration systems on Functionality
On-the-fly data incorporation
Flat file data process
Usability
Declarative interface
Low programming requirement
Performance
Incorporate indexing support

5. Approach Summary Metadata Code generation
Declarative description of data
Data mining algorithms for semi-automatic writing
Reusable by different requests on same data
Code generation
Request analysis and execution separated
General modules with plug-in data module

6. System Overview Understand Data Process Data Data File User Request
Metadata Description
Layout
Miner
Layout Descriptor
Schema Descriptor
Answer
Code
Generation
Request
Processor
Layout Descriptor
Schema Descriptor
Layout Descriptor
Schema Descriptor
Schema
Miner
Information Integration System

7. Advantages Simple interface General data model Low human involvement
At metadata level, declarative
General data model
Semi-structured data
Flat file data
Low human involvement
Semi-automatic data incorporation
Low maintenance cost
OK Performance
Linear scale guaranteed
Can improve by using indexing

8. System Components Understand data Process data Layout mining
Schema mining
Process data
Wrapper generation
Query Process
Query Process with indices

9. Data Process Overview Automatic code generation approach Input Output
Metadata about datasets involved
Optional:
Implicit data transformation task
Request by users
Indexing functions
Output
Executable programs
General modules
Task-specific data module

10. Metadata Description Two aspects of data in flat files
Logical view of the data
Physical data organization
Two components of every data descriptor
Schema description
Layout description
Design goals
Powerful
Easy for writing and interpretation

11. Schema Descriptors Follow XML DTD standard for semi-structured data
Simple attribute list for relational data
<?xml version='1.0' encoding='UTF-8'?>
<!ELEMENT FASTA (ID, DESCRIPTION, SEQ)>
<!ELEMENT ID (#PCDATA)>
<!ELEMENT DESCRIPTION (#PCDATA)>
<!ELEMENT SEQ (#PCDATA)>
[FASTA] //Schema Name
ID = string //Data type definitions
DESCRIPTION = string
SEQ = string

12. Layout Descriptors Overall structure (FASTA example)
DATASET “FASTAData” { //Dataset name
DATATYPE {FASTA} //Schema name
DATASPACE LINESIZE=80 {
// ---- File layout details goes here ---- }
DATA {osu/fasta} //File location

13. Wrapper Generation System Overview
Layout Descriptor
Schema Descriptors
Layout Parser
Mapping Generator
Mapping File
Mapping Parser
Data Entry
Representation
Schema Mapping
Wrapper generation
system
Application Analyzer
WRAPINFO
Source
Dataset
Target
Dataset
DataReader
DataWriter
Synchronizer
wrapper

14. Query With Indices Motivation
Goal
Improve the performance of query-proc program
Index
Maintain the advantages
Flat file based
Low requirement on programming

15. Challenges & Approaches
Various indexing algorithms for various biological data
User defined indexing functions
Standard function interfaces
Flat file data
Values parsed implicitly and ready to be indexed
Byte offset as pointer
Metadata about indices
Layout descriptor

16. Schema & Layout information
System Revisited
query
Source/target names
Query parser
Metadata
collection
Dataset
descriptors
Descriptor
parser
Schema & Layout information
mappings
Application analyzer
Query analysis
Query execution
QUERYINFOR
Source
data files
Target
data file
DataReader
DataWriter
Synchronizer
Index file
Index functions

17. Language Enhancement Describe indices
Indexing is a property of dataset
Extend layout descriptors
Maintain query format
DATASET “name”{ …
INDEX {attribute:index_file_loc:index_gen_fun:index_retr_fun:fun_loc
[, attribute:index_file_loc:index_gen_fun:index_retr_fun:fun_loc]} }
New meaning of “=“:
If index available, use index
retrieving function
Else, compare values directly
AUTOWRAP GNAMES
FROM CHIPDATA, YEASTGENOME
BY CHIPDATA.GENE = YEASTGENOME.ID
WHERE …

18. System Enhancement Metadata Descriptor Parser Application Analyzer
+ parse index information
Application Analyzer
+ index information: index look-up table
+ test condition: compare_field_indexing

19. Microarray Gene Information Look-up
Goal: gather information about genes (120)
Query: microarray output join genome database
Index: gene names in genome

20. BLAST-ENHANCE Query Goal: Add extra information to BLAST output
Query: BLAST output join Swiss-Prot database
Index: protein ID in Swiss-Prot

21. OMIM-PLUS Query Goal: add Swiss-Prot link to OMIM
Query: OMIM join Swiss-Prot
Index: protein ID in Swiss-Prot

22. Homology Search Query Goal: find similar sequences
Query: query sequence list * sequence database
Indexing algorithm
Sequence-based
Transformation of sub-string composition
Indexing n-D numerical values

23. Homology Search (1) Index (Singh’s algorithm) Data: yeast genome
wavelet coefficients
minimum bounding rectangles

24. Homology Search (2) Index (Ferhatosmanoglu’s algorithm) Data: GenBank
Wavelet coefficients
Scalar quantization
R-tree

25. Conclusions
A frame work and a set of tools for on-the-fly flat file data integration
New data source understood semi-automatically by data mining tools
New data processed automatically by generated programs
Support for indexing incorporated flexibly