e-Science Data Information and Knowledge Transformation BinX – A Tool for Binary File Access eDIKT project team Ted Wen Robert Carroll

What is BinX?  Binary in XML –Annotation language  Using XML  Descriptive  Low-level –Software components  BinX library  Generic utilities  API

How and Why BinX is used Special Application Program Special Application Program … … BinX Library Application Program Application Program Application Program Application Program Application Program Application Program

e-Science Data Information and Knowledge Transformation The BinX Language Annotating a binary data stream Mark up data types Mark up sequences Mark up arrays Complex structures

Primitive Data Types  Mark up data types FF 7F 7F FF FF FF C C

Abstract “struct” types  Mark up a sequence Screen descriptor in GIF: Screen width: unsigned short; Screen height: unsigned short; Packed field: a byte Background colour index: byte Pixel aspect ratio: byte

Abstract “array” types  Mark up an array A 2-dimensional array containing 10-by-100, 32-bit integers

Embedded abstract types  Complex structures

User-defined metadata  Label the data types and structures

Reusable type definitions  Define macros for reuse

Linking to binary data  Reference the binary data file … …

A BinX document  –  – –  –  –  Root element Data class section Data instance section Abstract data type

DataBinX DataBinX = BinX with Data

e-Science Data Information and Knowledge Transformation The BinX Library Core library Utilities Applications

BinX Components  The library has core functionality to support generic utilities and applications Applications Utilities BinX Library Core BinX core functionality Parse/Gen BinX doc Read/write binary data Parse/Gen DataBinX Generic tools DataBinx pack/unpack Extractor, Viewer BinX editor Applications Domain-specific

BinX application models  Data catalogue model  Data manipulation model  Data query model  Data service model  Data transportation model

Data catalogue model Primary storage Binary data files Metadata Syntactic annotation Semantic annotation Classification Domain specific Cross-reference XLink BinX 1.1 BinX 1.1 BinX BinX BinX BinX BinX BinX BinX 1.2 BinX 1.2 BinX 1 BinX 1 BINARY Detailed Abstract METADATA

Data manipulation model  Extraction –Subset of a dataset  Combination –Merge several datasets  Transformation –Conversion of data types –Change of sequence order –Transposition of array dimensions  Transparency –Automatic change of byte order

Data query model  In-dataset query –XPath against virtual XML  Cross-dataset query –Link into multiple datasets  Defining result format –XQuery-based return fragment  Output interface –SAX events Utility BinX library BinX data source BinX data source DataBinX SAX Events VOTable SAX Events APP VOTable APP DataBinx BinX data source BinX data source APP Custom XQuery SAX Events BinX data source BinX data source XPath BinX data source BinX data source XLink Transform

Data service model  Publishing logical datasets in BinX DB Client BinX Grid BinX Dataset from one binary file Dataset from several binary files Dataset from multiple data sources

Data transportation model DataBinX as interlingua XML document XML document DataBinX Schema BinX Schema BinX + Binary BinX + Binary ZIP (MIME) ZIP (MIME) XSLT BinX Util ZIP tool Send Receive XSLT BinX Util ZIP tool

e-Science Data Information and Knowledge Transformation Application in Astronomy Case Study Data Conversion Between FITS and VOTable

Application in astronomy  FITS and VOTable conversion DataBinX Utility BinX library Core SIMPLE = T … END SIMPLE = T … END <?xml version=. … <?xml version=. …

FITS file SIMPLE = T / file does conform to FITS standard BITPIX = 8 / number of bits per data pixel NAXIS = 1 / number of data axes … END 3D 4A 14 0F 1C FE … … XTENSION= ‘BINTABLE’ / binary table extension BITPIX = 8 / 8-bit bytes NAXIS = 2 / 2-dimensional binary table … END 7B 3E 40 2C E7 6F … … 0 79 Primary HDU Extension Header Data

VOTable Procyon

FITS →DataBinX →VOTable  FITS to VOTable conversion DataBinX Utility FITS Schema BinX Schema BinX Preprocessor DataBinX VOTable XSLT transformer

VOTable→DataBinX→FITS  VOTable to FITS conversion XSLT transformer VOTable XSLT DataBinX FITS Schema BinX Schema BinX DataBinX Utility Binary Data Binary Data Post processor FITS Header FITS Header

e-Science Data Information and Knowledge Transformation BinX Software Software library in C++ Documentation Utilities and Samples

Future releases  XPath-based data query  DFDL support  Output through SAX events  Output as XQuery return  Database interfacing  Java wrapper for utilities

Support  Information and software download: –  Questions:  Requirements and suggestions: