1. e-Science Data Information and Knowledge Transformation BinX – A Tool for Binary File Access eDIKT project team Ted Wen tedwen@nesc.ac.uktedwen@nesc.ac.uk Robert Carroll robertc@nesc.ac.ukobertc@nesc.ac.uk

2. www.edikt.org What is BinX?  Binary in XML –Annotation language  Using XML  Descriptive  Low-level –Software components  BinX library  Generic utilities  API

3. www.edikt.org How and Why BinX is used 0101010101 01010101010 10101010100 01000010111 01010101010 10101010110 Special Application Program Special Application Program … … BinX Library Application Program Application Program Application Program Application Program Application Program Application Program

4. e-Science Data Information and Knowledge Transformation The BinX Language Annotating a binary data stream Mark up data types Mark up sequences Mark up arrays Complex structures

5. www.edikt.org 1. 32767 2. 2147483647 3. 100.0 4. 100.0 Primitive Data Types  Mark up data types FF 7F 7F FF FF FF 00 00 C8 42 42 C8 00 00 1234

6. www.edikt.org Abstract “struct” types  Mark up a sequence Screen descriptor in GIF: Screen width: unsigned short; Screen height: unsigned short; Packed field: a byte Background colour index: byte Pixel aspect ratio: byte

7. www.edikt.org Abstract “array” types  Mark up an array A 2-dimensional array containing 10-by-100, 32-bit integers

8. www.edikt.org Embedded abstract types  Complex structures

9. www.edikt.org User-defined metadata  Label the data types and structures

10. www.edikt.org Reusable type definitions  Define macros for reuse

11. www.edikt.org Linking to binary data  Reference the binary data file … …

12. www.edikt.org A BinX document  –  – –  –  –  Root element Data class section Data instance section Abstract data type

13. www.edikt.org DataBinX DataBinX = BinX with Data 100 1000 5.257 1 2

14. e-Science Data Information and Knowledge Transformation The BinX Library Core library Utilities Applications

15. www.edikt.org BinX Components  The library has core functionality to support generic utilities and applications Applications Utilities BinX Library Core BinX core functionality Parse/Gen BinX doc Read/write binary data Parse/Gen DataBinX Generic tools DataBinx pack/unpack Extractor, Viewer BinX editor Applications Domain-specific

16. www.edikt.org BinX application models  Data catalogue model  Data manipulation model  Data query model  Data service model  Data transportation model

17. www.edikt.org Data catalogue model Primary storage Binary data files Metadata Syntactic annotation Semantic annotation Classification Domain specific Cross-reference XLink 0101 0101 01 BinX 1.1 BinX 1.1 BinX 1.2.1 BinX 1.2.1 BinX 1.2.2 BinX 1.2.2 BinX 1.2.3 BinX 1.2.3 0101 0101 01 BinX 1.2 BinX 1.2 BinX 1 BinX 1 BINARY Detailed Abstract METADATA

18. www.edikt.org Data manipulation model  Extraction –Subset of a dataset  Combination –Merge several datasets  Transformation –Conversion of data types –Change of sequence order –Transposition of array dimensions  Transparency –Automatic change of byte order

19. www.edikt.org Data query model  In-dataset query –XPath against virtual XML  Cross-dataset query –Link into multiple datasets  Defining result format –XQuery-based return fragment  Output interface –SAX events Utility BinX library 0101010 10 BinX data source BinX data source DataBinX SAX Events VOTable SAX Events APP VOTable APP DataBinx 0101010 10 BinX data source BinX data source APP Custom XQuery SAX Events 0101010 10 BinX data source BinX data source XPath 0101010 10 BinX data source BinX data source XLink Transform

20. www.edikt.org Data service model  Publishing logical datasets in BinX DB 0101 0101 01 Client BinX Grid 0101 0101 01 BinX Dataset from one binary file Dataset from several binary files Dataset from multiple data sources

21. www.edikt.org Data transportation model DataBinX as interlingua XML document XML document DataBinX Schema BinX Schema BinX + Binary BinX + Binary ZIP (MIME) ZIP (MIME) XSLT BinX Util ZIP tool Send Receive XSLT BinX Util ZIP tool

22. e-Science Data Information and Knowledge Transformation Application in Astronomy Case Study Data Conversion Between FITS and VOTable

23. www.edikt.org Application in astronomy  FITS and VOTable conversion DataBinX Utility BinX library Core SIMPLE = T … END 01010101 SIMPLE = T … END 01010101 <?xml version=. … <?xml version=. …

24. www.edikt.org FITS file SIMPLE = T / file does conform to FITS standard BITPIX = 8 / number of bits per data pixel NAXIS = 1 / number of data axes … END 3D 4A 14 0F 1C FE 25 04 … … XTENSION= ‘BINTABLE’ / binary table extension BITPIX = 8 / 8-bit bytes NAXIS = 2 / 2-dimensional binary table … END 7B 3E 40 2C 16 70 E7 6F … … 0 79 Primary HDU Extension Header Data

25. www.edikt.org VOTable Procyon 114.827 5.227 4 5 3 4 3 2 1 2 3 3 5 6

26. www.edikt.org FITS →DataBinX →VOTable  FITS to VOTable conversion DataBinX Utility FITS Schema BinX Schema BinX Preprocessor DataBinX VOTable XSLT transformer

27. www.edikt.org VOTable→DataBinX→FITS  VOTable to FITS conversion XSLT transformer VOTable XSLT DataBinX FITS Schema BinX Schema BinX DataBinX Utility Binary Data Binary Data Post processor FITS Header FITS Header

28. e-Science Data Information and Knowledge Transformation BinX Software Software library in C++ Documentation Utilities and Samples

29. www.edikt.org Future releases  XPath-based data query  DFDL support  Output through SAX events  Output as XQuery return  Database interfacing  Java wrapper for utilities

30. www.edikt.org Support  Information and software download: –http://www.edikt.org/binxhttp://www.edikt.org/binx  Questions: –support@edikt.orgsupport@edikt.org  Requirements and suggestions: –tedwen@edikt.orgtedwen@edikt.org –robertc@edikt.orgrobertc@edikt.org