VOTable: Tabular Data for Virtual Observatory François Ochsenbein Roy Williams Clive Davenhall, Daniel Durand, Pierre Fernique, Robert Hanisch, David Giaretta,

Slides:



Advertisements
Similar presentations
A VO experience: Lessons learnt from accessing SIA compliant image servers with the AVO prototype F.Bonnarel on behalf of M.Allen, T.Boch, M.Dolensky,
Advertisements

IVOA, Pune India September Data Access Layer Working Group Pune Workshop Summary Doug Tody National Radio Astronomy Observatory International.
VOTable 2005 Applications. Agenda Description of some applications Description of some applications Nilesh UrunkarAbout C++ Parser and CONVOT Nilesh UrunkarAbout.
September 13, 2004NVO Summer School1 VO Protocols Overview Tom McGlynn NASA/GSFC T HE US N ATIONAL V IRTUAL O BSERVATORY.
September 13, 2004NVO Summer School1 VO Protocols Overview Tom McGlynn NASA/GSFC T HE US N ATIONAL V IRTUAL O BSERVATORY.
VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.
CNES implementation of the ISO standard An extension of the current CNES implementation of the ISO metadata standard.
What does LOFAR have to do with the Virtual Observatory (VO)? LOFAR Science Day 16 December 2003 Melbourne David Barnes The University of Melbourne.
An International Virtual Observatory data exchange format VOTable Roy Williams François Ochsenbein Clive Davenhall Daniel Durand Pierre Fernique David.
28 March 2003e-MapScholar: content management system The e-MapScholar Content Management System (CMS) David Medyckyj-Scott Project Director.
Metadata in the TAP context (1) The Problem: learn about which tables, tablesets,... are available from a TAP server for each of the tables / tablesets,
Open Office.Org What is the Open Office.org Source Project? Open source project through which Sun Microsystems is releasing the technology for the popular.
XML/EDI Overview West Chester Electronic Commerce Resource Center (ECRC)
14 October 2003ADASS 2003 – Strasbourg1 Resource Registries for the Virtual Observatory R.Plante (NCSA), G. Greene (STScI), R. Hanisch (STScI), T. McGlynn.
BinX and Astronomy Bob Mann Institute for Astronomy and National e-Science Centre.
HDF 1 NCSA HDF XML Activities Robert E. McGrath Mike Folk National Center for Supercomputing Applications.
2003 April 151 Data Centres: Connecting to the Real World Clive Page.
S. Derriere et al., ESSW03 Budapest, 2003 May 20 UCDs - metadata for astronomy Sébastien Derriere François Ochsenbein Thomas Boch CDS, Observatoire astronomique.
12-Oct-03ADASS VO Tutorial 1 VOTable Format, I/O Libraries & Tools Participants- Francois Ochsenbein (CDS, Strasbourg) Mark Taylor (Starlink, UK) Pallavi.
Research paper: Web Mining Research: A survey SIGKDD Explorations, June Volume 2, Issue 1 Author: R. Kosala and H. Blockeel.
Chapter 16 The World Wide Web. 2 The Web An infrastructure of information combined and the network software used to access it Web page A document that.
16-1 The World Wide Web The Web An infrastructure of distributed information combined with software that uses networks as a vehicle to exchange that information.
Astronomical Data Query Language Simple Query Protocol for the Virtual Observatory Naoki Yasuda 1, William O'Mullane 2, Tamas Budavari 2, Vivek Haridas.
F. Genova, VOTECH kickoff meeting, 2004/11/ Interoperability standards F. Genova, M. Allen, T. Boch, F. Bonnarel, S. Derriere, P. Fernique, F. Ochsenbein,
Introduction to XML. XML - Connectivity is Key Need for customized page layout – e.g. filter to display only recent data Downloadable product comparisons.
XML BIS4430 – unit 10. XML Origins Extensible Markup Language (XML) 1998 Inspired by Standard Generalized Markup Language (SGML) and HTML. SGML defines.
DateADASS How to Navigate VO Datasets Using VO Protocols Ray Plante (NCSA/UIUC), Thomas McGlynn and Eric Winter NASA/GSFC T HE US N ATIONAL V IRTUAL.
November 1, 2006IU DLP Brown Bag : Fall Data Integrity and Document- centric XML Using Schematron for Managing Text Collections Dazhi Jiao, Tamara.
Hello!. International Virtual Observatory Alliance Ajit Kembhavi, IUCAA, Pune.
Introduction to Web Mining Spring What is data mining? Data mining is extraction of useful patterns from data sources, e.g., databases, texts, web,
AVO Status June 2002 Status M. Dolensky. AVO Status June 2002 WA0 - European Initiatives Enabling a Virtual Observatory for Europe EVOE –Submitted expression.
1 P.Quinn TIVO June Drivers, Status and Planning.
Science with the Virtual Observatory Brian R. Kent NRAO.
Chapter 27 The World Wide Web and XML. Copyright © 2004 Pearson Addison-Wesley. All rights reserved.27-2 Topics in this Chapter The Web and the Internet.
Prototype system of the Japanese Virtual Observatory The Japanese Virtual Observatory (JVO) aims at providing easy access to federated astronomical databases.
XML A web enabled data description language 4/22/2001 By Mark Lawson & Edward Ryan L’Herault.
1 CS 430 Database Theory Winter 2005 Lecture 17: Objects, XML, and DBMSs.
Softsmith Infotech XML. Softsmith Infotech XML EXtensible Markup Language XML is a markup language much like HTML Designed to carry data, not to display.
Strasbourg astronomical Data Centre (DS) Françoise GENOVA.
1 Metadata –Information about information – Different objects, different forms – e.g. Library catalogue record Property:Value: Author Ian Beardwell Publisher.
Federation and Fusion of astronomical information Daniel Egret & Françoise Genova, CDS, Strasbourg Standards and tools for the Virtual Observatories.
Introduction to XML This presentation covers introductory features of XML. What XML is and what it is not? What does it do? Put different related technologies.
A PPARC funded project Workflow and Job Control in Astrogrid Jeff Lusted Dept Physics and Astronomy University of Leicester.
Chapter 27 The World Wide Web and XML. Copyright © 2004 Pearson Addison-Wesley. All rights reserved.27-2 Topics in this Chapter The Web and the Internet.
Jennifer Widom XML Data Introduction, Well-formed XML.
16 October 2003Registry Interface CallsIVOA Interoperability, Strasbourg IVOA Interoperability Elizabeth Auden & Registry Workgroup 16 – 17 October 2003.
CS 157B: Database Management Systems II February 11 Class Meeting Department of Computer Science San Jose State University Spring 2013 Instructor: Ron.
The International Virtual Observatory Alliance (IVOA) interoperability in action.
Automatic Metadata Discovery from Non-cooperative Digital Libraries By Ron Shi, Kurt Maly, Mohammad Zubair IADIS International Conference May 2003.
Discussed in Kyoto Schema changes for the next version (Gerard Lemson)  will be included in VOTable1.2 Schema changes for the next version (Gerard Lemson)
F. Genova, VO as a Data Grid, 2003/06/301 Interoperability of astronomy data bases Françoise Genova, CDS.
12 Oct 2003VO Tutorial, ADASS Strasbourg, Data Access Layer (DAL) Tutorial Doug Tody, National Radio Astronomy Observatory T HE US N ATIONAL V IRTUAL.
April , 2006 HEASARC Users Group Tom McGlynn The HEASARC On-line Services Tom McGlynn.
Martin Kruliš by Martin Kruliš (v1.1)1.
Report of UCD Working Group Roy Williams Caltech.
Working with XML. Markup Languages Text-based languages based on SGML Text-based languages based on SGML SGML = Standard Generalized Markup Language SGML.
VOTable agenda Current VOTable status Current VOTable status News from Applications News from Applications Questions about VOTable schema Questions about.
VOTable agenda Current VOTable status News from Applications Referring STC (as a data model example) Relations between s Questions about VOTable schema.
Publishing Combined Image & Spectral Data Packages Introduction to MEx M. Sierra, J.-C. Malapert, B. Rino VO ESO - Garching Virtual Observatory Info-Workshop.
Evanthia Hatziminaoglou, ESO - Garching Virtual Observatory Info-Workshop, SOFIA January 2008 VO Tools Overview.
7 Dec 2009R. J. Hanisch: Astronomy Data Standards CERN 1 Data Standards in Astronomy Dr. Robert J. Hanisch Director, US Virtual Astronomical Observatory.
XML Extensible Markup Language
XML QUESTIONS AND ANSWERS
What is FITS? FITS = Flexible Image Transport System
XML in Web Technologies
XML Data Introduction, Well-formed XML.
XML Problems and Solutions
Google Sky.
VOTable Meetings: Conclusions
Presentation transcript:

VOTable: Tabular Data for Virtual Observatory François Ochsenbein Roy Williams Clive Davenhall, Daniel Durand, Pierre Fernique, Robert Hanisch, David Giaretta, Tom McGlynn, Alex Szalay, Andreas Wicenec

The Context Need of exchanging data in tabular form: Coming from a wide variety of data servers and archives (VO context) Must include the associated metadata in order to be interpretable by applications Must deal with potentially millions of records Existence of FITS

VOTable History Astrores at CDS/ESO (June 1999) XSIL at Caltech (June 2000) October 2001: first discussions December 2001: VOTable 0.1 January 2002: Interoperability meeting Strasbourg 15 April 2002: VOTable VOTable archives & discussion groups:

Why XML ? includes in a single document the data and their associated metadata (descriptive data) is of common usage since ~ 3 years can be interpreted  parsers and tools readily available can be visualized (XSL) can be encapsulated in messages

A “classical” XML Document William Herschel Procyon …..

Problems of “classical” XML Documents Each data element is, meaning: Huge overheads in terms of volume, required resources, and processing time  Not adapted to multi-million row tables Need to introduce new elements (tags) for each new parameter, or to cross-match a potentially large set of name spaces

The VOTable way The metadata part (data description), essentially as a set of and specifications The data part (serialisation), which may be in XML, FITS or binary. VOTables follow the classical tabular presentation where the columns are assumed to be homogeneous in terms of their associated metadata; a VOTable document contains:

This parameter is designed to store the observer's name Some bright stars Procyon Vega

… … … …

The and name column label unit standardized unit datatype computer type width character representation precision character representation Arraysize repetition factor ucd standardized parameter category Describe the metadata attached to columns or to the resource

The UCDs Interpretation of the table contents Decide whether values can be compared Data mining  S. Derrière's talk on Friday Unified Content Descriptor Categorisation of the parameters listed in the table

datatypeMeaningFITSBytes "boolean"LogicalL1 "bit"BitX* "unsignedByte"Byte (0 to 255)B1 "short"Short IntegerI2 "int"IntegerJ4 "long"Long integerK8 "char"ASCII CharacterA1 "unicodeChar"Unicode Character2 "float"Floating pointE4 "double"DoubleD8 "floatComplex"Float ComplexC8 "doubleComplex"Double ComplexM16

FITS Compatibility Compatible data types FITS keywords are represented as, e.g. width precision arraysize Array and variable-length arrays may link to existing FITS data sets VOTable was designed to be compatible with existing FITS data tables

Data Serialization FITS or BINARY data may be embedded in the document, or remote; compression/encoding may be applied.

Existing tools and Servers Several databases are delivering VOTables: HEASARC IPAC NOAO NRAO VizieR SIMBAD (cone search >50 services) VOTable parsers in Perl, Java, C (different types of parsers for different applications) VOTable validators XSLT basic XML/HTML translators

DTD or XML-Schema The VOTable rules are existing as a DTD (Document-Type Definition) and in the XML-Schema language (heavily used in developping WebServices applications)

VOTable appendices 1.The LINK conventions describing how to get the correlated data (explanations, images, spectra…) based on substitution of the column contents Astrores had two features not implemented in VOTables: … … … … photo/procyon.dat … … photo/vega.dat …

VOTable appendices (2) 2. The Query Mechanism using conventions similar to the HTML for retrieving the data from user-supplied constraints Some bright stars  toward more generic WDSL-like solutions ?

Conclusions Just version 1.0 … more to come Comments ? Proposals ?  Join the discussion group