12-Oct-03ADASS VO Tutorial 1 VOTable Format, I/O Libraries & Tools Participants- Francois Ochsenbein (CDS, Strasbourg) Mark Taylor (Starlink, UK) Pallavi Kulkarni (IUCAA, India) Sonali Kale (Persistent Systems,India)

12-Oct-032 ADASS VO Tutorial Agenda VOTable format I/O Libraries Plotting tools for VOTables Mirage VOPlot Topcat VOTable browsers Treeview

12-Oct-033 ADASS VO Tutorial Understanding VOTable François Ochsenbein on behalf of: Roy Willliams, Clive Davenhall, Daniel Durand, Pierre Fernique, David Giaretta, Robert Hanish, Tom McGlynn, Alex Szalay, Andreas

12-Oct-034 ADASS VO Tutorial Overview The VOTable format is a proposed XML standard for representing tabular data in the context of the Virtual Observatory VOTable was designed to be compatible with FITS Binary Tables (the data part can be addressed directly) VOTable is designed as a flexible storage and exchange format for tabular data, with particular emphasis on astronomical tables

12-Oct-035 ADASS VO Tutorial Why move to VOTable ? Interoperability is encouraged through the use of standard data structures and descriptions (metadata) FITS is a widely accepted structure, but FITS keywords are rarely shared beyond the basic ones. XML built-ins: easy validation of input document easy transformations and displays via XSLT engine VOTable can cope with very large datasets does not require the huge XML overhead direct access to binary files and existing FITS files

12-Oct-036 ADASS VO Tutorial Data Model VOTable = hierarchy of Metadata + Tables Metadata = Infos + Descriptions + Parameters + Links + Fields Resource = set of tables Table = list of Fields/Parameters + Data Data = stream of Rows (or binary stream (remote)) Row = list of Cells Cell = Primitive or variable-length list of Primitives or or multidimensional array of Primitives Primitive = integer, character, float, floatComplex

12-Oct-037 ADASS VO Tutorial A VOTable document DEFINITIONS typically about used coordinate system(s) RESOURCE contains a DESCRIPTION and a list of tables, eventually (recursively) other RESOURCEs TABLE contains: a textual DESCRIPTION of the table a list of FIELD (columns) which describe the table layout and eventually PARAMETER which may specify some constants the DATA part which can be present in the document, or remotely accessible in several formats including FITS, or absent (description of the data structure) possibly LINK for getting details, explanations, related data,... Example Contains the following elements:

12-Oct-038 ADASS VO Tutorial Table Element in Detail Written as DESCRIPTION and a collection of FIELD elements, plus possible PARAMETER and LINK elements, all representing the metadata; followed by the DATA Fields describe the nature of table columns Table data start with the DATA element, followed by the actual rows containing the values of the fields, in the same order as their description.

12-Oct-039 ADASS VO Tutorial FIELD in detail Has sub elements like DESCRIPTION, LINK, and possibly a VALUES element Provides information for a corresponding cell in the DATA element For identification, the FIELD has a name, and also an ID name is the field designation (column header) ID is an identifier which follows the XML rules (restricted character set, unicity in an XML document) The FIELD must contain a datatype attribute Each table cell may contain more than one of the specified datatype and is specified by arraysize, which can be variable and multidimensional ( 64x64x* )

12-Oct-0310 ADASS VO Tutorial Available Datatypes

12-Oct-0311 ADASS VO Tutorial Field cont.. unit attribute precises the units in a controlled vocabulary ucd (Unified Content Descriptors) This provides information about the semantics of the field expressed as a standardized string originally created at CDS and currently being reviewed. precision indicates the number of significant digits for edition purposes VALUES element Holds the domain information (min, max, null, set of available values) LINK Element It is used to provide pointers to other documents or data servers through URL.

12-Oct-0312 ADASS VO Tutorial Parameter definitions A PARAMETER is similar to a FIELD : it may contain a DESCRIPTION, the attributes unit, ucd, name, ID,... plus a value attribute may be considered as a constant column Typical examples: frequency or wavelength of flux measurements statistical error of results

12-Oct-0313 ADASS VO Tutorial Data Content The data content of the table is in a single DATA element and is organized in a reading order. Data can be stored or accessed in several formats: TABLEDATA introduces an XML coding of the table FITS refers to an external FITS file BINARY indicates a binary coding of the data used for its efficiency STREAM introduces remote or encoded data Data can be in the input stream or stored separately Example

12-Oct-0314 ADASS VO Tutorial VOTable Additions Several new features are being considered: GROUP of fields introduces a view of the fields as a structure utype attribute to characterize the role of a field in the context of a data model encoding attribute of a cell element in order to store e.g. images in table cells

12-Oct-0315 ADASS VO Tutorial VOTable I/O libraries Pallavi Kulkarni

12-Oct-0316 ADASS VO Tutorial I/O Libraries Why are parsers required ? Why are writers required ? Tree Structured Approach. Event Driven Approach. Tree Structured vs. Event Driven Approach. VOTable Parsers & Writers.

12-Oct-0317 ADASS VO Tutorial Why are parsers required ? Provide a library for API based access to VOTable files. These APIs can be directly used to develop VOTable applications without having to do raw VOTable processing.

12-Oct-0318 ADASS VO Tutorial Why are writers required ? Writer provides APIs for generating syntactically correct documents. The user doesn’t need to be aware of low level APIs. Simplifies the process of writing a document.

12-Oct-0319 ADASS VO Tutorial Tree Structured Approach Tree structured approach is a two step process. 1. The entire XML data is loaded in memory. 2. Operations are performed on the loaded data. VOTable Resource Table

12-Oct-0320 ADASS VO Tutorial Event Driven Approach Does not create a tree structure in memory. Single step process. Data is passed from XML document to the application on the fly.

12-Oct-0321 ADASS VO Tutorial Tree Structured vs. Event Driven Event driven approach is faster. Event driven approach is good for large documents because it takes comparatively less memory. With event driven approach, one can access the data but not modify it. With tree structured approach, one can modify data. Tree structured parsing allows back and forth traversal in the XML data. Event driven parsing can be stopped once desired XML data has been extracted.

12-Oct-0322 ADASS VO Tutorial VOTable Parsers & Writers JAVOT (NVO) SAVOT (European VO) VOTable Java Streaming Writer (VO-India) C++ Parser (VO-India) VOTable Perl Modules (NVO) VOTable::DOM (NVO)

12-Oct-0323 ADASS VO Tutorial JAVOT Developed at Caltech. Written in JAVA. Creates a tree structure in memory. Current version supports reading of VOTables. Editing and writing of tables not yet supported. More information can be found at:

12-Oct-0324 ADASS VO Tutorial SAVOT Developed at CDS. Written in JAVA. Supports reading, writing & editing of VOTables. Can work in both full (tree structured) & sequential (event driven) modes for parsing the data. Suitable for large VOTable files as well. More information can be found at:

12-Oct-0325 ADASS VO Tutorial Sample VOTable

12-Oct-0326 ADASS VO Tutorial Sample Code (Sequential mode) SavotResource currentResource = sb.getNextResource(); TRSet tr = currentResource.getTRSet(0); TDSet theTDs = tr.getTDSet(0); for (int k = 0; k < theTDs.getItemCount(); k++) { System.out.println(theTDs.getContent(k)); } Get resource element Fetch all the rows From first table Fetch first row from the set of rows. Print the data Field wise

12-Oct-0327 ADASS VO Tutorial VOTable JAVA Streaming Writer Developed at IUCAA and Persistent Systems. Written in JAVA. It supports only writing of data. Takes a streaming approach i.e. event based approach to write the data. More information can be found at: Writer.htmttp://vo.iucaa.ernet.in/~voi/votableStream Writer.htm

12-Oct-0328 ADASS VO Tutorial VOTable to be generated Mercury

12-Oct-0329 ADASS VO Tutorial Sample Code VOTableStreamWriter voWrite = new VOTableStreamWriter(prnStream) VOTable vot = new VOTable() voWrite.writeVOTable(vot) VOTableResource voResource = new VOTableResource() voWrite.writeResource(voResource) VOTableTable voTab = new VOTableTable() VOTableField voField = new VOTableField() voField.setName("Planet") voField.setDataType(“char”) voTab.addField(voField1) Initiallize the streaming writer Create & write VOTable element. Create & write Resource element. Create a table element Create a field & add it to the table.

12-Oct-0330 ADASS VO Tutorial Sample code contd. voWrite.writeTable(voTab) ; String [] firstRow = {"Mercury"} ; voWrite.addRow(firstRow, 1) ; voWrite.endTable() ; voWrite.endResource() ; voWrite.endVOTable() ; Write the table to output stream Create a row & write it to output stream End the respective elements.

12-Oct-0331 ADASS VO Tutorial C++ Parser Developed at Persistent Systems and IUCAA. Written in C++. Available as both streaming and non- streaming parser. It runs on Windows NT 4.0, Windows 2000, and Redhat Linux 7.1.

12-Oct-0332 ADASS VO Tutorial C++ Parser Currently, supports reading of VOTables and pure-xml table data. Currently, doesn’t support reading of binary or FITS data & doesn’t support writing of VOTables. More information can be found at: m m

12-Oct-0333 ADASS VO Tutorial VOTable PERL modules Developed by the ClassX project at GSFC (NVO). Written in PERL. Builds a tree structure in memory. Optimizations to handle large TABLEDATA elements. Can be used for parsing existing VOTables and creating new ones from scratch. More information can be found at:

12-Oct-0334 ADASS VO Tutorial PERL for formatting & printing Developed at NCSA (NVO). Written in PERL. It supports writing of VOTable documents. Takes a streaming i.e. event driven approach to write data. More information can be found at: Table-DOM.pm Table-DOM.pm

12-Oct-0335 ADASS VO Tutorial VOTable Applications: VOPLOT & MIRAGE Sonali Kale

12-Oct-0336 ADASS VO Tutorial Plotting Tools for VOTables VOPlot Developed under Virtual Observatory India initiative. Mirage Developed by Lucent Technologies, Bell Labs. Topcat Developed by Starlink, UK.

12-Oct-0337 ADASS VO Tutorial VOPlot – Introduction Visualization toolkit developed by Persistent Systems and IUCAA in collaboration with CDS. Motivation – Allow web-based visualization of astronomy data. Available as web-based version as well as a standalone desktop application. Web-based version is successfully integrated with Vizier Catalogue Service.

12-Oct-0338 ADASS VO Tutorial VOPlot: Visualization Has all the features of versatile graphical tool Scatter plots Connected plots Histograms Logarithmic axes Overlaying Auto-ranging

12-Oct-0339 ADASS VO Tutorial VOPlot – Features Column transformations based on expression. Data subset creation based on filter condition. Save graph in EPS format. Ability to select points on graph. View meta data and data in tabular and VOTable format.

12-Oct-0340 ADASS VO Tutorial VOPlot – Features Inter-operable with Aladin developed by CDS. This enables simultaneous visualization of some graph in VOPlot together with a representation as a sky plot in Aladin. Selecting some region in the graph highlights the corresponding points in the sky plot and vice versa.

12-Oct-0341 ADASS VO Tutorial VOPlot – Features VOPlot can be used for basic statistical analysis. Basic uni-variate and multivariate functions supported. Displays box plot. A box plot provides a visual summary important aspects of data distribution.

12-Oct-0342 ADASS VO Tutorial VOPlot – Features Allows overlaying from multiple catalogues. Allows drawing of error bars. Can be integrated with any web-based catalogue service that creates output in VOTable format. Example: Successfully integrated with LEDAS (UK).

12-Oct-0343 ADASS VO Tutorial Launch VOPlot from Vizier

12-Oct-0344 ADASS VO Tutorial Launch VOPlot from Vizier (cont.)

12-Oct-0345 ADASS VO Tutorial Launch VOPlot from Vizier (cont.) Choose Y column

12-Oct-0346 ADASS VO Tutorial VOPlot: Inter-operation with Aladin

12-Oct-0347 ADASS VO Tutorial Inter-operation with Aladin (cont.) Point highlighte d in VOPlot Object selected in Aladin.

12-Oct-0348 ADASS VO Tutorial VOPlot: References VOPlot Vizier Catalogue Service Aladin Virtual Observatory India

12-Oct-0349 ADASS VO Tutorial Mirage: Introduction Mirage is a Java-based tool for data visualization tool and exploratory analysis. Developed at Bell Laboratories, Lucent Technologies. Support for VOTables added in collaboration with John Hopkins University. Ray Plante produced an XSL stylesheet to convert from VOTable to Mirage format.

12-Oct-0350 ADASS VO Tutorial Mirage: Data Visualization Features Mirage provides multiple simultaneous views of data. Data visualization is provided through: Data matrix view Scatter Plots Histogram. Feature vector plot.

12-Oct-0351 ADASS VO Tutorial Mirage: Operations On Graph Points on plot can be selected using different selection methods (box, Bezier curve or free hand). Selection can be broadcasted to other views. Automatic walk in through graphs. Grid of plots can be created. Coloring of plot points.

12-Oct-0352 ADASS VO Tutorial Mirage: Command Interpreter Command line interpreter supports: Creation of new columns based on arithmetic expressions. Addition of new columns from other data file. Selection of points using a logical expression. Color selection of points.

12-Oct-0353 ADASS VO Tutorial Mirage: Data Matrix View One row for each entry and one column for each attribute. Data read from file

12-Oct-0354 ADASS VO Tutorial Mirage: Scatter Plot Choose a column for plotting Walking through a plot.

12-Oct-0355 ADASS VO Tutorial Mirage: Histogram Change bin width

12-Oct-0356 ADASS VO Tutorial Mirage: Multiple Simultaneous Views Broadcas t selection Make a selection

12-Oct-0357 ADASS VO Tutorial Mirage: Grid of plots Comman d to add column Zoom in and out

12-Oct-0358 ADASS VO Tutorial Mirage: References Mirage VO Enabled Mirage XSL Stylesheet to convert VOTable to Mirage format: vo.org/pub/articles/ScienceWithProtoVOtools /vot2mirage.xml

12-Oct-0359 ADASS VO Tutorial VOTable Applications : TOPCAT & Treeview Mark Taylor

12-Oct-0360 ADASS VO Tutorial TOPCAT Tool for OPerations on Catalogues And Tables Generic table read/view/edit/analyse/plot/write Format-neutral: VOTable, FITS, SQL and others Pure java Full online help Powerful expression language Supports large tables Under active development

12-Oct-0361 ADASS VO Tutorial TOPCAT capabilities View/edit table data View/edit table and column metadata Units, UCDs, type, array shape, format-specific items... Create new columns using algebraic expressions Powerful language – conditionals, nulls, string handling... Move/delete columns Sort rows Create row subsets in various ways Algebraic, graphical, boolean columns, selected rows Plot columns against each other Distinguish different defined subsets Calculate per-column statistics

12-Oct-0362 ADASS VO Tutorial Table I/O VOTable TABLEDATA, FITS, STREAM? FITS TABLE, BINTABLE SQL...others VOTable FITS SQL ASCII LaTeX Mirage …others TOPCAT (view/edit GUI) Tablecopy (command line) Tables class library (your code here) % tablecopy -ofmt fits my-votable.xml format-neutral view

12-Oct-0363 ADASS VO Tutorial

12-Oct-0364 ADASS VO Tutorial Treeview Find tables in deep hierarchies: Directory trees VOTable hierarchical structure Multi-extension FITS files Compression:.gz,.bz2,.Z Tar, zip, jar files FTP, HTTP Quick preview of VOTables XML, table data, table/column metadata Launch TOPCAT Recent versions of TOPCAT (v0.4) have embedded Treeview

12-Oct-0365 ADASS VO Tutorial References TOPCAT Treeview Starlink