Starlink VOTable software Author: Mark Taylor Open source Java software for table manipulation STIL: Table I/O & processing library TOPCAT: Graphical table view/edit/analysis application STILTS: STIL Tool Set

Software Features ● Java – J2SE 1.4/1.5, portable, easy deployment ● Open source – Licenced under the GPL ● Full documentation – Tutorial/reference user documents, javadocs ● Scalable – Designed for use with large tables, not memory-limited – Switch between memory & disk caching of bulk data using pluggable Storage Policies

STIL ● Generic table I/O & processing library ● Presents abstract view of table to programmer ● Format-neutral I/O – VOTable, FITS, SQL, ASCII, CSV... Starlink Tables Infrastructure Library

VOTable FITS SQL query ASCII CSV... VOTable FITS SQL table ASCII HTML LaTeX Mirage... Table metadata Column metadata Cell data Name URL Description PARAMs, INFOs... Names Descriptions Units, UCDs Types,... StarTable interface Cell values Table Processing Input handlers Output handlers random-access sequential

STIL VOTable features ● Fully implements VOTable standard – Understands TABLEDATA, BINARY, FITS (inline or by URL) – VOTable 1.0 or 1.1 versions ● Flexible options for writing VOTables – Can write TABLEDATA, BINARY, FITS (inline or by URL) ● Full access to structure of VOTable document – Table-aware SAX (streamed) parsing – Table-aware DOM (memory-based) processing ● Best-efforts parsing of non-conforming documents – Processing is mainly FIELD and DATA elements – Non-catastrophic schema changes easy to cope with

Table-aware DOM ● Parser builds normal org.w3c.dom document in memory ● Only small (non-DATA) elements are included ● For table data, use ((TableElement)el).getData() ● This gives you a StarTable which provides table elements ● Don't need to worry wheter it's TABLEDATA, BINARY or FITS <COOSYS equinox=”J2000.0” epoch=”J2000.0” system=”eq_FK5”/> AAAD0NoYXJsZXMgTWVzc2llc gAAAAtNYXJrIFRheWxvcg== getData() StarTable

Table-aware SAX ● Extends SAX (streaming) XML parser with VOTable- aware callback methods ● Parsing code receives notification for start, end and each row of TABLE data as well as XML elements, entities, etc public interface TableHandler { void startTable( StarTable metadata ); void rowData( Object[] row ); void endTable(); } public interface ContentHandler { void startDocument(); void endDocument(); void startElement( name, atts...); void endElement( name,... ); void characters( char-data,... ); } Standard SAX handler STIL extension for VOTables

Axis VOTable de/serialization ● STIL provides custom serializer/deserializers for use with Axis web services ● JAX-RPC calls send StarTable objects over the wire as VOTable documents ● In principle, streaming means this works with large tables ● (but there are problems at Axis 1.1)

TOPCAT ● View/analyse/edit/write tabular data – Manipulate table data and metadata – Plot, match, sort, calculate, select,... – Powerful and extensible expression language ● Based on STIL – Multi-format, including VOTable (all encodings) & SQL – Large tables (10^6 rows x 10^2 columns) Tool for OPerations on Catalogues And Tables

STILTS ● Command-line tools for table manipulation ● Generic table tools: – tcopy: format conversion – tpipe: table pipeline processing ● VOTable structure-sensitive tools – votcopy: DATA encoding conversion – votlint: VOTable document validation ● Current release is beta STIL Tool Set

votcopy ● Converts VOTable DATA encodings ● Can write all defined encodings: – TABLEDATA, BINARY, FITS ● Choose STREAM element output: – inline ( encoding='base64' ) – external ( href=' ' ) ● All non-data elements are untouched ● Useful for: – humans looking at STREAM data – conversion to STREAM prior to transmission –... <COOSYS equinox=”J2000.0” epoch=”J2000.0” system=”eq_FK5”/> % votcopy -f tabledata AAAD0NoYXJsZXMgTWVzc2llc gAAAAtNYXJrIFRheWxvcgQaY Ue0Gkyys/xso9PoiMSj4ZmZo +H753PxqfvggpdtyAQL5DtwA KQAEMkC+QADQGd/72lEZzAKQ... % votcopy -f binary % votcopy -f fits -href -base data

votlint ● Many VOTables are in violation of the standard – Makes it hard for parsing software – Data might get garbled ● DTD/Schema validation misses many errors: – Content of TD/STREAM elements can't be checked against FIELDs – arraysize attribute values may be meaningless – ref/ID crossreferences might make no sense – etc, etc, etc... ● votlint parses a VOTable document and warns of trouble – Doesn't pick up everything, but it's a lot better than not using it ● Useful for VOTable authors and parser developers ● If you are producing VOTables or VOTable-writing software - please use it to check your VOTables are legal!

votlint example <VOTABLE xmlns:xsi=” xsi:schemaLocation=” xmlns=” version=”1.1”> A very small table Fomalhaut d0 Running votlint on this valid document: gives these warnings: INFO (l.4): No arraysize for character, FIELD implies single character WARNING (l.15): Characters after first in char scalar ignored (missing arraysize?) ERROR(l.17): Bad double string d0 WARNING (l.19): Wrong number of TDs in row (expecting 3 found 4) ERROR (l.22): Row count (1) not equal to nrows attribute (2)

Summary ● STIL – Table-aware DOM – Table-aware SAX ● STILTS – votcopy – votlint ● Other features available – TOPCAT, generic parts of STIL & STILTS – Applications session tomorrow, or talk to me ● The future is uncertain...