Introduction to Observational Catalogues & Databases Duncan Law-Green (LEDAS Archive Scientist) 21st February 2007

Introduction What this seminar is about Introduction to selection of online astronomical data resources to help with your research. Outline of data formats and catalogue tools. Practical demos. What this seminar is about Introduction to selection of online astronomical data resources to help with your research. Outline of data formats and catalogue tools. Practical demos. What this seminar is not about Data reduction and analysis techniques. Computer science of databases, SQL programming, how to build your own databases. What this seminar is not about Data reduction and analysis techniques. Computer science of databases, SQL programming, how to build your own databases. Introduction to Observational Catalogues & Databases: 21/02/2007

Use of Catalogues/Archives Literature searches Build science case for your proposal Literature searches Build science case for your proposal Archive retrieval Previous observations of your objects Archive retrieval Previous observations of your objects Feasibility studies Is the observation possible with your instrument? Feasibility studies Is the observation possible with your instrument? Observation planning Source positions, instrument FOV, guide stars etc. Observation planning Source positions, instrument FOV, guide stars etc. Class studies Filter catalogues for interesting sources/outliers Class studies Filter catalogues for interesting sources/outliers Cross-correlation Compare source positions in different catalogues Cross-correlation Compare source positions in different catalogues

Data & Metadata Data A set of measured parameters for a source. Data A set of measured parameters for a source. Metadata Data about the dataset, provides vital context info. Examples include: coordinate epoch, date of observations, observing mode, filter bandpass, pipeline processing performed etc. Data without metadata is useless. Some data formats preserve metadata, some don't. Metadata Data about the dataset, provides vital context info. Examples include: coordinate epoch, date of observations, observing mode, filter bandpass, pipeline processing performed etc. Data without metadata is useless. Some data formats preserve metadata, some don't. Introduction to Observational Catalogues & Databases: 21/02/2007

Data Formats I ASCII (CSV/TSV: Comma/Tab-Separated Variables) Advantages: Easy to generate, easy to read. Will ingest directly into, e.g. Excel Disadvantages: No direct metadata support. No integrity support. No documentation. Bulky. ASCII (CSV/TSV: Comma/Tab-Separated Variables) Advantages: Easy to generate, easy to read. Will ingest directly into, e.g. Excel Disadvantages: No direct metadata support. No integrity support. No documentation. Bulky.

Data Formats II FITS (NOAO), HDS, NDF (Starlink) Advantages: Structured formats, include metadata. Multi-dimensional (tables, images, datacubes). Well-defined formats, good software support. Binary format, compact. Disadvantages: Varying compatibility between FITS, HDS, NDF. Conversion may affect metadata. No semantics. FITS (NOAO), HDS, NDF (Starlink) Advantages: Structured formats, include metadata. Multi-dimensional (tables, images, datacubes). Well-defined formats, good software support. Binary format, compact. Disadvantages: Varying compatibility between FITS, HDS, NDF. Conversion may affect metadata. No semantics.

Data Formats III VOTable (IVOA, AstroGrid) Advantages: Structured format, metadata+semantics. Human-readable, supported by modern software. Uses XML, existing tools to generate and check integrity. Disadvantages: Uses XML – very bulky. Multidimensional support awkward. Standards evolving. VOTable (IVOA, AstroGrid) Advantages: Structured format, metadata+semantics. Human-readable, supported by modern software. Uses XML, existing tools to generate and check integrity. Disadvantages: Uses XML – very bulky. Multidimensional support awkward. Standards evolving.

Semantics What a data column means in physical terms What a data column means in physical terms Unified Content Descriptors (UCDs) Created by IVOA (International Virtual Observatory Alliance) as standard controlled vocabulary of keywords to describe physical nature of table columns. Current system “UCD1+” R.A. (Main):pos.eq.ra;meta.main Source ID:meta.id;src Radio flux ratio: phot.flux;em.radio;arith.ratio UCDs are feature of well-constructed VOTables. Intended to ease automated data handling, “workflows” Unified Content Descriptors (UCDs) Created by IVOA (International Virtual Observatory Alliance) as standard controlled vocabulary of keywords to describe physical nature of table columns. Current system “UCD1+” R.A. (Main):pos.eq.ra;meta.main Source ID:meta.id;src Radio flux ratio: phot.flux;em.radio;arith.ratio UCDs are feature of well-constructed VOTables. Intended to ease automated data handling, “workflows”

VOTable Velocities and Distance estimations <PARAM name="Telescope" datatype="float" ucd="phys.size;instr.tel" unit="m" value="3.6"/> <FIELD name="RA" ID="col1" ucd="pos.eq.ra;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Dec" ID="col2" "pos.eq.dec;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Name" ID="col3" ucd="meta.id;meta.main" datatype="char" arraysize="8*"/> <FIELD name="RVel" ID="col4" ucd="src.veloc.hc" datatype="int" width="5" unit="km/s"/> <FIELD name="e_RVel" ID="col5" ucd="stat.error;src.veloc.hc" datatype="int" width="3" unit="km/s"/> <FIELD name="R" ID="col6" ucd="phys.distance" datatype="float" width="4" precision="1" unit="Mpc"> Distance of Galaxy, assuming H=75km/s/Mpc continued on next slide...

VOTable (cont.) N N N continued from previous slide...

Treeview File format viewer Can read multiple file formats, display hierarchical structures, expand and collapse nodes with click of mouse. Some basic plotting, image, stats routines. File format viewer Can read multiple file formats, display hierarchical structures, expand and collapse nodes with click of mouse. Some basic plotting, image, stats routines.

Literature Search ADS: NASA Astrophysics Data Service 3 bibliographic databases Astronomy & Astrophysics (1.2 million) Physics (3.6 million) ArXiv e-prints (400,000) Searchable by author, subject, title, object, abstract text, full-text... Links to full PDFs of articles, object catalogues, data tables. MyADS Update Service: subscribe to updates ADS: NASA Astrophysics Data Service 3 bibliographic databases Astronomy & Astrophysics (1.2 million) Physics (3.6 million) ArXiv e-prints (400,000) Searchable by author, subject, title, object, abstract text, full-text... Links to full PDFs of articles, object catalogues, data tables. MyADS Update Service: subscribe to updates Not peer-reviewed! adsabs.harvard.edu

“Telegrams” IAU Circulars: Central Bureau for Astronomical Telegrams Central clearinghouse for info on transient events (comets, solar system bodies, novae, supernovae etc.). Subscribe by . Search via ADS IAU Circulars: Central Bureau for Astronomical Telegrams Central clearinghouse for info on transient events (comets, solar system bodies, novae, supernovae etc.). Subscribe by . Search via ADS Astronomers' Telegram Primarily high-energy transient events (GRBs etc.). Subscribe via or RSS. Searchable web interface, LEDAS Astronomers' Telegram Primarily high-energy transient events (GRBs etc.). Subscribe via or RSS. Searchable web interface, LEDAS

Data Servers Catalogue Servers Surveys and article data. Searchable by position, filter by various parameters, output data in ASCII, FITS, VOTable. (examples: NED, HEASARC, LEDAS, ViZieR) Catalogue Servers Surveys and article data. Searchable by position, filter by various parameters, output data in ASCII, FITS, VOTable. (examples: NED, HEASARC, LEDAS, ViZieR) Image Servers Images of the sky at various wavelengths. Output data in bitmap (GIF,JPG,PNG) or FITS image. May or may not be “science grade”. (examples: DSS-I/II, SDSS, SkyView, Aladin) Image Servers Images of the sky at various wavelengths. Output data in bitmap (GIF,JPG,PNG) or FITS image. May or may not be “science grade”. (examples: DSS-I/II, SDSS, SkyView, Aladin) Archive Servers Repository of public data from particular observatory or mission. Various formats, may need specialist software or training to interpret. (examples: Hubble, MERLIN, Chandra, LEDAS). Archive Servers Repository of public data from particular observatory or mission. Various formats, may need specialist software or training to interpret. (examples: Hubble, MERLIN, Chandra, LEDAS).

Aladin Advanced image and catalogue search system Advanced image and catalogue search system Plot catalogue search results directly on image Plot catalogue search results directly on image Highly versatile, write scripts for repetitive operations. Highly versatile, write scripts for repetitive operations. Launch via CDS website, LEDAS site or directly on desktop. Launch via CDS website, LEDAS site or directly on desktop. RTFM! RTFM! Introduction to Observational Catalogues & Databases: 21/02/2007

Aladin Plane stack Object data Preview Button bar Main window

Aladin Introduction to Observational Catalogues & Databases: 21/02/2007 Multiview option Splitscreen button

TOPCAT Catalogue plotting, editing and filtering tool Catalogue plotting, editing and filtering tool Cross-correlations between catalogues Cross-correlations between catalogues Introduction to Observational Catalogues & Databases: 21/02/2007

TOPCAT

Cross-correlation Search for matching positions in 2 or more catalogues (e.g. “does this X-ray source have a radio counterpart?”) Search for matching positions in 2 or more catalogues (e.g. “does this X-ray source have a radio counterpart?”) Consider positional uncertainties, statistical probability of chance coincidence Consider positional uncertainties, statistical probability of chance coincidence Convenient tool for 2, 3, 4-way catalogue matches in TOPCAT (Joins -> Pair Match etc.) Convenient tool for 2, 3, 4-way catalogue matches in TOPCAT (Joins -> Pair Match etc.) Introduction to Observational Catalogues & Databases: 21/02/2007

Virtual Observatory (VO) International project to simplify access to astronomical catalogues and archives. Coordinated by IVOA. International project to simplify access to astronomical catalogues and archives. Coordinated by IVOA. Standard set of access commands (“protocols”), all databases “appear” the same on the network. Standard set of access commands (“protocols”), all databases “appear” the same on the network. UK VO project AstroGrid, developed additional software for distributed search, distributed storage, workflows etc. UK VO project AstroGrid, developed additional software for distributed search, distributed storage, workflows etc. Introduction to Observational Catalogues & Databases: 21/02/2007

AstroGrid Workbench Workbench: Desktop Java application for VO searches and data processing. Workbench: Desktop Java application for VO searches and data processing. Check availability of data Check availability of data Execute simultaneous searches across multiple catalogues/servers Execute simultaneous searches across multiple catalogues/servers Construct “workflows”: drag+drop editing of data gathering/reduction/analysis pipelines. Construct “workflows”: drag+drop editing of data gathering/reduction/analysis pipelines. Save results to “MySpace”, temp scratch space Save results to “MySpace”, temp scratch space Introduction to Observational Catalogues & Databases: 21/02/2007

VOPlot Product of VO India project Product of VO India project Reading, interactive plotting and filtering tool for catalogue data (primarily VOTables) Reading, interactive plotting and filtering tool for catalogue data (primarily VOTables) Introduction to Observational Catalogues & Databases: 21/02/2007

And finally... Stellarium ( Free planetarium software, impress your friends!... Questions to Seminar slides, URLs to appear on my webspace Stellarium ( Free planetarium software, impress your friends!... Questions to Seminar slides, URLs to appear on my webspace Introduction to Observational Catalogues & Databases: 21/02/2007