NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 1 Virtual Observatories...or how to do your research from a beach in the Bahamas rather than stuck up a cold mountain in the middle of the night...

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 2 Who I am John Taylor: Work on the AstroGrid and VOTech Virtual Observatory projects Member of Edinburgh's Wide Field Astronomy Unit – curates catalogue and image data from SuperCOSMOS and (soon) WFCAM and (less soon) VISTA Software engineer NOT Astronomer

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 3 Outline The data deluge Why science is changing The scale of the problem and technical issues Catalogues How to use them VOTables, FITS files Visualisation tools The Virtual Observatory AstroGrid Demonstration

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 4 I: The astronomer's world is changing... Previously an Astronomer was a “radio astronomer” or a “x-ray astronomer”. Now you need to be an expert in all wavelengths. The move towards Survey Astronomy Federating data from different instruments. Vast quantities of auto-generated data See:

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 5

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 6 Supernova Remnant Cassiopeia-A – a 300 year old Supernova Shocks seen in the X- ray Chandra image Heavy elements seen in the optical Dust shows in the IR Images from Chandra Science Centre Slide: Nic Walton, AstroGrid Project Scientist Mapping e¯s in the magnetic field as revealed by Radio data

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 7 Data Deluge (Ex-)Concorde (15 km) CD stack with 1 year LHC data (~ 20 km) SuperCOSMOS: 2.5 Tb WFCAM: 14Tb per year VISTA: 4Tb per week Physicists, Astronomers, Biologists all facing the same problem Steve Lloyd: /talks/Data_Deluge_Bru nel_2005.ppt

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 8 Nearby Spiral Galaxy x-ray optical radio

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 9 II: Catalogue and Image Servers Cone searches Data formats: FITS and VOTables Visualising the data: Topcat and Aladin More sophisticated searches using SQL Demo of the SuperCOSMOS Science Archive

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 10

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 11 Example: SuperCosmos Science Archive Scans of the UK Schmidt telescope plates Catalogue data & Image data 3 wavebands, 2 epochs Over 1 billion objects 2.5 terabytes in size

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 12 Data Formats FITS – the astronomer's jpeg images/spectra/tables Includes metadata VOTable “xml” format for tables The VO standard UCDs describe what a quantity “is”

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 13 Visualising the data TOPCAT For viewing and manipulating tables, especially VOTables Create new columns Filter the sources Output the table in different formats Aladin Good for viewing image files Overlay catalogue tables Create RGB images Access data archives such as Vizier Aims to be a general “Virtual Observatory” portal

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 14 Complex queries: SQL SQL is a general language for querying databases select ra,dec,mag FROM sources where ra>0 and ra -45 and dec 23 give me ra, dec and mag values (use * to get everything) table name (each db will have several) the following conditions are true

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 15 SQL II Plenty of resources on the web for learning SQL e.g. There is an astronomy variant of SQL called ADQL (astronomy data query language) It's basically SQL with a few extra bits thrown in (a CIRCLE keyword for doing cone searches, and an XMATCH keyword for cross-matching data between databases) [Cross Matching: Taking two or more different database tables from (maybe) different sources and joining them together based on their RA and DEC]

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 16 Some databases SIMBAD - Basic data on objectshttp://simbad.u- strasbg.fr/Simbadhttp://simbad.u- strasbg.fr/Simbad SuperCOSMOS Science Archive SDSS (Sloan Digital Sky Survey) 2mass (2 micron all sky survey) XMM sa/index.shtml sa/index.shtml

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 17 Databases of databases Vizier Datascope bin/vo/datascope/init.plhttp://heasarc.gsfc.nasa.gov/cgi- bin/vo/datascope/init.pl NED (Nasa Extragalactic Database) OpenSkyQuery

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 18 This presentation will be available at:

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 19 A recap on yesterday Is the SSA now behaving? Last night's observing group: how to use SQL to get your list of galaxies from SDSS select g.ra,g.dec,g.objId, s.z from Galaxy g, ELRedShift s where g.specObjID=s.specObjID and s.z>0 and s.z 0 and

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 20 III: Virtual Observatories: What's a virtual observatory? What do you need to make one? AstroGrid – glue for the Euro-VO Demo

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 21 The International Virtual Observatory

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 22 Virtual Observatory “The VO vision can be summed up as the desire to make all archives speak the same language − all searchable and analysable by the same tools, all data sources accessible through a common interface, all data held in distributed databases that appear as one.” Andy Lawrence, 09/2003 Take all the world's Astronomical data, and all the world's applications and computing power and make it appear that they reside on the user's desktop.

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 23 Science Cases – Brown Dwarf Location Brown Dwarfs are cool, faint and (if can see them) close Roughly speaking: Query optical and infrared catalogues for objects that are “bright” in the infra red, but not in the optical and have large proper motions. Requires use of more than one catalogue More: “The AstroGrid 10” Problems Problems

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 24 AstroGrid Components PAL CEA

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 25 Example workflow: calculating photometric redshifts

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 26 AstroGrid Components

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 27 Demo

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 28 Contacts and info Contacts: John Taylor Institute for Astronomy, Royal Observatoy of Edinburgh Background Info on AstroGrid Download the Euro-VO AstroGrid software from:

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 29 What now? AstroGrid Release 1 is up and running now

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 30 Locating data and applications IVO-standard registry Registries harvest from each other Xml-based query language ADQL

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 31 Workflow Orchestrate services Queries and workflows are designed using the portal – other clients on the way Work is run remotely and asynchronously Archives searched and results manipulated Results are stored in a virtual file system Queries and workflows can be re-used and shared

A PPARC funded project AstroGrid Intro & Demo John Taylor Institute for Astronomy, Edinburgh

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 33 Building a VO - Layer Diagram