AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick

AstroGrid – The UK’s Virtual Observatory Outline of this Seminar Describe the different types and volume of data that astronomers use Discuss the concept of Virtual Observatories and how they might solve some of these problems Introduce AstroGrid as the UK’s Virtual Observatory

AstroGrid – The UK’s Virtual Observatory The Types and Volume of Data in Astronomy

AstroGrid – The UK’s Virtual Observatory Ground Based Observatories Copyright RASC Calgary Centre

AstroGrid – The UK’s Virtual Observatory Space Observatories Copyright NASA Copyright ESA Copyright ESA/D. Ducros

AstroGrid – The UK’s Virtual Observatory Data Products Provided by Observatories For many years astronomers have been provided the raw data products and were required to reduce the data to useable forms themselves. Increasingly, images and spectra are provided in a reduced form as they are reduced automatically through a pipeline.

AstroGrid – The UK’s Virtual Observatory Astronomical Surveys Sloan Digital Sky Survey or SDSS –Aim to map 25% of the sky –Image (through 5 coloured filters) around 100 million objects –Take spectra of around 1 million objects Dark Energy Survey or DES –Plan to collect data on 300 million galaxies –520 Megapixel camera installed on the existing 4-metre Blanco Telescope in Chile –Survey to begin in 2009

AstroGrid – The UK’s Virtual Observatory How many bytes? Multimission Archive –Nearly 50 Tb, over half is from the HST (Aug 2007) –Approx. 24 Tb served from Jan-Aug 2007 European Southern Observatory Archive –Approx. 65 Tb (Aug 2007) –Over 12 Tb (public data) served every year Sloan Digitized Sky Survey –40 Tb when complete Visible and Infra-red Survey Telescope for Astronomy (VISTA) –315 Gb per typical night, est. 150 Tb per year

AstroGrid – The UK’s Virtual Observatory How many astronomers are accessing these data? American Astronomical Society –Approximately 6500 members (source AAS Nov 2007) Royal Astronomical Society –Approximately 3000 members (source RAS Nov 2007)

AstroGrid – The UK’s Virtual Observatory Accessing Data Digital Audio Tape (DAT) and Linear-Tape Open (LTO) CDs, DVDs and increasingly Blu-Ray and HD DVD USB Flash Drives/Sticks Online –Common Gateway Interface (CGI) –Web Services –HTML Forms

AstroGrid – The UK’s Virtual Observatory Online Access - Multimission Archive

AstroGrid – The UK’s Virtual Observatory Online Archive - ESO Archive

AstroGrid – The UK’s Virtual Observatory Online Access - SDSS Skyserver

AstroGrid – The UK’s Virtual Observatory Virtual Observatories and how they can help

AstroGrid – The UK’s Virtual Observatory What are Virtual Observatories (VOs)? “The VO vision can be summed up as the desire to make all archives speak the same language – all searchable and analyzable by the same tools, all data sources accessible through a common interface, all data held in distributed databases that appear as one.” September 2003 Professor Andy Lawrence FRSE Institute for Astronomy Royal Observatory Edinburgh

AstroGrid – The UK’s Virtual Observatory The Virtual Observatory Aim The astronomer should be able to easily search all astronomical data worldwide from anywhere –A single user interface which can find/search all available data It should be possible to view and analyse the data using a common set of tools –The returned data products should use common formats

AstroGrid – The UK’s Virtual Observatory Creating a Virtual Observatory - An International Effort The International Virtual Observatory Alliance "facilitate the international coordination and collaboration necessary for the development and deployment of the tools, systems and organizational structures necessary to enable the international utilization of astronomical archives as an integrated and interoperating virtual observatory."

AstroGrid – The UK’s Virtual Observatory As a result of the IVOA we have A common Resource Metadata A common data querying language –Astronomical Data Query Language (ADQL) –Based on Structured Query Language (SQL) Common table format –Virtual Observatory Table (.vot)

AstroGrid – The UK’s Virtual Observatory Identifying these resources International Virtual Observatory Resource Name or IVORN –Very similar to a Uniform Resource Location (URL) –Ivo://community.authority.id/resource/etc… Example 1 - ivo://nasa.heasarc/hstpaec –HST planned and archived exposures catalogue –Provided by NASA’s High Energy Astrophysics Science Archive Research Center Example 2 - ivo://wfau.roe.ac.uk/sdssdr5-dsa –SDSS Data Release 5 –Provided by Wide Field Astronomy Unit at the Royal Observatory, Edinburgh

AstroGrid – The UK’s Virtual Observatory AstroGrid

AstroGrid – The UK’s Virtual Observatory What is AstroGrid? AstroGrid is a UK government funded, open source project designed to create a working Virtual Observatory (VO) for UK and International astronomers. Working closely with other VO projects worldwide through the International Virtual Observatory Alliance (IVOA), internationally recognised interface standards are emerging which promote true scientifc interoperability of astronomical data and processing resources worldwide. AstroGrid is a leading member of this community.

AstroGrid – The UK’s Virtual Observatory Who’s Involved? And others………

AstroGrid – The UK’s Virtual Observatory How does it all work? Server side applications –Dataset Access (DSA) –Common Execution Architecture (CEA) –Community and VOSpace –Registry Client side application - VO Desktop –Graphical User Interface –Scripted access to the VO –Client-side workflow (Taverna)

AstroGrid – The UK’s Virtual Observatory Dataset Access - the DSA Provides access to catalogues Server application connects to Relational Database e.g. MySql, Postgres IVOA cone interface (positional search) ADQL query interface –More complex queries, similar to SQL

AstroGrid – The UK’s Virtual Observatory Common Execution Architecture - the CEA Provides a method for computation code to be executed –Any command-line application, although may need to be “wrapped” in additional code –HTTP Post/Get applications Limited success with executing jobs on the National Grid Service May look to running jobs on the UCL Condor Pool

AstroGrid – The UK’s Virtual Observatory Community and VOSpace Community –Used to give users an identity in VOSpace –A method of authentication for running queries/jobs VOSpace –Personal file storage –Attached to community identity –Can execute jobs and save results online –Can access data from any location –ivo://uk.ac.ucl.star/DuganWitherick#atable.vot

AstroGrid – The UK’s Virtual Observatory The Registry Perhaps most important server application Allows astronomer or service to be able to locate, get details of, and make use of, any resource located anywhere in the IVO space Two types: –Full registry - searches (or harvests) other registries and maintains the entries in a list which is searchable by users or other services –Publishing registry - tells full registries what is available under that authority

AstroGrid – The UK’s Virtual Observatory How everything communicates - an example Client Registry DSA Server CEA Server VO Space Invoke Monitor Notification Read Results Resolve Read Inputs Write Results

AstroGrid – The UK’s Virtual Observatory Client Side Application - The VO Desktop Application developed in Java Currently in alpha form but beta released end of month User is presented with a single button “Start” When running, provides XML-RPC interface to the Virtual Observatory

AstroGrid – The UK’s Virtual Observatory The All-VOScope

AstroGrid – The UK’s Virtual Observatory PLatform for Astronomical Tool InterConnection - PLASTIC Communication protocol for client side virtual observatory tool Essentially allows tools to transmit and receive data and instructions from other tools Tools can instruct each other (e.g.): –To load virtual observatory tables –Highlight subsets of rows –Load an image of a particular area of sky For more information visit

AstroGrid – The UK’s Virtual Observatory VO Explorer

AstroGrid – The UK’s Virtual Observatory VO Explorer

AstroGrid – The UK’s Virtual Observatory The Task Runner

AstroGrid – The UK’s Virtual Observatory Client-side Science Scripting Python - easy to learn high level programming language import sys, xmlrpclib, os, time prefix = file(os.path.expanduser("~/.astrogrid- desktop")).next().rstrip() acr = xmlrpclib.Server(prefix+"xmlrpc") ag = acr.astrogrid userInfo = ag.community.getUserInformation()

AstroGrid – The UK’s Virtual Observatory Client Side Workflow - Taverna

AstroGrid – The UK’s Virtual Observatory Client Side Workflow - Taverna

AstroGrid – The UK’s Virtual Observatory AstroGrid is not… True Grid Computing –Could be loosely referred to as a data grid –Efforts are being made to get it working with true computational grids A solution to the entire Virtual Observatory –Focus has been on infrastructure The only way for astronomers to access online data

AstroGrid – The UK’s Virtual Observatory But it…. Is probably the most advanced of the Virtual Observatories Does provide an easy/heterogenous method for astronomers to access their data Will continue to be improved upon by the dedicated team of programmers as members of AstroGrid and other astronomers increasingly use the software and provide critical user feedback