AstroGrid: Past, Present & Future Bob Mann Institute for Astronomy & National e-Science Centre University of Edinburgh

My background Research astronomer, not CS or IT Research astronomer, not CS or IT –Interested in evolution of galaxies & clusters Needs multiwavelength data Needs multiwavelength data Finding & matching catalogues is tedious! Finding & matching catalogues is tedious! –Want to automate data integration Hence interest in the nascent AstroGrid project Hence interest in the nascent AstroGrid project Half-time post in AstroGrid ( ) Half-time post in AstroGrid ( ) Research Leader at National e-Science Centre ( ), also half-time Research Leader at National e-Science Centre ( ), also half-time

Two talks at this workshop Now: Now: –AstroGrid: Past, Present & Future Tomorrow: Tomorrow: –Astronomy and e-Science: Opportunities & Challenges

AstroGrid Past, Present and Future

Outline of talk Past Past –Origins in UK e-Science Programme Present Present –Completion of AstroGrid-1 at end of 2004: a working data grid for UK astronomy Future Future –AstroGrid-2 and VOTech How might AstroGrid and Aus-VO mesh? How might AstroGrid and Aus-VO mesh?

Past Origins in the UK e-Science Programme

The UK e-Science Programme Major government initiative Major government initiative – : £118M = A$283M – : £115M = A$276M Funding: Funding: –e-Science Core Programme –Programmes for each Research Council Particle Physics & Astronomy: £56M=A$134M Particle Physics & Astronomy: £56M=A$134M –Astronomy: ~£12M=A$29M AstroGrid: ~£8M=A$19M AstroGrid: ~£8M=A$19M

e-Science Core Programme Network of Regional e-Science Centres and Centres of Excellence Open Middleware Infrastructure Institute Digital Curation Centre

AstroGrid Funded by PPARC e-Science programme Funded by PPARC e-Science programme –£3.7M = A$8.9M in (AG1) –£4M = A$9.6M in (AG2) to build a data grid for UK astronomy, which will form the UK contribution to a global Virtual Observatory AstroGrid is about VO infrastructure AstroGrid is about VO infrastructure

AstroGrid-1: AG1 Phase A: Sept 2001 – Dec 2002 AG1 Phase A: Sept 2001 – Dec 2002 –Analysis, R&D, Architecture AG1 Phase B: Jan 2003 – Dec 2004 AG1 Phase B: Jan 2003 – Dec 2004 –Build, test & deliver ~26 staff (23.4 FTE) ~26 staff (23.4 FTE) Edinburgh, Leicester, Cambridge, MSSL, Jodrell Bank, RAL, QUB Edinburgh, Leicester, Cambridge, MSSL, Jodrell Bank, RAL, QUB Part of EU-funded Astrophysical Virtual Observatory (AVO) project Part of EU-funded Astrophysical Virtual Observatory (AVO) project

Development Approach Iterative Iterative –full cycle each iteration –3 month(-ish) iterations Incremental Incremental –add functionality each iteration –refactor wherever necessary Use-case driven Use-case driven –from key 10 science cases Architecture led Architecture led

Some technical choices Open Source throughout Open Source throughout Modular architecture Modular architecture Write everything in Java Write everything in Java Everything browser-based Everything browser-based –no standalone applications Build with Web Services first Build with Web Services first –then move to Grid services (?) Open project Open project –all docs & most debate on wiki & forum

Present Completion of AstroGrid-1 at end of 2004: a working data grid for UK astronomy

Where are we now? Yellow = AG1 Yellow = AG1 Mauve = AG2 Mauve = AG2 Blue = external Blue = external We do have the yellow bits working now …sort of We do have the yellow bits working now …sort of

The pre-VO world NEDDB CDS Locally installed tools & local storage

AstroGrid Schematic NEDDB CDS Locally installed tools & local storage Registry Portal MySpace MS Compute Node MS Tool

AstroGrid Schematic NEDDB CDS Locally installed tools & local storage Registry Portal MySpace MS Compute Node MS Tool Job Execution

AstroGrid Schematic NEDDB CDS Locally installed tools & local storage Registry Portal MySpace MS Compute Node MS Tool Job Execution

MySpace Distributed file management system Distributed file management system –In the VO, you wont just use local disk –Allows third-party delivery of data User sees single logical view of all files User sees single logical view of all files –Manipulates them with file browser Implementation a little clunky now Implementation a little clunky now –Needs automation and scalability

Community & Auth/Auth Most astronomical data is open access Most astronomical data is open access –But the VO must support proprietary rights VO workflow model needs single sign-on VO workflow model needs single sign-on –Credentials must pass between services AstroGrid needs some Authentication and Authorization for the writable MySpace AstroGrid needs some Authentication and Authorization for the writable MySpace System is based on Communities from which individuals rights are derived System is based on Communities from which individuals rights are derived –Basic: more work on this in AstroGrid-2

Common Execution Architecture Remember: AstroGrid dont do tools Remember: AstroGrid dont do tools –But we want to use existing ones CEA provides web service wrapper to existing command line tools CEA provides web service wrapper to existing command line tools –CEA toolkit wraps and registers tools All CEA-wrapped services have same WSDL All CEA-wrapped services have same WSDL Easy to use – even by non-experts Easy to use – even by non-experts Developing this as an IVOA standard Developing this as an IVOA standard

Future AstroGrid-2 and VOTech

AstroGrid-2 & VOTech AstroGrid-2 AstroGrid-2 –£4M = A$9.6M for –New partners, bringing in new expertise: Bristol, Exeter, Leeds, Portsmouth Bristol, Exeter, Leeds, Portsmouth VOTech VOTech –3.3M = A$5.5M for –AstroGrid + ESO + French & Italian groups

Completing the picture

AstroGrid-2 plans Extension Extension –Infrastructure –Grid (GT4/WSRF/?) integration –Intelligent resource discovery (semantic grid) –Full security: Shibboleth/Internet2 integration? Shibboleth/Internet2 integration? Additional components Additional components –Data Mining & Visualization –Client-based access: VO Workbench – not just web-based portal VO Workbench – not just web-based portal

VOTech plans Design studies for a European VO Design studies for a European VO –New Infrastructure –New User Tools –Intelligent Resource Discovery –Data Exploration Coupling data mining and visualization Coupling data mining and visualization

How might AstroGrid and Aus-VO mesh?

AstroGrid vs Aus-VO AstroGrid AstroGrid –One of the big VO players –Aims to deliver full VO infrastructure –Doesnt do tools Aus-VO Aus-VO –Limited size – cant do everything –World-leading strengths in some areas e.g. radio, optical spectroscopy, visualization e.g. radio, optical spectroscopy, visualization The projects seem very complementary

How do we progress this? Download AstroGrid-1 (from mid-Dec) Download AstroGrid-1 (from mid-Dec) –AstroGrid-in-a-box or the full release –Deploy the bits you want – its all modular –Improve them (and give them back to us!) Its all open source Its all open source Set up an Aus-VO registry Set up an Aus-VO registry –of published datasets and of tools/services What else? What else?

Summary & Conclusions AstroGrid-1 finishes in December 2004 AstroGrid-1 finishes in December 2004 –It will deliver a working VO infrastructure AstroGrid-2 and VOTech will enhance that AstroGrid-2 and VOTech will enhance that –And add tools – e.g. for data exploration All AstroGrid code is open source All AstroGrid code is open source –Take what you want; tell us/fix whats bad We should collaborate more than we do We should collaborate more than we do

Portal and Workflow Supports workflow paradigm (from the simple to the complex) Supports workflow paradigm (from the simple to the complex) Queries and workflows are designed using the portal Queries and workflows are designed using the portal Work is run remotely & asynchronously Work is run remotely & asynchronously Archives searched & results manipulated Archives searched & results manipulated Results are stored in a virtual file system Results are stored in a virtual file system Workflows can be re-used and shared Workflows can be re-used and shared

Enabling Resource Discovery Types of Registries:– Full, Publish, Special Types of Registries:– Full, Publish, Special Registry is the main focal point for all Astrogrid components Registry is the main focal point for all Astrogrid components Agreed Standards with IVOA Agreed Standards with IVOA –Search and harvest interface –OAI standard for harvest interface Adheres to the Astrogrid CEA standard Adheres to the Astrogrid CEA standard Types of resource Types of resource –Generic services, web services, applications, … –Data collections –AstroGrid-specific resources (e.g. MySpace servers) Use of XQuery language with eXist XML database. Use of XQuery language with eXist XML database. Harvesting Harvesting –US NVO –VizieR

Access Control AstroGrid has taken the lead in AAA because our system is based on writable storage AstroGrid has taken the lead in AAA because our system is based on writable storage Implement single-sign-on (SSO) authentication Implement single-sign-on (SSO) authentication –Using a community service (so, account can be used with a number of portals or other UIs) Programme-to-programme authentication based on message security with SOAP headers and WS-Security Programme-to-programme authentication based on message security with SOAP headers and WS-Security Facade API to hide details from implementations Facade API to hide details from implementations –Underneath, use JAX-RPC/Axis handlers to secure messages. Little authorization support in 2004, more in 2005 Little authorization support in 2004, more in 2005 –groups, quotas, community authorization, file sharing Currently specific to AstroGrid Currently specific to AstroGrid –Working with IVOA to develop common standard

Common Execution Architecture Uniform interface for running applications in VObs Uniform interface for running applications in VObs Set of WSDL and schema Set of WSDL and schema –Define the interactions in language independent way Adapters (ExecutionControllers) implemented Adapters (ExecutionControllers) implemented –Using Java Web services –To run legacy command-line applications –To run legacy HTTP GET/POST apps within the framework ExecutionControllers ExecutionControllers –Allow asynchronous operation of the applications –Callback and polling mechanisms for delivery of results Tightly integrated with JES (Job Execution System) Tightly integrated with JES (Job Execution System) –Allows programmable workflows

Publisher's AstroGrid Library Task: What's involved in publishing data to the VO? Task: What's involved in publishing data to the VO? Library of components Library of components –Make a data owners life a (relatively) straight-forward transition to a data owner/publisher Variety of plugins Variety of plugins –Cope with different collections of data (modes of storage) To make life easier for installers To make life easier for installers and test how they might be presented to the VO and test how they might be presented to the VO Ready-made web UI Ready-made web UI –Suitable for astronomers to use, –Includes useful configuration & monitoring features VO-compatible interfaces for tools to access the data VO-compatible interfaces for tools to access the data Features to communicate with other VO services Features to communicate with other VO services –E.g. Registries