A PPARC funded project Workflow and Job Control in Astrogrid Jeff Lusted Dept Physics and Astronomy University of Leicester

7 Dec 2003Workflow and Job Control in Astrogrid 1 Astrogrid – UK's first Virtual Observatory To interrogate multiple data centers in a seamless and transparent way Provide powerful new analysis and visualization tools Give data centers and providers a standard framework for publishing and delivering services

7 Dec 2003Workflow and Job Control in Astrogrid 1 Standards Background Open source Web Services - Pluggable components Movement towards grid-based computing XML based Current implementation in Java Server side development GUI – browser based with some possible use of components applets (visualization tool)

7 Dec 2003Workflow and Job Control in Astrogrid 1 Problem Space: Developer's Large quantities of data Archives distributed around the world Historically based RDBMs of various persuasions or roll-your-own Different sources Data not necessarilly held in uniform manner

7 Dec 2003Workflow and Job Control in Astrogrid 1 Problem Space: Science – Astronomer's Brown Dwarf Selection What is the contribution of brown dwarfs to the stellar mass budget? Brown dwarfs are intrinsically faint and rare objects, so their detection is not straightforward Current solution requires local access to some of the relevant databases but also download of significant quantities of data from other datasets to a local workstation. VO Solution - the ability to make queries combining attributes of astronomical objects stored in different databases.

7 Dec 2003Workflow and Job Control in Astrogrid 1 Problem Space 2: Science – Layman's Cone Search Something interesting has happened in this area of the sky. Search the same area in other archives around the world Search other sources (xray, infrared) Obviously search is time based, with observations potentially extending over many years

7 Dec 2003Workflow and Job Control in Astrogrid 1 Simple Workflow Step One – search archive for some interesting data Step Two – search another archive for similar Step Three – merge the two sets of results in a meaningful way Step Four – analyse the results, perhaps for visualization

7 Dec 2003Workflow and Job Control in Astrogrid 1 Astrogrid Components VOSpace – Manager and Server Datacenter – Query tools and hosting Registry – publishing Portal – includes Workflow design and submission JES – the Job Entry System Application Controller Tools Message Log Community

7 Dec 2003Workflow and Job Control in Astrogrid 1 Workflow Designs XML based Designed within the portal environment Held as files within VOSpace On job submission, the relevant design is sent to the Job Entry System

7 Dec 2003Workflow and Job Control in Astrogrid 1 Workflow Example...

7 Dec 2003Workflow and Job Control in Astrogrid 1 Job Entry System

7 Dec 2003Workflow and Job Control in Astrogrid 1 Job Example xxxxxxx

7 Dec 2003Workflow and Job Control in Astrogrid 1 JES Components JobController Security check Inserts job data into the system Keeps an audit trail – the original job document JobScheduler Dispatches one or more job steps Interrogates Registry for location information JobMonitor A listener on the ApplicationController When job step finishes, decides whether job has finished The JES Database Required for reliability and recoverability Updates have transactional semantics

7 Dec 2003Workflow and Job Control in Astrogrid 1 JES Cooperators ApplicationController Deals with one job step Location dependent on tool type... Bottom leaf: Tool instance Command line tool colocated with ApplicationController Webservice based tool can be remote

7 Dec 2003Workflow and Job Control in Astrogrid 1 Interesting Points (one hopes) Workflow execution is managed centrally by JES A workflow design is both workflow descriptor and input message. Each step represents a tool invocation - “where” can be decided at run-time. Resources will often be references rather than in- stream. Referenced resources need to be managed To minimize data movement, data and tools should be co-located

7 Dec 2003Workflow and Job Control in Astrogrid 1 Astrogrid Home Page