Workflow Enactment in ICENI Dr Andrew Stephen M C Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre 2 nd September 2004

2 ICENI: Imperial College e-Science Network Infrastructure Collect and provide relevant Grid Meta-Data Pluggable architecture Test Architecture for Grid Research Foundation for higher-level Services and Autonomous Composition Integrated Grid Middleware Solution Interoperability between architectures, APIs Added value layer to other middleware Usability: Interactive Grid Workflows Role and policy driven security ICENI Open Source licence (extended SISSL) The Iceni, under Queen Boudicca, united the tribes of South-East England in a revolt against the occupying Roman forces in AD60.

3 Workflow Enactment  Applications consist of a number of components linked together in a dataflow manner  User specifies work as abstract workflow  The abstract workflow needs to be mapped down to a set of component implementations which will run on resources (concrete workflow) Mapping Workflow Graph over Resource Graph

4 Workflow Enactment Linear Equation Source Linear Equation Solver Display Vector Results General Equation generato r LU Factorisatio n Simple Vector Display

5 The Architecture Scheduler Launcher Reservation Service Performance Store Application Service Reservation Engine Workflow Grid Container

6 Enactment Pipeline Scheduling Framework Grid Container Launching Framework AdvertiserLauncher Concrete Workflow & JDML AM Concrete Workflow Abstract Workflow SRRIIMPR

7 Scheduling Service Scheduling Framework Application Mapper - Generates the possible mappings of Components to resources Scheduling Algorithm -Algorithm to select where to deploy components Listen out for services -Launcher Services -Reservation Services -Performance Services

8 Stages in the Scheduling Framework Abstract Workflow Workflow with Implemen- tations Workflow with Implemen- tations Workflow with Implemen- tations Application Mapper Workflow with Implementatio- ns & Resource Workflow with Implementatio- ns & Resource Workflow with Implementatio- ns & Resource Workflow with Implementatio- ns & Resource Workflow with Implementatio- ns & Resource Resource Mapper Workflow with Implementatio- ns & Resource Workflow with Implementatio- ns & Resource Workflow with Implementatio- ns & Resource Prune Resources Workflow with Implementatio- ns & Resource Concrete Workflows Scheduler Selects set of “best” Concrete Workflow Make Reservations JDML Generate JDML Scheduling Algorithm

9 Reservations & Delayed Execution Scheduling Framework Grid Container Launching Framework AdvertiserLauncher Concrete Workflow & JDML AM Concrete Workflow Abstract Workflow SRRIIMPR Launching&Res F-work AdvertiserLauncher Grid Container Green Room Reservation Service

10 Reservations time → Reservation Service Scheduler Workflow Reserve (workflow) HOLD Conflict HOLD Reserve WS-Agreement Request interval Linear Equation Source Linear Equation Solver Display Vector Results Reservations not possible on Users Desktop

11 Launcher -Converts a JDML document into a platform specific job Launcher -Converts a JDML document into a platform specific job Launching Service Launching Framework Reservation - Provides mechanism for reservations to be made Advertiser -Generate a document for each resource available from this Launcher Launcher -Converts a JDML document into a platform specific job Launcher Factory -Generates a Launcher for each job submitted to the Launching Service

12 Grid Container  Provided an environment for components to run within  Instantiates the components that should be launched locally  Performs the choreography between components  Responsible for inter-component communication

13 Grid Containers Grid Container Linear Equation Source Linear Equation Solver Vector Display

14 Summary  Have developed an end to end Pipeline for deploying workflows  Pluggable architecture for experimentation  ICENI is evolving Towards Web Services  Further research is required in scheduling algorithms and reservation techniques

15 Acknowledgements  Director: Professor John Darlington  Research Staff: Nathalie Furmento, Stephen McGough, William Lee Jeremy Cohen, Marko Krznaric, Murtaza Gulamali Asif Saleem, Laurie Young, Jeffery Hau David Mcbride, Ali Afzal  Support Staff Oliver Jevons, Sue Brookes, Glynn Cunin, Keith Sephton  Alumni: Steven Newhouse, Yong Xie, Gary Kong James Stanton, Anthony Mayer, Angela O’brien  Contact: 