RealityGrid: An Integrated Approach to Middleware through ICENI Prof John Darlington London e-Science Centre, Imperial College London, UK

Positioning Grids – transparent mapping of complex applications onto distributed machinery Routinely – for practising scientists c.f. heroic HPC

Requirements Simple application construction Automatic mapping to appropriate machines Automatic scheduling of activities Simple support for user interaction Simple support for collaboration Straightforward middleware installation and maintenance

No half-way house – need complete solution Requires automated use of considerable knowledge/intelligence previously provided manually Requires complete set of interoperable services for whole task Not necessarily monolithic single middleware solution (c.f. heroic middleware)

For Reality Grid Efficient execution of LB3D code in Grid environment Integrated support for collaborative steering and visualisation

The application Pipeline Deployment –Getting code and data to the resource Execution –Running the code –Steering Results Analysis –Visualisation Either after completion or real time

ICENI: Imperial College e-Science Network Infrastructure Integrated Grid Middleware Solution Interoperability between architectures, APIs Added value layer to other middleware Usability: Interactive Grid Workflows Deployment: Complete Install from Webstart Role and policy driven security Foundation for higher-level Services and Autonomous Composition ICENI Open Source licence (extended SISSL)

Usability Deployment ICENI Strands Component Programming Model Workflow Guided Scheduling Semantic Adaptation Role Based Access & Security Service Oriented Architecture ICENI

Deep Track: Tackle fundamental issues within Grids Focus on aspects relevant to RG scientists –Running jobs & selecting resources –Staging and managing data –Collaborative steering and visualisation –Controlled sharing of resources, data & knowledge Export solutions to other Grid activities –Promote best practice through RG experience –Lead & develop relevant grid standards

Development Infrastructure Project Website & mailing lists Daily build –Regression tests –On success binaries updated –Regenerated JavaDoc –Deployment tests CVS –Code split across multiple repositories & modules Documentation, manuals & user guides ICENI Open Source License (Extended SISSL) Java builds on Solaris & W2K Daily deployments on: rhea dirac Controlled open access to CVS source code Multiple repositories with defined release tags development branches Evolving 150+ page manual Installation & Configuration Deployment & Usage Developer & Contributors

Activity over the last year… Use ICENI to launch LB3D –Select which LB3D instance to use –Consider machine availability & basic performance –Wrapping LB3D as a binary component Visualise & steer LB3D through ICENI –Integration of fast track file-based steering ICENI testbed –Simplify deployment through Webstart –Wizards to simplify configuration

Resource Discovery & Initiation Select any machine with LB3D Select a specific machine with LB3D

Dynamic Discovery & Composition Register as running component services in the NetBeans user interface Deployed application Add new advertised components Drag-and-drop running component Execute to create new component instances and connect to application Application Visualisation Server

Collaborative Visualisation & Steering Integrated with ICENI Driven Access Grid! Visualisation server Application component Rendering engine 1 Rendering engine 2 Streamed to Access Grid Visualisation client 1 Visualisation client 2 Service Oriented Architecture Dataset A & B Dataset A Dataset B View of dataset A View of dataset B

Installation Mechanism and Control Centre Client Requirements: JRE Java Web Start (inc.) Internet Access Centralised configuration and service execution The ICENI Control Centre now has an installation ‘wizard’ that encapsulates configuration & execution for standard actions.

Wrapping up Legacy Code Legacy code can quickly be made available to the ICENI architecture

Story so far… within Grids Can submit & run jobs –But don’t necessarily know when they will run Collaborative visualisation & steering –Need to co-ordinate multiple resources Require predictability & guaranteed execution

Effective Interaction within the Grid Application Fabric Execution Performance model to predict resource requirements Obtain resource reservations Execution monitoring to build performance database

Service Architecture Scheduler Reservation Service Performance Store Launcher Application Service Reservation Engine

Scheduling Framework Application Mapper - Generates the possible mappings of Components to resources Scheduling Algorithm -Algorithm to select where to deploy components Listen out for services -Launcher Services -Reservation Services -Performance Services

Performance Store - Persistent Performance storage Performance Store - Persistent Performance storage Performance Repository Framework Performance Framework Performance Processing - Conversion of raw event times into performance data Data Collector -Collecting data on currently running applications (event times) Performance Store - Persistent Performance storage

Performance Events Events fired whenever ICENI components start or ports are accessed –Used to gather performance information about currently running application Events contain data relating to: –Time & application –Source component type, location & resource –Event type: start or port Events are serialised objects –Can be XML documents

Collection of Performance Results Data Collector Linear Equation Source Linear Equation Solver Display Vector Results Time Event 12:00 Linear Equation Source Start 12:04 Send out Equations 12:03 Linear Equation Solver Start 12:05 Receive Equations 12:12 ……….. Event: Start Linear Equation Source Performance Processing Workflow Performance Store

Fully exploit meta-data to infer temporal view of workflow Length: Execution Time Width: Resource Usage From the data flow and performance database infer the temporal workflow and thereby which component must be executed where and when. Reservations need to obtained from the grid fabric.

Launcher -Converts a JDML document into a platform specific job Launcher -Converts a JDML document into a platform specific job Launching Service Launching Framework Reservation - Provides mechanism for reservations to be made Advertiser -Generate a document for each resource available from this Launcher Launcher -Converts a JDML document into a platform specific job Launcher Factory -Generates a Launcher for each job submitted to the Launching Service

Launching Service Have a generic job submission system –This is being developed into an independent Web Service (GridSAM) GridSAM OMII Distribution SGE GridSAM Condor-G Tomcat & Axis GridSAM Shell Condor Application/ User

Evolution of ICENI

Summary Have demonstrated that transparent mapping is possible See the demo

Future work… Expansion of ICENI testbed with performance driven scheduling & reservations Use of simple WS to start jobs –Prototype in advanced development Integration of service based steering BINARY COMPONENT ICENI Steering BASIC APPLICATION FILE STEERED APPLICATION GS STEERED APPLICATION

Acknowledgements Director: Professor John Darlington Research Staff: –Nathalie Furmento, Stephen McGough, William Lee –Jeremy Cohen, Marko Krznaric, Murtaza Gulamali –Asif Saleem, Laurie Young, Jeffrey Hau –David McBride, Ali Afzal Support Staff: –Oliver Jevons, Sue Brookes, Glynn Cunin, Keith Sephton Alumni: –Steven Newhouse, Yong Xie, Gary Kong –James Stanton, Anthony Mayer, Angela O’Brien Contact: – 