ICENI Overview & Grid Scheduling Laurie Young London e-Science Centre Department of Computing, Imperial College

2 ICENI IC e-Science Networked Infrastructure Developed by LeSC Grid Middleware Group Collect and provide relevant Grid meta-data Use to define and develop higher-level services Interaction with other frameworks: OGSA, Jxta etc. The Iceni, under Queen Boudicca, united the tribes of South-East England in a revolt against the occupying Roman forces in AD60.

3 ICENI (The Big Picture) Private ResourceManager Policy Manager CR SR Identity Manager Domain Manager CR SR Gateway between private and public regions Public Resource Browser Public Computational Community SR CR Public Computational Community SR Private Administrative Domain SR CR Resource Broker Application Design Tools Component Design Tools Application Mapper Web Services Gateway Applicatio n Portal Computational Resource Software Resources Network Resources Storage Resources JavaCoG Globus

4 ICENI Stack Portal Interface Application Construction & Deployment ICENI Middleware Grid Fabric

5 Web Portals Handheld wireless devices become ubiquitous –Personal Digital Assistants, Mobile Phones –Secure access any time, any place, any where Use X.509 certificates embedded in a browser to authenticate user’s identity Integration portal infrastructure with ICENI –EPIC: Use component meta-data to build portal application Goal: Provide secure ‘one stop shop’ for e- science

6 EPIC: e-Science Portal at Imperial College Collaborative LeSC industrial project with Sun Microsystems Develop a secure portal infrastructure to: –Access your own personal environment –Applications to support day-to-day e-science –Interaction with other Grid infrastructures Allow role based access to resources –Anonymous: public web pages –Students: internal pages, , compute resources –Staff: restricted pages

7 ICENI Application Model Legacy code! Component Applications –Compose applications from many components –Component does work on data –Component communicates data

8 Component Motivation Logical application model Collaborative software authoring Promote component reuse and sharing Simplify application construction Enable deployment to diverse Grid resources: –Communication Selection –Implementation Selection

9 Layered Abstraction Meaning Behaviour dataflow abstract data types Implementation control flow threads etc. performance, architectures, concrete data type may have many may each have many Implementation

10 Component’s View of the Grid Other Code SOAP More Code RMI My Code You must implement a provided interface You may call methods provided by the middleware Context object

11 Visual Component Composition

12 Grid Container Deployment of Components Component Design Tools Scientist Application Design Tools End User Application Description Document Developer Implementation Annotating Tools Code Run-Time Representation Application Mapper RTR Code Access Resource Information APO Application Proxy Object Repository

13 SOAP RMI Component Execution Compute Resource Hardware RTR Code RTR Network Resource MPI APO Jini OGSA, Jxta, etc.

14 Components as Services Component Service interface SOAP (or other) protocol Context object

15 ICENI & Jini: P2P

16 Web Services Architecture

17 Synergy

18 Grid Service Contracts Jini Lookup Service DRMAA Resource DRMAA Client

19 Grid Service Contracts Jini Lookup Service DRMAA Resource User: A+B Duration:1hr DRMAA Client User:B Resource Browser

20 Grid Service Contracts Jini Lookup Service DRMAA Resource User: A+B Duration:1hr DRMAA Resource User:A Duration:10m DRMAA Client User:B DRMAA Client User:A

21 OGSA & Jini Integration Jini Lookup Service Gateway Manager GSI enabled Web Service Hosting Environment DRMAA Resource User: A+B Duration:1hr DRMAA Resource User:A Duration:10m

22 OGSA & Jini Integration Jini Lookup Service Gateway Manager GSI enabled Web Service Hosting Environment Jini Client Interface WSDL Interface DRMAA Resource User: A+B Duration:1hr DRMAA Resource User:A Duration:10m

23 OGSA & Jini Integration Jini Lookup Service Gateway Manager GSI enabled Web Service Hosting Environment GSI + SOAP Connection Jini Client Interface WSDL Interface DRMAA Resource User: A+B Duration:1hr DRMAA Resource User:A Duration:10m

24 OGSA & Jini Integration Jini Lookup Service Gateway Manager GSI enabled Web Service Hosting Environment GSI + SOAP Connection User Info SOAP->Java Jini Client Interface WSDL Interface DRMAA Resource User: A+B Duration:1hr DRMAA Resource User:A Duration:10m

25 Application Mapping (Scheduling) Architecture –How meta-data is collected –What meta-data is required Scheduling Algorithms –Map components onto resources for “best” results –Meta-data dependent decisions

26 Scheduling Architecture Resources ICENI App Builder (GUI) Component Repository Performance Models SchedulerLauncher

27 Multiple Metrics (1) “It is the goal of a scheduler to optimise one or more metrics” (Feitelson & Rudolph) Generally one metric is most important –Application Optimisation Execution time Execution cost –Host Optimisation Host utilisation Host throughput Interaction Latency

28 In a Grid Environment there are three application optimisation based important metrics –Start time ( ) –End time ( ) –Cost ( ) Relative importance varies on a user by user and application by application basis Multiple Metrics (2)

29 A Benefit Function maps the metrics we are interested in to a single Benefit Value metric Different benefit functions represent different optimisation preferences Combining Metrics – Benefit Fn

30 Optimisation Preferences Cost Optimisation Time Optimisation Cost/Time Optimisation

31 Schedule Benefit Each component and communication has a benefit function Each resource and network connection has a predicted time & cost for each component or communication that could be deployed Fit the tasks onto the resources to get the maximum Total Predicted Benefit

32 Graph Oriented Scheduling (1) Applications are described as a graph –Nodes represent application components –Edges represent component communication Resources are described as a graph –Nodes represent resources –Edges represent network connections

33 Graph Oriented Scheduling (2) Condor pool AtlasSaturn Viking DesignAnalyse Scatter Gather Mesh DRACS Mesh DRACS Mesh DRACS Factory

34 Graph Oriented Scheduling (3) Condor pool ScatterGather Design Atlas Factory Analyse Saturn Viking

35 Summary Component framework provides: –Rich application meta-data –Decoupled component definition and implementation Meta-Data: –Exploit performance information to map component implementation to the ‘best’ resources Resource Broker: –Resource selection through user defined policies: Minimise cost using computational economics Minimise execution time using the application mapper

36 Acknowledgements Director: Professor John Darlington Technical Director: Dr Steven Newhouse Research Staff: –Anthony Mayer, Nathalie Furmento –Stephen McGough, James Stanton –Yong Xie, William Lee –Marko Krznaric, Murtaza Gulamali –Asif Saleem, Laurie Young, Gary Kong Contact: – –