Resource Brokering on Complex Grids EUROGRID and GRIP Presented by John Brooke ESNW October 3/4 UK/Japan N+N

January 17, 2003Contents Aims of the resource broker Functionality of the ancestral broker EUROGRID broker Interoperability architecture (UNICORE-Globus) Towards a resource description ontology Relation to the GGF and OGSA

Organization Firewalls Users Virtual Organisation Brokers System Brokers Compute Resources Brokers as VOs

Client Broker Service Client Broker Service Resource Broker Service VO LayerSpecialist LayerSite Layer Replication Federated Brokering

Interoperability for Brokering We want to broker on Grids controlled by either UNICORE or Globus. In GRIP we developed two methods 1.Bifurcation, separate “sub-brokers” for a Globus or a Unicore Grid. This is achieved and is extensible to a limited extent. 2.Constructing an extendable resource broker utilising a Grid Resource Ontology to handle mappings of resource terms.

1 Ancestral EUROGRID Broker The API allows two levels of operation: Resource Checking: Static requirements, capability and capacity. QoS Checking: Performance vs cost. Tickets can be issued as a “guarantee”. Protocol can be used symmetrically by Broker. User Broker NJS Execution NJS Execution NJS 1 CheckQoS 2 CheckQoS 3 CheckQoS_Outcome 4 CheckQoS_Outcome

Brokering for Workflows Bezier SGI Manchester Vtk + VizServer Dirac SGI QMUL LB3D with RealityGrid Steering API Laptop SHU Conference Centre UNICORE Gateway and NJS Manchester Firewall SGI OpenGL VizServer Simulation Data VizServer client Steering GUI The Mind Electric GLUE web service hosting environment with OGSA extensions Single sign-on using UK e-Science digital certificates UNICORE Gateway and NJS QMUL Steering (XML)

Interoperable Broker – Method 1 1.The Network Job Supervisor (NJS) delegates the Resource Check to the Broker at the Vsite. 2.The UNICORE brokering track utilises the IDB exactly as for the ancestral broker. 3.The Globus track uses a translator of the QoS check object. The translation service is extendable. 4.The results of the translation are used to drive the LDAP search and the Globus broker then utilises MDS to perform this. UNICORE NJS 4.0 gave much greater power and flexibility in brokering for complex workflows.

NJS Broker Unicore Broker Globus Broker IDBTranslatorFilter Basic Translator MDS(GRIIS/GRIS) Delegates resource check Lookupresources Delegates translation Uses to drive LDAP search Performs Diagram Diagram Of Broker Architecture Architecture – Method 1

Pros and Cons A nice feature of Method 1 is that no alteration needs to be made to the client side of UNICORE, thus no alteration for application plugins or “expert” brokers Also no alterations need to be made to Globus. However the UNICORE description of Grid resources is very different from the MDS-2 description. MDS-2 does not publish software resource and user environment, Unicore does not check dynamic resource, e.g. machine loading. The need for resource description translation is thus highlighted.

NJS Broker Unicore Broker Globus Broker IDB Translator Filter Ontology engine Resource Discovery Service Delegates resource check Lookupresources Delegates translation Uses to drive MDS search Hierarchical Grid Search Diagram Diagram Of Broker Architecture Architecture – Method 2 Filter Uses to Drive MDS Search Hierarchical Grid Search OtherBrokers Resource Discovery Service

Ontologies Defines knowledge domain and allows reasoning on this domain. If we can create a Grid Resource Ontology, creation of specialist translation classes from basic Grid translator becomes possible. IDB at sites can be created via ontology, it contains site specific information which the clients job specification cannot do. So brokers take client request formulated in RR space, at each site use translator to convert to RR space, offers come back with capability and QoS.

BrokerNJS IDB TSI/HostGT3 Gateway Client Broker NJS Host Gateway Client R-GMA NJS Host Site-Wide Brokering Normal EUROGRID/GRIP Brokering Local Brokering Configurations

Persistent Virtual Environments Metascheduling Service Workflow Manager Clients Broker Chargeable Schedulable GridServices Other Brokers Site Feedback Policy Manager Resource Usage Monitor Banking Services Brokering and OGSA Services

1 Relevant GGF work Grid Protocol Architecture-RG : Core Grid Functions Grid Resource Allocation Agreement Protocol-WG :advanced reservation, co-allocation CIM-based schema-WG : successor to LDAP GESA-WG looking at economic issues of scheduling The recently-formed Semantic Grid RG is very interested in the Grid Resource Ontology idea.

1 Points for Discussion What is the relationship between brokering and scheduling? How to deal with legacy (not Grid-aware) schedulers? How to relate the ontologies from the application side (Resource Requestor) to the service provision side (Resource Provider)? How does a broker estimate upper and lower bounds for turnaround time? How far does the broker trust information from the service provider. Should it monitor running workflows?

January 17, 2003GRIP First Project Review1 RR space RP space RR space request Request referral sync Figure 1: Request from RR space at A mapped into resource providers at B and C, with C forwarding a request formulated in RR space to RP space at D. B and C synchronize at end of workflow before results returned to the initiator A. A B C D RR and RP Spaces