Fujitsu Laboratories of Europe © 2003 A Broker/Scheduler Architecture for Grid Services Dr. David Snelling Fujitsu Laboratories of Europe Open Issues in Grid Scheduling October 21-22, 2003 How to Build a Schprokerer

Fujitsu Laboratories of Europe © 2003 What is a Schprokerer? The Function of Optimizing Workload on the Grid. It is not Controlling or Monitoring that Workload. Not the Unicore NJS or GRAM. Does Include: Bids, Optimization, SLAs, etc... What is Workload? Resource usage over time Workload Profile Not Necessarily Workflow See sidebar.

Fujitsu Laboratories of Europe © 2003 Workflow Sidebar What is Workflow? A description workload consisting of more than one activity. Activity = task | job | process | function A net to describe the dependencies between activities. DAGs, DFGs, Pert Charts, BPDs,... Categories of Workflow Task or Job (partial) sequences Data/Task/Message Flow Workflow Description Types Static vs. Dynamic (changing) Persistent vs. Dynamic (on demand) Explicit (DAGs etc.) vs. Implicit (message sets) Stateful vs. Stateless

Fujitsu Laboratories of Europe © 2003 Workflow Sidebar Continued Extended Functions Loops and Conditionals Error management Workload Exertion Types Deployment InvocationEntities Jobs Processes or Services Other workflows People Tokens Implicit or explicit on dependency arcs. From BPMN © BPMI

Fujitsu Laboratories of Europe © 2003 Functions - 1 Resource Publication Resource Abstraction Compute # of atoms to processor time on machine X. Resource Reservation Issue of Tickets Generation of Bids for Resource Use Cost + QoS User Incarnation Authorization checkingNegotiation SLAs, WS-Agreement,...

Fujitsu Laboratories of Europe © 2003 Functions - 2 Gather Performance Information Gather Adherence Information Were SLAs Honored? Gather Load Information Locally and Grid wide. Make Commitments on behalf of other Schprokerers Optimize Workload versus Policies A scheduler for example.

Fujitsu Laboratories of Europe © 2003 Functions - 3 Translate Information Models Ontology for resource description Convert Grid Currency between VOs Decide to Migrate Workload Compute/Data workload and network workload Not actually migrate the workload Decide to Preempt Workload Not actually preempt the workload Locate Necessary Resources

Fujitsu Laboratories of Europe © 2003 Functions - 4 Aggregate Workflow Task Bids E.g. Bidding for complete Workflows Predicting Actual Workload of a WL description E.g. Workflow loop, conditionals, etc. Topology checking Data not at the end of a thin pipe. Prediction of Resource Loading

Fujitsu Laboratories of Europe © 2003 OGSA Perspective - I

Fujitsu Laboratories of Europe © 2003 OGSA Perspective - II

Fujitsu Laboratories of Europe © 2003 OGSA Perspective - III

Fujitsu Laboratories of Europe © 2003 Architectural Assumption Use the WS-Agreement Approach Distributed Scheduling with Local Information From the Grid perspective, we can only plan Not Central scheduling based on distributed information The Grid cannot control all the resources therefore we cant schedule.

Fujitsu Laboratories of Europe © 2003 Class Relationships WorkloadSource Demand side Initiate Agreements Consign Work Workload Describes the work WorkloadDescription Describes resource requirements Term in agreement WS-Agreement Manifest the contract WorkloadManagement Supply side Provide load info. InfoService Publish information OntologyService Translation of resource description WorkloadOptimizer Implement workload management policy Broker Supply vs Demand Policy Optimization Scheduler Collective, Temporal Optimization Dynamic Runtime workload optimization

Fujitsu Laboratories of Europe © 2003 CRC Analysis - 1 WorkloadSource: Demand resources Initiate Agreements Consign workload to ResourceManager Create workload Workload: The work Fully describe the work to be done Provide WorkloadDescriptions Subtypes: Workflow, Job, Task, WS-deployment, WS- invocation

Fujitsu Laboratories of Europe © 2003 CRC Analysis - 2 WorkloadDescriptions: Resource request Define the (abstract) resource requirements of a workload Part of an Agreement initiation WS-Agreement: Contract Encapsulate agreed terms of usage contract Define contract between WorkloadSource and ResourceManager ResourceManager: Supplier of resources Access control and management of resources Supply load information etc. to InformationService Notify GridMonitor of workload deltas etc.

Fujitsu Laboratories of Europe © 2003 CRC Analysis - 3 InformationService: Global Information Source Maintain and publish information Gather data from: ResourceManager LoadPredictor GridMonitor GridSpies Filter information via ResourceOntologyMapper ResourceOntologyMapper: Translation Translate representations of resources Provide to InformationService only?

Fujitsu Laboratories of Europe © 2003 CRC Analysis - 4 WorkloadOptimizer: Meet policy requirements Implement workload management policy Create/Negotiate Agreements with WorkloadSources Check with AuthorizationService Subtypes: Broker: Supply vs. Demand policy agreements Check with the GridBank (exchange rates only) Scheduler: Collective, temporal optimization DynamicOptimizer: Runtime workload adaptation

Fujitsu Laboratories of Europe © 2003 Functions - 1 Resource Publication - InformationService Resource Abstraction - Broker Compute # of atoms to processor time on machine X. Resource Reservation - ResourceManager Issue of Tickets - ResourceManager Generation of Bids for Resource Use - ResourceManager Cost + QoS User Incarnation - AuthorizationService Authorization checking Negotiation - WorkLoadOptimizer and WorkloadSource SLAs, WS-Agreement,...

Fujitsu Laboratories of Europe © 2003 Functions - 2 Gather Performance Information - GridMonitor Gather Adherence Information - GridSpies Were SLAs Honored? Gather Load Information - GridMonitor Locally and Grid wide. Make Commitments on behalf of other Schprokerers - Broker Optimize Workload versus Policies - WorkloadOptimizer A scheduler for example.

Fujitsu Laboratories of Europe © 2003 Functions - 3 Translate Information Models - ResourceOntologyMapper Ontology for resource description Convert Grid Currency between VOs - GridBank Decide to Migrate Workload - DynamicWorkloadOptimizer Compute/Data workload and network workload Not actually migrate the workload Decide to Preempt Workload - DynamicWorkloadOptimizer Not actually preempt the workload Locate Necessary Resources - InformationService

Fujitsu Laboratories of Europe © 2003 Functions - 4 Aggregate Workflow Task Bids - Scheduler E.g. Bidding for complete Workflows Predicting Actual Workload of a WL description - Scheduler E.g. Workflow loop, conditionals, etc. Topology checking - Scheduler Data not at the end of a thin pipe. Prediction of Resource Loading - LoadPredictor