C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Project: C3-Grid Collaborative Climate Community Data and Processing Grid Scheduling in C3-Grid: Requirements, Concepts, Realization Christian Grimme, Alexander Papaspyrou

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Outline Why C3-Grid? –Applications –Goals Architecture Applied technologies Scheduling- and data management- aware workflows –Orchestration of workflow components –Implementation details

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Why C3-Grid? Applications: –Earth System Science investigates anthropogenous influence on climate –Analysis of simulation data and field measurements –Huge amounts of inhomogenous data distributed over various sites Goals: –Development of a Grid-based infrastructure for Data discovery Data processing Data analysis

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Why C3-Grid? Simulation Data E E E E E ……… 2.11E E E …… 1.22E E E …… different format

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Architecture Nutzer User Interface API (Web Services)GUI DBMS/ File Available Resources Data Transfer Service Distributed Grid Infrastructure DMS (global) Meta Data Job Data Archive Interface DMS (local) Site Grid Infrastructure Distributed Processing Resources Distributed Data Archives Workflow Scheduler Pre- Processing User Grid Workspace Resource Information Service Resource Scheduler DIS Base Data & Meta Data File Management Data Archive Staging Job Submission Monitoring Search Harvesting Task Execution Matchmaking

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Applied Technologies Gridsphere as framework for portal (UI) services Grid core services with Globus Toolkit 4 Web services and WSRF GSI and Shibboleth for security JSDL and home-grown Workflow Specification Language (WSL)

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Scheduling- and Data Management- aware Workflows Jobs –Single (atomic) execution units –Execution in one step –Input data locally available –Output data stored locally Files –Datasets in workflow, that are created new (output) are used for job execution (input) –possibly reusable

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Workflow: Restrictions Data Job Data Job Data Job Data … … … Job ? X X Allowed workflow: Prohibited elements in a workflow: Circles, back references Conditionals

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Workflows: Orchestration Meta- Data Workspace Results Workspace DIS Harvester Portal Selection- Wizard Selection- Wizard WSL Scheduler WS GRAM DMS Data Provider 1 2a 2b Transfer HTTP Daemon HH Köln Data Job Data Job Data Job

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Details on Scheduling and Data Management Scheduler Info Service DMS Work- space Staging Infos & Functionality Sites & Programs Computing Resources MDS D RM Resource Management WS-GRAM Resource Scheduler OfferExeOfferData Exe + Data input Portal Interface Interface to Data Provider virtual

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop WSL: Workflow Specification Language