The Knowledge-based Workflow System for Grid Applications Ladislav Hluchý, Viet Tran, Ondrej Habala II SAS, Slovakia

May 9-11, 2007, ManchesterEGEE User Forum2 K-Wf Grid: Partners  Fraunhofer FIRST (Berlin, Germany)  UIBK (Innsbruck, Austria)  IISAS (Bratislava, Slovak Republic)  CYFRONET (Cracow, Poland)  LogicDIS S.A. (Athens, Greece)  Softeco Sismat SpA (Genoa, Italy)

May 9-11, 2007, ManchesterEGEE User Forum3 K-Wf Grid: Consortium K-Wf Grid: Consortium Bratislava Athens Genova Kraków Innsbruck Berlin

May 9-11, 2007, ManchesterEGEE User Forum4 K-Wf Grid: Objectives  Objectives of K-Wf grid To enable users to create complex workflows and use grid resources without detailed knowledge of grid To enable users to create complex workflows and use grid resources without detailed knowledge of grid To construct workflows optimized for underlying infrastructure, using its advantages and avoiding its bottlenecks To construct workflows optimized for underlying infrastructure, using its advantages and avoiding its bottlenecks To (semi-)automatically construct workflows based on user’s requirements, using semantic annotation of services, data, applications and resources To (semi-)automatically construct workflows based on user’s requirements, using semantic annotation of services, data, applications and resources To constantly renew information about the grid by using complex monitoring network – to learn from experience To constantly renew information about the grid by using complex monitoring network – to learn from experience To provide simple, easy-to-use interface to K-Wf Grid services To provide simple, easy-to-use interface to K-Wf Grid services

May 9-11, 2007, ManchesterEGEE User Forum5 Knowledge-based Workflow System for Grid Applications – K-Wf Grid Execute workflow Capture knowledge Reuse knowledge Monitor environment Analyze information Construct workflow

May 9-11, 2007, ManchesterEGEE User Forum6 K-Wf Grid: Advantages  Composition of workflow from a set of services System composes the workflow for you – just tell him what you want to get at the end System composes the workflow for you – just tell him what you want to get at the end System uses services which are available at the time and which are expected (based on past experience) to provide good results (good = what you want) System uses services which are available at the time and which are expected (based on past experience) to provide good results (good = what you want)  Usability Less grid language, more application domain language Less grid language, more application domain language Integrated collaboration interfaces Integrated collaboration interfaces  Reuse of components K-Wf Grid is based on respected standards K-Wf Grid is based on respected standards

May 9-11, 2007, ManchesterEGEE User Forum7 K-Wf Grid: Workflow Refinement Web Service Classes Web Service Operations Abstract Operation Grid Resources User Request “ ” “I want this data” “ ” “There is my data!”

May 9-11, 2007, ManchesterEGEE User Forum8 K-Wf Grid: Pilot applications  PAs as domain-related instances of a typical K-Wf Grid-based application system Common horizontal features / Specific vertical customization Common horizontal features / Specific vertical customization  Flood Forecasting Simulation Cascade (FFSC, PA1) Chain of simulations, targeting hydraulic simulation of a flooded area Chain of simulations, targeting hydraulic simulation of a flooded area Stateful WSRF services implementation in Java, using Globus Toolkit 4 Stateful WSRF services implementation in Java, using Globus Toolkit 4  Enterprise Resource Planning (ERP, PA2) Stateless web services in Java (Tomcat/Axis) Stateless web services in Java (Tomcat/Axis)  Coordinated Traffic Management (CTM, PA3) Traffic simulation in Genoa, Italy Traffic simulation in Genoa, Italy Developed as stateless web services in Perl Developed as stateless web services in Perl

May 9-11, 2007, ManchesterEGEE User Forum9 K-Wf Grid: PA1 – FFSC  Flood Forecasting Simulation Cascade a cascade of simulation models for weather forecasting, hydrological and hydraulic simulations, able to produce predictions of possible floods in target areas a cascade of simulation models for weather forecasting, hydrological and hydraulic simulations, able to produce predictions of possible floods in target areas  Input – map of area, precipitation, river water level etc.  Output – water flow in target area  Suggestions / user support – applicability of different models to specific scenarios (QoS) – comparison between stored predictions and real data – future dispersion forecasting

May 9-11, 2007, ManchesterEGEE User Forum10 Flood Forecasting Application (PA1-FFSC)  Flood forecasting based on a series of simulations  Several simulation models for Meteorology Meteorology Hydrology Hydrology Hydraulics Hydraulics  2D/3D visualization  Implementation using GT4 WSRF  Being extended More models More models More services – towards risk management More services – towards risk management

May 9-11, 2007, ManchesterEGEE User Forum11 Replica location Metadata catalog Central services Information index

May 9-11, 2007, ManchesterEGEE User Forum12 Replica location Metadata catalog Central services Information index Monitoring service

May 9-11, 2007, ManchesterEGEE User Forum13 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management

May 9-11, 2007, ManchesterEGEE User Forum14 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management Portal

May 9-11, 2007, ManchesterEGEE User Forum15 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management Portal Execute workflow Monitor environment Analyze information Capture knowledge Reuse knowledge Construct workflow

May 9-11, 2007, ManchesterEGEE User Forum16 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management Portal Execute workflow Monitor environment Analyze information Capture knowledge Reuse knowledge Construct workflow

May 9-11, 2007, ManchesterEGEE User Forum17 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management Portal Execute workflow Monitor environment Analyze information Capture knowledge Reuse knowledge Construct workflow

May 9-11, 2007, ManchesterEGEE User Forum18 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management Portal Execute workflow Monitor environment Analyze information Capture knowledge Reuse knowledge Construct workflow

May 9-11, 2007, ManchesterEGEE User Forum19 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management Portal Execute workflow Monitor environment Analyze information Capture knowledge Reuse knowledge Construct workflow

May 9-11, 2007, ManchesterEGEE User Forum20 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management Portal Execute workflow Monitor environment Analyze information Capture knowledge Reuse knowledge Construct workflow

May 9-11, 2007, ManchesterEGEE User Forum21 Porting to gLite  In K-Wf grid, applications are implemented as WSRF services  In gLite, the services are replaced by a set of worker jobs that are submited by workflow manager via gLite WMS When started, the worker jobs will contact to workflow manager for executing tasks When started, the worker jobs will contact to workflow manager for executing tasks

May 9-11, 2007, ManchesterEGEE User Forum22 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management Portal Execute workflow Monitor environment Capture knowledge Reuse knowledge Construct workflow gLite WMS

May 9-11, 2007, ManchesterEGEE User Forum23 Replica location Metadata catalog Central services Information index Monitoring service Knowledge store Information processing Workflow management Portal Execute workflow Monitor environment Capture knowledge Reuse knowledge Construct workflow gLite WMS

May 9-11, 2007, ManchesterEGEE User Forum24 Int.eu.grid  Goals Deploy and operate an interoperable production-level e-Infrastructure for demanding interactive applications Deploy and operate an interoperable production-level e-Infrastructure for demanding interactive applications Distributed (MPI-) parallel interactive computing and storage at Tera level Distributed (MPI-) parallel interactive computing and storage at Tera level User friendly access through a interactive Grid desktop User friendly access through a interactive Grid desktop Support for virtual organizations at all levels: Support for virtual organizations at all levels:  Setup, Collaborative environment, Grid enhancement of applications, Execution and monitoring tools  Project details Project start: May 1, 2006 Project start: May 1, 2006 Project duration: 24 months Project duration: 24 months  Contribution of UI SAV Operation of computer cluster with 32 CPUs and 32GB RAM as part of the project production testbed Operation of computer cluster with 32 CPUs and 32GB RAM as part of the project production testbed User support, organization of tutorials and users’ trainings User support, organization of tutorials and users’ trainings Support for environmental applications and their users Support for environmental applications and their users Implementation of interactivity to flood simulation application Implementation of interactivity to flood simulation application Support for gridification and interactivity of radiocative pollution simulation aplication (owned by MicroStep-MIS) Support for gridification and interactivity of radiocative pollution simulation aplication (owned by MicroStep-MIS)

May 9-11, 2007, ManchesterEGEE User Forum25 Int.eu.grid - Partners  BIFI, Zaragoza/Spain  CESGA, Santiago de Compostela/Spain  CSIC-IFCA, Santander/Spain  CYFRONET, Cracow/Poland  FZK, Karlsruhe/Germany  GUP, Linz/Austria  HLRS, Stuttgart/Germany  ICM, Warsaw/Poland  LIP, Lisbon/Portugal  PSNC, Poznan/Poland  TCD, Dublin/Ireland  UAB, Barcelona/Spain  UI SAV, Bratislava/Slovakia

May 9-11, 2007, ManchesterEGEE User Forum26 Int.eu.grid - Interactivity Migrating Desktop Remote Access Server Job Submission Services gLogin CrossBroker Computing Element Gatekeeper LRMS Worker node 0 Worker node 1 Worker node N … JDL Job control Interactive channels Interactive input Interactive output

May 9-11, 2007, ManchesterEGEE User Forum27 Control Output/visualizations /state

May 9-11, 2007, ManchesterEGEE User Forum28 Control Output/visualizations

May 9-11, 2007, ManchesterEGEE User Forum29 Flood Interactivity in Int.eu.grid  Job is internally controlled by a workflow manager Job steps can be added/removed during runtime Job steps can be added/removed during runtime Can be paused/restarted/modified Can be paused/restarted/modified  Job outputs/visualized data are available through the MD interface  Job may be cloned State = workflow state + intermediate data, may be easily transferred State = workflow state + intermediate data, may be easily transferred Good for parameter studies Good for parameter studies

May 9-11, 2007, ManchesterEGEE User Forum30 Conclusion  Current research focuses on SOKU (Service Oriented Knowledge Utilities) K-Wf Grid is one of the early implementations of SOKU concepts K-Wf Grid is one of the early implementations of SOKU concepts How to adapt infrastructure research to this shift in paradigm? How to adapt infrastructure research to this shift in paradigm?  Application developers & end users need easy access to grid infrastructure SOKU is the way to achieve this SOKU is the way to achieve this How to extend gLite towards SOKU? How to extend gLite towards SOKU?