Workflow Management Concepts and Requirements For Scientific Applications

The Two Tier View of a Workflow Management system (generic)

Workflow layers –Control-flow layer –Application and Software Tools layer –I/O System layer –Storage and Network Resource layer

Anatomy of a scientific workflow management system Control Flow Layer Applications & Software Tools Layer I/O System Layer Storage & Network Resouces Layer Flow Tier Work Tier + Data Mover Simulation Program Parallel R Post Processing Terascale Browser Task A: Generate Time-Steps Task B: Move TS Task C: Analyze TS Task D: Visualize TS Parallel NetCDF PVFSLN HDF5 Libraries SRM

Architecture of a workflow management system Design time (Process modeling) Run time Post run time Process Design Tool Process Database Workflow engine Invoke Tasks Tracking Tools Workflow execution history database Data Mining & Analysis Tool Application code Software Tools

Main Services Offered by Workflow Management Systems Workflow design tools (GUI interfaces) Automatic sequencing of component invocation Synchronizing data flow between components Tracking and reporting mechanisms

Discussion items Control flow tier –Granularity of tasks, sub-workflows –Task Invocation - Web Services, Corba, Wrappers, Callbacks –Human tasks: Notifications and alerts, steering –Dataflow streaming granularity –Performance expectations (provoke alerts) Work Tier –Workflow engine for scientific applications –Integrated dataflow management –Failure detection and recovery –Data-driven flow control –Performance-driven flow control –Workflow optimization –Run-time resource coordination

Discussion items - Granularity of tasks, sub-workflows - Task Invocation - Human tasks: Notifications, alerts, steering - Dataflow streaming granularity - Performance expectation - Workflow engine for scientific applications - Integrated dataflow management - Failure detection and recovery - Data-driven flow control - Performance-driven flow control - Workflow optimization - Run-time resource coordination Research and Development Hardening and Packaging Deployment and maintenance X X X X X X X X X X X X

Other topic discussed Simulations setup –reserve resources, schedule run Performance monitoring needed –e.g. Disks getting full –e.g. Some task is stalled or too slow Interoperation of components –Match outputs to inputs (e.g. CCA) Allow feedback loops How to specify what to do in case of failures/exceptions

Example Workflow Systems Considered Scientific –Ptolemy/Kepler – UC Berkeley –DAGMAN – U Wisc, used in Griphyn –SciRun – U Utah, collaborate with CCA –(Taverna) –(Triana) Commercial –FileNet –Oracle Workflow –IBM’s MQ flow