Workflow languages and engines breakout Matthew Addis IT Innovation 5 December 2003 NeSC workshop on workflow services

Workflow languages and engines breakout Objective:Objective: –Better understand the requirements for workflow languages and engines in scientific applications –Allow comparison of work already done by projects employing scientific workflow/dataflow ApproachApproach –Identify the different areas/types of requirements –Identify the different levels at which these requirements might exist using a stack type approach

Requirements areas PerformancePerformance SchedulingScheduling DiscoveryDiscovery Events/monitoring/reportingEvents/monitoring/reporting Fault toleranceFault tolerance ScalabilityScalability Launching/invocation/executionLaunching/invocation/execution Steering/interaction/controlSteering/interaction/control ManageabilityManageability

Stack

Positioning of existing systems

Some characteristics to consider when differentiating existing approaches Execution policies/approaches, e.g. data flowsExecution policies/approaches, e.g. data flows Models and structures: e.g. DAGsModels and structures: e.g. DAGs Data model and typesData model and types We need to pull together and reuse existing body of work in this areaWe need to pull together and reuse existing body of work in this area

Plumbing group Streams are important in scienceStreams are important in science –Not supported in commercial systems DependabilityDependability –Detection of failures –Propagation of exceptions –Handling OptimisationOptimisation –Quality of service –Networks, computers, sets of resources –Choices of data formats and transfer mechanisms ManageabilityManageability –Monitor whats going on –Control over execution and services –Dynamic adaptation of workflow

Next steps Establish discussion groupEstablish discussion group More work on describing existing systems with respect to the areas weve identifiedMore work on describing existing systems with respect to the areas weve identified ReportReport Ideally:Ideally: –Common research and development –What is the smallest reference architecture that satisfies most of what we want