© 2006 Open Grid Forum Workflow Management Research Group - WFM-RG q Chairs: Ian Taylor and Ewa Deelman Secretaries: Andrew Harrison and Matthew Shields GridNet2 Activities for WFM Research Group Matthew Shields 18th October 2007

© 2006 Open Grid Forum 2 Group Focus The Group was pretty quiet OGF 17, 18 and 19 Taylor became co-chair in April Shields and Harrison became secretaries New focus for the group based on use-case gathering in two specific areas: Workflow Sharing (OGF 20) Workflow Interoperability (OGF 21)

© 2006 Open Grid Forum 3 Group New Focus There’s a number of options for standards Standardise scientific workflow somehow Or accept there are a number of co- existing workflow systems Encourage reuse Encourage sharing What interfaces are needed for this? Are there use cases for sharing and interoperability?

There are many Workflow systems… 27 Chapters - from applications and environments BPEL, Taverna, Triana, Pergasus, Kepler, P-Grade, Sedna, ICENI, Java CoG Workflow, Condor, ASKALON, Swift, Petrinets, and so on … Successfully used - choice depends on requirements and politics :)

© 2006 Open Grid Forum 5 Motivation Focus Accept co-existing workflow representations/ environments How do we share/reuse workflows? Focus on the scientist performing the experiment How can sharing help him/her Workflow Interoperability Do we need this - use cases? Workflow embedding?

© 2006 Open Grid Forum 6 Interoperability Levels Enactment level Triana, Kepler etc. Representation level BPEL, SCUFL etc. Data level Files, XML Schema etc. Metadata level Provenance, data description etc. Community level cross-domain algorithms, results sharing etc.

© 2006 Open Grid Forum 7 Enactment Level Workflow enactment engines are complex. Usually have: Their own component architecture. Their own flow mechanisms e.g dataflow, control flow, mixture of the two. Their own history! Many person hours Specialisms in certain domains Designers/developers with particular research interests. Usually tightly coupled to their chosen workflow representation.

© 2006 Open Grid Forum 8 Representation Level Workflow languages for distributed computing differ, particularly when it comes to control structures e.g. ASKALAN’s Abstract Grid Workflow Language (AGWL) supports if, forEach, while. Triana taskgraphs are really just component dependency graphs They have different levels of abstraction And therefore are coupled to their target environment in different ways This is not bad - it means some languages are better suited to certain environments, other are not. The tight coupling between enactment engine and workflow representation means changing the enactment engine if the representation changes. Homogenising existing systems - destruction of history and domain and environment specific strengths

© 2006 Open Grid Forum 9 Data Level Data level interoperability allows workflows to be treated as black boxes What does it consume? What does it spit out? Data level is not Service level How do I get it consume and spit out? Data level is the pivot between the other levels: Once you have interoperable data you can: Represent and enact a workflow in your own favorite way You can contextualize the data with metadata, e.g provenance The data and metadata, can be shared in a community.

© 2006 Open Grid Forum 10 Workflow Interoperability Enactment level Representation level Data level Metadata level Community level Private Public

© 2006 Open Grid Forum 11 More Use Cases in The Area In OGF 20, there were 7 talks focused on sharing workflows In OGF 21, there are 4 talks on interoperability - workflow embedding Recently, there was an NSF workshop on interoperability We provide a summary of these ideas/ examples/workshops here:

© 2006 Open Grid Forum 12 OGF 20 - Manchester Standards work from WfMC, Oasis, OMG Gateways -> shared definition (XPDL) -> protocol compatibility (Wf-XML/ASAP) Sharing abstract vs concrete, taskgraph vs services Data intensive workflows, share optimisation information Sharing through social network Shibboleth extensions for workflow and service security Common model for securing service across VO QoS parameters across federated providers

© 2006 Open Grid Forum 13 OGF 21 - Seattle NSF/Mellon Workshop on Scientific and Scholarly Workflows Users don’t always want interoperability Academic exercise? Users do need to know capabilities of systems Challenges Fault tolerance, Parallelism Long running workflows Driving forces in interoperability Scientific use case, hybrid coupled model Author in one system, execute in another

© 2006 Open Grid Forum 14 Workflow Interoperability through data interoperability - Our perspective Focus on sharing data Make getting and sending it as simple as possible Make the data available in as flexible way as possible We are exploring RESTful approaches: Expose and provide access to data simply - CRUD Makes no demands on how the data is interpreted Data is just ‘stuff’ at an address Does not enforce typing data (like WS interfaces) I can choose whether I want to perceive the data as a stream, a file, a programming language object etc. Good at binary data Atom feed format and publishing protocol allows more complex interactions beyond simple request/response pairs E.g. tracking job status through time

© 2006 Open Grid Forum 15 Sharing and interoperability - Cardiff’s contributions Workflows Hosted in Portals (WHIP) is addressing both sharing and interoperability themes

© 2006 Open Grid Forum 16 WHIP - Sharing Extending myExperiment to support: Exposing Triana workflows Launching Triana locally Passing workflows to local Triana Uploading workflows to the server from local Triana Integrating WHIP archiving format enables upload/download of compound objects (workflow description, executable code, metadata) Enables signing of workflow archives using X-509 certificates.

© 2006 Open Grid Forum 17 WHIP - Interoperability Use-case: Kepler want to use Triana data mining workflow tools from within the Kepler environment. These components are part of larger workflows that are ‘native’ to Kepler. We are looking at embedding workflows as a solution. Does not impose alien workflow representations onto enactment engines. Uses the WHIP archiving format. Using RESTful approach e.g. Atom syndication format and publishing protocol for sending and retrieving data and job/workflow status.

© 2006 Open Grid Forum 18 Use Cardiff Sharing There are a number of projects that are looking at sharing workflows E.g. myExperiment At Cardiff we have WHIP, Music Information Retrieval in Triana (DART), Omer (CATNETS, FAEHIM, The Provenance Project) Interoperability At Cardiff we have identified many levels of interoperability for Triana, for example: Workflow embedding: looking into Kepler-Triana interoperability Kepler wants to make use of Triana’s data mining tools - DMG, FAEHIM (Rana) Reuse existing work by embedding Triana rather than re-inventing the wheel Graphical Workflow Editing: Triana Pegasus integration Triana used to graphically edit DAGMan workflows. Current and on-going Might extend to execute Pegasus workflows also

© 2006 Open Grid Forum 19 WFM-RG WFM-RG has started a research document Shields editor to gather more use cases from the community Send us yours ! Current on-going activity for this group use cases to: