1. “Workflow” in Data Access and Integration An OGSA-DAI/DAIS Perspective Mario Antonioletti EPCC mario@epcc.ed.ac.uk

2. e-Science Workflow Services - www.ogsadai.org.uk2 Talk Overview l Background: OGSA-DAI and DAIS l Motivation and Definitions l Hierarchies of Service Coordination l Conclusions

3. e-Science Workflow Services - www.ogsadai.org.uk3 OGSA-DAI and DAIS l GGF DAIS WG u Database Access and Integration Services u Attempting to standardise interfaces based on OGSI l OGSA-DAI u Aim to provide an implementation of DAIS u Serve UK e-Science Community l OGSA-DAI and DAIS u Currently not aligned l Data service interface in OGSA-DAI coarse grained u Based on an earlier version of DAIS l Data service interface in DAIS currently fine grained u Scope for more coarse grained interfaces u OGSA-DAI will realign DAIS once the latter stabilizes

4. e-Science Workflow Services - www.ogsadai.org.uk4 OGSA-DAI Project Partners Powered by ….

5. e-Science Workflow Services - www.ogsadai.org.uk5 Data Resource 1. Provides access to a data resource. Simple Data Service Scenario Client Data Service Data Resource 2. May provide integration of several data resources.

6. e-Science Workflow Services - www.ogsadai.org.uk6 Some Definitions l Data Resource u An object that can source/sink data u Currently databases in scope l Files and file systems may come in scope l Data Services u Grid services u Provides common interface to data resources u Exposes some capabilities of a data resource l SQL Queries, XPath, BinX, … u Can also provide additional capabilities l Transformations, Third party data delivery, etc …

7. e-Science Workflow Services - www.ogsadai.org.uk7 Motivation l Want common interfaces for: u Data access u Data integration l As requests to data service may produce lots of data u Want to minimise data movement l Hence encapsulate interactions with service u Serialise multiple interactions into one interaction u Abstract each interaction into an “activity” u Data flows between activities u Use a document mechanism to describe this l DAIS and OGSA-DAI u Concerned with data flow u Currently do not have control constructs l No looping, conditionals, splits, joins, …

8. e-Science Workflow Services - www.ogsadai.org.uk8 Service Coordination Patterns Client Data Service 1. Coordinate of activities performed at one Data Service. Data Service 2. Client choreographs a set of services to work together. Service … or a service may orchestrate on behalf of the client. 3. Orchestration of services using a document directed to one service. 4. Possibly interface with standard workflow languages, e.g. BPEL4WS, WSCI, …

9. e-Science Workflow Services - www.ogsadai.org.uk9 Coordination Hierarchies l Service coordination may take place: u Intra service l Document based u Inter services – application driven l Choreographed/orchestrated by a client or service u Inter service – document driven l Orchestration l Ideally would look the same as the intra service document based interface u Combined with other workflow languages

10. e-Science Workflow Services - www.ogsadai.org.uk10 Intra Service Processing l Service processing described by a document l Possible activities (OGSA-DAI perspective): u Statement l SQL Query, XPath Query u Delivery l Input data from third party l Output data to a third party l Deliver data in the response u Transformations l XSL Transformations, compression l OGSA-DAI has produced a framework for this

11. e-Science Workflow Services - www.ogsadai.org.uk11 Simple Example: no data flow sqlQueryStatement DeliverToURL select * from myTable where id=10 ftp://anon:frog@ftp.example.com/home

12. e-Science Workflow Services - www.ogsadai.org.uk12 Simple Example: with data flow DeliverToURL select * from myTable where id=10 ftp://anon:frog@ftp.example.com/home sqlQueryStatement

13. e-Science Workflow Services - www.ogsadai.org.uk13 The Perform Document <gridDataServicePerform xmlns="http://ogsadai.org.uk/namespaces/2003/07/gds/types" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ogsadai.org.uk/namespaces/2003/07/gds/types../../../../schema/ogsadai/xsd/activities/activities.xsd"> This example performs a simple select statement to retrieve one row from the test database. The results are delivered within the response document. select * from littleblackbook where id=10 ftp://anon:frog@ftp.example.com/home

14. e-Science Workflow Services - www.ogsadai.org.uk14 Predefined Building Blocks sqlQueryStatement sqlStoredProcedure sqlUpdateStatement sqlBulkLoadRowset xPathStatement xUpdateStatement xQueryStatement xmlResourceManagement xmlCollectionManagement relationalResourceManager gzipCompression zipArchive xslTransform inputStream outputStream DeliverFromURL DeliverToURL DeliverToGFTP DeliverFromGFTP DeliverToStream DeliverFromGDT DeliverToGDT

15. e-Science Workflow Services - www.ogsadai.org.uk15 Activities: positives l Simple sequence pattern u Data-flow l Avoid multiple message exchanges l Minimise data movement l Extensible u XML Schema excerpt gives syntax u Associate an implementation with activity u Done at configuration l Allows optimisation u Enactment engine can optimise interaction

16. e-Science Workflow Services - www.ogsadai.org.uk16 Activities: negatives l Incomplete syntax u Activity inputs and outputs are not typed u No typing of data streams u Possible issue in coming up with a sensible document l Activity implementation & XML schema loosely coupled u Keeping activity and implementation in synch l Semantics are not specified l Puts work load on the server u Workloads on the server may need to be managed l Activities not exposed at the interface level u This may change in line with DAIS l Perform document factored out from DAIS base specs u Standardisation to become a DAIS informational document u Scope may be bigger than DAIS

17. e-Science Workflow Services - www.ogsadai.org.uk17 Inter Service Application Defined "Workflow" l Services stitched together by an application u Could be a client l Use the OGSA-DAI GridDataTransport (GDT) portType u Could be another service l Distributed Query Processing (DQP) l Service configured separately u Each performs its part in the workflow

18. e-Science Workflow Services - www.ogsadai.org.uk18 Client Driven Scenario (aka poor man's data integration) Client Data Service … … GDT Client creates Data Services.

19. e-Science Workflow Services - www.ogsadai.org.uk19 Service Driven Scenario Client Query planning, compilation, scheduling, evaluation, partitioning GDQSGQES Evaluate sub-queries Distributed Query Processing

20. e-Science Workflow Services - www.ogsadai.org.uk20 More Complex DQP Scenario

21. e-Science Workflow Services - www.ogsadai.org.uk21 Application Driven "Workflow" l Labour intensive u Client driven (service choreography) l Restricted to small numbers of services u Need tooling u Even then this is best done through other means u Service driven (service orchestration) l DQP hides details l There may be other examples … l Need to explore this space further u Can probably accommodate these patterns in an existing workflow language l For more general data integration need: u Describe more sophisticated behaviour

22. e-Science Workflow Services - www.ogsadai.org.uk22 Inter Service Document Coordination l Currently evolving l Document describes: u Sequence of operations that may span multiple services l Single document includes enough information to: u Run an expression on a source data service u Deliver the results to a target data service u Run and expression on the target data service l Informational document to be presented at GGF10

23. e-Science Workflow Services - www.ogsadai.org.uk23 A Dataset Example Client Data Service Request DataRequest.xsd … RemoteRequiredTable DataAccessRecipe.xsd … … Data Service

24. e-Science Workflow Services - www.ogsadai.org.uk24 Document Driven "Workflow" l Work in this area is tentative u No implementations as yet l OGSA-DAI needs to see how it matures u Shows versatility l Carries over some of the OGSA-DAI activity framework u Focused on data l Can track provenance in the dataSet l Needs to be positioned against general workflow languages

25. e-Science Workflow Services - www.ogsadai.org.uk25 Traditional Workflow l OGSA-DAI has not explored this space … yet u May need such a framework to facilitate data integration l Traditionally workflow: u Revolves around the execution of atomic activities u Use a processing model, e.g. WfMC based l Akin to how people talk about service orchestration l Want to use existing frameworks as far as possible u OGSA-DAI does not want to define its own workflow u DAIS may come up with something l Clearly: u Activity model can be used to implement a workflow u Collecting use cases

26. e-Science Workflow Services - www.ogsadai.org.uk26 Workflow Issues l OGSA-DAI needs to play to see what works l Standards still evolving u IP rights: l BPEL4WS u Royalty-free … ? l WSCI u Royalty-free l Need workflow engines l Tooling to construct workflow u Ptolemy II … Triana … ?

27. e-Science Workflow Services - www.ogsadai.org.uk27 Summary & Conclusions l Base standards in a state of flux u DAIS not settled down yet l If you don't like what you see get involved and change it u Document based interface needs to be re-worked l OGSA-DAI implemented simple "workflow" patterns u Successful for data access u Shied away from real workflow u Should try to use emerging standards if possible l Data integration will require workflow patterns u Need to examine use cases l Positioning of OGSA-DAI u Want it to be the leaves of your complex workflow graphs u Wrap your data sources and sinks l Try OGSA-DAI and feedback!

28. e-Science Workflow Services - www.ogsadai.org.uk28 Further information l The OGSA-DAI Project Site: u http://www.ogsadai.org.uk l The DAIS-WG site: u http://cs.man.ac.uk/grid-db l OGSA-DAI Users Mailing list u users@ogsadai.org.uk u General discussion on grid DAI matters l Formal support for OGSA-DAI releases u http://www.ogsadai.org.uk/support u support@ogsadai.org.uk l OGSA-DAI training courses