1. June 25, 2007 WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 GRIDCC: Real-time Workflow system A.Stephen McGough, Asif Akram, Li Guo, Marko Krznaric, Luke Dickens David Colling, Janusz Martyniak Roger Powell, Paul Kyberd Constantinos Kotsokalis

2. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 What is GRIDCC Adding real time control of instruments to the Grid What makes this unique:  Instruments may only be ready at specific times Reservations  Need to ensure other Grid services are available with instruments Reservations / SLA’s  Real-time visualisation of live results and steering Through Virtual Control Room (VCR) Reservations / SLA’s Grid enabled Remote Instrumentation with Distributed Control and Computation

3. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 GRIDCC – Overview VCR GRID Problem Solver Real Time QoS

4. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Motivation QoS is important in Workflows  Just running a workflow and hoping is not enough We need more than WS-Agreement  Though WS-Agreement is very important to us We want to allow the user to specify their QoS requirements over a workflow Either as hard constraints  This must finish by time x As soft constraints  This should finish by time x in y% of cases

5. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Aim of GRIDCC Editor Portal Based Editor Drag and Drop Hiding Complexities Ease of use Web Service Registry QoS Constraint XML Schema Registry (To be Implemented) Workflow Monitoring (To be Implemented) Standard Based Workflow Engine Independent Based on Flex 2 for User Interface, Java on Server Side

6. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007

7. Adding QoS to a Workflow XPath Partner Link

8. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Less than 15 mins (80% confidence) Must run in less than 10 mins A BPEL workflow … … … Reserve a resource for this Using XPath pointers into BPEL document

9. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Simple Example /process/flow 2048000 100 The whole (Work)flowAll 2Ghz BoxesTotal Time 100s 100% Reliable resources

10. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 /process/flow/sequence[1]/sequence[1]/invoke[1] /process/flow/sequence[1]/sequence[1]/invoke[2] 2048000 100 Refer to multiple parts of document

11. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 /process/flow/sequence[1]/sequence[1]/invoke[1] 2006-10-02T23:00:00.0Z 2006-10-02T23:59:59.0Z cpu1.doc.ic.ac.uk 1024000 80 100 /process/flow/sequence[1]/sequence[1]/invoke[2] 2048000 20 50 2006-10-02T23:00:00.0Z 2006-10-02T23:59:59.0Z cpu3.doc.ic.ac.uk dveinvunvuin42235454 Make a reservation through Agreement Service (WS-Agreement) Store result here

12. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Workflow Management Service (WfMS) Architecture

13. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Architecture Planner  Determines if a workflow can be executed  Decides the best resources to use for the workflow Workflow Engine  Engine to enact the workflow determined within the Planner  ActiveBPEL Temporary WfWS  Instance of a workflow deployed to the Workflow Engine Agreement Service  Negotiation service for agreements with the CEs, SEs, IEs, NEs

14. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Architecture Observer  Monitors the temporary WfWS as it executes checking that it is performing within parameters  Can be seen as an instance of a “Global Problem Solver” Delegation Service  Provides a mechanism for delegating credentials through the WfMS Workload Management System  Standard gLite WMS

15. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 What is the Planner? Makes a “plan” for how to execute a workflow  Can QoS requirements be met Which resources to use Whether reservations are required in order to meet QoS  Does not override the right of a user to request reservations Manipulation of the workflow  The expectation for how the workflow will progress What will be executed and where BPEL & QoS Planner BPEL & QoS Information Service Performance Information Agreement Service

16. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Flow of Documents

17. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 WfMS Planner BPEL + QoS Observer BPEL Engine WS Interface Performance Repository Agreement Service IECESE FaçadeF FF Validator Reserver Workflow Optimiser Resource Selector Resource Pruning GRIDCC WfMS SEDA based architecture

18. WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Pipeline Stages for WfMS XML to Object Stream  Convert the XML document into a set of objects Object Stream to XML  Convert Objects into XML document Reservation maker  Make reservations through the Agreement Service Workflow fiddly bits (add in stuff we need)  Automatically deal with adding in security elements to workflow  Insert error checking into workflow, Performance logging of workflows Workflow Validator Workflow manipulator  Modify a workflow to improve performance QoS validate  Validate that the workflow can meet the QoS requirements placed on it Workflow Resource Selector  Select the resources within the Grid which will allow QoS to be met. JDL Manipulator - Manipulate JDL documents to use resources better

19. June 25, 2007 WORKS 07, HPDC 07, Monterey Bay California, June 25 2007 Questions