June 25, 2007 WORKS 07, HPDC 07, Monterey Bay California, June GRIDCC: Real-time Workflow system A.Stephen McGough, Asif Akram, Li Guo, Marko Krznaric, Luke Dickens David Colling, Janusz Martyniak Roger Powell, Paul Kyberd Constantinos Kotsokalis

WORKS 07, HPDC 07, Monterey Bay California, June What is GRIDCC Adding real time control of instruments to the Grid What makes this unique:  Instruments may only be ready at specific times Reservations  Need to ensure other Grid services are available with instruments Reservations / SLA’s  Real-time visualisation of live results and steering Through Virtual Control Room (VCR) Reservations / SLA’s Grid enabled Remote Instrumentation with Distributed Control and Computation

WORKS 07, HPDC 07, Monterey Bay California, June GRIDCC – Overview VCR GRID Problem Solver Real Time QoS

WORKS 07, HPDC 07, Monterey Bay California, June Motivation QoS is important in Workflows  Just running a workflow and hoping is not enough We need more than WS-Agreement  Though WS-Agreement is very important to us We want to allow the user to specify their QoS requirements over a workflow Either as hard constraints  This must finish by time x As soft constraints  This should finish by time x in y% of cases

WORKS 07, HPDC 07, Monterey Bay California, June Aim of GRIDCC Editor Portal Based Editor Drag and Drop Hiding Complexities Ease of use Web Service Registry QoS Constraint XML Schema Registry (To be Implemented) Workflow Monitoring (To be Implemented) Standard Based Workflow Engine Independent Based on Flex 2 for User Interface, Java on Server Side

WORKS 07, HPDC 07, Monterey Bay California, June

Adding QoS to a Workflow XPath Partner Link

WORKS 07, HPDC 07, Monterey Bay California, June Less than 15 mins (80% confidence) Must run in less than 10 mins A BPEL workflow … … … Reserve a resource for this Using XPath pointers into BPEL document

WORKS 07, HPDC 07, Monterey Bay California, June Simple Example /process/flow The whole (Work)flowAll 2Ghz BoxesTotal Time 100s 100% Reliable resources

WORKS 07, HPDC 07, Monterey Bay California, June /process/flow/sequence[1]/sequence[1]/invoke[1] /process/flow/sequence[1]/sequence[1]/invoke[2] Refer to multiple parts of document

WORKS 07, HPDC 07, Monterey Bay California, June /process/flow/sequence[1]/sequence[1]/invoke[1] T23:00:00.0Z T23:59:59.0Z cpu1.doc.ic.ac.uk /process/flow/sequence[1]/sequence[1]/invoke[2] T23:00:00.0Z T23:59:59.0Z cpu3.doc.ic.ac.uk dveinvunvuin Make a reservation through Agreement Service (WS-Agreement) Store result here

WORKS 07, HPDC 07, Monterey Bay California, June Workflow Management Service (WfMS) Architecture

WORKS 07, HPDC 07, Monterey Bay California, June Architecture Planner  Determines if a workflow can be executed  Decides the best resources to use for the workflow Workflow Engine  Engine to enact the workflow determined within the Planner  ActiveBPEL Temporary WfWS  Instance of a workflow deployed to the Workflow Engine Agreement Service  Negotiation service for agreements with the CEs, SEs, IEs, NEs

WORKS 07, HPDC 07, Monterey Bay California, June Architecture Observer  Monitors the temporary WfWS as it executes checking that it is performing within parameters  Can be seen as an instance of a “Global Problem Solver” Delegation Service  Provides a mechanism for delegating credentials through the WfMS Workload Management System  Standard gLite WMS

WORKS 07, HPDC 07, Monterey Bay California, June What is the Planner? Makes a “plan” for how to execute a workflow  Can QoS requirements be met Which resources to use Whether reservations are required in order to meet QoS  Does not override the right of a user to request reservations Manipulation of the workflow  The expectation for how the workflow will progress What will be executed and where BPEL & QoS Planner BPEL & QoS Information Service Performance Information Agreement Service

WORKS 07, HPDC 07, Monterey Bay California, June Flow of Documents

WORKS 07, HPDC 07, Monterey Bay California, June WfMS Planner BPEL + QoS Observer BPEL Engine WS Interface Performance Repository Agreement Service IECESE FaçadeF FF Validator Reserver Workflow Optimiser Resource Selector Resource Pruning GRIDCC WfMS SEDA based architecture

WORKS 07, HPDC 07, Monterey Bay California, June Pipeline Stages for WfMS XML to Object Stream  Convert the XML document into a set of objects Object Stream to XML  Convert Objects into XML document Reservation maker  Make reservations through the Agreement Service Workflow fiddly bits (add in stuff we need)  Automatically deal with adding in security elements to workflow  Insert error checking into workflow, Performance logging of workflows Workflow Validator Workflow manipulator  Modify a workflow to improve performance QoS validate  Validate that the workflow can meet the QoS requirements placed on it Workflow Resource Selector  Select the resources within the Grid which will allow QoS to be met. JDL Manipulator - Manipulate JDL documents to use resources better

June 25, 2007 WORKS 07, HPDC 07, Monterey Bay California, June Questions