Workflow Optimisation Services for e-Science Applications David W. Walker Cardiff University

WOSE Overview Draws together JISGA and Triana work at CU, with ICENI at IC, and portal expertise at DL. Topics addressed –Service aggregation and deployment –Runtime discovery and late binding of services –Service discovery and selection from multiple semantically equivalent services

Workflow Optimisation Types of workflow optimisation –Through service selection –Through workflow re-ordering –Through exploitation of parallelism When is optimisation performed? –At design time (early binding) –Upon submission (intermediate binding) –At runtime (late binding)

Service Binding Models Late binding of abstract service to concrete service instance means: –We use up-to-date information to decide which service to use when there are. multiple semantically equivalent services –We are less likely to try to use a service that is unavailable.

Late Binding Case Search registry for all services that are consistent with abstract service description. Select optimal service based on current information, e.g, host load, etc. Execute this service. Doesn’t take into account time to transfer inputs to the service. In early and late binding cases we can optimise overall workflow.

WOSE Architecture Work at Cardiff has focused on implementing a late binding model for dynamic service discovery, based on a generic service proxy, and service discovery and optimisation services. History database Proxy Configuration script Workflow script User ConverterActiveBPEL workflow engine Web service instance Discovery Service Optimization Service Registry services (such as UDDI) Performance Monitor Service

WOSE Sequence Diagram CCGrid2005 Wip section 12 May, 2005

Service Discovery Issues Service discovery and optimisation is based on service metadata. Could store in a database. Could obtain by interrogating service.

Optimisation by Re-Ordering Work at Imperial has looked at static optimisation –Optimise the runtime execution of workflow before it is executed –Achieves the goal through: Re-ordering of components Addition of components Substitution of components Pruning of the workflow Performance and workflow aware Scheduling Runtime Optimisation –through monitoring, check-pointing and migration

Component Manipulation Re-ordering: Workflows (often composed from composite workflows) may contain non-optimal ordering of components –Use re-ordering to improve performance multiply Matrix Gen Vector Gen Matrix Gen multiply Matrix Gen Vector Gen Matrix Gen multiply

Component Addition Addition: For a component requiring a specific format of data as input, a transformer component could be added to achieve the desired format. –Allows more optimal components to be used together Input required in MPS format Output in LP format C 1LP to MPSC 2

Component Substitution Substitution: –A Jacobi Iteration linear solver replaced by Conjugate Gradient linear solver according to the output of the Discretizer (FEM) Based on observing the meta-data associated with previous components b A (sparse and diagonally dominant) FEM JI linear solver

Pruning Workflow Pruning: –Workflows may contain unused components. Especially when composed from other sub-workflows Remove redundant components Not needed ab f c e g d h  

Performance Aware Scheduling Scheduler Globus Resource s Globus Launcher Single Resource Launcher Component Repository JSDL Query Request Reservation Performance Repository SGE Resource s Reservation Launcher Reservation Service Query Negotiate Reservation WS-Agreement

Execution Pipeline JSDL for different VMs of components Optimised Workflow Abstract Workflow Concrete Workflow GRID Scheduler Performance Aware Deployment Service (DS) Application Co-ordination Service (ACS) Static Optimiser OS VM Hardware Service OS VM Hardware Service

Explanation ACS (Application Co-ordination Service) –Abstract layer which coordinates checkpointing and migration –makes Web Service calls to Deployment Service (DS) which then co-ordinates with the services running on resources in order to interact with the VM This is the execution environment of the component ACS temporarily suspends all the VMs in which the components of a workflow execute –Web Service call to DS The DS causes each VM to take its snapshot –ACS fetches the state snapshots and maintains the most recent copy If Migrating - stages the VMs needing migration to the new resources as decided by the scheduler Only VM state needs migration ACS resumes the VMs –Web Service call to DS

Work Done at Daresbury Daresbury started working with evaluating different BPEL engines to have proper understanding of BPEL specification i.e. ActiveBPEL, Oracle BPEL engine initially Collaxa BPEL engine. Most projects at CCLRC are portal based, so we studied the relevance of Portal based workflow monitoring and controlling solution. Web Services for Remote Portlets (WSRP) is an emerging specification, with limited support - seems to bridge Portal and Web Services and a lot of work has been done, with demo prototype. WSRF is extension of existing Web Services and provides extra layer of functionality without changing the Web Service implementation. WSRF extension for Web Service starts from modification of existing WSDL file of specific Web Service. WSRF extension of WSDL is not straight forward and needs a thorough understanding of WSDL and WSRF. Daresbury has built working prototype for testing Web Services based on WSDL.

Main Prototype Screen Text Field to enter WSDL location Drop Down Menu for Services & Operations Buttons to Analyse and Add WS JTree for adding WS, with user interaction Display Message content for WS Operation Call

Dynamic Swing Form created for testing WS Prototype can handle every complex and composite data types

Adding Different WS and SOAP Message Different WS added and populated drop down menu Contents of SOAP Method Call

Prototype Prototype to test WS is modular in nature and is quite stable. Different components of WSDL like messages, types, ports, services are implemented in different classes for maximum flexibility. Prototype can handle composite and complex data type of significant complexity. Prototype is using in-memory representation of WSDL, rather than creating stubs and skeletons. Another prototype based on stubs and skeletons is created which uses WSDL2Java class to create stubs and skeletons from WSDL and Ant tool to compile them, but it has not been thoroughly tested. Purpose of two prototypes is to compare which technique of calling WS is more efficient. Initial test shows for calling small number of WS’s in memory technique is efficient, but for large workflows with many WS’s creating stubs and skeletons should be preferred.

Future Work Prototype is stable but still it needs a lot of improvements. Improvements in Graphical User Interface for better user experience. Adding new features for handling WSDL files, which imports external WSDL files or XML Schema for data types. Extending prototype to create WSRF compatible WSDL file from given WSDL file according to user requirements:  Adding appropriate WSRF related namespaces to WSDL  Importing appropriate WSRF related XML Schema  Adding Resources to WSDL  Adding Resource Properties to WSDL  Adding support to WS-Addressing  Adding Support to WS-Notification Making prototype as First Tool to test WSRF compliant WS’s. Evaluating BPELJ/ WSIF (Web Service Invocation Framework) for extending BPEL for dynamic WS selection, flow decision and data conversion.

Future Work Complete WOSE prototype with dynamic scheduling. Compare dynamic and static approaches at Cardiff and Imperial. Improve Discovery Service mechanisms. Investigate different approaches to optimisation.