Multiple Views of Workflow through ICENI John Darlington, Nathalie Furmento, William Lee, Anthony Mayer, Steve McGough, Steven Newhouse London e-Science Centre

Different Views of “Workflow”: Aggregation of Elementary Tasks Loosely defined: “Programming the Grid” Composition of Services Scripting languages … more tightly defined: Ordering of Execution (often DAGs) Ordering of Communication similar in purpose… but certainly not identical in semantics

Interactive vs Batch Grid Services Execution time occurs after design time. Need to express ordering of actions. Interactive: Execution time and design time overlap, execution time passes while one designs. Expressing temporal ordering is tricky.

Expressions of Aggregation B A B A interacts with B Application = A  B Spatial Relationship A then B do A ; do B Temporal Relationship

Spatial Expressions Defined (observed) as: interactions are NOT temporally ordered Implicit assumption: Entities may be concurrent, Do not occupy same resources Usually dataflow relations Multiple models of computation Used when considering components and composition Usually graphical, certainly declarative

Temporal Expressions Defined (observed) as: interactions with temporal ordering Previous assumptions are relaxed Entities could occupy same space, (as long as it’s at a different time) Used when considering activities, tasks Workflow, imperative Usually textual, procedural

ICENI: Imperial College e-Science Network Infrastructure Integrated Grid Middleware Solution Interoperability between architectures, APIs Added value layer to other middleware Usability: Interactive Grid Workflows Deployment: Complete Install from Webstart Role and policy driven security Foundation for higher-level Services and Autonomous Composition ICENI Open Source licence (extended SISSL) The middleware we are developing at the centre is called ICENI. Deployability and installability are very important in the Grid. During today’s talk we will install ICENI and demonstrate some of its capab Ilities. The aim of ICENI is to provide a integrated and interoperable framework for grid application to be deployed. It is going to be an end-to-end solution where we cater for the need of the developers, the administrator and the day-to-day scientists using the system by developing the middleware as the foundation and a set of tools build on top of ICENI. ICENI is a service oriented architecture which I will explain in more depth later. It has a rich set of metadata description for many aspects of services, such as performance, composable interfaces. These services can be federated across multiple domains govern by its service level agreement and usage policy. These rich description will hopefully give us a foundation for building higher-level services and possibly autonomous computing. http://www.lesc.ic.ac.uk/iceni/ ICENI Release 1.0 available for download

The ICENI Stack Domain & Identity Management Service API Discovery API Client Side Tools: Netbeans / Portal Higher Level Services Runtime Component Framework Execution Mechanism OGSA Gateway Security Layer Domain & Identity Management Service API Discovery API Service Oriented Architecture Core API Jini Jxta OGSI Implementation Fabric

Augmented Component Programming Model Linear Solver Matrix Vector Meaning How components can be linked together Jacobi Matrix Vector LU Matrix Vector Push Model Behaviour Pull Model How they interact with each other Parallel LU Sequential LU Implementation How they will perform on different resources

Netbeans GUI Plug In: Drag & Drop Composition

Focus on Usability: ICENI Portal

Utilising the Spatial View: Some advantages… Raising the level of abstraction Semantic adaptation Interactivity through dynamic composition

ICENI: Utilising the Spatial View (1) Raising the level of abstraction Collaborations Composition CDL: Meaning Implementations may be resource specific High level of abstraction means we can select implementation More options for scheduler! Behavioural Annotations Implementation Annotations

ICENI: Utilising the Spatial View (2): Semantic Adaptation Metadata Space 2. Gather metadata Grid Service Semantic Matching Service Grid Service requirement Semantic Description 1. publish User 3. Produce Adaptation Proxy 4. Invoke service The ICENI Service Adaptation Framework\cite{isaf} builds on top of ICENI middleware to provide ways of annotating services using Resource Description Framework (RDF) and The Web Ontology Language (OWL). Semantic annotated services enables users to search through capability rather than static interface definitions. Once user's requirement is semantically matched with a smenatic service, an adaptation proxy conforming to user's interface requirement is automatically generated. The adaptation proxy provides a implementation and architecture independent way of invoking the required functionality. Grid Service Adaptation Service Adaptation Proxy

ICENI: Utilising the Spatial View (3): Dynamic Composition Drag-and-drop running component Deployed application Application Visualisation Server Register as running component services in the NetBeans user interface Execute to create new component instances and connect to application Add new advertised components

Delivering e-Science: Case Study of Interactive Dataflows LB3D – (Lattice-Boltzmann 3D) ICENI provides collaborative visualisation and steering across the Access Grid So, who are using ICENI – Early adopters of the Application and computer Science communities are already using ICENI, helping provide feedback and suggestions to this developing system. A number of universities have shown interest at both the Computer Science and applications level in this emerging technology and have already downloaded v1.0.1 . This is not a huge list, but has allowed the developers to react to the needs of the project researchers and increase ICENI’s functionality. The two projects I’m going to talk about here are RealityGrid’s LB3D and the GENIE project RealityGrid is looking into methods of collaborative visualisation. Using Lattice-Boltzmann equations to model mixing fluids as the test bed, we have been able to display images and manipulate the data processing over the access grid at remote locations. This requires a lot of work to compose the work flow in ICENI in order to send the data off to separate machines for processing, rendering, visualisation and steering, but shows off nicely the power of GRID technology. GENIE is more about mass data processing and recursively analysing data in order to model the complexities of the atmosphere and sea interactions. Already is has come up with some hugely important results on freshwater transport and is looking into the potential effects of atmospheric variation through such things as global warming.

Capturing the Information: Workflow from Dataflow B Each component has it’s own thread A and B are partners in the transaction A does work, then communicates with its partner, by sending a message B does nothing… … until it receives control and subsequently does work Data is worked on by both partners

Utilisation of Temporal View: Informed Scheduling (1) B Sequence: A then B Execution Time: T(A) + T(B) Data Constraint: M is shared M

Utilisation of Temporal View: Informed Scheduling (2) Both components share a single resource Opportunities to interleave components on resources Model of Execution time + Individual Performance of Activities = Composite Model of Application A B

ICENI: Temporal Expression Annotations component-by-component Inferred temporal ordering Information provided to scheduler Spatial Composition Temporal Expression: Workflow Behavioural Annotations Scheduling Services Implementation Annotations

ICENI Workflow: Elementary Units Activity Receive Start Description Duration Resource Usage Port Port Send Stop

ICENI Workflow: Splits & Joins And Split Or Split Conditions Or Join And Join

Workflow graph built recursively Start Activity Send Receive Activity Stop

Why graphs? Lot of theory Looks good Underlying is XML liveness properties deadlock Looks good Feedback to user is important Underlying is XML Experiments with modifications to existing workflow languages Could (easily?) use a scripting language

Netbeans GUI (workflow view)

Information from the temporal view Start Length: Execution Time Activity And Split Width: Resource Usage Send Send Receive Receive Activity Activity Stop Stop

Loops! Generate Sorted List Process List Start Activity: Generate Receive Generate Sorted List Or Join Activity: Sort Process List Send Receive Activity: Process Or Split Stop

… not the end of the world Loop detection eased by join nodes Performance Model in terms of loop terms: <iteration>s are not ordered, so terms are not ordered in general But where a single dominant <iteration>, we can produce an ordering on its factor

Useful information despite loops Performance models used for comparison implementation selection resource selection So in our example: T(Generate) + y. (T(Sort) + T(Process)) thus we can order implementations on (T(Sort) + T(Process)) Surprisingly common!

Delivering e-Science: Case Study of Iterative Workflows GENIE – (Grid ENabled Integrated Earth system model) Has already produced major scientific results, showing the fragility of the freshwater transport system.

Example: GENIE Spatial Composition Setup Control Integrate Display Land Atmos Sea Ice Ocean

Example: GENIE Inferred Temporal Information Setup Control Integrate <S> Sea Ice Atmos Ocean <F> Display

Future Developments… More user involvement in the behavioural descriptions (control flow editing) Not too hard Looking at data concerns (protocols etc) Harder Multiple forms of expression Scripting, GUI tools, portals, Standards for reuse, sharing Technically easier, but requires community effort (shocking hard)

Summary: Separation of Concerns Capturing user requirements: spatial view Allows dataflow models of computation Interactive applications High level of abstraction Mapping application to grid: temporal view Workflow Performance Modelling Optimised Scheduling ICENI infers temporal from spatial by exploiting component meta-data

Development Infrastructure Project Website & mailing lists Daily build Regression tests On success binaries updated Regenerated JavaDoc Deployment tests CVS Code split across multiple repositories & modules Documentation, manuals & user guides ICENI Open Source License (Extended SISSL)

Acknowledgements London e-Science Centre Director: Professor John Darlington Technical Director: Dr Steven Newhouse Research Staff: Anthony Mayer, Nathalie Furmento Stephen McGough, James Stanton Yong Xie, William Lee Marko Krznaric, Murtaza Gulamali Asif Saleem, Laurie Young, Gary Kong Support Staff: Keith Sephton (Systems Manager) Oliver Jevons (Operations Manager) Susan Brookes (Administrative Assistant) And acknowledge the contribution into this work of the research team here at Imperial. E-Science is a collaborative multi-disciplinary activity

ICENI: An integrated Grid Middleware http://www.lesc.ic.ac.uk/iceni/ ICENI Release 1.0 available !!! ICENI Open Source licence (extended SISSL)