1. Multiple Views of Workflow through ICENI
John Darlington, Nathalie Furmento, William Lee, Anthony Mayer, Steve McGough, Steven Newhouse
London e-Science Centre

2. Different Views of “Workflow”: Aggregation of Elementary Tasks
Loosely defined:
“Programming the Grid”
Composition of Services
Scripting languages
… more tightly defined:
Ordering of Execution (often DAGs)
Ordering of Communication
similar in purpose… but certainly not identical in semantics

3. Interactive vs Batch Grid Services
Execution time occurs after design time.
Need to express ordering of actions.
Interactive:
Execution time and design time overlap, execution time passes while one designs.
Expressing temporal ordering is tricky.

4. Expressions of AggregationB A B
A interacts with B
Application = A  B
Spatial Relationship
A then B
do A ; do B
Temporal Relationship

5. Spatial Expressions Defined (observed) as:
interactions are NOT temporally ordered
Implicit assumption:
Entities may be concurrent,
Do not occupy same resources
Usually dataflow relations
Multiple models of computation
Used when considering
components and composition
Usually graphical, certainly declarative

6. Temporal Expressions Defined (observed) as:
interactions with temporal ordering
Previous assumptions are relaxed
Entities could occupy same space, (as long as it’s at a different time)
Used when considering activities, tasks
Workflow, imperative
Usually textual, procedural

7. ICENI: Imperial College e-Science Network Infrastructure
Integrated Grid Middleware Solution
Interoperability between architectures, APIs
Added value layer to other middleware
Usability: Interactive Grid Workflows
Deployment: Complete Install from Webstart
Role and policy driven security
Foundation for higher-level Services and Autonomous Composition
ICENI Open Source licence (extended SISSL)
The middleware we are developing at the centre is called ICENI. Deployability and installability are very important in the Grid. During today’s talk we will install ICENI and demonstrate some of its capab
Ilities.
The aim of ICENI is to provide a integrated and interoperable framework for grid application to be deployed. It is going to be an end-to-end solution where we cater for the need of the developers, the administrator and the day-to-day scientists using the system by developing the middleware as the foundation and a set of tools build on top of ICENI.
ICENI is a service oriented architecture which I will explain in more depth later. It has a rich set of metadata description for many aspects of services, such as performance, composable interfaces. These services can be federated across multiple domains govern by its service level agreement and usage policy. These rich description will hopefully give us a foundation for building higher-level services and possibly autonomous computing.
ICENI Release 1.0 available for download

8. The ICENI Stack Domain & Identity Management Service API Discovery API
Client Side Tools: Netbeans / Portal
Higher
Level
Services
Runtime Component Framework
Execution Mechanism
OGSA Gateway
Security
Layer
Domain & Identity Management
Service API
Discovery API
Service
Oriented
Architecture
Core API
Jini
Jxta
OGSI
Implementation Fabric

9. Augmented Component Programming Model
Linear
Solver
Matrix
Vector
Meaning
How components can
be linked together
Jacobi
Matrix
Vector LU
Matrix
Vector
Push Model
Behaviour
Pull Model
How they interact
with each other
Parallel LU
Sequential LU
Implementation
How they will perform
on different resources

10. Netbeans GUI Plug In: Drag & Drop Composition

11. Focus on Usability: ICENI Portal

12. Utilising the Spatial View: Some advantages…
Raising the level of abstraction
Semantic adaptation
Interactivity through dynamic composition

13. ICENI: Utilising the Spatial View (1) Raising the level of abstraction
Collaborations
Composition
CDL:
Meaning
Implementations may be
resource specific
High level of abstraction means
we can select implementation
More options for scheduler!
Behavioural
Annotations
Implementation
Annotations

14. ICENI: Utilising the Spatial View (2): Semantic Adaptation
Metadata Space
2. Gather metadata
Grid
Service
Semantic
Matching
Service
Grid
Service
requirement
Semantic
Description
1. publish
User
3. Produce Adaptation Proxy
4. Invoke
service
The ICENI Service Adaptation Framework\cite{isaf} builds on top of ICENI
middleware to provide ways of annotating services using Resource Description
Framework (RDF) and The Web Ontology Language (OWL). Semantic annotated services
enables users to search through capability rather than static interface
definitions. Once user's requirement is semantically matched with a
smenatic service, an adaptation proxy conforming to user's interface requirement
is automatically generated. The adaptation proxy provides a implementation and
architecture independent way of invoking the required functionality.
Grid
Service
Adaptation
Service
Adaptation
Proxy

15. ICENI: Utilising the Spatial View (3): Dynamic Composition
Drag-and-drop running component
Deployed application
Application
Visualisation
Server
Register as running component services in the NetBeans user interface
Execute to create new component instances and connect to application
Add new advertised components

16. Delivering e-Science: Case Study of Interactive Dataflows
LB3D – (Lattice-Boltzmann 3D)
ICENI provides collaborative visualisation and steering across the Access Grid
So, who are using ICENI –
Early adopters of the Application and computer Science communities are already using ICENI, helping provide feedback and suggestions to this developing system.
A number of universities have shown interest at both the Computer Science and applications level in this emerging technology and have already downloaded v
This is not a huge list, but has allowed the developers to react to the needs of the project researchers and increase ICENI’s functionality.
The two projects I’m going to talk about here are RealityGrid’s LB3D and the GENIE project
RealityGrid is looking into methods of collaborative visualisation. Using Lattice-Boltzmann equations to model mixing fluids as the test bed, we have been able to display images and manipulate the data processing over the access grid at remote locations.
This requires a lot of work to compose the work flow in ICENI in order to send the data off to separate machines for processing, rendering, visualisation and steering, but shows off nicely the power of GRID technology.
GENIE is more about mass data processing and recursively analysing data in order to model the complexities of the atmosphere and sea interactions.
Already is has come up with some hugely important results on freshwater transport and is looking into the potential effects of atmospheric variation through such things as global warming.

17. Capturing the Information: Workflow from DataflowB
Each component has
it’s own thread
A and B are partners in the transaction
A does work, then communicates with its partner, by sending a message
B does nothing…
… until it receives control
and subsequently does work
Data is worked on by both partners

18. Utilisation of Temporal View: Informed Scheduling (1)B
Sequence: A then B
Execution
Time:
T(A) + T(B)
Data Constraint: M is shared M

19. Utilisation of Temporal View: Informed Scheduling (2)
Both components share a single resource
Opportunities to interleave components on resources
Model of Execution time +
Individual Performance of Activities =
Composite Model of Application A B

20. ICENI: Temporal Expression
Annotations component-by-component
Inferred temporal ordering
Information provided to scheduler
Spatial Composition
Temporal Expression:
Workflow
Behavioural
Annotations
Scheduling
Services
Implementation
Annotations

21. ICENI Workflow: Elementary Units
Activity
Receive
Start
Description
Duration
Resource Usage
Port
Port
Send
Stop

22. ICENI Workflow: Splits & Joins
And Split
Or Split
Conditions
Or Join
And Join

23. Workflow graph built recursively
Start
Activity
Send
Receive
Activity
Stop

24. Why graphs? Lot of theory Looks good Underlying is XML
liveness properties
deadlock
Looks good
Feedback to user is important
Underlying is XML
Experiments with modifications to existing workflow languages
Could (easily?) use a scripting language

25. Netbeans GUI (workflow view)

26. Information from the temporal view
Start
Length:
Execution
Time
Activity
And Split
Width:
Resource
Usage
Send
Send
Receive
Receive
Activity
Activity
Stop
Stop

27. Loops! Generate Sorted List Process List Start Activity: Generate
Receive
Generate
Sorted List
Or Join
Activity: Sort
Process List
Send
Receive
Activity: Process
Or Split
Stop

28. … not the end of the world
Loop detection eased by join nodes
Performance Model in terms of loop terms:
<iteration>s are not ordered, so terms are not ordered in general
But where a single dominant <iteration>, we can produce an ordering on its factor

29. Useful information despite loops
Performance models used for comparison
implementation selection
resource selection
So in our example:
T(Generate) + y. (T(Sort) + T(Process))
thus we can order implementations on
(T(Sort) + T(Process))
Surprisingly common!

30. Delivering e-Science: Case Study of Iterative Workflows
GENIE – (Grid ENabled Integrated Earth system model)
Has already produced major scientific results, showing the fragility of the freshwater transport system.

31. Example: GENIE Spatial Composition
Setup
Control
Integrate
Display
Land
Atmos
Sea Ice
Ocean

32. Example: GENIE Inferred Temporal Information
Setup
Control
Integrate
<S>
Sea Ice
Atmos
Ocean
<F>
Display

33. Future Developments…
More user involvement in the behavioural descriptions (control flow editing)
Not too hard
Looking at data concerns (protocols etc)
Harder
Multiple forms of expression
Scripting, GUI tools, portals,
Standards for reuse, sharing
Technically easier, but requires community effort (shocking hard)

34. Summary: Separation of Concerns
Capturing user requirements: spatial view
Allows dataflow models of computation
Interactive applications
High level of abstraction
Mapping application to grid: temporal view
Workflow
Performance Modelling
Optimised Scheduling
ICENI
infers temporal from spatial
by exploiting component meta-data

35. Development Infrastructure
Project Website &
mailing lists
Daily build
Regression tests
On success binaries updated
Regenerated JavaDoc
Deployment tests
CVS
Code split across multiple repositories & modules
Documentation, manuals & user guides
ICENI Open Source License (Extended SISSL)

36. Acknowledgements London e-Science Centre
Director: Professor John Darlington
Technical Director: Dr Steven Newhouse
Research Staff:
Anthony Mayer, Nathalie Furmento
Stephen McGough, James Stanton
Yong Xie, William Lee
Marko Krznaric, Murtaza Gulamali
Asif Saleem, Laurie Young, Gary Kong
Support Staff:
Keith Sephton (Systems Manager)
Oliver Jevons (Operations Manager)
Susan Brookes (Administrative Assistant)
And acknowledge the contribution into this work of the research team here at Imperial. E-Science is a collaborative multi-disciplinary activity

37. ICENI: An integrated Grid Middleware
ICENI Release 1.0 available !!!
ICENI Open Source licence (extended SISSL)