1. OGSA-DAI: Patterns and Functions Supercomputing ‘06 AIST Stand Malcolm Atkinson Director e-Science Institute & e-Science Envoy www.nesc.ac.uk 14 th November 2006

2. Outline of Talk Reminder about OGSA-DAI Functionality, Extensibility & Patterns of Use Users OGSA-DAI Team The new architecture Motivation Features Status The Road ahead Engineering Challenges Research Challenges

3. OGSA-DAI OGSA-DAI provides: WS framework for wrapping data resources Extensible: Functionality can be added Client library Move computation to be close to the data Have a workflow like language  Encapsulate multiple interactions in one-interaction Works with multiple Grid Fabrics OMII-UK container, GT4.*, Tomcat + Axis Soon (OMII-Europe project): Unicore, gLite 3.0

4. Simple DB operations SQL Data Resource Accessor Relational XMLDB Data Resource Accessor Data Service Resource Data Service Data Service XPath or XQuery

5. 5 Exposing Databases Data Resource Accessor Relational XMLDB Data Resource Accessor Data Service Resource Files Data Service Resource SQLOne XMLOne FilesOne Data Service

6. Relational Multi-resources Multiple Data Resource Accessor Multi Resource Data Service Resource MultiOne Data Service Data Service Relational SQL Query Multi Results SQL Query Results x M Data Service Resource Data Service Relational SQL Query Results SQL Query Results SQL Query Results

7. Delegate DAI task completion

8. Multi-Service Integration e.g. DQP

9. 9 Clients & Authorisation Data Resource Accessor Data Service Data Service Resource Client Perform Document SQLOne Relational Perform Document SQL Query ResultSet SQL Query Results Response Document Authorization points

10. Used by a number of projects OGSA-DAI More info from: http://www.ogsadai.org.uk/about/projects.php

11. The OGSA-DAI Team IBM Development Team, Hursley NEReSC, Newcastle NeSC, Edinburgh ESNW, Manchester IBM Dissemination Team EPCC Team, Edinburgh

12. International Cooperation and Recognition USA: o Globus Alliance o IBM Corporation o caBIG o BIRN o Indiana University o GridSphere o GEON o LEAD o MCS o NCSA o Secure Data Grid o UNC Japan: o AIST o BioGrid o NAREGI Europe: o CERN o DataMiningGrid o GridMiner o GridSphere o inteligrid o N2Grid o OntoGrid o Provenance o SIMDAT UK: o OMII o NGS o NCeSS o NIeeS o AstroGrid o BioSimGrid o BRIDGES o CancerGrid o ConvertGrid o eDiaMonD o EDINA o First Group plc o Fujitsu Labs Europe o GEDDM o GeneGrid o Genomic Technology and Informatics o GOLD o Human Genetics Unit o IBM UK o my Grid o Oracle UK China: o CAS o ChinaGrid o cnGrid o INWA Australia: o Curtin Business School o INWA Tutorials BostonCambridge CERNChicago EdinburghLondon San FranciscoSeattle SeoulSingapore TokyoISSGC 03 to 06 DIALOGUE workshops Columbus, Edinburgh, Indiana, Vienna Chicago, Manchester, San Diego South Korea: o KISTI 1485 registered users 5250+ downloads

13. LEAD GeneGrid caBIG BRIDGES OGSA WebDB FirstDIG ConvertGrid eDiaMoND OGSA-DQP Grid Miner Meeting User Requirements

14. Outline of Talk Reminder about OGSA-DAI Functionality, Extensibility & Patterns of Use Users OGSA-DAI Team The new architecture Motivation Features Status The Road ahead Engineering Challenges Research Challenges

15. Features of Future OGSA-DAI Comprehensive library of Integration Activities Putting the I in OGSA-DAI You will still add your own Perhaps less often With more examples to start from With a new OGSA-DAI Engine for support With new Activity base classes Connected by “pipes” carrying Streams of values – scalability is a major motivator Each with own threads Exploiting concurrent processing Possibly in different VMs – resilience a major motivator

16. Support for Iteration Expect Request documents To contain more sophisticated DAGs Branching and merging activities Synchronous and dynamic data consumption Auto-iteration over (Lists of) values Iteration activities Manage the iteration of a controlled graph of activities

17. 17 Multi-resource requests Data Request Execution Service Data Request Execution Resource Client Data Resource Data Data Resource Data Data Resource Data MyDRER One Two Three Perform Document myQuery(One) myQuery(Two) myIntegrateData deliverGridFTP Client-server interaction Service-resource association Resource-resource association Resource-database association

18. 18 An expanded resource model Data Request Execution Service Data Request Execution Resource Client Data Resource Information Service Data Resource Session Management Service Session Data Data Resource Data Data Resource Data Session Request Request Management Service MySession123 MyDRER One Two Three MyRequest123456

19. 19 Persistence OGSA-DAI State Data Request Execution Service Data Request Execution Resource Client Data Resource Information Service Data Resource Session Management Service Session Data Session Request Request Management Service OGSA-DAI Server Another OGSA-DAI Server

20. 20 Persistence l Caching u Reduce overheads of OGSA-DAI-database communications l Allows configuration and state to persist between container shutdowns and restarts l Sticky resources u e.g. data sources and sinks u In-memory – only available via server that created them l API u File-based implementation for backwards compatibility u Relational implementation for compatibility with common OGSA-DAI databases u Extensibility point

21. 21 Scalability and robustness OGSA-DAI State Client OGSA-DAI cluster Front-end Router Monitor & Restart Warning: not in next release – ODE contains the foundations

22. 22 Scalability and robustness l Release 3 will afford future provision of clustering and load balancing l Clustered OGSA-DAI servers u Different JVM u Different machine l Load balancer u Simple e.g. round robin u Complex e.g. decide on basis of current load of each server l Scalable u Increases number of requests that OGSA-DAI can handle concurrently l Robust u Seamless operation in the face of errors e.g. server failures l Sticky sessions and resources u Session and resource state may be non-serializable or be lightweight in-memory components u Need to dispatch resources to their “owner” JVM

23. 23 Other planned features l Support for registries u Guidelines on how to best use standard registries (e.g. GRIMOIRES or GT4 ServiceGroups) and tools to help you do so within OGSA-DAI l Improved support for message-level and transport-level security including delegation l Concurrent or closely-following releases of u A new implementation of DAIS l OGSA-DAIR and OGSA-DAIX u A re-factored OGSA-DQP with a Java-compliant query compiler and which runs under Linux, Solaris or Windows

24. Support for generic activities Powerful general purpose activities Parameterised by Java methods Any data transformation or integration Task for application partitioned to writing simple methods Iteration and data management provided Examples TupleGenericSortTupleGenericProductSelectProject TupleGenericSelectProjectTupleGenericUnion TupleGenericMultiSelectProjectTupleGenericIntersection TupleGenericJoinSelectProjectTupleGenericDifference TupleGenericGroupBy

25. Outline of Talk Reminder about OGSA-DAI Functionality, Extensibility & Patterns of Use Users OGSA-DAI Team The new architecture Motivation Features Status The Road ahead Engineering Challenges Research Challenges

26. Engineering Challenges Integrating / supporting all activities Choices of internal data formats OMII-UK & OGSA-DAI code Contributors code Robust & scalable ODE implementations Continuous operation deployment, configuration, security & management Supporting but containing users’ code Integration with other data services Replica & cashing services Grid file services Storage management services

27. Research Challenges High-level notations Auto-Optimisation, Partitioning & Parallelisation Meta-data driven DAI Automatic generation of DAI workflow detail Visual Comprehension Visual forms of DAI workflows Visualisation of enactment progress Provision of control and diagnosis aids Relationship with other workflows

28. 28 Further information l The OGSA-DAI project site u http://www.ogsadai.org.uk l The DAIS-WG site u http://forge.gridforum.org/projects/dais-wg l Recent talks http://www.ogsadai.org.uk/documentation/presentations/GW2006GGF18// l Design in Progress http://www.ogsadai.org.uk/documentation/Design_documents/ l OGSA-DAI users mailing list u users@ogsadai.org.uk u General discussion on OGSA-DAI, data and the grid l Formal support for OGSA-DAI releases u http://www.ogsadai.org.uk/support u support@ogsadai.org.uk l OGSA-DAI training courses