EGEE is a project funded by the European Union under contract IST NA3 Strategy for Future Grids Malcolm Atkinson Director of the National e-Science Centre Induction Course, Edinburgh, 27 April

NA3 Induction Course Development, Edinburgh, 27 April Contents What is the Web Service Resource Framework  Why has it emerged?  What has it to do with Grids?  What are the parts of WSRF?  What is the status of WSRF? Standards process Implementations Globus Alliance Plans WSRF in Perspective  What is important?

NA3 Induction Course Development, Edinburgh, 27 April Reminder: what are our goals? Address the challenge  build inter-enterprise systems NOT just connecting systems WORK together to build infrastructure  That persistently and adaptively supports multiple Virtual Organisations  VOs that span organisational structures  Distributed implementation and operation Pioneering new ways of working  John Taylor’s vision We pioneer & transfer results to industry Easy tasks can use any technology Only some of our goals align with industry

NA3 Induction Course Development, Edinburgh, 27 April Why OGSI adopted web services Expectation: WS would meet several Grid needs  E.g. Standard interface definition language Foundation for better engineering  E.g. Standard invocation mechanism Foundation for interoperability But other channels used for performance  Good commercial tooling (eventually) Reliability and performance Service-Oriented Architecture  Has valuable scalability and durability properties E.g. ICENI using Jini

NA3 Induction Course Development, Edinburgh, 27 April Web Services components and framework Not a silver bullet or a complete solution!  Most of the engineering effort What you do when you get a message Not how you address, package and deliver it  Most of the standardisation effort Agreeing how to factor large systems and the semantics of services Agreeing conventions for information in messages  Confusing & Rival standards proposals  Limited quality public implementations Don’t give up – engage and help fix it?  Is this the role of EGEE?  Is there just one answer? Incremental adoption of WS-I, WS-Security, WS-Addressing Incremental adoption of WSRF as it emerges We all agree that it is a good strategy to use web services. The issue are: Which ones to adopt when? and What conventions organise our systems?

NA3 Induction Course Development, Edinburgh, 27 April OGSI & GT3 investment Three forms of investment Architectural effort  OGSA: Use cases, Design Patterns, …  Factoring & describing a complex engineering domain Standardisation effort  OGSI, DAIS, WS-Agreement, etc.  WSDL 2.0, WSDM, WS-Security, etc. Implementation effort  Combined OGSI & Grid component work This investment carries forward into WSRF

NA3 Induction Course Development, Edinburgh, 27 April Grid Web Several (some partial) implementations Issues: technical, political & commercial Successes: a number of operational grids Started far apart in apps & tech OGSI GT2 GT1 HTTP WSDL, WS-* WSDL 2, WSDM Have been converging ? Combining Grid and Web Services – First try Technology intercept is not easy People accepted OGSI

NA3 Induction Course Development, Edinburgh, 27 April Not the only possible technical solution Combining Grid and Web Services: Second try Grid Web WSRF Started far apart in apps & tech OGSI GT2 GT1 HTTP WSDL, WS-* WSDL 2, WSDM Have been converging Support from major WS vendors especially service management suppliers e.g., CA, HP, IBM, Fujitsu, BEA, SAP, … Technology intercept is still not easy

NA3 Induction Course Development, Edinburgh, 27 April Core Ideas in WSRF Preserves OGSI functionality  Lifetime, properties, notification, error types, … Separates service from resource  Service is static and stateless  Resource is dynamic and stateful Builds on WS-Addressing Is WS-I compliant  But note that WS-I alone doesn’t make the problems go away, still need to worry about how to manage lifetime, naming, state, …

NA3 Induction Course Development, Edinburgh, 27 April “Components” of WSRF WS-AddressingMarch 04 www-106.ibm.com/developerworks/library/specification/ws-add/ WSRF White paper on modelling stateful resources WS-ResourceLifetimeMarch 04 WS-ResourcePropertiesMarch 04 WS-BaseFaultsMarch 04 WS-RenewableReferences March 04 WS-ServiceGroupMarch 04 WS-Notification WS-BaseNotificationMarch 04 …/specification/ws-notification/ WS-TopicsMarch 04 …/specification/ws-topics/ WS-BrokeredNotification March 04 …/specification/ws-pubsub/

NA3 Induction Course Development, Edinburgh, 27 April From OGSI to WSRF: Refactoring and Evolution OGSIWSRF Grid Service ReferenceWS-Addressing Endpoint Reference Grid Service HandleWS-Addressing Endpoint Reference HandleResolver portTypeWS-RenewableReferences Service data defn & accessWS-ResourceProperties GridService lifetime mgmtWS-ResourceLifetime Notification portTypesWS-Notification Factory portTypeTreated as a pattern ServiceGroup portTypesWS-ServiceGroup Base fault typeWS-BaseFaults Identity & naming is being done by OGSA

NA3 Induction Course Development, Edinburgh, 27 April GW GT3.2 Improved robustness, scalability, performance, usability 3.2 March 4.0  Q2 4.0 Q3 4.2 Q2 ‘05 Numerous new WSRF-based services GT4.2 GT4.0 WSRF; some new functionality; further usability, performance enhancements Not waiting for finalisation of WSRF specs. Use as submitted GT & WSRF Timeline OASISGGF10interopTC 1 techPre

NA3 Induction Course Development, Edinburgh, 27 April Components in GT 3.0 GSI WS-Security Security WS Core Resource Management Data Management RFT (OGSI) RLS WU GridFTP JAVA WS Core (OGSI) OGSI C Bindings Information Services MDS2 WS-Index (OGSI) Pre-WS GRAM WS GRAM (OGSI)

NA3 Induction Course Development, Edinburgh, 27 April Components in GT 3.2 GSI WS-Security CAS (OGSI) SimpleCA Security Data Management RFT (OGSI) RLS OGSA-DAI WU GridFTP XIO Information Services MDS2 WS-Index (OGSI) Resource Management Pre-WS GRAM WS GRAM (OGSI) WS Core JAVA WS Core (OGSI) OGSI C Bindings OGSI Python Bindings (contributed) pyGlobus (contributed)

NA3 Induction Course Development, Edinburgh, 27 April Planned Components in GT 4.0 GSI WS-Security CAS (WSRF) SimpleCA Security Authz Framework Data Management RFT (WSRF) RLS OGSA-DAI New GridFTP XIO WS Core JAVA WS Core (WSRF) C WS Core (WSRF) Information Services MDS2 WS-Index (WSRF) Resource Management Pre-WS GRAM WS-GRAM (WSRF) CSF (contribution) pyGlobus (contributed)

NA3 Induction Course Development, Edinburgh, 27 April Importance of collaboration: VDT A highly successful collaborative effort  VDT Working Group  VDS (Chimera/Pegasus) team Provides the “V” in VDT  Condor Team  Globus Alliance  NMI Build and Test team  EDG/LCG/EGEE Middleware, testing, patches, feedback …  PPDG Hardening and testing  Pacman Provides easy installation capability Currently Pacman 2, moving to Pacman 3 soon Used by many projects Systematic testing Rich integration of components EGEE is part of this – exploit test bed contribute components Thanks to Miron Livny Collaboration is a two way street – or should be!

NA3 Induction Course Development, Edinburgh, 27 April VDT Timeline Highlights Fall 2001: VDT started by GriPhyN and iVDGL  Supported US-CMS testbed in 2002 March 2002: VDT support system inaugurated Early 2003: Adopted by European Data Grid & LHC Computing Grid April 2003: VDT Testers group started Fall 2003: Supporting Grid3 Fall 2003: Adopted by Particle Physics Data Grid Nov 2003: Nightly test infrastructure deployed Thanks to Miron Livny

NA3 Induction Course Development, Edinburgh, 27 April VDT Growth VDT 1.0 Globus 2.0b Condor VDT 1.1.3, & pre-SC 2002 VDT Switch to Globus 2.2 VDT Grid2003 VDT First real use by LCG Thanks to Miron Livny

NA3 Induction Course Development, Edinburgh, 27 April Tools in the VDT Condor Group  Condor/Condor-G  DAGMan  Fault Tolerant Shell  ClassAds Globus Alliance  Job submission (GRAM)  Information service (MDS)  Data transfer (GridFTP)  Replica Location (RLS) EDG & LCG  Make Gridmap  Certificate Revocation List Updater  Glue Schema/Info prov. ISI & UC  Chimera & Pegasus NCSA  MyProxy  GSI OpenSSH  UberFTP LBL  PyGlobus  Netlogger Caltech  MonaLisa VDT  VDT System Profiler  Configuration software Others  KX509 (U. Mich.) Thanks to Miron Livny

NA3 Induction Course Development, Edinburgh, 27 April Relative Importance What envelopes you put your messages in  How they are delivered  Infrastructure to organise a common technical platform – the foundations of communication

NA3 Induction Course Development, Edinburgh, 27 April Relative Importance What envelopes you put your messages in  How they are delivered  Infrastructure to organise a common technical platform – the foundations of communication What information you send in your messages  Their patterns of Use - sequences that mean something  Their Contents  The Grammar and Vocabulary of Communication  Agreed Interpretations

NA3 Induction Course Development, Edinburgh, 27 April Relative Importance What envelopes you put your messages in  How they are delivered  Infrastructure to organise a common technical platform – the foundations of communication What information you send in your messages  Their patterns of Use - sequences that mean something  Their Contents  The Grammar and Vocabulary of Communication  Agreed Interpretations What you do when you get a message  The Application Code you Execute  The Middleware Services Security, Privacy, Authorisation, Accounting, Registries, Brokers, …  Integration Services Multi-site Hierarchical Scheduling, Data Access & Integration, …  Portals, Workflow Systems, Virtual Data, Semantic Grids  Tools to support Application Developers, Users & Operations Incremental deployment tools, diagnostic aids, performance monitoring, … Technical Experts

NA3 Induction Course Development, Edinburgh, 27 April Relative Importance What envelopes you put your messages in  How they are delivered  Infrastructure to organise a common technical platform – the foundations of communication What information you send in your messages  Their patterns of Use - sequences that mean something  Their Contents  The Grammar and Vocabulary of Communication  Agreed Interpretations What you do when you get a message  The Application Code you Execute  The Middleware Services Security, Privacy, Authorisation, Accounting, Registries, Brokers, …  Integration Services Multi-site Hierarchical Scheduling, Data Access & Integration, …  Portals, Workflow Systems, Virtual Data, Semantic Grids  Tools to support Application Developers, Users & Operations Creative Actions and Judgements of Researchers, Designers & Clinicians  Data, Models & Analyses  In Silico Experiments, Design, Diagnosis & Planning  Creating the Scientific Record Domain Experts

NA3 Induction Course Development, Edinburgh, 27 April Conclusions - Strategy If you are making long-term plans  Plan to use WSRF If you develop or research middleware  Engage with groups developing WSRF  If you need those or related functions E.g. to notify or handle state without incremental resource loss If you run distributed Grid operations  Plan to use WSRF  But only when components using it are robust  Incremental transition is possible – even necessary

NA3 Induction Course Development, Edinburgh, 27 April Conclusions - Tactics Value your team’s skills & momentum Change only if necessary If you’re an applications researcher  Stay with what you have working WS-I +GSI; GT2, VDT, LCG2, GT3, … If you’re a computing researcher  If your platform serves your investigation stay on it WS-I +GSI; GT2, VDT, LCG2, GT3, … If you’re doing middleware R&D  Hard choices & frustrating times – keep going  Those who understand the new order will reap advantage  Therefore engage with WSRF WSRF principles and design patterns  Are a useful guide to building distributed infrastructure  Adopt them, even while WSRF details are being agreed To ease transition

NA3 Induction Course Development, Edinburgh, 27 April Conclusions - Final WSRF  Good enough for recurrent platform requirements  Has significant commercial and technical momentum  Improves engagement with industry  Only sensible flag to rally behind Must collaborate internationally  Scale of challenge & international virtual organisations Discourage localised alternatives  Avoid effort fragmentation and unnecessary arguments Coping well with transitions … Is a primary Darwinian selector!