1. Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com

2. Sept 8, 2004 Workflow + Globus Update 2 Agenda Workflow activity Globus Toolkit status CA Activity (Yonsook)

3. Sept 8, 2004 Workflow + Globus Update 3 Agenda Workflow activity Globus Toolkit status CA Activity (Yonsook)

4. Sept 8, 2004 Workflow + Globus Update 4 Workflow Activity Began in late 2003 Goal was to be able to chain processing among participants in a single workflow Held a series of teleconferences http://grid- tech.ceos.org/gridwiki/WorkflowTigerTeamhttp://grid- tech.ceos.org/gridwiki/WorkflowTigerTeam (Ask for username/password)

5. Sept 8, 2004 Workflow + Globus Update 5 Workflow activity - 1 First, we learned about workflow and about CEOS Grid Participants workflow tools…

6. Sept 8, 2004 Workflow + Globus Update 6 Workflow overview The next slides are taken from Tech Team members presentations. Annotations that were added for this presentation look like this Annotation box…

7. Sept 8, 2004 Workflow + Globus Update 7 What is a Workflow? Dutch Space has renamed their workflow tools to GridAssist…

8. Sept 8, 2004 Workflow + Globus Update 8 Workflow Levels Application Domain Abstract Workflow Concrete Workflow Execution Environment Application Component Selection Resource Selection Data Replica Selection Transformation Instance Selection = Data Transfer FFT FFT filea host2 host1 Retry Pick Different Resources Specify a Different Workflow copy host1://home/filea to host2://u1/file1 /usr/local/bin/fft /u1/file1 From Pegasus presentation by Ewa Deelman, ISI This material was redrawn to enhance visibility, its from Ewa Deelmans workflow presentation…

9. Sept 8, 2004 Workflow + Globus Update 9 Typical Workflow Environment

10. Sept 8, 2004 Workflow + Globus Update 10 Workflow example This is a domain (or application level) workflow…

11. Sept 8, 2004 Workflow + Globus Update 11 Workflow detailed example This is an abstract workflow…

12. Sept 8, 2004 Workflow + Globus Update 12 GMU Virtual Product Concept Virtual data concept requires workflow tools in order to be built…

13. Sept 8, 2004 Workflow + Globus Update 13 Workflow progression Different levels of workflow… Domain… Abstract… Concrete…

14. Sept 8, 2004 Workflow + Globus Update 14 Workflow tools DutchSpace - Grid Assist –Developed in-house, workflow engine and client tools. (Runs on Globus Toolkit 2.x) NCSA (UAH partner) - Open GCE Runtime Engine –Based on Apache Ant (not Globus dependent) ISI - Pegasus –Developed as part of Earth Systems Grid work w/Argonne (Runs on Globus 2.x, 3.x?) GMU - GeoTree –Based on BPEL (Business Process Execution Language) (Runs on Globus 3.2.x)

15. Sept 8, 2004 Workflow + Globus Update 15 Workflow Activity - 2 Then we investigated workflow interoperability. The plan was to 1.Ask participants for use cases 2.Decompose use cases into workflow patterns

16. Sept 8, 2004 Workflow + Globus Update 16 Use case descriptions Identifier Description Actors Goal Preconditions Assumptions Frequency Basic Course Alternate Courses Included Use Cases Extended Use Cases Issues Decisions Workflows can be derived from these descriptions of actions

17. Sept 8, 2004 Workflow + Globus Update 17 Workflow Patterns http://tmitwww.tm.tue.nl/research/patterns/ Provides a good explanation and definition of 21 workflow patterns. Next slide shows some of the simple ones as examples.

18. Sept 8, 2004 Workflow + Globus Update 18 Basic workflow patterns A B C Sequence: B is started after A completes. C is started after B completes. A AND B C Parallel Split: B and C are begun any time after A has completed. C AND A B Synchronization: C is not begun until A and B have both completed. A XOR B C Exclusive Choice: Following A, a choice is made to pursue B or C but not both. C XOR A B Simple Merge: C activated once for every completion of A or B

19. Sept 8, 2004 Workflow + Globus Update 19 Use case analysis Use cases were received from –USGS –UAH/NOMADS –Dutch Space USGS and UAH/NOMADS use cases were refined via teleconference + email All are available on Workflow web page

20. Sept 8, 2004 Workflow + Globus Update 20 Initial Findings Workflow tools are a necessary component! –Grid applications are partioned into work units that run on many machines. –Runtime error handling is a vital part of workflow engine capability Grid workflows + tools are complex –Tools take a long time to learn –Grid workflows support complex processing tasks Many (~30) different Grid workflow systems exist –Interoperability among workflow tools is currently nonexistent

21. Sept 8, 2004 Workflow + Globus Update 21 Use case findings - 1 Based on the USGS and NOMADS/UAH Use Cases Found that use cases tended to have many hands-on steps where scientists would make choices –This is one of the key areas the Grid workflow community has identified as needing work. Use of formal workflow tools may be more work than its worth for the use cases –Application projects are using the Grid mainly for data transfer, not computing –Workflows include some automation areas (reprojection, reformatting, subsetting).

22. Sept 8, 2004 Workflow + Globus Update 22 Use case findings - 2 Based on the Dutch Space Application description Dutch Space application is well suited for formal tool Dutch Space has implemented their own tool in response to their need and the (then) lack of existing tools. OMI application has long processing chain (c.f. earlier slides)

23. Sept 8, 2004 Workflow + Globus Update 23 Workflow Conclusions Workflow community is starting to talk about interoperable workflows –http://www.extreme.indiana.edu/groc/ ggf10-ww/index.html Perhaps cross-workflow interoperability is not an immediate need –A single workflow tool can be used to control remote job execution without needing to interoperate with other workflow tools. –CEOS Grid activity should wait until Application Projects have a clear need for cross-tool workflow interoperability before doing further work.

24. Sept 8, 2004 Workflow + Globus Update 24 Agenda Workflow activity Globus Toolkit status CA Activity (Yonsook)

25. Sept 8, 2004 Workflow + Globus Update 25 Globus Toolkit (GT) Update Current GT version is 3.2 –Most CEOS Globus users are still using 2.4 GT version 4.0 release is scheduled for January 31, 2005 –http://www-unix.globus.org/toolkit/GT4Facts/http://www-unix.globus.org/toolkit/GT4Facts/ GT4 will be backwards compatible with GT3.2 Next 5 slides taken from the presentation given in Tromso, also available at http://grid-tech.ceos.org/gridwiki/GlobusVersions

26. Sept 8, 2004 Workflow + Globus Update 26

27. Sept 8, 2004 Workflow + Globus Update 27

28. Sept 8, 2004 Workflow + Globus Update 28

29. Sept 8, 2004 Workflow + Globus Update 29

30. Sept 8, 2004 Workflow + Globus Update 30

31. Sept 8, 2004 Workflow + Globus Update 31 GT 4.0 schedule and content Next two slides have material from http://www-unix.globus.org/toolkit/GT4Facts/

32. Sept 8, 2004 Workflow + Globus Update 32 GT4 Release Schedule

33. Sept 8, 2004 Workflow + Globus Update 33 GT4 Components

34. Sept 8, 2004 Workflow + Globus Update 34 Globus Toolkit Recommendations If you are using –GT 2.4 Migrate to 4.0 Dont go to 3.2 as intermediate step –GT 3.2 Migrate to 4.0 4.0 will be backwards compatible

35. Sept 8, 2004 Workflow + Globus Update 35 Agenda Workflow activity Globus Toolkit status CA Activity (Yonsook)

36. Sept 8, 2004 Workflow + Globus Update 36 CA Activity Following earlier work on Certificate Authorities (CA), NASA IPG (Information Power Grid) volunteered to provide CA services for CEOS Grid. NASA IPG funding has been cut –CEOS Grid cannot rely on IPG CA anymore We decided to establish a CEOS CA on a NASA machine at GSFC

37. Sept 8, 2004 Workflow + Globus Update 37 CA Activity - cont. Ananth Rao has installed a test CA and has produced a Certificat Request tool similar to the one provided by IPG This tool will undergo further testing after WGISS-18 Once we are satisfied with it, we will provide it to the CEOS Grid