1. Job Submission Condor, Globus, Java CoG Kit Young Suk Moon

2. Workflow The computerized facilitation or automation of a business process, in whole or part. W i T = ( G r T, G s T, Q u T, W m T ) – W i = Workflow instantiation – G r = Grid resources – G s = Grid services – Q u = Quality expectations from the user – W m = Workflow model. http://grid.rit.edu2 von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing.

3. Condor High throughput computing system – Job management – Scheduling policy – Priority scheme – Resource monitoring / management Checkpointing – Fault tolerance – Migration 3http://grid.rit.edu

4. Condor Kernel architecture Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. 4http://grid.rit.edu

5. Condor Job submission in a condor pool Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. 5http://grid.rit.edu

6. Condor Gateway flocking Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. 6http://grid.rit.edu

7. Condor-G Interacting with GRAM Disadvantage – the agent does not know which resource is available in the foreign system Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. 7http://grid.rit.edu

8. Condor – Gliding In Step 1 – submit Condor servers as jobs Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. 8http://grid.rit.edu

9. Condor – Gliding In Step 2 – submitted servers form an ad-hoc Condor pool with the matchmaker Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. 9http://grid.rit.edu

10. Condor – Gliding In Step 3 – use the Condor pool as normal Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. 10http://grid.rit.edu

11. Condor - Problem Solvers Master-Worker – Master process assigns jobs to worker processes – Suitable for a large problem with no dependency DAGMan (The Directed Acyclic Graph Manager) – For multiple jobs with dependancy Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. A Directed Acyclic Graph 11 http://grid.rit.edu

12. Condor Universes Define execution environments – Standard (default) : checkpointing & remote system calls – Vanila : run programs which cannot be re-linked – PVM : run Parallel Virtual Machine programs – MPI : run MPICH programs – Globus : use a Globus system – Java : run on JVM – Scheduler : executes jobs without waiting to be matched to resources. Used for DAGMan 12http://grid.rit.edu

13. Condor Example submit description file – submit.hello Executable = hello Universe = standard Output = hello.out Log = hello.log Queue To submit the job – > condor_submit submit.hello 13http://grid.rit.edu

14. Globus GRAM Grid Resource Allocation and Management Program execution requirements – State – User executables – Staging of input and output – Streaming output – Control – Scheduler – Monitoring 14http://grid.rit.edu

15. Globus GRAM State transition diagram for GRAM jobs Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4. 15http://grid.rit.edu

16. Globus GRAM Client APIs – Four portTypes: Managed Job Factory, Managed Job, Delegation Factory, Delegation Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 operations interfaces resources 16http://grid.rit.edu

17. Globus GRAM GT4 Container – Hosts Web Services Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 17http://grid.rit.edu

18. Globus GRAM GRAM implementation structure Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 18http://grid.rit.edu

19. Globus GRAM Job submission % globusrun-ws -submit -job-command /bin/program arg Submitting job…Done Job ID: uuid:c51fe35a-4fa3-11d9-9cfc-000874404099 Termination time: 12/17/2004 20:47 GMT Current job state: Active Current job state: CleanUp Current job state: Done Destroying job...Done. Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 19http://grid.rit.edu

20. Globus GRAM Job submission with job description files % globusrun-ws -submit -job-description-file touch.xml -factory-epr-file gram.epr #touch.xml /bin/program arg #gram.epr https://viz-login.isi.edu:9000/wsrf/services/ManagedJobFactoryService PBS Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 20http://grid.rit.edu

21. Java CoG Kit Java Commodity Grid Toolkit Maps Grid functionality into a commodity framework (Java) Extends the use of Globus Purpose – To use Grid services more efficiently by using / developing Java classes 21http://grid.rit.edu

22. Java CoG Kit Workflow management layer – It supports multiple Grid middleware Grid Middleware + CoG Kit providers GT2, SSH, GT4 WS-RF, WebDAV Grid Middleware + CoG Kit providers GT2, SSH, GT4 WS-RF, WebDAV Job submission file transfer & authentication Workflow Abstractions, Queue/Set Abstractions Job submission file transfer & authentication Workflow Abstractions, Queue/Set Abstractions Task & workflow management Karajan (GridAnt, ant) Task & workflow management Karajan (GridAnt, ant) Workflow monitoring Workflow Portlet & visualizer/Editor Workflow monitoring Workflow Portlet & visualizer/Editor Application Nano Materials, Bio-Informatics, etc. Application Nano Materials, Bio-Informatics, etc. 22http://grid.rit.edu

23. Java CoG Kit - Job Submission (GUI) http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf 23http://grid.rit.edu

24. Java CoG Kit – Command Line Job execution – globusrun File transfer – globus-url-copy GASS (Global Access to Secondary Storage) – globus-gass-server – globus-gass-server-shutdown 24http://grid.rit.edu

25. Java CoG Kit - Shell http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf 25 http://grid.rit.edu

26. References Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://www.cs.wisc.edu/condor/manual/v6.4/2_4Road_map_Running.html http://www.cs.wisc.edu/condor/quick-start.html Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing. http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf von Laszewski, G., Foster, I., Gawor, J., & Lane, P. (2001). A Java Commodity Grid Kit. Concurrency and Computation: Practice and Experience, 13(8-9), 643-662. 26http://grid.rit.edu