Job Submission Condor, Globus, Java CoG Kit Young Suk Moon

Workflow The computerized facilitation or automation of a business process, in whole or part. W i T = ( G r T, G s T, Q u T, W m T ) – W i = Workflow instantiation – G r = Grid resources – G s = Grid services – Q u = Quality expectations from the user – W m = Workflow model. von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing.

Condor High throughput computing system – Job management – Scheduling policy – Priority scheme – Resource monitoring / management Checkpointing – Fault tolerance – Migration 3http://grid.rit.edu

Condor Kernel architecture Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), http://grid.rit.edu

Condor Job submission in a condor pool Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), http://grid.rit.edu

Condor Gateway flocking Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), http://grid.rit.edu

Condor-G Interacting with GRAM Disadvantage – the agent does not know which resource is available in the foreign system Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), http://grid.rit.edu

Condor – Gliding In Step 1 – submit Condor servers as jobs Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), http://grid.rit.edu

Condor – Gliding In Step 2 – submitted servers form an ad-hoc Condor pool with the matchmaker Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), http://grid.rit.edu

Condor – Gliding In Step 3 – use the Condor pool as normal Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), http://grid.rit.edu

Condor - Problem Solvers Master-Worker – Master process assigns jobs to worker processes – Suitable for a large problem with no dependency DAGMan (The Directed Acyclic Graph Manager) – For multiple jobs with dependancy Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), A Directed Acyclic Graph 11

Condor Universes Define execution environments – Standard (default) : checkpointing & remote system calls – Vanila : run programs which cannot be re-linked – PVM : run Parallel Virtual Machine programs – MPI : run MPICH programs – Globus : use a Globus system – Java : run on JVM – Scheduler : executes jobs without waiting to be matched to resources. Used for DAGMan 12http://grid.rit.edu

Condor Example submit description file – submit.hello Executable = hello Universe = standard Output = hello.out Log = hello.log Queue To submit the job – > condor_submit submit.hello 13http://grid.rit.edu

Globus GRAM Grid Resource Allocation and Management Program execution requirements – State – User executables – Staging of input and output – Streaming output – Control – Scheduler – Monitoring 14http://grid.rit.edu

Globus GRAM State transition diagram for GRAM jobs Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4. 15http://grid.rit.edu

Globus GRAM Client APIs – Four portTypes: Managed Job Factory, Managed Job, Delegation Factory, Delegation Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 operations interfaces resources 16http://grid.rit.edu

Globus GRAM GT4 Container – Hosts Web Services Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 17http://grid.rit.edu

Globus GRAM GRAM implementation structure Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 18http://grid.rit.edu

Globus GRAM Job submission % globusrun-ws -submit -job-command /bin/program arg Submitting job…Done Job ID: uuid:c51fe35a-4fa3-11d9-9cfc Termination time: 12/17/ :47 GMT Current job state: Active Current job state: CleanUp Current job state: Done Destroying job...Done. Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 19http://grid.rit.edu

Globus GRAM Job submission with job description files % globusrun-ws -submit -job-description-file touch.xml -factory-epr-file gram.epr #touch.xml /bin/program arg #gram.epr PBS Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 20http://grid.rit.edu

Java CoG Kit Java Commodity Grid Toolkit Maps Grid functionality into a commodity framework (Java) Extends the use of Globus Purpose – To use Grid services more efficiently by using / developing Java classes 21http://grid.rit.edu

Java CoG Kit Workflow management layer – It supports multiple Grid middleware Grid Middleware + CoG Kit providers GT2, SSH, GT4 WS-RF, WebDAV Grid Middleware + CoG Kit providers GT2, SSH, GT4 WS-RF, WebDAV Job submission file transfer & authentication Workflow Abstractions, Queue/Set Abstractions Job submission file transfer & authentication Workflow Abstractions, Queue/Set Abstractions Task & workflow management Karajan (GridAnt, ant) Task & workflow management Karajan (GridAnt, ant) Workflow monitoring Workflow Portlet & visualizer/Editor Workflow monitoring Workflow Portlet & visualizer/Editor Application Nano Materials, Bio-Informatics, etc. Application Nano Materials, Bio-Informatics, etc. 22http://grid.rit.edu

Java CoG Kit - Job Submission (GUI) 23http://grid.rit.edu

Java CoG Kit – Command Line Job execution – globusrun File transfer – globus-url-copy GASS (Global Access to Secondary Storage) – globus-gass-server – globus-gass-server-shutdown 24http://grid.rit.edu

Java CoG Kit - Shell

References Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing. von Laszewski, G., Foster, I., Gawor, J., & Lane, P. (2001). A Java Commodity Grid Kit. Concurrency and Computation: Practice and Experience, 13(8-9), http://grid.rit.edu