3-2.1 Topics Grid Computing Meta-schedulers –Condor-G –Gridway Distributed Resource Management Application (DRMAA) © 2010 B. Wilkinson/Clayton Ferner. Spring 2010 Grid computing course. Modification date: Feb 15, 2010

Meta-schedulers Schedule jobs across distributed sites Highly desirable in a Grid computing environment. For a Globus installation, interfaces to local Globus GRAM installation, which in turn interfaces with local job scheduler Uses whatever local scheduler present at each site 3-2.2

3-2.3 Fig 3-17 Meta-scheduler interfacing to Globus GRAM

Condor-G A version of Condor that interfaces to Globus environment. Jobs submitted to Condor through Grid universe and directed to Globus job manager (GRAM) Fig 3-18

Simple job description file universe = grid grid_resource = gt4 wsrf/services/ManagedJobFactoryService Fork executable = /usr/bin/uptime log = condor_test1.log output = condor_test1.out error = condor_test1.error should_transfer_files = YES when_to_transfer_output = ON_EXIT queue 3-2.5

3-2.6 Communication between user, myProxy server, and Condor-G for long-running jobs Fig 3.19

Gridway A meta-scheduler designed specifically for a Grid computing environment Interfaces to Globus components. Project began in Now open source. Became part of Globus distribution from version onwards (June 2007)

Has ability to match jobs to resources using both static information about job and resources, and dynamic information (resource load) Dynamic scheduling Automatic job migration, including controlled by the job during execution Checking for both fault tolerance and dynamic job migration Reporting and accounting facilities Basic job dependencies (workflow) 3-2.8

Can be installed on: Client machines to interact with a distributed system or Server where multiple users access it. Uses file transfer, execution management, and information services of Globus 3-2.9

Globus components used with Gridway Fig 3-20

Submitting a Gridway Job Jobs described in a Gridway job template (GWJT). Sample EXECUTABLE = /bin/ls EXECUTABLE = /usr/bin/uptime STDOUT_FILE = stdout.${JOB_ID} STDERR_FILE = stderr.${JOB_ID} If above saved as file myJob.jt, command: gwsubmit -t myJog.jt would cause it to be submitted to be executed on a Grid resource

Job matching Uses REQUIREMENTS and RANK (similar to Condor) REQUIREMENTS expression has to evaluate to TRUE for the execution host to be consider at all for job. RANK expression computed for each host and host with higher ranks used first for job

Example: RANK = CPU_MHZ

Array Jobs Specifying multiple instances of job, with different arguments. EXECUTABLE = myJob.exe ARGUMENTS = ${TASK_ID} STDOUT_FILE = stdout_file.${TASK_ID} STDERR_FILE = stderr_file.${TASK_ID} RANK = CPU_MHZ RANK expression - processors with higher clock frequency preferred (should be coupled with processor type if higher clock frequency to mean higher performance.) Array of 10 instances of myJob.exe could be submitted with: gwsubmit -t myJob.jt -n

Distributed Resource Management (DRM) systems Term used to cover job schedulers and the like. Several choices of DRM’s for a system, each having different characteristics and modes of operation and different commands and APIs

Distributed Resource Management Application (DRMAA) (pronounced “drama”) Standard set of API’s for submission and control of jobs to DRM’s Bindings in C/C++, Java, Perl, Python, and Ruby for a range of DSMs including (Sun) Grid Engine, Condor, PBS/Torque, LSF and Gridway

Scheduler with DRMAA interface Fig 3.21

Example of the use of DRMAA Fig 3.22

Questions