1. Batch Systems
P. Nilsson, PROOF Meeting, October 18, 2005

2. Overview Sun’s Grid Engine [Open Source]
What is the Grid Engine project?
Daemon components
Execution daemon
Priorities
Job life cycle
Platform LSF [Commercial system]
What is Platform LSF? CERN Batch Service
Product features
Architecture
Load sharing
Other Batch Systems
OpenPBS, Condor, BQS, …
Maui Scheduler [Open Source]
October 18, 2005
Paul Nilsson

3. Sun’s Grid Engine What is the Grid Engine project?
“An open source community effort to facilitate the adoption of distributed computing solutions”, sponsored by Sun
The project provides distributed resource management software for wide ranging requirements from compute farms to grid computing
The Grid Engine has been ported to many operating systems, including Sun Solaris, Linux, SGI IRIX, Compaq/HP Tru64, IBM AIX, HP HP/UX, Apple Mac OS/X and others. The project welcomes those who are interested in implementing new ports or in taking over the maintenance of an existing port Good documentation…
More information at
October 18, 2005
Paul Nilsson

4. Grid Engine Components
Sun’s Grid Engine
Grid Engine Components
Qmaster [“queue master”] controls the overall behavior in a cluster, responsible for answering requests from clients and for delivering dispatched jobs to the assigned Execd's
Schedd [“scheduling daemon”] gets notified about all scheduling relevant information. Resulting scheduling decisions are sent as orders to Qmaster
Execd [“execution daemon”] provides Qmaster with information about utilization and availability of resources. A job sent to Execd is started by writing all relevant information into files describing the job and forking a “Shepherd”. After Shepherd’s termination, Execd reports back to Qmaster
Shepherd starts all kinds of jobs according to what he finds in the per-job configuration files written by Execd
Commd [“communication daemon”] handles network communication in a cluster
Shadowd [“shadow daemon”] detects failures of the Qmaster and starts a new Qmaster if necessary
October 18, 2005
Paul Nilsson

5. Execd – The Execution Daemon
Sun’s Grid Engine
Execd – The Execution Daemon
The Execution Daemon is the instance that:
Starts jobs
Controls jobs (e.g. it can suspend/unsuspend a job, reprioritize the processes associated with a job, etc)
Gathers information about jobs (e.g. resource usage, exit code, etc)
Gathers information about the execution host it controls (e.g. load, free memory, etc)
There is one execd on each host of a cluster
October 18, 2005
Paul Nilsson

6. Much more information about the scheduler is in the documentation…
Sun’s Grid Engine
On Priorities…
The Grid Engine has the feature of a share-based scheduler, where each job gets a certain share of the system resources
The sum of all shares for a job is expressed in tickets. A job has a certain number of tickets enabling it to run with certain process priorities
If multiple jobs are running concurrently on a host, their different share of system resources (their different number of tickets) can be mapped to priorities in the OS
Setting priorities in the OS is done by either setting the nice value for all processes of a job or by using special priority mapping facilities provided by the OS
The Grid Engine reassigns the number of tickets per job in a regular interval. It then maps the number of tickets of a job to nice values (or another operating system priority representation) and renices all processes of the job
Much more information about the scheduler is in the documentation…
October 18, 2005
Paul Nilsson

7. Job Life Cycle Execds report load information to Qmaster
Sun’s Grid Engine
Job Life Cycle
Execds report load information to Qmaster
User submits job using qsub command
Qmaster notifies Schedd about new job
Schedd dispatches job to an Execd
Qmaster delivers job to Execd; Execd starts job using Shepherd
At job end Execd notifies Qmaster about job finish
Qmaster feeds jobs resource consumption in accounting database
Execd1
Execd2
ExecdN 1 1 1 5 4 6
Schedd
Qmaster 3 7 2
qsub
October 18, 2005
Paul Nilsson

8. Platform LSF What is Platform LSF? CERN Batch Service
“Platform LSF (Load Sharing Facility) is a [commercial] workload management solution that optimizes the use of enterprise-wide resources by providing transparent, on demand access to valuable computing resources”
CERN Batch Service
CERN Batch Service provides an “LSF” farm with 1500 dual-processor machines for data analysis and simulation (used CPU time is accounted to the experiments!!!)
LXPLUS is used for the public logon (i.e. for the job submits)
Depending on the load and resource requirements of jobs up to 3 jobs are running in parallel on the same node
More information at
October 18, 2005
Paul Nilsson

9. Platform LSF
Product features
Dynamic Load Balancing by continuously monitoring of system resources; CPU and memory usage, swap space, software license availability (!)
Resource-based Queuing and Scheduling. Resources are dynamically managed based on policies, schedules and thresholds; jobs submitted to any network-based queues are automatically processed as resources become available
Optimal Resource Sharing. Continuous resource management, even in the event of host failures: failed jobs are automatically re-run and failed servers restarted
Administrative Control and Policies. Admins can suspend, stop, and submit jobs from any node in a network; users can modify their own jobs once submitted to queues. Varied options are available for configuring workload policies, supporting resource sharing by users, user groups, and projects
October 18, 2005
Paul Nilsson

10. Load Sharing Load index Description
Platform LSF
Load Sharing
To achieve load sharing, LSF must have up to date information about the load on each machine in a cluster. The Load Information Manager (LIM) component is responsible for this.
A LIM daemon runs on each host of the cluster. It gathers information about its host and makes the information available to all hosts. The information is organized as a load vector. The load vector comprises a number of load indices as described in the following table
Load index Description
r15s Load average for last 15 seconds
r1m Load average for last minute exponentially averaged CPU run queue lengths
r15m Load average for last 15 minutes
ut Percent CPU utilization averaged over last minute
pg Paging (in/out) activity over last 20 seconds
ls Number of login sessions
it Idle time - number of minutes since last keyboard or mouse activity
tmp Available space (MB) in /tmp file system
swp Available swap space (MB)
mem Available real memory (MB)
October 18, 2005
Paul Nilsson

11. Job Life Cycle 1 Platform LSF User submits a job to LSF for execution
The submitted job proceeds through the batch library to the Load Information Manager (LIM)
LIM communicates the job's information to the cluster's master LIM. Periodically, the LIM on individual machines gathers its 12 built-in load indices and forwards this information to the master LIM [see previous slide]
The master LIM determines the best host to run the job and sends this information back to the submission host's LIM
(Information about the chosen execution host is passed through the batch library)
Information about the host to execute the job is passed back to the bsub process or lsb_submit() function
To enter the batch system, bsub or lsb_submit() sends the job to the batch library
Using batch library services, the job is sent to the mbatchd running on the cluster's master host 1
October 18, 2005
Paul Nilsson

12. Platform LSF
Job Life Cycle
The mbatchd puts the job in an appropriate queue and waits for the appropriate time to dispatch the job. User jobs are held in batch queues by mbatchd, which checks the load information on all candidate hosts periodically
The mbatchd dispatches the job when an execution host with the necessary resources becomes available where it is received by the host's sbatchd
sbatchd controls the execution of the job and reports the job's status to mbatchd. The sbatchd creates a child sbatchd to handle job execution
The child sbatchd sends the job to the Remote Execution Server (RES)
The RES creates the execution environment to run the job
The job is run in the execution environment
The results of the job are sent to the system
The system sends the job's results to the user 2
October 18, 2005
Paul Nilsson

13. Other Batch Systems What other batch systems are on the market?
OpenPBS - Open source Portable Batch System [Unsupported version – development stopped in 1999]. “Flexible batch queuing system developed for NASA in the early to mid-1990s. It operates on networked, multi-platform UNIX environments”. Developed into a commercial PBS Pro version ( compilation requires hacker intervention..).
Public home:
Condor – “Specialized workload management system for compute-intensive jobs. Like other full-featured batch systems, Condor provides a job queuing mechanism, scheduling policy, priority scheme, resource monitoring, and resource management. Users submit their serial or parallel jobs to Condor, Condor places them into a queue, chooses when and where to run the jobs based upon a policy, carefully monitors their progress, and ultimately informs the user upon completion” (
BQS - Batch Queuing System. Some information at CC-IN2P3 web site ( …
October 18, 2005
Paul Nilsson

14. Maui Scheduler
Open source batch queuing and scheduling software designed to schedule parallel jobs
Maui can schedule the order of job execution for queued jobs (from other batch systems)
Has lots of scheduling concepts; FIFO (First-in first-out) like reservations, back-filling of jobs, job priorities, time-of-day scheduling, etc
Written in Java
Maui Scheduler has been designed to communicate directly with a database through an abstraction layer BUT currently MySQL is the only one implemented [MySQL is required]
Considered for CASTOR2 (until LSF was chosen)
What is “backfilling”?
Maui allows a lower priority job to be executed before a higher priority job if it does not delay the start of the prioritized job [apparently not found in other schedulers]
More information at
October 18, 2005
Paul Nilsson