OpenPBS – Distributed Workload Management System Presented by - Anup Bhatt and William Schneble

PBS – Portable Batch System[2,3] Distributed workload management system – has threading and MPI integration Job management Queuing Scheduling Monitoring Uses Message orientated Middleware (MoM) that runs on each execution host Manages jobs by starting, tracking, stages files, returns output and runs cleanup on host

What is a job? [1] Job: task in form of shell script or batch file with PBS directives and application to run Directives come before program and commands and specify shell, resources, time and paths Shell specification PBS directive for nodes and number of CPUs PBS directive request for time Program directory Analyze program Run program Submit job to PBS server using qsub Request time, nodes, and other resources with submission

Typical Lifecycle of Job[3] Write job script and submit PBS accepts job and returns ID Scheduler finds execution host and sends job Licenses obtained PBS creates job-specific staging and execution directory on hosts Input files and directories copied to primary execution host CPUsets created (if needed) Job runs Output files and directories copied to specified locations Temp files and directories cleaned up Licenses returned CPUsets deleted

Checkpointing[4] So PBS shutdown or crashed… PBS automatically checkpoints jobs during shutdown Checkpoint creates image of job – has all necessary information for recovery Will not checkpoint jobs submitted with qsub –cn option If checkpoint space is exceeded then un-checkpointed jobs are lost Need manual checkpointing for crashes Use the qsub –c interval or write explicit code in program Why checkpoint? If the system crashes, PBS cannot automatically checkpoint the running processes. You might have a job you want to run in stages. Even if PBS checkpoints a running job, the PBS recovery of the job could fail. This could result from a lack of disk space when PBS is writing  a checkpoint file for your job; or it could fail if you modify or remove one of the files associated with one of the processes of the job before the job is recovered. Your job might exceed the CPU time limit specified when you submitted it for processing (a recovered job has the same time left that it had when it was checkpointed). You might have a calculation within a job and want to change the parameters of this calculation as the job proceeds. When checkpointing a job for this purpose, you may have to save more data.

Virtual Nodes[2,3] Virtual nodes: abstract object representing set of resources on a machine Can be an entire host, nodeboard or a blade and a single host can have multiple vnodes Host with multiple vnodes must have a natural vnode Defines invariant host information and dynamic and shared resources Chunks can be broken up across vnodes on same host

Message orientated Middleware[2,3] MoM: daemon that runs on each execution host and manages jobs Runs any prologs scripts and epilogue scripts At job start, creates new session that is identical to user’s Performs communication with job and other MoMs MoM on first node on which job is running manages all communication with others and is called Mother Superior

PBS Server[2,3] PBS Server: maintains file of nodes managed by PBS On startup MoM sends list of managed vnodes to server Scheduling: basic policy is job placement at host that can satisfy resource requirements – works down the lists of host in order Priority can be modified for execution, when to run each job, and preemption, which queued job can preempt running jobs

OpenPBS Extensions/Utilities[5] Maui Scheduler Updates state information Refreshes reservations Schedules reserved jobs Schedules priority jobs Backfills jobs Updates statistic Handles user requests: Any call to the system Perform next scheduling cycle Each iteration, the scheduler contacts the resource manager and requests up to date information on computer resources, workload, and policy configuration.  6. pestat - Node Resource Monitor collects node stats to check for availability 7. Any call requesting state information, configuration changes, or job or resource manipulation commands. Underlord 1. The system administrator can set thresholds to vary the significance of this stage in relationship with other stages.

UnderLord Scheduler Job Age Stage:  This stage considers the time that each job has been waiting in the queue and then assigns a weight based on that value. Job Duration Stage: This stage considers the projected duration of each job and assigns a weight based on that value. In systems running parallel jobs, the administrator can configure this stage to optionally multiply the duration by the number of processors requested. Queue Priority Stage: This stage evaluates the priority specified for each queue on the PBS server, as well as that queues historical system utilization. Each job from the corresponding queue is then provided a weight based on this value.

Underlord Scheduler - User Stages User Share Stage: Each user's fair share of system resources is specified within the configuration file. User Priority Stage: A user can specify a priority for each job submitted to the system. 1.The scheduler will consider the user's historic utilization (in terms of CPU time , wall time, and job count) and will provide the job with an appropriate weight. In the absence of other more significant weight factors, the user should receive his fair share of system resources over time.  2. If a user has multiple weighting jobs, this stage will alter their weight to favor the job with the highest priority. This stage should have no impact on the ordering of jobs submitted by different users. 

OpenPBS Industry Usage[7] NASA’s Pleiades supercomputer used OpenPBS for workload management Each NASA mission has a certain % of CPUs on Pleiades supercomputer A job cannot start if the action causes the mission to exceed its share of computing power PBS was originally developed for NASA under a contract project that began on June 17, 1991 Altair Engineering currently owns and maintains the intellectual property associated with PBS, and also employs the original development team from NASA.

Variants of PBS[6] There are several variants of PBS in the industry One such system is Altair PBS, used to schedule jobs in the CRAY supercomputer company to simulate crash/safety parameter evaluation in vehicles Other versions include TORQUE and PBS Professional They demonstrate the versatility of PBS

Work Cited https://hpcc.usc.edu/support/documentation/running-a-job-on-the-hpcc-cluster-using-pbs/ http://www.pbsworks.com/pdfs/PBSProAdminGuide13.1.pdf http://www.pbsworks.com/pdfs/PBSProUserGuide13.1.pdf http://www.tifr.res.in/~cc/pbs/pbs_checkpt.html http://www.mcs.anl.gov/research/projects/openpbs/ http://www.altairhyperworks.com/product/RADIOSS https://www.nas.nasa.gov/hecc/support/kb/portable-batch-system-(pbs)-overview_126.html