Jefferson Lab and the Portable Batch System Walt Akers High Performance Computing Group

Jefferson Lab and PBS:Motivating Factors New Computing Cluster –Alpha Based Compute Nodes –16 XP1000 Single Processor Nodes (LINPACK 5.61 GFlop/Sec) –8 UP2000 Dual Processor Nodes (LINPACK 7.48 GFlop/Sec) Heterogeneous Job Mix –Combination of Parallel and Non-Parallel Jobs –Job execution times range from a few hours to weeks –Data requirements range from minimal to several gigabytes Modest Budget –Much of our funding was from internal sources –Initial hardware expense was relatively high Expandability –Can the product be expanded from a few nodes to hundreds

Jefferson Lab and PBS:Alternative Systems PBS - Portable Batch System –Open Source Product Developed at NASA Ames Research Center DQS - Distributed Queuing System –Open Source Product Developed by SCRI at Florida State University LSF - Load Sharing Facility –Commercial Product from Platform Computing –Already Deployed by the Computer Center at Jefferson Lab Codine –Commercial Version of DQS from Gridware, Inc. Condor –A Restricted Source ‘Cycle Stealing’ Product From The University of Wisconsin Others To Numerous To Mention

Jefferson Lab and PBS:Why We Chose PBS? Portability –The PBS distribution compiled and ran immediately on both the 64 bit Alpha and 32 bit Intel platforms. Documentation –PBS comes with comprehensive documentation including an Administrators Guide, External Reference, and Internal Reference. Active Development Community –There is a large community worldwide that continues to improve and refine PBS. Modularity –PBS is a component oriented system. –A well defined API is provided to allow components to be replaced with locally defined modules. Open Source –The source code for the PBS system is available without restriction. Price –Hey, its free…

Jefferson Lab and PBS:The PBS View Of The World PBS Server –Mastermind of the PBS System –Central Point of Contact PBS Scheduler –Prioritizes Jobs –Signals Server to Start Jobs Machine Oriented Mini-Server (MOM) –Executes Scripts on Compute Nodes –Performs User File Staging

Jefferson Lab and PBS:The PBS Server Routing Queues –Can move jobs between multiple PBS Servers Execution Queues –Defines default characteristics for submitted jobs –Defines a priority level for queued jobs –Holds jobs before, during and after execution Node Capabilities –The server maintains a table of nodes, their capabilities and their availability. Job Requirements –The server maintains a table of submitted jobs that is independent of the queues. Global Policy –The server maintains global policies and default job characteristics.

Jefferson Lab and PBS:The PBS Scheduler Prioritizes Jobs –Called periodically by the PBS Server –Downloads job lists from the server sorts them based on locally defined requirements. Tracks Node Availability –Examines executing jobs to determine projected availability time for nodes. –Using this data the scheduler can calculate future deployments and determine when back-filling should be performed. Recommends Job Deployment –At the end of the scheduling cycle, the scheduler will submit a list of jobs that can be started immediately to the server. –The PBS Server is responsible for verifying that the jobs can be started, and then deploying them.

Jefferson Lab and PBS:Machine Oriented Mini-Server Executes Scripts –At the direction of the PBS Server, MOM executes the user provided scripts –For parallel jobs, the primary MOM (Mother Superior) starts the jobs on itself and all other assigned nodes. Stages Data Files –Prior to script execution, the MOM is responsible for remotely copying user specified data files to Mother Superior. –Following execution, the resultant data files are remote copied back to the user specified host. Tracks Resource Usage –MOM tracks the cpu time, wall time, memory and disk that has been used by the job. Kills Rogue Jobs –Kills jobs at the PBS Server’s request

Jefferson Lab and PBS:Our Current Implementation

Jefferson Lab and PBS:What We’ve Learned So Far PBS Is Reasonably Reliable, But Has Room For Improvement –PBS Server and PBS Scheduler components work well and behave predictably –PBS MOM works okay, but behaves bizarrely in certain situations Disk full = chaos Out of process slots = chaos Improper file transfer or staging = chaos Note: The first two can be avoided by conspicuous system management, the last is the responsibility of the job submitter. Red Hat Linux 6.2 –We’ve seen many problems associated with NFS. After upgrading to Kernel many of these problems went away. –Klogd occasionally spins out of control and uses all available CPU cycles. –Sshd on SMP machines dies for no apparent reason. –Crontab works intermittently on SMP nodes. We’re considering experimenting with True 64 Unix to see if these problems exist there. Writing a Scheduler Is Hard Work –We have developed two interim schedulers and are now working on the ‘final’ implementation.

Jefferson Lab and PBS:Ongoing Development Underlord Scheduling System –Built on the existing PBS Scheduler Framework Plug-in replacement for the default scheduler Uses an object oriented interface to the PBS Server –Comprehensive match making scheme Starts from an ordered list of jobs Works with a collection of homogeneous or heterogeneous nodes Locates the optimal node or combination of nodes where a job should be deployed Uses user specified job parameters to project future job deployment Uses future job scheduling in combination with backfilling to maximize system utilization. –Multi-layered job sorting algorithm Time in queue Projected execution time Number of processors requested Queue priority Progressive user share (similar to the LSF scheme) –Generates a projection table Allows users to determine when their job is projected to start

Jefferson Lab and PBS:Future Directions Data Grid Server –In order to provide greater flexibility to the Batch System and allow it to accommodate data provided through the proposed Data Grid system, a Data Grid Server will be added to the existing system components. –This module will have the following capabilities Will provide time projections for when data will be available Will perform data migration to a script accessible host Will provide mechanisms to transfer resultant data to a specified location Will replace the existing staging capabilities of the PBS Server and PBS MOM. PBS Meta-Facility - The Overlord Scheduler –The Overlord Scheduler will be a centralized location where jobs are submitted that can be forwarded to other PBS Clusters for execution. The Overlord Scheduler will have the following capabilities. Will prioritize and sort all jobs based on global Meta-Facility rules Will consider job requirements, data location and network throughput and will forward each job to the PBS Server where it will be scheduled earliest. Will not forward jobs to one of the ‘Underlord’ systems until it is eligible for immediate execution there. –We don’t have all of this figured out yet… but, we are confident.

Jefferson Lab and PBS:Places On The Web Jefferson Lab HPC Home Page – Currently we have most of the PBS documentation and some statistics about our cluster and its development. PBS Home Page – Register and download PBS and all documentation from this site`