Pre-GDB on Batch Systems (Bologna)11 th March Torque/Maui PIC and NIKHEF experience C. Acosta-Silva, J. Flix, A. Pérez-Calero (PIC) J. Templon (NIKHEF)

Pre-GDB on Batch Systems (Bologna)11 th March Outline ‣ System overview ‣ Successful experience (NIKHEF and PIC) ‣ Torque/Maui current situation ‣ Torque overview ‣ Maui overview ‣ Outlook

Pre-GDB on Batch Systems (Bologna)11 th March System overview ‣ TORQUE is a community and commercial effort based on OpenPBS project. It improves scalability, enables fault tolerance and many other features ‣ ‣ Maui Cluster Scheduler is a job scheduler capable of supporting multiple scheduling policies. It is free and open- source software ‣

Pre-GDB on Batch Systems (Bologna)11 th March System overview ‣ TORQUE/Maui system has the usual batch system capabilities: ‣ Queues definition (routing queues) ‣ Accounting ‣ Reservation/QOS/Partition ‣ FairShare ‣ Backfilling ‣ Handling of SMP and MPI jobs ‣ Multicore allocation and job backfilling ensure that Torque/Maui is capable of supporting multicore jobs

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience ‣ NIKHEF and PIC are multi-VO sites with local & Grid users ‣ Succesful experience during first LHC run with Torque/Maui system ‣ Currently, both are running Torque Maui ‣ NIKHEF: 30% non-HEP, 55% WLCG, rest non-WLCG HEP or local jobs. Highly non-uniform workload ‣ 3800 jobs slots ‣ 97.5% utilization (last 12 months) ‣ 2000 waiting jobs (average)

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience NIKHEF: running jobs (last year) NIKHEF: queued jobs (last year)

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience ‣ PIC: 3% non-HEP, 83% Tier-1 WLCG, 12% ATLAS Tier-2, rest local jobs (ATLAS Tier-3, T2K, MAGIC,…) ‣ 3500 jobs slots ‣ 95% approx utilization (last 12 months) ‣ 2500 waiting jobs (average)

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience PIC: running jobs (last year)

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience PIC: queued jobs (last year)

Pre-GDB on Batch Systems (Bologna)11 th March Torque overview ‣ Torque has a very active community: ‣ Mailing list: ‣ Total free support from Adaptive Computing ‣ New releases each year (approx. or less) and frequent new patches ‣ is the last release of branch 2.5.X

Pre-GDB on Batch Systems (Bologna)11 th March Torque overview

Pre-GDB on Batch Systems (Bologna)11 th March Torque overview ‣ Torque is well integrated with EMI middleware ‣ Vastly used in WLCG Grid sites (~75% of sites in BDii -pbs-) ‣ No complex to install, configure and manage: ‣ via qmgr tool ‣ plain text accounting ‣ … ‣ Torque scalability issues ‣ Reported for branch 2.5.X ‣ Not detected at our scale ‣ Branch 4.2.X presents significant enhancements to scalability for large environments, responsiveness, reliability, …

Pre-GDB on Batch Systems (Bologna)11 th March Maui overview ‣ Support: Maui is no longer supported by Adaptive Computing ‣ Documentation: ‣ Poor documentation causes initial complexity to install it ‣ Things do not always work like the documentation suggests ‣ Scalability issues: ‣ At ~8000 queued jobs, Maui hangs ‣ MAXIJOBS parameter can be adjusted to limit the number of jobs consider for scheduling ‣ This solves this issue (currently in production in NIKHEF)

Pre-GDB on Batch Systems (Bologna)11 th March Maui overview ‣ Moab is the non-free scheduler supported by Adaptive Computing and based in Maui ‣ Aims to increase the scalability ‣ It is a continued commercial support ‣ Configuration files are very similar to the ones in Maui: ‣ Feedback from sites running Torque/Moab would be a good complement to this review

Pre-GDB on Batch Systems (Bologna)11 th March Outlook ‣ Torque/Maui scalability issues ‣ Only relevant for larger sites ‣ feasible option for small-medium size sites ‣ Might be well solved in 4.2.X branch and tunning Maui options ‣ Actually, multicore jobs reduces the number of jobs to be handled by the system ‣ for sites that are predominantly WLCG (eg PIC at 95%), switching to a pure multicore load would further reduce scheduling issues at the site level. ‣ for sites that are much less WLCG dominated (eg Nikhef at 55%), a switch to pure multicore load might actually increase scheduling issues at the site level, as this move would remove much of the entropy which allows reaching 97% utilization. ‣ Another concern is the support for the systems, being Maui the weakest link for the Torque/Maui combination

Pre-GDB on Batch Systems (Bologna)11 th March Outlook ‣ Some future options ‣ Change from Maui to Moab (but, it is not free!) ‣ Setting up a kind of “OpenMaui” project within WLCG-sites as a community effort to provide support and improvements to Maui ‣ Integrate with another scheduler. Which one? ‣ Complete change to another system (SLURM, HTCondor, …) ‣ “Do nothing” until a real problem arrives ‣ Currently, just a worry, no real problem detected so far in PIC/NIKHEF ‣ Improvements from migrating to another system unclear

Pre-GDB on Batch Systems (Bologna)11 th March Outlook ‣ Questions: ‣ If decided for WLCG sites to move away from Torque/Maui, would it be feasible before the LHC Run2? ‣ Migration to a new batch system requires time and effort, thus manpower and expertise, in order to reach and adequate performance for a Grid site ‣ Not clear if needed before Run2 ‣ What happens with sites shared with non-WLCG VOs? ‣ Impact on other users (NIKHEF 45%) ‣ For PIC, several disciplines rely on local job submissions. A change on the batch system affects many users, and requires re-education, changes, and tests of their submission tools to adapt to an eventual new system