Pre-GDB on Batch Systems (Bologna)11 th March 2014 1 Torque/Maui PIC and NIKHEF experience C. Acosta-Silva, J. Flix, A. Pérez-Calero (PIC) J. Templon (NIKHEF)

Slides:



Advertisements
Similar presentations
1 User Analysis Workgroup Update  All four experiments gave input by mid December  ALICE by document and links  Very independent.
Advertisements

Presented by: Priti Lohani
Scheduling under LCG at RAL UK HEP Sysman, Manchester 11th November 2004 Steve Traylen
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
S. Gadomski, "ATLAS computing in Geneva", journee de reflexion, 14 Sept ATLAS computing in Geneva Szymon Gadomski description of the hardware the.
Presented by Scalable Systems Software Project Al Geist Computer Science Research Group Computer Science and Mathematics Division Research supported by.
Workload Management Workpackage Massimo Sgaravatto INFN Padova.
A Grid Resource Broker Supporting Advance Reservations and Benchmark- Based Resource Selection Erik Elmroth and Johan Tordsson Reporter : S.Y.Chen.
6/2/20071 Grid Computing Sun Grid Engine (SGE) Manoj Katwal.
Workload Management Massimo Sgaravatto INFN Padova.
J. Templon Nikhef Amsterdam Physics Data Processing Group Scheduling multicore workload on shared multipurpose clusters A. Perez-Calero Yzquierdo, J.Flix.
Next Generation of Apache Hadoop MapReduce Arun C. Murthy - Hortonworks Founder and Architect Formerly Architect, MapReduce.
SEG Software Maintenance1 Software Maintenance “The modification of a software product after delivery to correct faults, to improve performance or.
DONVITO GIACINTO (INFN) ZANGRANDO, LUIGI (INFN) SGARAVATTO, MASSIMO (INFN) REBATTO, DAVID (INFN) MEZZADRI, MASSIMO (INFN) FRIZZIERO, ERIC (INFN) DORIGO,
Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES News on monitoring for CMS distributed computing operations Andrea.
Assessment of Core Services provided to USLHC by OSG.
Self-Organizing Agents for Grid Load Balancing Junwei Cao Fifth IEEE/ACM International Workshop on Grid Computing (GRID'04)
Workload Management WP Status and next steps Massimo Sgaravatto INFN Padova.
Cluster Reliability Project ISIS Vanderbilt University.
HTCondor at the RAL Tier-1 Andrew Lahiff, Alastair Dewhurst, John Kelly, Ian Collier pre-GDB on Batch Systems 11 March 2014, Bologna.
Tier-1 Batch System Report Andrew Lahiff, Alastair Dewhurst, John Kelly, Ian Collier 5 June 2013, HEP SYSMAN.
© 2008 IBM Corporation ® Atlas for Lotus Connections Unlock the power of your social network! Customer Overview Presentation An IBM Software Services for.
Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.
Scalable Systems Software Center Resource Management and Accounting Working Group Face-to-Face Meeting October 10-11, 2002.
DOSAR Workshop, Sao Paulo, Brazil, September 16-17, 2005 LCG Tier 2 and DOSAR Pat Skubic OU.
Grid job submission using HTCondor Andrew Lahiff.
WLCG Service Report ~~~ WLCG Management Board, 1 st September
Quick Introduction to NorduGrid Oxana Smirnova 4 th Nordic LHC Workshop November 23, 2001, Stockholm.
Multi-core jobs at the RAL Tier-1 Andrew Lahiff, Alastair Dewhurst, John Kelly February 25 th 2014.
Issues Autonomic operation (fault tolerance) Minimize interference to applications Hardware support for new operating systems Resource management (global.
Dan Tovey, University of Sheffield User Board Overview Dan Tovey University Of Sheffield.
GLite – An Outsider’s View Stephen Burke RAL. January 31 st 2005gLite overview Introduction A personal view of the current situation –Asked to be provocative!
OSG Tier 3 support Marco Mambelli - OSG Tier 3 Dan Fraser - OSG Tier 3 liaison Tanya Levshina - OSG.
Review of Condor,SGE,LSF,PBS
Scalable Systems Software for Terascale Computer Centers Coordinator: Al Geist Participating Organizations ORNL ANL LBNL.
Enabling Grids for E-sciencE SGE J. Lopez, A. Simon, E. Freire, G. Borges, K. M. Sephton All Hands Meeting Dublin, Ireland 12 Dec 2007 Batch system support.
Ruth Pordes November 2004TeraGrid GIG Site Review1 TeraGrid and Open Science Grid Ruth Pordes, Fermilab representing the Open Science.
CERN IT Department CH-1211 Genève 23 Switzerland t Frédéric Hemmer IT Department Head - CERN 23 rd August 2010 Status of LHC Computing from.
A PanDA Backend for the Ganga Analysis Interface J. Elmsheuser 1, D. Liko 2, T. Maeno 3, P. Nilsson 4, D.C. Vanderster 5, T. Wenaus 3, R. Walker 1 1: Ludwig-Maximilians-Universität.
6/23/2005 R. GARDNER OSG Baseline Services 1 OSG Baseline Services In my talk I’d like to discuss two questions:  What capabilities are we aiming for.
Slide David Britton, University of Glasgow IET, Oct 09 1 Prof. David Britton GridPP Project leader University of Glasgow UK-T0 Meeting 21 st Oct 2015 GridPP.
OPERATING SYSTEMS CS 3530 Summer 2014 Systems and Models Chapter 03.
Jan 2010 OSG Update Grid Deployment Board, Feb 10 th 2010 Now having daily attendance at the WLCG daily operations meeting. Helping in ensuring tickets.
A Year of HTCondor at the RAL Tier-1 Ian Collier, Andrew Lahiff STFC Rutherford Appleton Laboratory HEPiX Spring 2014 Workshop.
Workload management, virtualisation, clouds & multicore Andrew Lahiff.
6 march Building the INFN Grid Proposal outline a.ghiselli,l.luminari,m.sgaravatto,c.vistoli INFN Grid meeting, milano.
MND review. Main directions of work  Development and support of the Experiment Dashboard Applications - Data management monitoring - Job processing monitoring.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI MPI and Parallel Code Support Alessandro Costantini, Isabel Campos, Enol.
Testing CernVM-FS scalability at RAL Tier1 Ian Collier RAL Tier1 Fabric Team WLCG GDB - September
J. Templon Nikhef Amsterdam Physics Data Processing Group Nikhef Multicore Experience Jeff Templon Multicore TF
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks EGEE Operations: Evolution of the Role of.
2004 Queue Scheduling and Advance Reservations with COSY Junwei Cao Falk Zimmermann C&C Research Laboratories NEC Europe Ltd.
Considerations on Using CernVM-FS for Datasets Sharing Within Various Research Communities Catalin Condurache STFC RAL UK ISGC, Taipei, 18 March 2016.
DIRAC for Grid and Cloud Dr. Víctor Méndez Muñoz (for DIRAC Project) LHCb Tier 1 Liaison at PIC EGI User Community Board, October 31st, 2013.
1 Performance Impact of Resource Provisioning on Workflows Gurmeet Singh, Carl Kesselman and Ewa Deelman Information Science Institute University of Southern.
Job Priorities and Resource sharing in CMS A. Sciabà ECGI meeting on job priorities 15 May 2006.
J. Templon Nikhef Amsterdam Physics Data Processing Group Large Scale Computing Jeff Templon Nikhef Jamboree, Utrecht, 10 december 2012.
Ian Bird, CERN WLCG Project Leader Amsterdam, 24 th January 2012.
HTCondor Accounting Update
Workload Management Workpackage
Scheduling systems Carsten Preuß
HTCondor and TORQUE/Maui
Benchmarking Changes and Accounting
Brief overview on GridICE and Ticketing System
The CREAM CE: When can the LCG-CE be replaced?
CREAM-CE/HTCondor site
Practical aspects of multi-core job submission at CERN
Quattor Usage at Nikhef
Material Requirements Planning (MRP)
Presentation transcript:

Pre-GDB on Batch Systems (Bologna)11 th March Torque/Maui PIC and NIKHEF experience C. Acosta-Silva, J. Flix, A. Pérez-Calero (PIC) J. Templon (NIKHEF)

Pre-GDB on Batch Systems (Bologna)11 th March Outline ‣ System overview ‣ Successful experience (NIKHEF and PIC) ‣ Torque/Maui current situation ‣ Torque overview ‣ Maui overview ‣ Outlook

Pre-GDB on Batch Systems (Bologna)11 th March System overview ‣ TORQUE is a community and commercial effort based on OpenPBS project. It improves scalability, enables fault tolerance and many other features ‣ ‣ Maui Cluster Scheduler is a job scheduler capable of supporting multiple scheduling policies. It is free and open- source software ‣

Pre-GDB on Batch Systems (Bologna)11 th March System overview ‣ TORQUE/Maui system has the usual batch system capabilities: ‣ Queues definition (routing queues) ‣ Accounting ‣ Reservation/QOS/Partition ‣ FairShare ‣ Backfilling ‣ Handling of SMP and MPI jobs ‣ Multicore allocation and job backfilling ensure that Torque/Maui is capable of supporting multicore jobs

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience ‣ NIKHEF and PIC are multi-VO sites with local & Grid users ‣ Succesful experience during first LHC run with Torque/Maui system ‣ Currently, both are running Torque Maui ‣ NIKHEF: 30% non-HEP, 55% WLCG, rest non-WLCG HEP or local jobs. Highly non-uniform workload ‣ 3800 jobs slots ‣ 97.5% utilization (last 12 months) ‣ 2000 waiting jobs (average)

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience NIKHEF: running jobs (last year) NIKHEF: queued jobs (last year)

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience ‣ PIC: 3% non-HEP, 83% Tier-1 WLCG, 12% ATLAS Tier-2, rest local jobs (ATLAS Tier-3, T2K, MAGIC,…) ‣ 3500 jobs slots ‣ 95% approx utilization (last 12 months) ‣ 2500 waiting jobs (average)

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience PIC: running jobs (last year)

Pre-GDB on Batch Systems (Bologna)11 th March Succesful experience PIC: queued jobs (last year)

Pre-GDB on Batch Systems (Bologna)11 th March Torque overview ‣ Torque has a very active community: ‣ Mailing list: ‣ Total free support from Adaptive Computing ‣ New releases each year (approx. or less) and frequent new patches ‣ is the last release of branch 2.5.X

Pre-GDB on Batch Systems (Bologna)11 th March Torque overview

Pre-GDB on Batch Systems (Bologna)11 th March Torque overview ‣ Torque is well integrated with EMI middleware ‣ Vastly used in WLCG Grid sites (~75% of sites in BDii -pbs-) ‣ No complex to install, configure and manage: ‣ via qmgr tool ‣ plain text accounting ‣ … ‣ Torque scalability issues ‣ Reported for branch 2.5.X ‣ Not detected at our scale ‣ Branch 4.2.X presents significant enhancements to scalability for large environments, responsiveness, reliability, …

Pre-GDB on Batch Systems (Bologna)11 th March Maui overview ‣ Support: Maui is no longer supported by Adaptive Computing ‣ Documentation: ‣ Poor documentation causes initial complexity to install it ‣ Things do not always work like the documentation suggests ‣ Scalability issues: ‣ At ~8000 queued jobs, Maui hangs ‣ MAXIJOBS parameter can be adjusted to limit the number of jobs consider for scheduling ‣ This solves this issue (currently in production in NIKHEF)

Pre-GDB on Batch Systems (Bologna)11 th March Maui overview ‣ Moab is the non-free scheduler supported by Adaptive Computing and based in Maui ‣ Aims to increase the scalability ‣ It is a continued commercial support ‣ Configuration files are very similar to the ones in Maui: ‣ Feedback from sites running Torque/Moab would be a good complement to this review

Pre-GDB on Batch Systems (Bologna)11 th March Outlook ‣ Torque/Maui scalability issues ‣ Only relevant for larger sites ‣ feasible option for small-medium size sites ‣ Might be well solved in 4.2.X branch and tunning Maui options ‣ Actually, multicore jobs reduces the number of jobs to be handled by the system ‣ for sites that are predominantly WLCG (eg PIC at 95%), switching to a pure multicore load would further reduce scheduling issues at the site level. ‣ for sites that are much less WLCG dominated (eg Nikhef at 55%), a switch to pure multicore load might actually increase scheduling issues at the site level, as this move would remove much of the entropy which allows reaching 97% utilization. ‣ Another concern is the support for the systems, being Maui the weakest link for the Torque/Maui combination

Pre-GDB on Batch Systems (Bologna)11 th March Outlook ‣ Some future options ‣ Change from Maui to Moab (but, it is not free!) ‣ Setting up a kind of “OpenMaui” project within WLCG-sites as a community effort to provide support and improvements to Maui ‣ Integrate with another scheduler. Which one? ‣ Complete change to another system (SLURM, HTCondor, …) ‣ “Do nothing” until a real problem arrives ‣ Currently, just a worry, no real problem detected so far in PIC/NIKHEF ‣ Improvements from migrating to another system unclear

Pre-GDB on Batch Systems (Bologna)11 th March Outlook ‣ Questions: ‣ If decided for WLCG sites to move away from Torque/Maui, would it be feasible before the LHC Run2? ‣ Migration to a new batch system requires time and effort, thus manpower and expertise, in order to reach and adequate performance for a Grid site ‣ Not clear if needed before Run2 ‣ What happens with sites shared with non-WLCG VOs? ‣ Impact on other users (NIKHEF 45%) ‣ For PIC, several disciplines rely on local job submissions. A change on the batch system affects many users, and requires re-education, changes, and tests of their submission tools to adapt to an eventual new system