Scheduling under LCG at RAL UK HEP Sysman, Manchester 11th November 2004 Steve Traylen

Slides:



Advertisements
Similar presentations
Batch System Operation & Interaction with the Grid LCG/EGEE Operations Workshop May 25 th 2005 CERN.ch.
Advertisements

Andrew McNab - Manchester HEP - 17 September 2002 Putting Existing Farms on the Testbed Manchester DZero/Atlas and BaBar farms are available via the Testbed.
Steve Traylen Particle Physics Department Experiences of DCache at RAL UK HEP Sysman, 11/11/04 Steve Traylen
HTCondor and the European Grid Andrew Lahiff STFC Rutherford Appleton Laboratory European HTCondor Site Admins Meeting 2014.
1 User Analysis Workgroup Update  All four experiments gave input by mid December  ALICE by document and links  Very independent.
Beowulf Supercomputer System Lee, Jung won CS843.
CREAM John Gordon GDB November CREAM number of sites now – gstat2 says 24. Batch systems supported Experiment Tests Feedback from sites. Evaluation.
Pre-GDB on Batch Systems (Bologna)11 th March Torque/Maui PIC and NIKHEF experience C. Acosta-Silva, J. Flix, A. Pérez-Calero (PIC) J. Templon (NIKHEF)
HTCondor within the European Grid & in the Cloud
Andrew McNab - Manchester HEP - 22 April 2002 UK Rollout and Support Plan Aim of this talk is to the answer question “As a site admin, what are the steps.
The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.
London Tier 2 Status Report GridPP 12, Brunel, 1 st February 2005 Owen Maroney.
5 November 2001F Harris GridPP Edinburgh 1 WP8 status for validating Testbed1 and middleware F Harris(LHCb/Oxford)
Workload Management WP Status and next steps Massimo Sgaravatto INFN Padova.
Andrew McNab - Manchester HEP - 5 July 2001 WP6/Testbed Status Status by partner –CNRS, Czech R., INFN, NIKHEF, NorduGrid, LIP, Russia, UK Security Integration.
03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio.
Tier-1 Batch System Report Andrew Lahiff, Alastair Dewhurst, John Kelly, Ian Collier 5 June 2013, HEP SYSMAN.
1 BIG FARMS AND THE GRID Job Submission and Monitoring issues ATF Meeting, 20/06/03 Sergio Andreozzi.
:: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: GridKA School 2009 MPI on Grids 1 MPI On Grids September 3 rd, GridKA School 2009.
Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.
12th November 2003LHCb Software Week1 UK Computing Glenn Patrick Rutherford Appleton Laboratory.
Monitoring the Grid at local, national, and Global levels Pete Gronbech GridPP Project Manager ACAT - Brunel Sept 2011.
ATLAS DC2 seen from Prague Tier2 center - some remarks Atlas sw workshop September 2004.
11/30/2007 Overview of operations at CC-IN2P3 Exploitation team Reported by Philippe Olivero.
Steve Traylen Particle Physics Department EDG and LCG Status 9 th December 2003
Multi-core jobs at the RAL Tier-1 Andrew Lahiff, Alastair Dewhurst, John Kelly February 25 th 2014.
Steve Traylen PPD Rutherford Lab Grid Operations PPD Christmas Lectures Steve Traylen RAL Tier1 Grid Deployment
Grid Operations Centre LCG Accounting Trevor Daniels, John Gordon GDB 8 Mar 2004.
Certification and test activity IT ROC/CIC Deployment Team LCG WorkShop on Operations, CERN 2-4 Nov
Resource Management Task Report Thomas Röblitz 19th June 2002.
Enabling Grids for E-sciencE SGE J. Lopez, A. Simon, E. Freire, G. Borges, K. M. Sephton All Hands Meeting Dublin, Ireland 12 Dec 2007 Batch system support.
Tier1A Status Andrew Sansum 30 January Overview Systems Staff Projects.
Presenter Name Facility Name UK Testbed Status and EDG Testbed Two. Steve Traylen GridPP 7, Oxford.
Your university or experiment logo here Tier1 Deployment Steve Traylen.
Proposal for a IS schema Massimo Sgaravatto INFN Padova.
1 Andrea Sciabà CERN Critical Services and Monitoring - CMS Andrea Sciabà WLCG Service Reliability Workshop 26 – 30 November, 2007.
Jens G Jensen RAL, EDG WP5 Storage Element Overview DataGrid Project Conference Heidelberg, 26 Sep-01 Oct 2003.
Derek Ross E-Science Department DCache Deployment at Tier1A UK HEP Sysman April 2005.
2-Sep-02Steve Traylen, RAL WP6 Test Bed Report1 RAL and UK WP6 Test Bed Report Steve Traylen, WP6
Andrew McNab - Manchester HEP - 17 September 2002 UK Testbed Deployment Aim of this talk is to the answer the questions: –“How much of the Testbed has.
Automatic Resource & Usage Monitoring Steve Traylen/Flavia Donno CERN/IT.
A Year of HTCondor at the RAL Tier-1 Ian Collier, Andrew Lahiff STFC Rutherford Appleton Laboratory HEPiX Spring 2014 Workshop.
Workload management, virtualisation, clouds & multicore Andrew Lahiff.
15-Feb-02Steve Traylen, RAL WP6 Test Bed Report1 RAL/UK WP6 Test Bed Report Steve Traylen, WP6 PPGRID/RAL, UK
Report on Installed Resource Capacity Flavia Donno CERN/IT-GS WLCG Management Board, CERN 25 November 2008.
Accounting in LCG/EGEE Can We Gauge Grid Usage via RBs? Dave Kant CCLRC, e-Science Centre.
EGEE is a project funded by the European Union under contract INFSO-RI Grid accounting with GridICE Sergio Fantinel, INFN LNL/PD LCG Workshop November.
WP1 Status and plans Francesco Prelz, Massimo Sgaravatto 4 th EDG Project Conference Paris, March 6 th, 2002.
8 August 2006MB Report on Status and Progress of SC4 activities 1 MB (Snapshot) Report on Status and Progress of SC4 activities A weekly report is gathered.
The Grid Storage System Deployment Working Group 6 th February 2007 Flavia Donno IT/GD, CERN.
LCG Pilot Jobs + glexec John Gordon, STFC-RAL GDB 7 December 2007.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks EGEE Operations: Evolution of the Role of.
John Gordon Grid Accounting Update John Gordon (for Dave Kant) CCLRC e-Science Centre, UK LCG Grid Deployment Board NIKHEF, October.
The RAL PPD Tier 2/3 Current Status and Future Plans or “Are we ready for next year?” Chris Brew PPD Christmas Lectures th December 2007.
Installation Accounting Status Flavia Donno CERN/IT-GS WLCG Management Board, CERN 28 October 2008.
Software Management Workshop Steve Traylen. Software Management(WG5) The aim of the working group is to look at deficiencies in deployed and upcoming.
Integrating HTCondor with ARC Andrew Lahiff, STFC Rutherford Appleton Laboratory HTCondor/ARC CE Workshop, Barcelona.
LCG/EGEE Operational Issues Stephen Burke RAL. November 1 st 2004LCG Operations - Issues Introduction List of problems to initiate discussion –A personal.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarksEGEE-III INFSO-RI MPI on the grid:
ECGI meeting on job priorities on May 15th 2006, CNAF Bologna How LHCb thinks to use/integrate g-PBox (or single components) and when Gianluca Castellani.
Job Priorities and Resource sharing in CMS A. Sciabà ECGI meeting on job priorities 15 May 2006.
BaBar & Grid Eleonora Luppi for the BaBarGrid Group TB GRID Bologna 15 febbraio 2005.
First proposal for a modification of the GIS schema
Connecting LRMS to GRMS
Towards GLUE Schema 2.0 Sergio Andreozzi INFN-CNAF Bologna, Italy
Summary on PPS-pilot activity on CREAM CE
CREAM Status and Plans Massimo Sgaravatto – INFN Padova
Moving from CREAM CE to ARC CE
Developments in Batch and the Grid
Presentation transcript:

Scheduling under LCG at RAL UK HEP Sysman, Manchester 11th November 2004 Steve Traylen

RAL, LCG, torque and Maui Observations of RAL within LCG vs. traditional batch. Various issues that arose and what was done to tackle this. Upcoming changes in the next release. Some items still to be resolved.

LCG Grid vs. Traditional Batch Observations from the LHCb vs. Atlas period earlier this year. Matters in common for LCG Grid and Batch. –RAL must provide 40% to Atlas, 30% to LHCb, … as dictated by GridPP. Differences for LCG Grid and Batch. –Batch queued for 400 job slots. –LCG - often < 5 queued for 400 job slots.

LCG Grid vs. Traditional Batch Providing allocations difficult with LHCb submitting at a faster rate. RAL only received LHCb jobs. Only solution with OpenPBS to just hard limit LHCb. –Idle CPUs a waste of money. –Always better to give allocation as soon as possible. –LHCb jobs pile up due to apparent free resource. –RAL becomes unattractive (via ETT) to Atlas.

Queues per VO Many sites, CNAF, LIP, NIKHEF, … moved to queues per VO. Advantages –Estimated traversal calculation orthogonal for each VO. While piling up LHCb jobs Atlas jobs are still attracted, hopefully always one queued job available. –Queue lengths can be customised. Disadvantages –Change in farm just to fit into to LCG. –Adding VOs becomes harder.

Queues per VO(2) ETT calculation rudimentary, just increases as jobs are queued on a per queue basis. –RAL only gives 1 CPU to Zeus and 399 to Atlas ETT calculation does not reflect this really. In fact RAL’s queues now have a zero FIFO component. But it still works once Zeus jobs pile up they stop coming.

Take-up within LCG

CPU Scaling CPU variation. –Can be removed now within the batch farm by configuring pbs_mom to normalise CPU time. –Normalised speed published into info System. Walltime scaling more confusing. –RAL does, we fairshare the whole farm on wall time. –However what we advertise is a lie.

CPU Scaling and ETT Only at RAL the scaling is extreme. –Normalised on Pentium 450 –Nodes are scaled from 4.7 to 5.0 at present. –So CPUlimits and walltimes are very long (9 days). Once jobs are queued RAL became very unattractive. We modified info provider to make the “ETT” comparable to other sites. We will renormalize at some point soon.

OpenPBS to Maui/Torque OpenPBS (as in LCG today) hangs when one node crashes. –Torque is okay (most of the time). Torque is just a new version of OpenPBS maintained by –No integration required. Active user community and mailing list. Maintained well, bug fixes and patches are accepted and added regularly. Maui is a more sophisticated scheduler capable of fairshare for instance.

Fairshare with Maui The default is FIFO and so the same as the default PBS scheduler. Maui supports fairshare on Walltime. –E.g., Consider last 7 days of operation. –Give Atlas 50%, CMS 20%. –Give lhcbsgm a huge priority but limit “them” to one job. –Reserve one CPU for a 10 minute queue. Monitoring jobs. Maui will strive to reach these targets. –Tools exist to diagnose and understand why Maui is not doing what you hope for allowing tuning.

Heterogeneous Clusters Many farms currently have mixed memory, local disk space, ….. Glue contains sub clusters but they are currently identified by only the hostname. GlueSubClusterUniqueID=lcgce02.gridpp.rl.ac.uk So only one hardware type per CE possible. Does RB joins the SubCluster against a GlueCE object.

Heterogeneous Clusters(2) It would seem easy to describe a second sub cluster and use a unique key. GlueSubClusterUniqueID=lcgce02.gridpp.rl.ac.uk-bigmem Different GlueCEs can then join on this? –Does this work? –Information providers may need tweaking. –Will the RB do this, what else will break. –Can the JobManager support different attributes per queue to target nodes. –Advertising fake queues possible.

Future Possibilities One queue per VO per memory size per local disk space per time period …. = a lot of queues. Some sites only have one queue and insist on users setting requirements per job. –It is a good a idea within the batch farm. –The Resource Broker does not pass this on to gram transfer, how much do we want this?

Maui based Info Provider Current info provider only interrogates PBS. PBS has no idea what is going to happen next. A Maui based provider could calculate the ETT better. But it may difficult to port to LSF, BQS, ….

Conclusions Moving to torque and Maui is transparent. Maui and queues per VO will introduce more control of resources within a site and increase their occupancy. Major adjustments are needed to existing queue infrastructures. Heterogeneous cluster support within LCG…. As LCG resources merge with other EGEE resources new challenges arise such as running parallel jobs in production – Crossgrid?

References Maui and Torque homepages including documentation. – Maui/Torque RPMS appearing in LCG. – support/faq/torque.html More Maui/Torque RPMS and qstat cache mechanism. –