HTCondor-CE for USATLAS Bob Ball AGLT2/University of Michigan OSG AHM March, 2015 Bob Ball AGLT2/University of Michigan OSG AHM March, 2015

Why HTCondor-CE? Bob Ball University of Michigan OSG AHM – March 2015 Slide 2 USATLAS has made it a goal for the next run to use a more highly scalable GRID access method. The GRAM protocol that has been in use can strain gatekeeper resources without reaching the high number of batch jobs needed as our clusters scale up in size. OSG has developed and provided HTCondor-CE to fill this order “The biggest difference you will see between an HTCondor CE and a GRAM CE is in the way that jobs are submitted to your batch system; HTCondor CE uses the built-in JobRouter daemon whereas GRAM CE uses jobmanager scripts written in Perl. Customizing your site’s CE now requires editing configuration files instead of editing jobmanager scripts.“ OSG has published a calendar that shows GRAM support dropped by ansitionPlan

What is Condor-CE? Bob Ball University of Michigan OSG AHM – March 2015 Slide 3 Replacement for GRAM Far better scalability Available with OSG-CE software 3.2.x Incoming jobs interpreted by JobRouter Direct insertion from there into standard batch system queue Works well with HTCondor, PBS/slurm LSF work at WT2 and SWT2 in progress Working with SGE at BU Correlation between condor_ce_q and batch queue accessed via ClassAd variables RoutedToJobId and RoutedFromJobId

What is Condor-CE? Bob Ball University of Michigan OSG AHM – March 2015 Slide 4 Configuration macros for JobRouter control insertion into scheduler.

Transition Steps Bob Ball University of Michigan OSG AHM – March 2015 Slide 5 Initially run HTCondor-CE in parallel with GRAM Low rate while confirming operations Later on move to full HTCondor-CE submission Retain GRAM functionality SAM tests will still use GRAM for a while Full transition -- Disable GRAM support once full usage of HTC-CE Future (2016) OSG updates will remove GRAM Included in distributions but not configured Full removal

Initial Implementation Bob Ball University of Michigan OSG AHM – March 2015 Slide 6 Quoting John Hover, “Panda functionality for fully property-based job submission doesn’t exist yet” Start with “queue” based submission queue = “prod” at AGLT2 is legacy central Production queue Possible to set similar values, but not yet implemented in BNL factories Due to manual configuration Values that can be passed in Condor-G submit +remote_queue = “queue name” +maxMemory = value in MB, eg, xcount = number of cores, eg, 1 +maxWallTime = max minutes of wall time There are no IO-related parameters (now)

Proposed Standard Queues Bob Ball University of Michigan OSG AHM – March 2015 Slide 7 On March 3, the following “standards” were proposed by John Hover Production: queue = “prod”, xcount = 1 MCORE: queue = “prod”, xcount = 8 Analysis: queue = “analy”, xcount = 1 maxWallTime and maxMemory to be taken from AGIS for a site Queue values may vary from site to site THESE SHOULD BE STANDARDIZED This full standard set is not yet easily possible Need to extract from AGIS, etc.

Current USATLAS Participation Bob Ball University of Michigan OSG AHM – March 2015 Slide 8 AGLT2_SL6 AGLT2_TEST AGLT2_MCORE ANALY_AGLT2_SL6 ANALY_AGLT2_TIER3_TEST ANALY_BNL_SHORT-grigk07 ANALY_BNL_LONG-gridgk07 ANALY_MWT2_SL6-uct2-gk MWT2_SL6-uct2-gk MWT2_MCORE-uct2-gk ANALY_SLAC_SHORT_1HR OU_OSCER_ATLAS NET2 ready, but not yet receiving pilots These are all either testing or fully stable

TimetableTimetable Bob Ball University of Michigan OSG AHM – March 2015 Slide 9 Goal is to have full HTCondor-CE usage by the time the run starts Reality is that neither AGIS nor factories are ready, not to mention the sites. Phase in as follows Phase 1. Queue only defined, with xcount=8 for MCORE Phase 1.5. Standardize queue names. Possibly pass maxMemory and wall time by queue name, but also accept those specific parameters Phase x. Full acceptance of all 4 possible parameters, with simple set of standard queues Phase y. Should “queue” be eliminated?

What do we need? Bob Ball University of Michigan OSG AHM – March 2015 Slide 10 HTCondor-CE reporting to BDII just added. Is this working correctly? SAM must work with HTCondor-CE Standardize on combinations of queue, xcount, etc Still testing with LSF, PBS/slurm, SGE working well Do we need an IO Factor? If we implement an IO Factor, can queue be discarded? Understanding by site admins on writing JobRouter This is non-trivial, some things are not very obvious. Lots of discovered hints on Twiki pages

JobRouter Sample Coding Bob Ball University of Michigan OSG AHM – March 2015 Slide 11 JOB_ROUTER_ENTRIES = \ /* ***** Route no 1 ***** */ \ /* ***** Analysis queue ***** */ \ [ \ GridResource = "condor localhost localhost"; \ eval_set_GridResource = strcat("condor ", "$(FULL_HOSTNAME)", " $(JOB_ROUTER_SCHEDD2_POOL)"); \ Requirements = target.queue=="analy"; \ Name = "Analysis Queue"; \ eval_set_RequestMemory = ifThenElse(maxMemory isnt undefined, \ ifThenElse(maxMemory <= 4096, 3968, maxMemory), 3968 ); \ eval_set_RequestCpus = ifThenElse(xcount isnt undefined, xcount, 1); \ ] \ /* ***** Route no 6 ***** */ \ /* ***** mp8 queue ***** */ \ [ \ GridResource = "condor localhost localhost"; \ eval_set_GridResource = strcat("condor ", "$(FULL_HOSTNAME)", " $(JOB_ROUTER_SCHEDD2_POOL)"); \ Requirements = ifThenElse(target.queue is undefined, \ false, \ ifThenElse(target.xcount is undefined, \ false, \ target.queue=="prod" && target.xcount==8)); \

JobRouter Sample Coding (cont) Bob Ball University of Michigan OSG AHM – March 2015 Slide 12 Name = "MCORE Queue"; \ eval_set_RequestMemory = ifThenElse(maxMemory isnt undefined, \ ifThenElse(maxMemory <= 32768, 32640, maxMemory), ); \ eval_set_RequestCpus = ifThenElse(xcount isnt undefined, xcount, 8); \ ] \ /* ***** Route no 8 ***** */ \ /* ***** Default queue for usatlas1 user ***** */ \ [ \ GridResource = "condor localhost localhost"; \ eval_set_GridResource = strcat("condor ", "$(FULL_HOSTNAME)", " $(JOB_ROUTER_SCHEDD2_POOL)"); \ Requirements = ifThenElse(target.queue is undefined, \ regexp("usatlas1",target.Owner), \ ifThenElse(target.xcount is undefined, \ regexp("usatlas1",target.Owner), \ target.queue=="prod" && target.xcount==1)); \ Name = "ATLAS Production Queue"; \ eval_set_varTest = target.queue; \ eval_set_RequestMemory = ifThenElse(maxMemory isnt undefined, \ ifThenElse(maxMemory <= 4096, 3968, maxMemory), 3968 ); \ eval_set_RequestCpus = ifThenElse(xcount isnt undefined, xcount, 1); \ ] Note the insertion in the ClassAd of the variable varTest in Route no 8. This is a good mechanism to see if what you are doing evaluates correctly; just look at the ClassAd via “condor_q –long” after the job is submitted.

Some Discussion Points Bob Ball University of Michigan OSG AHM – March 2015 Slide 13 Should “queue” be retained? Or should an IO factor be implemented? Standardize queue names to a simple set What differentiations in job types are relevant? Possible IO parameter What PanDA functionality is still needed? And what is impacted by that lack? Interaction between AGIS and the pilot factories to supply needed parameters Non-Condor batch systems LSF seems well in hand

Some Discussion Points (2) Bob Ball University of Michigan OSG AHM – March 2015 Slide 14 Interaction with BDII (and status) Interaction with SAM tests (and status) What setup files should be sourced? There is a long thread on this topic $OSG_GRID/setup.sh $OSG_APP/atlas_app/atlas_rel/cctools/latest/set up.sh $OSG_APP/atlas_app/atlaswn/setup.sh $OSG_APP/atlas_app/atlas_rel/local/setup.sh

ReferencesReferences Bob Ball University of Michigan OSG AHM – March 2015 Slide 15 lease3/HTCondorCEOverview stallHTCondorCE 43SfH9D1ERgnRp8InxBtmxEsdS4S5frVtwXn8v0/edit?pli=1#slid e=id.g79f425739_030 lease3/JobRouterRecipes lease3/TroubleshootingHTCondorCE lease3/SubmittingHTCondorCE