Multicore Accounting John Gordon, STFC-RAL WLCG Operations Coordination, 1 st October 2015

Outline History Recent Progress Current Status Issues Plan John Gordon, MB July2015

History I first raised the issue of sites publishing details of the cores used per job at the EGI OMB last December with an update in January. There was some initial improvement but then progress flattened off. WLCG are now running many more multicore jobs and wish to see this reflected in accounting. Knowing the number of cores used is important in calculating the effective wallclock time and thus the overall occupancy of a cluster. John Gordon, MB July 2015

Recent Progress At the June meeting of the WLCG Grid Deployment Board I reported that 87% of LHC CPU use was now reported as coming from Sites/CEs which reported the number of cores per job. Since there were some obvious omissions from important sites and countries I was asked to address this. I raised tickets against all NGIs and gave them a link to the publishing of cores for June by their sites which run LHC work. This has mainly been successful. By the end of June we had 95% publishing By mid August 99.5% of cpu time was accounted by sites reporting cores. John Gordon, MB July 2015

Status 99.5% of CPU time has cores reported There are still about 60 sites who have published jobs without cores in the last few days but there is a long tail of failed jobs and rogue CEs that don’t amount to significant CPU use. There is a smaller number of sites with some or all CEs not reporting cores. There are few with problems not under their control Many have never responded to tickets. John Gordon, MB July 2015

Sept 2015 – 99.5% of cpu with cores

Germany John Gordon, MB July 2015 WLCG View

Within a Site John Gordon, MB June 2015

Issues DESY-HH – Outstanding ticket with ARC team – CREAM PBS publishes Processors=1 for multicore jobs. This may also be an issue at other sites. – MPPMU have the same ARC issue – DESY have patched the PBS parser and have successfully published cores. Currently cleaning up their old data. Insert footer here

SubmitHostCores=1 ce01.tier2.hep.manchester.ac.uk:8443/cream-pbs-gpu atlas-ce-02.roma1.infn.it:8443/cream-lsf-atlasgshort atlas-creamce-01.roma1.infn.it:8443/cream-lsf-atlasgshort ce03.clumeq.mcgill.ca:8443/cream-pbs-atlas_mcore ce02.clumeq.mcgill.ca:8443/cream-pbs-atlas_mcore grid-cr3.desy.de:8443/cream-pbs-mcore grid-cr0.desy.de:8443/cream-pbs-mcore grid-cr2.desy.de:8443/cream-pbs-mcore grid-cr1.desy.de:8443/cream-pbs-mcore gb-ce-amc.amc.nl:8443/cream-pbs-express ce.lsg.psy.vu.nl:8443/cream-pbs-express gb-ce-emc.erasmusmc.nl:8443/cream-pbs-long gb-ce-tud.ewi.tudelft.nl:8443/cream-pbs-long gb-ce-lumc.lumc.nl:8443/cream-pbs-medium gb-ce-amc.amc.nl:8443/cream-pbs-long ce.lsg.bcbr.uu.nl:8443/cream-pbs-long ce.lsg.psy.vu.nl:8443/cream-pbs-infra gridce03.ifca.es:8443/cream-sge-biomed dc2-grid-66.brunel.ac.uk:8443/cream-pbs-biomed gridce02.ifca.es:8443/cream-sge-biomed gb-ce-rug.sara.usor.nl:8443/cream-pbs-infra gb-ce-tud.ewi.tudelft.nl:8443/cream-pbs-infra ce.irb.egi.cro-ngi.hr:8443/cream-pbs-sunx cert-37.pd.infn.it:8443/cream-lsf-grid ce04.ncg.ingrid.pt:8443/cream-sge-opsgrid ce05.ncg.ingrid.pt:8443/cream-sge-opsgrid grid002.jet.efda.org:8443/cream-pbs-biomed ce06.ncg.ingrid.pt:8443/cream-sge-opsgrid ce02.lip.pt:8443/cream-sge-opsgrid cirigridce01.univ-bpclermont.fr:8443/cream-pbs-lhcb ce05.ncg.ingrid.pt:8443/cream-sge-dteamgrid Veracity of PBS cores CEs with eff>100% cpu/(wall*cores) >1200 CEs reporting Top 3 probably GPUs Then DESY Then NL sites, prob not HEP Several biomed CEs PT and NL sites also reporting cores>1 Worth investigation

What are yours? Those Were My Issues

Soon the dev portal will hold all historical APEL data. – This will only contain cores from when sites started publishing it. When this id done the Portal will use this data including cores for the production T1 and T2 reports and show the current dev tree(EMI3) in parallel with the current production EGI views. The portal is currently undergoing a major rewrite. When this is released next April it will use the data with cores only. WLCG is encouraged to track the portal developments. There will be a prototype demo at the EGI meeting in Bari with a webcast. Timescales

Summary John Gordon, MB July 2015