Improving ARC backends: Condor and SGE/GE LRMS interface Adrian Taga University of Oslo

LRMS bakends in ARC supported: PBS&variants, LSF, Condor, SGE, LL perl/sh wrappers around the command line interface of LRMS advantage: easy to modify No clear structure Scripts for: Populating InfoSystem Job control (submit, get status, kill) Automated system like atlas production needs to clearly distinguish failures of the job from failures related to LRMS. In special, failures due to exceeded resource limits.

Queues in Condor Why have queues? Inhomogeneous clusters Problem: Condor has no notion of queues Solution: use ClassAd mechanism to partition the cluster. Example: Queues defined based on memory size So far, Condor backend did not support queues. Why d we need queues? - Condor pools tend to be quite heterogeneous. The way heterogenity is handled innordugrid is by dividing the cluster into separate queues of more or less identical machines. However, condor does not have the notion of queues like other batch systems have. Still, it's possible to divide up the condor pool in virtual 'queues' by expploiting the flexible classAd mechanism. [queue/large] requirements="(Opsys == "linux" && Arch == "intel"" requirements=" && (Disk > 30000000 && Memory > 2000)" [queue/small] requirements="(Opsys == "linux" && Arch == "intel”" requirements=" && (Disk > 30000000 && Memory <= 2000 && Memory > 1000)"

Error reporting Many LRMS backends Inconsistent reporting of LRMS errors Clear diagnosis is needed for ATLAS prodsys Scenario: job exceeded resource limits PBS & Co – natively reports WallTime, CpuTime, Memory limit exceeded Exitcode < 256 from job, Exitcode ≥ 256 resource limit hit Condor, SGE - provide no clear diagnosis, workarounds needed Automated system like atlas production needs to clearly distinguish failures of the job from failures related to LRMS. In special, failures due to exceeded resource limits.

Thank you