1. Condor Administration in the Open Science Grid
University of Wisconsin, Madison
Dan Bradley
Representing Condor & DISUN (CMS) Teams

2. What is Condor to OSG? Batch System Multi-purpose Job Queue
Flexible
Multi-purpose Job Queue
Submit jobs to local cluster
Submit jobs to the grid (a.k.a. Condor-G)
Queue for fork jobs (managed-fork)
Grid Overlay
Condor Glidein
Grid-wide Collector/Negotiator
OSG ReSS service

3. Example: Submitting Jobs within UW Campus Grid
Requirements = Arch == “INTEL”
&& Disk >= DiskUsage
&& Memory*1024 >= ImageSize
ImageSize =
DiskUsage =
OSG_VO = “uscms”
Requirements = MY.PassedTest =!= False
Rank = OSG_VO =?= “uscms”
Arch = “INTEL”
Disk =
Memory = 2000
PassedTest = True
HEP matchmaker
CS matchmaker
GLOW
matchmaker
schedd
(Job caretaker)
condor_submit
job ClassAd
flocking
startd
(Job Executor)
machine ClassAd
job runs

4. Example: Submitting jobs through OSG to UW Campus
HEP matchmaker
CS matchmaker
GLOW
matchmaker
flocking
Globus gatekeeper
schedd
(Job caretaker)
condor_submit
schedd
(Job caretaker)
job ClassAd
startd
(Job Executor)
machine ClassAd
condor
gridmanager
job runs

5. condor_status gives information about the pool:
Name OpSys Arch State Activ LoadAv Mem ActvtyTime
perdita.cs.wi LINUX INTEL Owner Idle :28:42
coral.cs.wisc LINUX INTEL Claimed Busy :27:21
doc.cs.wisc.e LINUX INTEL Unclaimed Idle :20:04
dsonokwa.cs.w LINUX INTEL Claimed Busy :01:45
ferdinand.cs. LINUX INTEL Claimed Suspe :00:55
To inspect full ClassAds: condor_status -long

6. condor_submit & condor_q
% condor_submit sim.submit
Submitting job(s).
1 job(s) submitted to cluster 1.
% condor_q
-- Submitter: perdita.cs.wisc.edu : < :1027> :
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
frieda /16 06: :00:00 I sim.exe
1 jobs; 1 idle, 0 running, 0 held %

7. View the full ClassAd % condor_q -long
-- Submitter: perdita.cs.wisc.edu : < :1027> :
MyType = “Job”
TargetType = “Machine”
ClusterId = 1
QDate =
CompletionDate = 0
Owner = “frieda”
RemoteWallClockTime =
LocalUserCpu =
LocalSysCpu =
RemoteUserCpu =
RemoteSysCpu =
ExitStatus = 0 …

8. View Specific Job Attributes
The unix way:
% condor_q -long | grep UserLog
/home/frieda/a.log
/home/frieda/b.log
The C way:
% condor_q -format “ %s\n” UserLog
5.0 /home/frieda/a.log
6.0 /home/frieda/b.log
The XML way … (and there’s a direct SOAP interface too)
% condor_q -xml …

9. Looking at Held Jobs See why jobs are on hold:
% condor_q –held
-- Submiter: x.cs.wisc.edu : < :510> :x.cs.wisc.edu
ID OWNER HELD_SINCE HOLD_REASON
6.0 frieda /20 13:23 Error from starter on
9 jobs; 8 idle, 0 running, 1 held
See full details for a job:
% condor_q –long 6.0

10. The Job’s “User Log” Look in the job log for clues: % cat b.log
000 ( ) 04/20 14:47:31 Job submitted from host: < :48740>
...
007 ( ) 04/20 15:02:00 Shadow exception!
Error from starter on gig06.stat.wisc.edu: Failed to open '/scratch.1/frieda/workspace/v67/condor-test/test3/run_0/b.input' as standard input: No such file or directory (errno 2)
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job

11. Using Constraints
% condor_q -constraint ‘JobStatus == 5 && HoldReasonCode == 14 && HoldReasonSubCode == 2’
-- Submitter: login03.hep.wisc.edu : < :32769> : login03.hep.wisc.edu
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
frieda /19 11: :00:02 H hello_world
frieda /19 11: :00:01 H hello_world
% sudo condor_rm -constraint ‘JobStatus == 5 && HoldReasonCode == 14 && HoldReasonSubCode == 2’
Jobs matching constraint (JobStatus == 5 && HoldReasonCode == 14 && HoldReasonSubCode == 2) have been marked for removal
The queue super user:
QUEUE_SUPER_USERS = root, condor

12. Job History % condor_history % condor_history -long 2.0
1.0 frieda /20 01: :15:55 C 4/20 04:19 /home/frieda/a
2.0 frieda /20 01: :38:01 C 4/20 04:20 /home/frieda/b
Extract ClassAd details just like condor_q. Example:
% condor_history -long 2.0
Configurable history depth:
MAX_HISTORY_LOG = (20MB)

13. Resource Allocation

14. Fair Share User priority Example: two users, 60 batch slots
Inversely proportional to fair share
Example: two users, 60 batch slots
priority 50 - gets 40 slots
priority gets 20 slots

15. Fair Share Dynamics User priority changes over time Example:
wants to be equal to number of slots in use
Example:
User steadily running 100 jobs: priority 100
Stops running jobs:
1 day later: priority 50
2 days later: priority 25
Configure speed of adjustment: PRIORITY_HALFLIFE = 86400

16. Modified Fair Share User Priority Factor Example:
multiplies the “real user priority”
result is called “effective user priority”
Example:
condor_userprio -setfactor cdf 4.0
condor_userprio -setfactor cms 1.0
cdf steadily uses 10 slots - priority 40
cms steadily uses 20 slots - priority 20

17. Reporting Condor Pool Usage
% condor_userprio -usage -allusers
Last Priority Update: 7/30 09:59
Accumulated Usage Last
User Name Usage (hrs) Start Time Usage Time …
/18/ :37 7/30/ :24
/03/ :56 7/30/ :59
/03/ :56 7/30/ :59
/03/ :54 7/30/ :59
Number of users: /03/ :56 7/29/ :00
When upgrading Condor, preserve the central manager’s AccountantLog
Happens automatically if you follow general rule: preserve Condor’s LOCAL_DIR

18. A Flexible Mechanism for Resource Allocation: The ClassAd Way

19. Machine Rank Numerical expression: Example:
higher number preempts lower number
rank trumps user priority considerations, (including PREEMPTION_REQUIREMENTS)
Example:
CMS gets 1st prio, CDF gets 2nd, others 3rd
RANK = 2*(Owner == +
1*(Owner ==

20. Another Rank Example
Rank = (Group =?= "LMCG") * ( RushJob)

21. Note on Scope of Condor Policies
pool-wide scope: example negotiator
user priorities, factors, etc.
preemption policy related to user priority
steering jobs via negotiator job rank
execute machine/slot scope: startd
machine rank, requirements
preemption/suspension policy
customized machine ClassAd values
submit machine scope
automatic additions to job rank, requirements, and insertion of arbitrary ClassAd attributes
personal scope
environmental configurations: _CONDOR_<config val>=value

22. Preemption Policy
Should Condor jobs yield to non-condor activity on the machine?
Should some types of jobs never be interrupted? After 4 days?
Should some jobs immediately preempt others? After 30 minutes?
Is suspension more desirable than killing?
Can need for preemption be decreased by steering jobs towards the right machines?

23. Example Preemption Policy
When a claim is preempted, do not allow killing of jobs younger than 4 days old.
MaxJobRetirementTime = 3600 * 24 * 4
Applies to all forms of preemption:
user priority, machine rank, machine activity

24. Another Preemption Policy
Expression can refer to attributes of batch slot and job, so can be highly customized.
MaxJobRetirementTime = 3600 * 24 * 4 * (OSG_VO =?= “uscms”)

25. Preemption Policy Pitfall
If you disable all forms of preemption, you probably want to limit lifespan of claims:
PREEMPTION_REQUIRMENTS = False
PREEMPT = False
RANK = 0
CLAIM_WORKLIFE = 3600
Otherwise, reallocation of resources will not happen until a user’s jobs dry up.

26. Resource Allocation: Group Accounting

27. Fair Sharing Between Groups
Useful when:
multiple user ids belong to same group
group’s share of pool is not tied to specific machines
# Example group settings
GROUP_NAMES = group_cms, group_cdf
GROUP_QUOTA_group_cms = 200
GROUP_QUOTA_group_cdf = 100
GROUP_AUTOREGROUP = True
GROUP_PRIO_FACTOR_group_cms = 10
GROUP_PRIO_FACTOR_group_cdf = 10
DEFAULT_PRIO = 100

28. Group Sharing cntd There are different modes of group accounting
users within group limited by same shared quota
users in group also share same priority/usage history
The job advertises its own group identity:
+AccountingGroup = “group_cms.cmsprod”
+AccountingGroup = “group_cms”
For OSG jobs, that means modifying the Globus jobmanager for Condor.
$(OSG_LOCATION)/globus/lib/perl/Globus/GRAM/JobManager/condor.pm

29. More Condor Jobmanager Hacks

30. condor.pm hacks
Add an attribute to all jobs, specifying the VO affiliation of the user.
Use Condor’s file staging to reduce reliance on NFS (UCSD’s NFSLite)
Add a wrapper script to the job.
Clean up orphaned jobs.

31. Before hacking…
Keep in mind the condor submit configuration options that exist:
APPEND_REQUIREMENTS
APPEND_RANK
SUBMIT_EXPRS
If you use managed fork, you probably do not want to append the same requirements to “vanilla” and “local” universe jobs. Use APPEND_REQ_VANILLA

32. Example: dedicated batch slot
I want software installation jobs from CMS VO to run immediately
on specific, dedicated machines.
On the execute node:
IsCMSSoftSlot = True
STARTD_ATTRS = IsCMSSoftSlot
START = Owner == "osg_cmssoft"
On the submit node (i.e. the CE)
APPEND_REQ_VANILLA =
MY.Owner != "osg_cmssoft" ||
MY.Owner == "osg_cmssoft" && TARGET.IsCMSSoftSlot =?= True

33. Alternative to dedicated slot
Rather than preempting other jobs, suspend them while high priority job runs.
More complex, but doable
Details in Condor manual: Job Suspension

34. My Favorite Condor Feature
Condor is not perfect. But we try to be directly engaged with users and move it in a helpful direction.

35. Need scalability tuning advice for Condor 6.8?