Condor DAGMan Warren Smith

12/11/2009 TeraGrid Science Gateways Telecon2 Basics Condor provides workflow support with DAGMan Directed Acyclic Graph –Each node has a Condor job –Edge edge is a dependency Example DAGMan file: Job A setup.condor Job B sweep1.condor Job C sweep2.condor Job D analyze.condor Parent A Child B C Parent B C Child D Job is used to name condor submit scripts Parent/Child specifies dependencies Node D Node A Node BNode C

12/11/2009 TeraGrid Science Gateways Telecon3 Managing a DAG condor_submit_dag –Creates a local job to manage the DAG Monitors jobs that make up the DAG Submits jobs when dependencies are satisfied lslogin2% condor_submit_dag example8.dag Checking all your submit files for log file names. This might take a while... Done File for submitting this DAG to Condor : example8.dag.condor.sub Log of DAGMan debugging messages : example8.dag.dagman.out Log of Condor library output : example8.dag.lib.out Log of Condor library error messages : example8.dag.lib.err Log of the life of condor_dagman itself : example8.dag.dagman.log Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster condor_q -dag and condor_q to monitor condor_rm to remove the DAG or individual jobs in the DAG

12/11/2009 TeraGrid Science Gateways Telecon4 DAGMan Node Each node can have pre- and post-scripts –Run on your submit system –SCRIPT PRE JobName ExecutableName [arguments] –SCRIPT POST JobName ExecutableName [arguments] Condor job runs if PRE succeeds POST runs if node executes Result of POST is result of node Job A setup.condor Job B sweep1.condor Job C sweep2.condor Job D analyze.condor Script PRE A retrieve.sh Script POST B check.sh $JOB $RETURN Script POST C check.sh $JOB $RETURN Script POST D archive.sh Parent A Child B C Parent B C Child D Node D Node A Node BNode C PRE script POST script Job B

12/11/2009 TeraGrid Science Gateways Telecon5 Managing Failures Retry statement for any node Retry B 3 POST script can analyze what happened and try to correct –Can be used with Retry ABORT-DAG-ON to exit immediately if can’t recover –On the exit code of a node ABORT-DAG-ON B 12 Rescue DAG –Condor executes a DAG as far as it can, even when individual nodes failure –Rescue DAG is generated if DAG didn’t fully complete Includes comments and marks which nodes completed Can be resubimitted as is or edited and submitted

12/11/2009 TeraGrid Science Gateways Telecon6 TeraGrid Condor-G Matchmaking Matchmaking selecting a resource for a job –A job provides requirements and preferences for a host –A resource provides them for jobs –Jobs are paired to resources Satisfy all requirements of both job and resource Optimize preferences of job and resource TeraGrid supports matchmaking of Condor-G jobs –Can be used with DAGMan Can’t express everything you might want –For example, “run job A on the same machine as job B” Available from several TeraGrid systems – matchmaking Your Condor install can be authorized

12/11/2009 TeraGrid Science Gateways Telecon7 System Information You can find information about systems using condor_status –Each row describes a queue on a system lslogin2% condor_status Name OpSys Arch State Activity LoadAv Mem ActvtyTime … tacc.lonestar.deve LINUX X86_64 Unclaimed Idle :00:00 tacc.lonestar.high LINUX X86_64 Unclaimed Idle :00:00 tacc.lonestar.norm LINUX X86_64 Unclaimed Idle :00:00 tacc.lonestar.seri LINUX X86_64 Unclaimed Idle :00:00 … Load average tries to describe how busy a queue is –(slots used + slots requested) / slots used

12/11/2009 TeraGrid Science Gateways Telecon8 First Job with Matchmaking executable = /bin/hostname arguments = --fqdn transfer_executable = false output = example2.out error = example2.err log = example2.log requirements = (Name=="tacc.lonestar.development") universe = grid x509userproxy=/home/teragrid/tg458637/.globus/userproxy.pem grid_resource = $$(GramResource) globusrsl = (maxWallTime=5)(count=1)(queue=$$(Queue)) queue

12/11/2009 TeraGrid Science Gateways Telecon9 Notes on First Job x509userproxy –Don’t have to specify when not matchmaking –Need to when matchmaking Requirements –What the job requires of any machine it gets matched to –Boolean –Variables “Name” are for the machine being matched to Variables “$$(Queue)” are for the machine being matched to –$$() only needed outside of requirements

12/11/2009 TeraGrid Science Gateways Telecon10 Second Job with Matchmaking executable = /bin/hostname arguments = --fqdn transfer_executable = false output = example2-$(CLUSTER).$(PROCESS).out error = example2-$(CLUSTER).$(PROCESS).err log = example2-$(CLUSTER).$(PROCESS).log requirements = ((Name=="tacc.lonestar.development") || \ (Name=="tacc.ranger.development") || \ (Name=="loni-lsu.queenbee.workq") || \ (Name=="ncsa.abe.debug") || \ (Name=="ncsa.dtf.debug") || \ (Name=="sdsc.dtf.dque") || \ (Name=="purdue.steele.tg_workq")) rank = LoadAvg - CurMatches * 0.25 universe = grid x509userproxy=/home/teragrid/tg458637/.globus/userproxy.pem grid_resource = $$(GramResource) globusrsl = (maxWallTime=5)(count=1)(queue=$$(Queue)) queue 10

12/11/2009 TeraGrid Science Gateways Telecon11 Notes on Second Job $(Cluster) –The ID for the set of jobs submitted by this script $(Process) –The ID (0 - (n-1)) of a job within a cluster Requirements is (mostly) a boolean expression –() to group, || for or, && for and –, >=, ==, != –A few others since expressions actually have 3 values True, false, undefined Rank expression used to identify best machine –Higher rank is better –CurMatches is number of jobs matched to that machine in current round –100 is the max load average queue 10 –Submits 10 copies of this job

12/11/2009 TeraGrid Science Gateways Telecon12 Ranking Machines It’s a bit of an art at this point –Let me know what works for you Let Warren know if additional information is needed Working on providing queue wait time predictions –QBETS predictions have been available –Most likely moving to a new technology over the next few months

12/11/2009 TeraGrid Science Gateways Telecon13 Additional Information Condor User Guide – TeraGrid –Condor-G page –Condor-G matchmaking wiki condorg_userguide