Douglas Thain Computer Sciences Department University of Wisconsin-Madison October Condor by Example

Lecture Format: › In each lecture:  Lecture to whole group.  Workshop and examples at computer. › Oops!  Some items are filled in at the last minute.  Please fill the _______ with notes.

Outline › Overview › Submitting Jobs, Getting Feedback › Setting Requirements with ClassAds › Which Universe? › Move to Workshop

What is Condor? › Condor converts a collection of unrelated workstations into a high- throughput computing facility. › Condor uses matchmaking to make sure that everyone is happy.

What is High-Throughput Computing? › High-performance: CPU cycles/second under ideal circumstances.  “How fast can I run simulation X on this machine?” › High-throughput: CPU cycles/day (week, month, year?) under non-ideal circumstances.  “How many times can I run simulation X in the next week using all available machines?”

What is High-Throughput Computing? › Condor does whatever it takes to run your jobs, even if some machines…  Crash!  Are disconnected  Run out of disk space  Are removed or added from the pool  Are put to other uses

What is Matchmaking? › Condor uses Matchmaking to make sure that work gets done within the constraints of both users and owners. › Users (jobs) have constraints:  “I need an Alpha with 256 MB RAM” › Owners (machines) have constraints:  “Only run jobs when I am away from my desk and never run jobs owned by Bob.”

Who uses Condor? › Hundreds of universities and companies around the world! › University of Wisconsin, USA  682 CPUs in one building  Computer architecture simulations › National Institute of Physics, Italy  200 CPUs in many cities  Reconstruction of collider events › And many others!

What can Condor do for me? Condor can… › …increase your throughput. › …do your housekeeping. › …improve reliability. › …give performance feedback.

Cluster Overview Server 512 MB 800 MHz 100 Mb/s network 20 GB Client 128 MB 666 MHz Client 128 MB 666 MHz Client 128 MB 666 MHz Client 128 MB 666 MHz Client 128 MB 666 MHz 10 GB

How many machines now? › The map is out of date! › The system is always changing. › First example: What machines (and of what kind) are in the pool now?

How Many Machines? % condor_status Name OpSys Arch State Activity LoadAv Mem lxpc1.na.infn LINUX-GLIBC INTEL Unclaimed Idle axpd21.pd.inf OSF1 ALPHA Owner Idle vlsi11.pd.inf SOLARIS26 SUN4u Claimed Busy Machines Owner Claimed Unclaimed Matched Preempting ALPHA/OSF INTEL/LINUX INTEL/LINUX-GLIBC SUN4u/SOLARIS SUN4u/SOLARIS SUN4u/SOLARIS SUN4x/SOLARIS Total

Machine States › Most machines will be:  Owner: The machine’s owner is busy at the console, so no Condor jobs may run.  Claimed: Condor has selected the machine to run jobs for other users.

Machine States › Only a few should be:  Unclaimed: The owner is gone, but Condor has not yet selected the machine.  Matched: Between claimed and unclaimed.  Preempting: Condor is busy removing a job.

More Things to Try % condor_status -help % condor_status –avail % condor_status –run % condor_status –total % condor_status –pool condor.cs.wisc.edu

Submitting Jobs

Steps to Running a Job › Re-link for Condor. › Submit the job. › Watch the progess. › Receive when done.

Example Job Integrate sin(x) from 0 to 10, using 10 million slices. Simple program takes a few seconds. %./integrate

PROGRAM INTEGRATE CHARACTER STR*10 REAL X, SLICES, LIMIT CALL GETARG(1,STR) READ (STR,*) LIMIT CALL GETARG(2,STR) READ (STR,*) SLICES TOTAL=0 STEP=LIMIT/SLICES DO X=0, LIMIT, STEP TOTAL = TOTAL + SIN(X)*STEP END DO PRINT *, TOTAL END

Re-link for Condor › If you normally compile like this:  g77 integrate.f -o integrate › Then compile for Condor like this:  condor_compile g77 integrate.f -o integrate

Submit the Job › Create a submit file: emacs integrate.submit & › Submit the job: condor_submit integrate.submit Executable = integrate Arguments = Output = integrate.out Log = integrate.log queue

Watch the Progress % condor_q -- Submitter: axpbo8.bo.infn.it : : ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 5.0 thain 6/21 12: :00:15 R fib 40 Each job gets a unique number. Status: Unexpanded, Running or Idle Size of program image (MB)

Receive When Done This is an automated from the Condor system on machine "axpbo8.bo.infn.it". Do not reply. Your condor job /tmp_mnt/usr/users/ccl/thain/test/fib 40 exited with status 0. Submitted at: Wed Jun 21 14:24: Completed at: Wed Jun 21 14:36: Real Time: 0 00:11:54 Run Time: 0 00:06:52 Committed Time: 0 00:01:37...

Running Many Processes › 100 processes are almost as easy as !. › Each condor_submit makes one cluster of one or more processes. › Add the number of processes to run to the Queue statement. › Use the $(PROCESS) variable to give each process slightly different instructions.

Running Many Processes › Perform the same program on 50 different intervals. › Output goes in integrate.out.1, integrate.out.2, and so on… Executable = integrate Arguments = $(PROCESS) Output = integrate.out.$(PROCESS) Log = integrate.log Queue 50

Running Many Processes % condor_q -- Submitter: axpbo8.bo.infn.it : ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 9.3 thain 6/23 10: :05:40 R fib thain 6/23 10: :05:11 R fib thain 6/23 10: :05:09 R fib jobs; 2 idle, 19 running, 0 held Cluster number Process number

Where Are They Running? › condor_q –run - Submitter: axpbo8.bo.infn.it : : ID OWNER SUBMITTED RUN_TIME HOST(S) 9.47 thain 6/23 10: :07:03 ax4bbt.bo.infn.it 9.48 thain 6/23 10: :06:51 pewobo1.bo.infn.it 9.49 thain 6/23 10: :06:30 osde01.pd.infn.it Current Location

Help! I’m buried in ! › By default, Condor sends one for each completed process. › Add these to your submit file:  notification = error  notification = never › To send it to someone else:  notify_user =

Removing Processes › Remove one process:  condor_rm 9.47 › Remove a whole cluster:  condor_rm 9 › Remove everything!  condor_rm -a

Getting Feedback

What have I done? › The user log file (fib.log) shows a chronological list of everything important that happened to a job. 001 ( ) 06/21 17:03:44 Job executing on host: 004 ( ) 06/21 17:04:58 Job was evicted. 009 ( ) 06/21 17:05:10 Job was aborted by the user.

What have I done? % condor_history ID OWNER SUBMITTED CPU_USAGE ST COMPLETED CMD 9.3 thain 6/23 10: :00:00 C 6/23 10:58 fib thain 6/23 10: :00:24 C 6/23 10:59 fib thain 6/23 10: :00:00 C 6/23 11:01 fib thain 6/23 10: :05:45 C 6/23 11:01 fib thain 6/23 10: :00:00 C 6/23 11:01 fib 7

Brief I/O Summary % condor_q –io -- Schedd: c01.cs.wisc.edu : ID OWNER READ WRITE SEEK XPUT BUFSIZE BLKSIZE joe KB KB KB/s KB 32.0 KB joe KB KB B /s KB 32.0 KB joe 44.7 KB 22.1 KB B /s KB 32.0 KB 3 jobs; 0 idle, 3 running, 0 held

Complete I/O Summary in Your condor job "/usr/joe/records.remote input output" exited with status 0. Total I/O: KB/s effective throughput 5 files opened 104 reads totaling KB 316 writes totaling 1.2 MB 102 seeks I/O by File: buffered file /usr/joe/input opened 2 times 100 reads totaling KB 311 write totaling 1.2 MB 101 seeks (Only since Condor Version )

Complete I/O Summary in › The summary helps identify performance problems. Even advanced users don't know exactly how their programs and libraries operate.

Complete I/O Summary in › Example:  CMSSIM - collider simulation  “Why is this job so slow?”  Data summary: read 250 MB from 20 MB file.  Very high SEEK total -> random access.  Solution: Increase buffer to 20 MB.

Who Uses Condor? % condor_q –global -- Schedd: to02xd.to.infn.it : ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD garzelli 6/21 18: :18:16 R tosti2trisdn -- Schedd: quark.ts.infn.it : ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD dellaric 4/10 14: :20:31 R john p2.dat dellaric 6/2 11: :27:30 R john p1.dat pamela 6/20 09: :41:43 R montepamela

Who uses Condor? % condor_status –submitters Name Machine Running IdleJobs MaxJobsRunning decux1.pv quark.ts.i to05xd.to RunningJobs IdleJobs Total 59 86

Who Uses Condor? % condor_userprio Last Priority Update: 6/23 16:27 Effective User Name Priority Number of users shown: 8

Who Uses Condor? › The user priority is computed by Condor to estimate how much of the pool’s CPU resources have been used by each submitter. › Lighter users receive a lower priority: they will be allocated CPUs before heavy users. › Users consuming the same amount of CPU will be allocated an equal amount.

Measuring Goodput › Goodput is the amount of time a workstation spends making forward progress on work assigned by Condor. › This is a big topic all by itself:

Measuring Goodput % condor_q –goodput -- Submitter: coral.cs.wisc.edu : : coral.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME GOODPUT CPU_UTIL Mb/s thain 6/23 07: :47: % 87.6% thain 6/23 07: :38: % 99.8% thain 6/23 07: :38: % 98.7% thain 6/23 07: :10: % 99.8% 0.00

Setting Requirements › We believe that Condor must allow both users (jobs) and owners (machines) to set requirements. › This is an absolute necessity in order to convince people to participate in the community.

ClassAds › ClassAds are a simple language for describing both the properties and the requirements of jobs and machines. › Condor stores nearly everything in ClassAds -- use the –l option to condor_q and condor_submit to get the full details.

ClassAd for a Machine › condor_status –l axpbo8 MyType = "Machine" TargetType = "Job" Name = "axpbo8.bo.infn.it" START = TRUE VirtualMemory = Disk = Memory = 160 Cpus = 1 Arch = "ALPHA" OpSys = "OSF1“

ClassAd for a Job › condor_q –l 9.49 MyType = "Job" TargetType = "Machine" Owner = "thain" Cmd = "/tmp_mnt/usr/users/ccl/thain/test/fib" Out = “fib.out.49” Args = “49” ImageSize = 2544 DiskUsage = 2544 Requirements = (Arch == "ALPHA") && (OpSys == "OSF1") && (Disk >= DiskUsage) && (VirtualMemory >= ImageSize)

Default Requirements › By default, Condor assumes the requirements for your job are: “I need a machine with…”  The same operating system and architecture as my workstation.  Enough disk to store the program.  Enough virtual memory to run the program.

ClassAd Requirements › Similar to C/C++/Java expressions:  Symbols: Arch, OpSys, Memory, Mips  Values: 15, 6.5, “LINUX”  Operators: ==,, = &&, || ( )

Adding Requirements › In the submit file, add a line beginning with “requirements = “ Executable = fib Arguments = 40 Output = fib.out Log = fib.log Requirements = (Memory > 64) queue

Example Requirements › (Memory>64) › (Machine == “axpbo3.bo.infn.it” ) › (Mips>100) || (Kflops>10000) › (Subnet != “ ”) && (Disk > )

Preferences › Condor assumes that any machines that match your requirements are suitable. › However, you may prefer some machines over others. (100 Mips is better than 10) › To indicate a preference, you may provide a ClassAd expression which ranks all matches.

Rank › The rank expression is evaluated into a number for every potential matching machine. › A machine with a higher number will be preferred over a machine with a lower number.

Rank Examples › Prefer machines with more Mips: Rank = Mips › Prefer machines with a high ratio of memory to cpu performance: Rank = Memory/Mips › Prefer more memory, but add 100 to the rank if the machine is Solaris 2.7: Rank = Memory + 100*(OpSys==“SOLARIS27)”

Standard or Vanilla?

Which Universe? › Each Condor universe provides different services to different kinds of programs:  Standard – Relinked UNIX programs  Vanilla – Unmodified UNIX programs  PVM  Scheduler (Not described here)  Globus

Standard Universe › Submit a specially-linked UNIX application to the Condor system. › Advantages:  Checkpointing for fault tolerance.  Remote I/O services: Friendly environment anywhere in the world. Data buffering and staging. I/O performance feedback. User remapping of data sources.

Standard Universe › Disadvantages:  Must statically link with Condor library.  Limited class of applications: Single-process UNIX binaries. Certain system calls prohibited.

System Call Limitations › Standard universe does not allow:  Multiple processes: fork(), exec(), system()  Inter-process communication: semaphores, messages, shared memory  Complex I/O: mmap(), select(), poll(), non-blocking I/O, …  Kernel-level threads (User level threads are OK.)

System Call Limitations › Too restrictive?  Use the vanilla universe.

Vanilla Universe › Submit any sort of UNIX program to the Condor system. › Advantages:  No relinking required.  Any program at all, including Binaries Shell scripts Interpreted programs (java, perl) Multiple processes

Vanilla Universe › Disadvantages:  No checkpointing.  Very limited remote I/O services. Specify input files explicitly. Specify output files explicitly.  Condor will refuse to start a vanilla job on a machine that is unfriendly. ClassAds: FilesystemDomain and UIDDomain

Which Universe? › Standard:  Good for mixed Condor pools, flocked pools, and the Grid at large. › Vanilla:  Good for a Condor pool of identical machines.

Conclusion › Condor expands your reach to many CPUs – even those you cannot log in to. › Condor makes it easy to run and manage large numbers of jobs › Good candidates for the standard universe are single-process CPU-bound jobs with simple I/O. › Too restrictive? Use the vanilla universe, but fewer available machines.

Move to Workshop Meet again in room ____ at _____. Bring printouts to follow along.