Using the NSF TeraGrid for Parametric Sweep CMS Applications Jeffrey P. Gardner Edward Walker Vladimir Litvin Pittsburgh Supercomputing Center Texas Advanced Computing Center Texas Advanced Computing Center California Institute of Technology

The NSF TeraGrid Links nine resource provider sites via a high speed 10/30 Gbps network backbone, providing, in aggregate, 20 teraflops of computing power, 1 petabyte of storage capacity, and high-end facilities for remote visualization of computing results. Compute resources composed of a heterogeneous mix of clusters with different architectures and operating systems, running different workload management systems, with different local configuration, specifying different queues, with different job submission and run- time limits.

Requirements Schedulers at each Teragrid site –Should be able to gracefully handle ~1000 single- processor jobs at a time Metascheduler (Teragrid doesn’t have metascheduler) –distributes jobs across the Teragrid –Metascheduler must be able to handle ~100,000 jobs

Solution: PBS/LSF? In theory, existing Teragrid schedulers like PBS or LSF should provide the answer. In practice, this does not work. –Teragrid nodes are multiprocessor –Only 1 PBS job per node –Teragrid machines frequently restrict the number of jobs a single user may run –Asking for many processors at once communicates your actual resource requirements.

Solution: Clever shell scripts? We could submit a single PBS job that uses many processors. –Now we have a reasonable number of PBS jobs. –Scheduling priority would reflect our actual resource usage. This still has problems. –Each job takes a different amount of time to run: we are using resources inefficiently.

Metacheduler Solution: Condor-G? Condor-G will schedule an arbitrarily large number of jobs across multiple grid resources using Globus. However, 1 serial Condor-G job = 1 PBS job, so we are left with the same PBS limitations as before: –Teragrid nodes are multiprocessor –Only 1 PBS job per node –Teragrid machines frequently restrict the number of jobs a single user may run.

The Real Solution: Condor + GridShell The real solution is to submit one large PBS on each Teragrid node, then use a private scheduler to manage serial work units within the PBS job. Vocabulary: JOB: (n) a thing that is submitted via Globus or PBS WORK UNIT: (n) An independent unit of work (usually serial), such as the analysis of a single run PBS Job PE Serial Work Units Private Scheduler

The Real Solution: Condor + GridShell The real solution is to submit one large PBS on each Teragrid node, then use a private scheduler to manage serial work units within the PBS job. Vocabulary: JOB: (n) a thing that is submitted via Globus or PBS WORK UNIT: (n) An independent unit of work (usually serial), such as the analysis of a single run PBS Job PE Serial Work Units Private Scheduler Condor GridShell

Features: GridShell/Condor on the TeraGrid Automatically throttles condor_startd job submissions based on configurable local site policies. Tolerates and recovers in the presence of transient faults in the WAN and login nodes. Balances condor_startd jobs between sites based on queue wait times. Shunts condor_startd jobs off temporary faulty sites. Automatically renews grid proxy credentials across sites within the command environment. Current users: Caltech CMS and NVO. SUBMITTED 70,000 jobs through clusters on TeraGrid to date.

Why GridShell/Condor? scalability - the actual parametric job submission is done directly to the compute nodes and not through the front- end node of the cluster; fault-tolerance –agents at a front-end node of a cluster maintain the condor_startd submissions locally, allowing transient WAN outages and periodic front-end node reboots to be resolved independently, in isolation from the rest of the system; and usability – the entire Condor submission, monitor, and control infrastructure is leveraged as a common job management environment for the user.

GridShell/Condor Process Architecture

Condor Overview Condor was first designed as a CPU cycle harvester for workstations sitting on people’s desks. Condor is designed to schedule large numbers of jobs across a distributed, heterogeneous and dynamic set of computational resources.

Advantages of Condor Condor is flexible –Resources can be any mix of architectures –Resources do not need a common filesystem –Resources do not need common user accounting Condor is dynamic –Resources can disappear and reappear Condor is fault-tolerant –Jobs are automatically migrated to new resources if existing one become unavailable.

Central Manager collector Condor Daemon Layout (very simplified) Submission Machine schedd Execution Machine startd Startd sends system specifications (ClassAds) and system status to Central Manager negotiator (To simplify this example, the functions of the Negotiator are combined with the Collector)

Condor Daemon Layout (very simplified) Central Manager collector Submission Machine schedd Execution Machine startd Schedd sends job info to Central Manager User submits Condor job

Central Manager collector Condor Daemon Layout (very simplified) Submission Machine schedd Execution Machine startd Central Manager uses information to match Schedd jobs to available Startds

Condor Daemon Layout (very simplified) Submission Machine schedd Execution Machine startd Schedd sends job to Startd on assigned execution node Central Manager collector

“Personal” Condor on a Teragrid Platform Condor daemons can be run as a normal user. Condor “GlideIn”™ ability supports the ability to launch condor_startd’s on nodes within an LSF or PBS job.

“Personal” Condor on a Teragrid Platform (Condor runs with normal user permissions) Central Manager collector Submission Machine schedd Execution PE startd Execution PE startd Execution PE startd Login Node PBS Job on Compute Nodes- GlideIn

GridShell Overview Allows users to interact with distributed grid computing resources from a simple shell-like interface. extends TCSH version 6.12 to incorporates grid-enabled features: –parallel inter-script message-passing and synchronization –output redirection to remote files –parametric sweep

GridShell Examples Redirecting the standard output of a command to a remote file location using GlobusFTP: a.out > gsiftp://tg-login.ncsa.teragrid.org/data Message passing between 2 parallel tasks: if ( $_GRID_TASKID == 0) then echo "hello" > task_1 else Set msg=`cat < task_0` endif Executing 256 instances of a job: a.out on 256 procs

Merging GridShell with Condor Use GridShell to launch Condor GlideIn jobs at multiple grid sites All Condor GlideIn jobs report back to a central collector This converts the entire Teragrid into your own personal Condor pool!

Merging GridShell with Condor Login Node Gridshell event monitor User starts GridShell Session at TACC TACC (IA32) Purdue (IA32) UC (IA32) GridShell process

Merging GridShell with Condor Login Node Gridshell event monitor Login Node Gridshell event monitor Login Node Gridshell event monitor GridShell session starts event monitor on remote login nodes via Globus TACC (IA32) Purdue (IA32) UC (IA32) GridShell process Condor process

Merging GridShell with Condor Login Node collectorschedd Gridshell event monitor Login Node Gridshell event monitor Login Node Gridshell event monitor Local event monitor starts condor daemons on login node TACC (IA32) Purdue (IA32) UC (IA32) GridShell process Condor process

Login Node collectorschedd Gridshell event monitor PBS Job Login Node Gridshell event monitor PBS Job Login Node Gridshell event monitor PBS Job gtcsh-ex UC (IA32) TACC (IA32) Purdue (IA32) All event monitors submit PBS/LSF jobs. These jobs start GridShell gtcsh-exec on all processors All event monitors submit PBS/LSF jobs. These jobs start GridShell gtcsh-exec on all processors gtcsh-ex Master gtcsh-exec GridShell process Condor process gtcsh-ex Master gtcsh-exec gtcsh-ex Master gtcsh-exec

Login Node collectorschedd Gridshell event monitor PBS Job Login Node Gridshell event monitor PBS Job Login Node Gridshell event monitor PBS Job gtcsh-ex UC (IA32) TACC (IA32) Purdue (IA32) gtcsh-exec on each processor starts a Condor startd. Heartbeat is maintained between all gtcsh-exec processes gtcsh-exec on each processor starts a Condor startd. Heartbeat is maintained between all gtcsh-exec processes gtcsh-ex Master gtcsh-exec GridShell process Condor process gtcsh-ex Master gtcsh-exec gtcsh-ex Master gtcsh-exec startd “Heartbeat”

Login Node collectorschedd Gridshell event monitor PBS Job Login Node Gridshell event monitor PBS Job Login Node Gridshell event monitor PBS Job gtcsh-ex UC (IA32) TACC (IA32) Purdue (IA32) gtcsh-exec on each processor starts a Condor startd gtcsh-ex Master gtcsh-exec GridShell process Condor process gtcsh-ex Master gtcsh-exec gtcsh-ex Master gtcsh-exec startd

Login Node collectorschedd Gridshell event monitor PBS Job Login Node Gridshell event monitor PBS Job Login Node Gridshell event monitor PBS Job gtcsh-ex UC (IA32) TACC (IA32) Purdue (IA32) Condor schedd distributes Condor jobs to compute nodes gtcsh-ex Master gtcsh-exec GridShell process Condor process gtcsh-ex Master gtcsh-exec gtcsh-ex Master gtcsh-exec startd

CMS production chain (preparations) First of all we are installing all CMS software on clusters and pileup at lonestar (we don’t have enough disk space to set up pileup everywhere) We are defining the name of the clusters because now we have only one cluster where digitization with pileup and final analysis must be done – lonestar –Copy both the condor_config.master.template and condor_config.glidein.template to our own private directory. Then in our login script (.cshrc or.bashrc), set the environment variable _GRID_TEMPLATE_DIR to point to this directory –Add the unique definition of the cluster in condor_config.glidein.template

CMS production chain (preparations) Define specific variables in.cshrc which will be different from cluster to cluster: $WORKDIR - where temporary subdir will be created and everything will be run $WWORKDIR - where temporary subdir will be created and everything will be run, as it is originally defined in OS. Usually it is the same with $WORKDIR, but sometimes it is differ. It is needed for FCrenamePFN to get rid of the absolute paths before tarring files and moving them out. $OSCAR_SRC - main OSCAR source directory where "eval `scram runtime -csh ` must be done to set everything up. $ORCA_SRC - main ORCA source directory where "eval `scram runtime - csh ` must be done to set everything up. $TOPRELEASE - where all releases of CMS software were installed. $VCAL - place where VCAL shower library was placed $DETGEOMFILE - place where cms133.rz is placed $PILEUP - where are the pileup data and PoolFileCatalog_pileup.xml resided

Start GridShell Session Write a simple GridShell configuration script: # vo.conf: # A GridShell config script tg-login.uc.teragrid.org tg-login.purdue.teragrid.org Start GridShell session. This submits PBS jobs at each site and starts local Condor daemons % vo-login –n 4:16 –H ~/vo.conf –G –W 600 Spawning on tg-login.uc.teragrid.org Spawning on tg-login.purdue.teragrid.org waiting for VO participants to callback... ###########Done. -n 4:16 Start 4 PBS job per Teragrid machine, each with 16 processors -W 600 Each PBS job has a wallclock limit of 600 minutes -H vo.conf Configuration file

CMS production chain One main preparation script which massaging the scripts and putting all run numbers etc in place (walltime is in secs): –./createCondorSubmit.csh –This script has all info hardcoded inside – number of events per run, dataset and owner names, condor submission parameters, etc. So, if you need to check this out and change the parameters – this is the place to look at. –It invokes./createCondorJob.csh which is making preparations for single job with specified run number.

CMS production chain The main part of createCondorJob.csh:./makeDAG.pl $titles $counter >>! $TOPDIR/work/${titles}.dag./makeOscarInitCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $wall_time_oscar_init >! $TOPDIR/work/${titles}_oscar_init_$counter.cdr./makeOscarJobCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $wall_time_oscar_job $PYTHIA_DATASET_NAME >! $TOPDIR/work/${titles}_oscar_job_$counter.cdr./makeOrcaInitCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $ORCA_OWNER $ORCA_DATASET $wall_time_orca_init >! $TOPDIR/work/${titles}_orca_init_$counter.cdr./makeOrcaJobCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $ORCA_OWNER $ORCA_DATASET $wall_time_orca_job >! $TOPDIR/work/${titles}_orca_job_$counter.cdr./makeTTCDR.pl $titles $counter $ORCA_OWNER $ORCA_DATASET $wall_time_tt >! $TOPDIR/work/${titles}_tt_$counter.cdr

CMS production chain DAG template JOB OSCAR_INIT /home/tg456083/Prod2005/aajj/work/aajj_oscar_init.cdr JOB OSCAR_JOB /home/tg456083/Prod2005/aajj/work/aajj_oscar_job.cdr JOB ORCA_INIT /home/tg456083/Prod2005/aajj/work/aajj_orca_init.cdr JOB ORCA_JOB /home/tg456083/Prod2005/aajj/work/aajj_orca_job.cdr JOB TT /home/tg456083/Prod2005/aajj/work/aajj_tt.cdr PARENT OSCAR_INIT CHILD OSCAR_JOB PARENT OSCAR_JOB CHILD ORCA_INIT PARENT ORCA_INIT CHILD ORCA_JOB PARENT ORCA_JOB CHILD TT RETRY OSCAR_INIT 0 RETRY OSCAR_JOB 0 RETRY ORCA_INIT 0 RETRY ORCA_JOB 0

CMS production chain CDR template # Common job definitions" Executable = /home/tg456083/Prod2005/aajj/startup/orca_job_wrapper Getenv = True Notification = Never universe = vanilla Error = /home/tg456083/Prod2005/aajj/work/err_orca_job.err Output = /home/tg456083/Prod2005/aajj/work/out_orca_job.out Log = /home/tg456083/Prod2005/aajj/logs/log_orca_job.log should_transfer_files = TRUE when_to_transfer_output = ON_EXIT arguments = $titles $counter $orca_owner $orca_dataset requirements = TARGET.TimeToLive > _WALLTIME_ORCA_JOB_ Requirements = TARGET.Project == “TACC_RESOURCES" Queue

CMS production chain *_wrapper is doing all job (orca_job_wrapper e.g.): –Checking all setup variables –Transferring files from previous step to the area where this job will be executed –Running the job –Attaching run and making FixColl –Moving data out

CMS production chain Submit the whole DAG to condor # Filename: /home/tg456083/Prod2005/aajj/work/aajj.dag.condor.sub # Generated by condor_submit_dag /home/tg456083/Prod2005/aajj/work/aajj.dag universe = scheduler executable = /home/tg456083/condor2//bin/condor_dagman getenv = True output = /home/tg456083/Prod2005/aajj/work/aajj.dag.lib.out error = /home/tg456083/Prod2005/aajj/work/aajj.dag.lib.out log = /home/tg456083/Prod2005/aajj/work/aajj.dag.dagman.log remove_kill_sig = SIGUSR1 arguments = -f -l. -Debug 3 -Lockfile /home/tg456083/Prod2005/aajj/work/aajj.dag.lock Condorlog /home/tg456083/Prod2005/aajj/logs/personal_aajj.log –Dag /home/tg456083/Prod2005/aajj/work/aajj.dag -Rescue /home/tg456083/Prod2005/aajj /work/aajj.dag.rescue -MaxJobs 200 -MaxPre 200 -MaxPost 200 -NoPostFail environment =_CONDOR_DAGMAN_LOG=/home/tg456083/Prod2005/aajj/work/aajj.dag.dagman.out;_CONDOR_ MAX_DAGMAN_LOG=0 queue

Conclusion We have used GridShell to turn the TeraGrid into our own personal Condor pool We can submit Condor jobs, and Condor will schedule these jobs across multiple TeraGrid site TeraGrid sites do not need to share architecture or queuing systems GridShell also allows us to use TeraGrid protocols to transfer our input and output data All of this fits into existing Teragrid software. We can run CMS production chain through this system –40000 were done –Will be demonstrated at SC2005 in Seattle in November 2005