1. Running FTKSIM on the grid 1. Summary of experience 2. Technical tutorial

2. Non-technical summary ● Most technical difficulties have been solved – “-L ANALY_MWT2” is ready to run ftksim jobs – Other sites may or may not work ● Can submit 32 parallel jobs (one per subreg) – All output is registered under a single dataset – Saves ASCII tracks, root files, and logs ● Features and Limitations – Time: long queue (default) is 120 hours (!!!) – Space: 2 gb of space per job (hard limit)

3. Getting data to compute nodes ● Must feed ftksimwrap_raw.dat files into ftksim ● Tried the following options (worst to best): – Wget (zipped).dat files from ftp/http server – Globus-url-copy zipped.dat files from TIER3 – dq2-get from a registered dataset – Arrange for data to be in MWT2 dcache ● Last method is the best: input data can be read directly from /pnfs/... filesystem – Note that wrapper jobs submitted to ANALY_MWT2 are automatically available

4. Tutorial / implementation details ● Set up and source recent athena (eg, 14.0.23) ● Download & install prun – a pathena-like job submitter – https://twiki.cern.ch/twiki/bin/view/Atlas/PandaRun https://twiki.cern.ch/twiki/bin/view/Atlas/PandaRun – Don't forget to set PATHENA_GRID_SETUP_SH – Don't forget to source panda_setup.sh ● Download & extract ftksim submission scripts – http://hep.uchicago.edu/~antonk/gridftk.tar.bz2 http://hep.uchicago.edu/~antonk/gridftk.tar.bz2 ● Main script is prun_job.sh; it has some settings near the top – Choose # events to process (safe to have it large) ● A file “command” contains sample submission commands

5. What the script does ● Parse command line: which reg and subreg ● Source environment on tier2 compute node ● If ftksim.tar.bz2 not in $PWD, download it ● Compile ftksim with either gzip or libdcap support ● If config.tar.bz2 not in $PWD, download it ● Create a list of input data from contents of data/ directory ● Loop over all requested regions/subregions and: – Download & extract the banks – Run ftksim in each subregion ● Archive all output and let pathena register it in the catalog You can use custom versions of ftksim or config scripts & maps. Simply put your own version of ftksim.tar.bz2 and config.tar.bz2 into the directory from which you submit the prun jobs!

6. Getting the data ● Getting data is decoupled from the main script ● prepare_data.sh must create a data/ directory with all inputs ● Look through prepare_data.sh for examples ● Caveat: unless the data is read directly from tier2 /pnfs, it will be downloaded and will contribute to the 2gb space limit! – Can't run over too many input files at once – But ftksim can read zipped files (“gzip filename.dat”) ● Recommended way: read data from /pnfs on TIER2 – Example in prepare_data.sh: ● simply symlinks /pnfs/ /*.dat to data/ ● prun_script.sh then directly uses them as input!

7. Useful commands ● Use globus-job-run to find your files on tier2, or make directories: ● globus-job-run uct2-grid6.uchicago.edu /bin/ls -lsh /pnfs/uchicago.edu/data/tier3/user/user.AntonKapliy/ ● Use globus-url-copy to transfer files to or from tier2: ● globus-url-copy gsiftp://uct2- dc1.uchicago.edu/pnfs/uchicago.edu/atlasuserdisk/user09.AntonKapliy/user09.AntonKapliy.ftksim.wh1 20bb34_10evts.01_sub04102486/user09.AntonKapliy.ftksim.wh120bb34_10evts.01._24691412.log.tgz file:///tmp/myfile.tgz file:///tmp/myfile.tgz ● Download ftksim run results: ● dq2-get -L ROAMING user09.AntonKapliy.ftksim.WH120bb34.2000evts.05 ● There are 3 *.tbz (tar xfj) archives for each subregion: *log*.tbz, *root*.tbz, *track*.tbz ● There are also a few *.tgz archives (tar xfz) that contain general logs

8. In case of errors ● Some jobs may fail, since not all tier2 nodes are homogeneous. I've seen the following: – Some missed aclocal & GNU autotools – Some missed libdcap – These particular problems are now fixed ● prun_job.sh has some simple error reporting – Look for athena_stdout.txt in *XYZ_log.dat.tgz ● If a new node problem is found, email me – kapliy@gmail.com