1. FCC HtCondor Submission:
Scripts to support
production of a large number of FCC events:
part of fcc_datasets
7 December 2017
Alice Robson

2. Example Example submission:
source fcc_condor_submit.sh –p input_parameters.yaml –e –r 100
Number of events per job
Input parameters yaml
Number of condor jobs

3. Input Parameters yaml (1)
Label
Directory for outputs
name: "CMS_eeZZ”
rate:
base_outputdir: outputs/
gaudi_command: ''' $FCCSWBASEDIR/run fccrun.py simple_papas_condor.py rpythiainput ee_ZZ.txt routput output.root rmaxevents '''
Main command
Expected events per hour => choose queue

4. Input Parameters yaml (2)
events per job
name: "CMS_eeZZ”
events: 10
runs: 2
rate:
base_outputdir: /eos/experiment/fcc/ee/datasets/papas/
xrdcp_base: root://eospublic.cern.ch/
input: $FCCDATASETS/htcondor/examples/papas/ee_ZZ.txt
script: $FCCDATASETS/htcondor/examples/papas/simple_papas_condor.py
gaudi_command: ''' $FCCSWBASEDIR/run fccrun.py {} rpythiainput {} --routput output.root rmaxevents {}''.format(condor_pars["script"], condor_pars["input"], condor_pars["events"])'
number of jobs
for xrdcp with eos
Input parameters used by gaudi_command
source fcc_condor_submit.sh –p input_parameters.yaml –e –r 100
overrides values in yaml

5. FCC_condor_submit.sh What does fcc_condor_submit.sh do?
Calls fcc_condor_setup.py:-
creates uniquely named directory inside
- working directory (not EOS)
- output directory (may be EOS)
writes a parameter.yaml file
creates error/log/output directories in working directory (for condor)
copies across files needed for condor runs
chooses the queue type (timing)
writes a condor dag.sub to submit several jobs (run.sub/run.sh/run.py)
sets final summary stage of condor dag (finish.sub/finish.sh/finish.py)
Unsets some local python related env variables
Submits dag job
Resets env variables

6. Working Directory
CMS_eeZZ_ _e100000_r10_0 NAME_YYYYMMDD_eEVENTS_rRUNS_COUNTER
Unique working directory name
creates final summary info.yaml
used by condor
Main condor dag submission file
submission for each run
Working directory after completion of runs
All other files are logs/errors/outputs from condor

7. Dag submission file Automatically generated Dag submission file
######DAG file
Job A0 run.sub
Vars A0 runnumber="0"
Job A1 run.sub
Vars A1 runnumber="1"
Job A2 run.sub
...
Job A9 run.sub
Vars A9 runnumber="9"
FINAL FO finish.sub
runs when all the other jobs have finished and
produces a summary info.yaml file

8. Outputs (EOS or otherwise)
CMS_eeZZ_ _e100000_r10_0
Summary yaml file
Output files
Other files are parameter/configuration files

9. Summary info.yaml file Run details unique directory_name
parameters:
  base_outputdir: /eos/experiment/fcc/ee/datasets/papas/
  events: 3
  gaudi_command: '''LD_PRELOAD=$FCCSWBASEDIR/build.$BINARY_TAG/lib/libPapasUtils.so
    $FCCSWBASEDIR/run  fccrun.py {}   --rpythiainput  {} --routput output.root --rmaxevents
    {}''.format(condor_pars["script"], condor_pars["input"], condor_pars["events"])'
  input: /afs/cern.ch/work/a/alrobson/papasdagruns/fcc-ee-higgs/ee_ZZ.txt
  name: CMS_ee_ZZ
  parameters: papas_CMS_ee_ZZ.yaml
  rate: 50000
  runs: 2
  script: $FCCDATASETS/htcondor/examples/papas/simple_papas_condor.py
  subdirectory: CMS_ee_ZZ_ _e3_r2_20
  xrdcp_base: root://eospublic.cern.ch//
sample:
  id: !!python/object:uuid.UUID
    int:
  jobtype: fccsw
  mother: null
  nevents: 6
  nfiles: 2
  ngoodfiles: 2
  pattern: '*.root'
software:
  fccdag: /cvmfs/fcc.cern.ch/sw/0.8.1/dag/0.1/x86_64-slc6-gcc62-opt
  fccedm: /cvmfs/fcc.cern.ch/sw/0.8.1/fcc-edm/0.5.1/x86_64-slc6-gcc62-opt
  fccpapas: /cvmfs/fcc.cern.ch/sw/0.8.1/papas/1.2.0/x86_64-slc6-gcc62-opt
  fccphysics: /cvmfs/fcc.cern.ch/sw/0.8.1/fcc-physics/0.2.1/x86_64-slc6-gcc62-opt
  fccsw: !!python/unicode 'dddd362ea142b51c25d11eb357155fcd2a19a38a'
  fccswstack: /cvmfs/fcc.cern.ch/sw/0.8.1
  podio: /cvmfs/fcc.cern.ch/sw/0.8.1/podio/0.7/x86_64-slc6-gcc62-opt
  pythia8: /cvmfs/sft.cern.ch/lcg/views/LCG_88/x86_64-slc6-gcc62-opt
  root: /cvmfs/sft.cern.ch/lcg/releases/ROOT/ c8fb4/x86_64-slc6-gcc62-opt
Run details
unique directory_name
How many events were successfully
produced
Software versions

10. Not just for Papas: delphes example
source fcc_condor_submit.py –p delphes_parameters.yaml –e –r 10
default_parameters:
name: delphes
events: 10
  runs: 2
  rate: 10000
  base_outputdir: /eos/experiment/fcc/ee/datasets/papas/
xrdcp_base: root://eospublic.cern.ch
  script: $FCCDATASETS/htcondor/examples/delphes/PythiaDelphes_config.py
  gaudi_command: ''' $FCCSWBASEDIR/run  fccrun.py  {} --nevents {} ''.format(
condor_pars["script"], condor_pars["events"])'

11. How to get going Install FCCSW >source init.sh Install fcc_datasets
Create a base working directory
Create input_parameters.yaml (see examples in fcc_datasets)
Submit:
source fcc_condor_submit.sh –p input_parameters.yaml –e –r 10
See also: fcc_datasets/htcondor/CondorSubmit.md

12. How fast? Pythia Generation with Papas Runs
Depends on configuration... and on batch machine...
Typical so far: events/hour (25 events/sec)
(roughly between 50 and 20 events/sec)
So with 100 runs of events run overnight could produce
events.
NB Condor time not normalized: a job in the « 1 hour » queue finishes 1 hour after execution starts (real time)

13. Condor Comments Documentation is not great
Need to be careful with python environment
Cannot have the condor working directory on EOS
Condor not stable:-
issues with failed submissions (store_cred)
eos out of order for several days
several machines await a cvfms patch and cause jobs to fail
... but hopefully scripts are now more robust 