1. 26 April 2005Steve Armstrong1 High Level Trigger Software Tutorial Steve Armstrong Brookhaven National Laboratory 26 April 2005 ATLAS Trigger/DAQ Week at CERN

2. 26 April 2005Steve Armstrong2 OVERVIEW OF THIS TUTORIAL Welcome! Main Goal: Convert newcomers or currently inactive HLT members into active software developers – hope to break the prevalent addiction to avoiding trigger software development! Minimal knowledge of Unix/Linux, C++, shell scripts, cmt, Athena, ATLAS, paw or ROOT is required; DISCLAIMER: this is first tutorial of this type given, so please give me feedback on what is good/bad/clear/not clear about it! ATLAS Software is amazingly capricious – be prepared for this, have patience; Never be afraid to ask questions: frequently things are genuinely wrong and need to be fixed by experts; Impossible to document every step and allow for every contingency, even for simple examples. Be aware of good web-based resources: http://hepunx.rl.ac.uk/atlasuk/simulation/level2/doc/PesaCore/ https://uimon.cern.ch/twiki/bin/view/Atlas/ReleaseRecipes https://uimon.cern.ch/twiki/bin/view/Atlas/RunningReconstruction

3. 26 April 2005Steve Armstrong3 TUTORIAL CONVENTIONS X.Y IS A STEP X.SUB-STEP Y IN THE TUTORIAL. red text are commands you type at shell prompt blue text is something that should be printed out from Linux/Athena underlined and italicized text is not literal - replace it with something like your afs username purple text is something that needs to be changed/checked

4. 26 April 2005Steve Armstrong4 STEP 0. WHAT YOU SHOULD HAVE BEFORE YOU START 0.1 We will work on CERN LXPLUS machines running Scientific Linux SLC 3.0.3. Therefore you must have: 0.1.1 An active LXPLUS account; 0.1.2 A means to ssh to LXPLUS from your laptop/computer. 0.2 Please create a new session on LXPLUS: ssh -l yourusername lxplus.cern.ch 0.3 Please use a common ATLAS shell (e.g., tcsh, bash, zsh). Check your shell by doing: ps PID TTY TIME CMD 5939 pts/26 00:00:00 tcsh 10861 pts/26 00:00:00 ps This tells you which shell you are using.

5. 26 April 2005Steve Armstrong5 1.1 Do you have a scratch0 or maxidisk area with some space on it? 1.1.1 If yes, you should see/do: ls mail public scratch0 util www cd scratch0 fs listquota Volume Name Quota Used %Used Partition s.atlas.sarmstro.0 1500000 634754 42% 88% 1.1.2 If no, you should use temporary “/tmp” space: cd /tmp mkdir yourusername cd /tmp/yourusername STEP 1. CREATE A DIRECTORY TO DEVELOP SOFTWARE Your scratch0 area exists! If not, goto 1.1.2 Your quota in kBytes. Make sure (Quota – Used) > 100000 If you have to do this, please, in subsequent transparencies, replace “~/scratch0” with “/tmp/yourusername”

6. 26 April 2005Steve Armstrong6 STEP 1. CREATE A DIRECTORY TO DEVELOP SOFTWARE (continued) 1.2 Make a directory in your scratch area and change into it: cd ~/scratch0 ; mkdir Tutorial ; cd Tutorial Here, I have chosen the directory name “Tutorial”. You can choose another name, just be careful to replace “Tutorial” in the subsequent slides with the name you have chosen.

7. 26 April 2005Steve Armstrong7 STEP 2. CONFIGURE CMT TO USE RELEASE 10.0.1 2.3 Edit the cmt requirements file to look something like this: set CMTSITE CERN \ cmt_atlas "ATLAS" macro ATLAS_DIST_AREA "/afs/cern.ch/atlas/software/dist" # # Select releases as they are available # macro ATLAS_RELEASE "10.0.1" #otherwise use nightly #macro ATLAS_RELEASE "/nightlies/rel/atlrel_3" use AtlasSettings v* $(ATLAS_DIST_AREA) #use HLTSettings HLTSettings-* $(HLT_DIST_AREA)/releases/$(HLT_RELEASE) # # Prepend your test area(s) according to your choices # path_remove CMTPATH /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial path_prepend CMTPATH /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial 2.1 In your Tutorial area, make a “cmtdir” and change into it: cd ~/scratch0/Tutorial ; mkdir cmtdir; cd cmtdir 2.2 Get a copy of a cmt requirements file: cp ~sarmstro/public/requirements./ Make sure this says “10.0.1” Change these TWO lines to make it agree with the full path of YOUR Tutorial area

8. 26 April 2005Steve Armstrong8 STEP 3. SETUP CMT AND DOUBLE-CHECK THINGS 3.2 Make sure your cmt is working and configured properly: cmt config ------------------------------------------ Configuring environment for standalone package. CMT version v1r16p20040901. System is i686-slc3-gcc323-dbg ------------------------------------------ Creating setup scripts. Creating cleanup scripts. 3.1 Make sure you are in your Tutorial cmtdir and Issue CMT setup command: cd ~/scratch0/Tutorial/cmtdir source /afs/cern.ch/sw/contrib/CMT/v1r16/mgr/setup.[c]sh 3.3 Execute cmt setup script: source setup.[c]sh -tag=opt 3.4 Make sure the cmt path is sensible: echo $CMTPATH /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial:/afs/cern.ch/atlas/software/di st/10.0.1:/afs/cern.ch/atlas/offline/external/Gaudi/0.14.6.14pool201:/afs/ cern.ch/atlas/offline/external/LCGCMT/LCGCMT_33 “dbg” means we are going to compile a debug version of the code which is significantly slower than “opt” or optimized version

9. 26 April 2005Steve Armstrong9 4.2 For demonstration purposes, let us also explicitly check out an HLT algorithm, T2CALO. Normally this would be done only if you were modifying it. cmt co -r TrigT2Calo-00-03-03 Trigger/TrigAlgorithms/TrigT2Calo 4.1 Change directory to the top of your Tutorial area and check out a specific tag of the TriggerRelease package cd ~/scratch0/Tutorial cmt co -r TriggerRelease-00-02-22 Trigger/TriggerRelease STEP 4. CHECK OUT TRIGGER SPECIFIC PACKAGES This tag, 00-02-22, is more recent than the one in Release 10.0.1 – it contains some minor fixes Again, we are using a specific tag version. If we were really modifying T2CALO in a development context, we would most certainly work with the “head” cvs version and in careful coordination with the other developers of this algorithm.

10. 26 April 2005Steve Armstrong10 STEP 5. MODIFY PACKAGE CMT REQUIREMENTS FILES 5.1 Within each Athena package, there is a cmt directory containing a requirements file for that package. We need to modify the one for TriggerRelease to let it know we have explicitly checked out T2CALO and may be modifying it: cd ~/scratch0/Tutorial/Trigger/TriggerRelease/TriggerRelease-00-02-22/cmt emacs -nw requirements 5.2 Within the TriggerRelease requirements file there are many “use” statements that are commented with a “#”. Look for the one for T2CALO and uncomment it (i.e., delete the “#”)... #use TrigAlgExample TrigAlgExample-* Trigger/TrigAlgorithms #use TrigIDSCAN TrigIDSCAN-* Trigger/TrigAlgorithms #use TrigMoore TrigMoore-* Trigger/TrigAlgorithms #use TrigSiTrack TrigSiTrack-* Trigger/TrigAlgorithms #use TrigSiTreeAlg TrigSiTreeAlg-* Trigger/TrigAlgorithms use TrigT2Calo TrigT2Calo-* Trigger/TrigAlgorithms #use TrigTRTLUT TrigTRTLUT-* Trigger/TrigAlgorithms #use TrigTRTxK TrigTRTxK-* Trigger/TrigAlgorithms...

11. 26 April 2005Steve Armstrong11 STEP 6. CONFIGURE AND COMPILE PACKAGES 6.1 From the cmt directory of TriggerRelease, setup things and then issue a cmt “broadcast” command to build packages cd ~/scratch0/Tutorial/Trigger/TriggerRelease/TriggerRelease-00-02-22/cmt cmt broadcast cmt config source setup.[c]sh cmt broadcast gmake and wait a few minutes...

12. 26 April 2005Steve Armstrong12 STEP 7. PREPARE THE RUN DIRECTORY 7.1 Create a run directory and cd to it: mkdir ~/scratch0/Tutorial/run cd ~/scratch0/Tutorial/run 7.2 Setup run area: source /afs/cern.ch/atlas/software/dist/10.0.1/Control/AthenaRunTime/*/cmt/setup.[c]sh 7.3 Setup RecExCommon: RecExCommon_links.[c]sh 7.4 Are there files in your run directory? ls dc1.001003.simul.redig.00001.test.h130_2e2mu_extract.zebra PDGTABLE.MeV ZEBRA_HEEMM.P ZEBRA_ZEE_LUMI10.P dc2_mixing_all_phys_0003621_file100002.data PoolFileCatalog.xml ZEBRA_HEEMM_TDR.P ZEBRA_ZEE.P FastShowerHistograms PoolFileCatalog.xml.BAK ZEBRA_JET_25.P jobOptions.py RawEvent_hllll_g3_extract.re ZEBRA.P myTopOptions.py ZEBRA_HEEMM_EXTRACT.P ZEBRA_ZEE_LUMI02.P

13. 26 April 2005Steve Armstrong13 STEP 8. COPY HLT SOFTWARE CONTROL FILES 8.1 Make sure you are still in the run directory: cd ~/scratch0/Tutorial/run 8.2 Use the “get_files” command to get files: get_files -copy -xmls LVL1triggermenu.xml LVL1triggerthresholds.xml Finding LVL1triggermenu.xml Finding LVL1triggerthresholds.xml Copying file LVL1triggermenu.xml from /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial Copying file LVL1triggerthresholds.xml from /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial get_files -copy -xmls HLTsequence.xml HLTsignature.xml Finding HLTsequence.xml Finding HLTsignature.xml Copying file HLTsequence.xml from /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial Copying file HLTsignature.xml from /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial

14. 26 April 2005Steve Armstrong14 STEP 9. EXAMINE/MODIFY HLT SEQUENCE FILE 9.1 Make sure you are in run directory and edit HLTsequence.xml file: cd ~/scratch0/Tutorial/run emacs -nw HLTsequence.xml 9.2 Modify your HLTsequence.xml file to look like this: <SEQUENCE level="L2" input="EMTAUROI" algorithm=" T2Calo/T2Calo/g4 TrigIdScan::IdScanMain/IdScan_IdScanMain/1 TrigSiTrack/SiTrack/SiTrack TrigEgammaNtuple/TrigEgammaNtuple/1 " output="LVL2out" /> L2 and EMTAUROI means that this sequence is for Level-2 will be executed upon receipt of a Level-1 accept for EMTAU triggers This specified the exact sequence of HLT algorithms that will be run everytime we have an RoI from Level-1

15. 26 April 2005Steve Armstrong15 STEP 10. GET AND MODIFY JOBOPTIONS 10.1 Make sure you are still in the run directory: cd ~/scratch0/Tutorial/run 10.2 Copy Trigger jobOptions using the “get_files” command: get_files -copy Trigger_topOptions_rome.py 10.3 Examine jobOptions: emacs -nw Trigger_topOptions_rome.py Locate and make note of the following lines: doTrigger = True EvtMax = 2 PoolRDOInput = [ "LFN:ele25_Rome_RDO_extract.pool" ] doCBNT = True This tells Athena that we are using Trigger software. This tells Athena how many events to process. Change to a low number (2) at first to make sure job runs! This is our MC file containing single electrons of 25 GeV. You can change it to your own favorite file, but you must make sure it is known in Pool file catalog in your run directory. This tells Athena that we want a combined Ntuple (CBNT) output. Note: these days, jobOptions in Athena are in python, which is powerful, but yet another language to learn.

16. 26 April 2005Steve Armstrong16 STEP 10. GET AND MODIFY JOBOPTIONS (continued) 10.4 For this tutorial, we want to run the Level-2 Algorithm T2CALO. Let us make sure the jobOptions is reading from our HLTsequence file. So we modify jobOptions: emacs -nw Trigger_topOptions_rome.py Comment the following lines: # JobO with TrigCaloRec #TriggerConfig.sequenceListFileLocation = "HLTsequenceLVL2_EF.xml" #TriggerConfig.signatureListFileLocation = "HLTsignatureLVL2_EF.xml" Uncomment the following lines: # JobO without TrigCaloRec TriggerConfig.sequenceListFileLocation = "HLTsequence.xml" TriggerConfig.signatureListFileLocation = "HLTsignature.xml" 10.5 Are you uncomfortable with using ROOT to look at Ntuples? If so, you may insert the following line in your jobOptions: useROOTNtuple=False but make sure you put it in only AFTER the line include ("RecExCommon/RecExCommon_flags.py") so you can use PAW instead of ROOT to look at ntuples.

17. 26 April 2005Steve Armstrong17 10.6 Insert the default data file into the pool catalog: pool_insertFileToCatalog ${ATLASTESTDATA}/ele25_digi_Rome_10ev_test.pool.root FCregisterLFN -p ${ATLASTESTDATA}/ele25_digi_Rome_10ev_test.pool.root -l \ ele25_Rome_RDO_extract.pool STEP 10. GET AND MODIFY JOBOPTIONS (and put data file into POOL catalog) (continued)

18. 26 April 2005Steve Armstrong18 STEP 11. ENHANCE MEMORY AND RUN ATHENA! 11.1 Make sure you are still in the run directory: cd ~/scratch0/Tutorial/run 11.3 Run athena with the Trigger jobOptions: athena.py -bs Trigger_topOptions_rome.py >& out1.log & 11.2 Enhance shell virtual memory: (for tcsh): limit vmemoryuse 1300000 (for zsh): limit addressspace 1300000 (for bash): ulimit -Sv 1300000 11.4 Monitor progress: tail -f out1.log The “-bs” flag tells Athena to print out ALL jobOptions used (i.e., the included jobOptions from included jobOptions...) to configure your job. This will result in a HUGE but very instructive log file. We do it here to demonstrate how highly configured and configurable Athena is. Frequently, problems running an Athena job are due to mistakes in jobOptions rather than C++ source code.

19. 26 April 2005Steve Armstrong19 STEP 12. UNDERSTAND OUTPUT FILE 12.1 As noted before, we are running athena with “-bs” option to explicitly show all jobOptions being used. Hence, in the log file, you will have literally thousands of lines looking like: Athena INFO including file "AthenaCommon/Compat.py" Athena INFO including file "AthenaCommon/Bootstrap.py" -+- 0 #************************************************************** -+- 1 # -+- 2 # Athena bootstrap options file -+- 3 # -+- 4 #============================================================== -+- 5 -+- 6 #-------------------------------------------------------------- -+- 7 # Atlas specific event loop and output stream -+- 8 #-------------------------------------------------------------- -+- 9 A theApp.EventLoop = "AthenaEventLoopMgr" -+- 10 A theApp.OutStreamType = "AthenaOutputStream“ Line number of current jobOptions file being included. Important! Only lines with “A” (for Active) are used – without “A” means it is a comment or part of a conditional statement which was not satisfied. 12.2 After lots of lines (about 8000) you should see lines showing what was defined by the HLTsequence.xml file: SequenceTable INFO *** Sequence table for step 1 *** Sequence INFO - Level: L2 Input(s): EMTAUROI Algo(s): T2Calo/T2Calo/g4 TrigIdScan::IdScanMain/IdScan_IdScanMain/1 TrigSiTrack/SiTrack/SiTrack TrigEgammaNtuple/TrigEgammaNtuple/1 Output: LVL2out

20. 26 April 2005Steve Armstrong20 STEP 12. UNDERSTAND OUTPUT FILE (continued) 12.3 Event processing starts at about line 8300 of the log file with the lines: AthenaEventLoopMgr INFO ===>>> start of run 2885 <<<=== AthenaEventLoopMgr INFO ===>>> start of event 1 <<<=== 12.4 If everything worked OK, you should see at the end of your output file: NTupleSvc INFO NTuples saved successfully ApplicationMgr INFO Application Manager Finalized successfully ApplicationMgr INFO Application Manager Terminated successfully

21. 26 April 2005Steve Armstrong21 STEP 13. LOOK AT RESULTS! 13.1 A successful athena job produces a CBNT ntuple output file as specified by the default jobOptions (you can change if you want): RootNtupleOutput = ntuple.root HbookNtupleOutput = ntuple.hbook 13.2 The content of the LVL2 Calorimeter Ntuple block is defined in the package Trigger/TrigAnalysis/TrigNtCalo. You can have a look at it here: http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAnalysis/TrigNtCalo/src/TrigNtCaloBlock.cxx 13.3 Let’s make a quick plot of Electromagnetic Cluster Energy, T2CaEmE using paw or root on the ntuple file…

22. 26 April 2005Steve Armstrong22 STEP 14. LEARN MORE ABOUT AN HLT ALGORITHM! 14.1 Examine the code of T2CALO using the LXR browser: http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/ 14.2 Note the 4 main “tools” that actually do the work of T2CALO: CaloSamp2Fex.cxx which calculates cluster position and energy in EM Sampling 2: http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloSamp2Fex.cxx CaloSamp1Fex.cxx which calculates strip profile and energy in EM Sampling 1: http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloSamp1Fex.cxx CaloEmEnFex.cxx which calculates energy in EM Presampler and Sample 3: http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloEmEnFex.cxx CaloHadEnFex.cxx which calculates energy in Tile and HEC: http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloHadEnFex.cxx 14.3 Examine the T2CALO jobOptions which are read in and control T2CALO operation, found in the T2CALO./share area: http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/share/jobOfragment_TrigT2Calo.py and make special note of the line that defines which tools and their order of execution: T2Calo_g4_L2.T2CaloTools=["CaloSamp2Fex","T2CaloL1Sim","CaloSamp1Fex","CaloEmEnFex","CaloHadEnFex"]