1. High Level Triggering
Fred Wickens

2. High Level Triggering (HLT)
Introduction to triggering and HLT systems
What is Triggering
What is High Level Triggering
Why do we need it
Case study of ATLAS HLT (+ some comparisons with other experiments)
Summary

3. Why do we Trigger and why multi-level
Over the years experiments have focussed on rarer processes
Need large statistics of these rare events
DAQ system (and off-line analysis capability) under increasing strain
limiting useful event statistics
Aim of the trigger is to record just the events of interest
i.e. Trigger selects the events we wish to study
Originally - only read-out the detector if Trigger satisfied
Larger detectors and slow serial read-out => large dead-time
Also increasingly difficult to select the interesting events
Introduced: Multi-level triggers and parallel read-out
At each level apply increasingly complex algorithms to obtain better event selection/background rejection
These have:
Led to major reduction in Dead-time – which was the major issue
Managed growth in data rates – this remains the major issue

4. Summary of ATLAS Data Flow Rates
From detectors > 1014 Bytes/sec
After Level-1 accept ~ 1011 Bytes/sec
Into event builder ~ 109 Bytes/sec
Onto permanent storage ~ 108 Bytes/sec  ~ 1015 Bytes/year

5. The evolution of DAQ systems

6. TDAQ Comparisons

7. Level 1 Time: few microseconds Hardware based
Using fast detectors + fast algorithms
Reduced granularity and precision
calorimeter energy sums
tracking by masks
During Level-1 decision time store event data in front-end electronics
at LHC use pipeline - as collision rate shorter than Level-1 decision time
For details of Level-1 see Dave Newbold talk

8. High Level Trigger - Levels 2 + 3
Level-2 : Few milliseconds (10-100)
Partial events received via high-speed network
Specialised algorithms
3-D, fine grain calorimetry
tracking, matching
Topology
Level-3 : Up to a few seconds
Full or partial event reconstruction
after event building (collection of all data from all detectors)
Level-2 + Level-3
Processor farm with Linux server PC’s
Each event allocated to a single processor, large farm of processors to handle rate

9. Summary of Introduction
For many physics analyses, aim is to obtain as high statistics as possible for a given process
We cannot afford to handle or store all of the data a detector can produce!
The Trigger
selects the most interesting events from the myriad of events seen
I.e. Obtain better use of limited output band-width
Throw away less interesting events
Keep all of the good events(or as many as possible)
must get it right
any good events thrown away are lost for ever!
High level Trigger allows:
More complex selection algorithms
Use of all detectors and full granularity full precision data

10. Case study of the ATLAS HLT system
Concentrate on issues relevant for ATLAS (CMS very similar issues), but try to address some more general points

11. Starting points for any Trigger system
physics programme for the experiment
what are you trying to measure
accelerator parameters
what rates and structures
detector and trigger performance
what data is available
what trigger resources do we have to use it
Particularly network b/w + cpu performance

12. Physics at the LHC
7 TeV
Interesting events are buried in a sea of soft interactions
B physics
High energy QCD jet production
top physics
Higgs production

13. The LHC and ATLAS/CMS LHC has This results in
Design luminosity 1034 cm-2s-1
In 2010 from 1027 – 2x1032 ; 2011 up to 2x1033
Design bunch separation 25 ns (bunch length ~1 ns)
This results in
~ 23 interactions / bunch crossing
~ 80 charged particles (mainly soft pions) / interaction
~2000 charged particles / bunch crossing
Total interaction rate sec-1
b-physics fraction ~ sec-1
t-physics fraction ~ sec-1
Higgs fraction ~ sec-1

14. Physics programme
Higgs signal extraction important - but very difficult
There is lots of other interesting physics
B physics and CP violation
quarks, gluons and QCD
top quarks
SUSY
‘new’ physics
Programme will evolve with: luminosity, HLT capacity and understanding of the detector
low luminosity ( )
high PT programme (Higgs etc.)
b-physics programme (CP measurements)
high luminosity (2013 or 2014?)
searches for new physics

15. Trigger strategy at LHC
To avoid being overwhelmed use signatures with small backgrounds
Leptons
High mass resonances
Heavy quarks
The trigger selection looks for events with:
Isolated leptons and photons,
-, central- and forward-jets
Events with high ET
Events with missing ET

16. Example Physics signatures
Objects
Physics signatures
Electron 1e>25, 2e>15 GeV
Higgs (SM, MSSM), new gauge bosons, extra dimensions, SUSY, W, top
Photon 1γ>60, 2γ>20 GeV
Higgs (SM, MSSM), extra dimensions, SUSY
Muon 1μ>20, 2μ>10 GeV
Jet 1j>360, 3j>150, 4j>100 GeV
SUSY, compositeness, resonances
Jet >60 + ETmiss >60 GeV
SUSY, exotics
Tau >30 + ETmiss >40 GeV
Extended Higgs models, SUSY

17. ARCHITECTURE Trigger DAQ 40 MHz ~1 PB/s (equivalent) ~ 200 Hz
Three logical levels
Hierarchical data-flow
~1 PB/s (equivalent)
LVL1 - Fastest: Only Calo and Mu Hardwired
On-detector electronics: Pipelines
~2.5 ms
LVL2 - Local: LVL1 refinement + track association
Event fragments buffered in parallel
~40 ms
LVL3 - Full event: “Offline” analysis
Full event in processor farm
~4 sec.
~ 200 Hz
~ 300 MB/s
Physics

18. Selected (inclusive) signatures

19. Trigger design – Level-1
sets the context for the HLT
reduces triggers to ~75 kHz
Uses limited detector data
Fast detectors (Calo + Muon)
Reduced granularity
Trigger on inclusive signatures
muons;
em/tau/jet calo clusters; missing and sum ET
Hardware trigger
Programmable thresholds
CTP selection based on multiplicities and thresholds

20. Level-1 Selection The Level-1 trigger
an “or” of a large number of inclusive signals
set to match the current physics priorities and beam conditions
Precision of cuts at Level-1 is generally limited
Adjust the overall Level-1 accept rate (and the relative frequency of different triggers) by
Adjusting thresholds
Pre-scaling (e.g. only accept every 10th trigger of a particular type) higher rate triggers
Can be used to include a low rate of calibration events
Menu can be changed at the start of run
Pre-scale factors may change during the course of a run

21. Trigger design - HLT strategy
Level 2
confirm Level 1, some inclusive, some semi-inclusive, some simple topology triggers, vertex reconstruction (e.g. two particle mass cuts to select Zs)
Level 3
confirm Level 2, more refined topology selection, near off-line code

22. Trigger design - Level-2
Level-2 reduce triggers to ~2 kHz
Note CMS does not have a physically separate Level-2 trigger, but the HLT processors include a first stage of Level-2 algorithms
Level-2 trigger has a short time budget
ATLAS ~40 milli-sec average
Note for Level-1 the time budget is a hard limit for every event, for the High Level Trigger it is the average that matters, so OK for a small fraction of events to take times much longer than this average
Full detector data is available, but to minimise resources needed:
Limit the data accessed
Only unpack detector data when it is needed
Use information from Level-1 to guide the process
Analysis proceeds in steps with possibility to reject event after each step
Use custom algorithms

23. Regions of Interest
The Level-1 selection is dominated by local signatures (I.e. within Region of Interest - RoI)
Based on coarse granularity data from calo and mu only
Typically, there are 1-2 RoI/event
ATLAS uses RoI’s to reduce network b/w and processing power required

24. Trigger design - Level-2 - cont’d
Processing scheme
extract features from sub-detectors in each RoI
combine features from one RoI into object
combine objects to test event topology
Precision of Level-2 cuts
Limited (although better than at Level-1)
Emphasis is on very fast algorithms with reasonable accuracy
Do not include many corrections which may be applied off-line
Calibrations and alignment available for trigger not as precise as ones available for off-line

25. ARCHITECTURE Trigger DAQ LVL1 H L T ~ 1 PB/s 40 MHz 75 kHz LVL2 ROS
Calo MuTrCh
Other detectors
~ 1 PB/s
40 MHz
40 MHz
75 kHz
~2 kHz
~ 200 Hz
LVL1
2.5 ms
Calorimeter
Trigger
Muon
FE Pipelines
2.5 ms
LVL1 accept
Read-Out Drivers
ROD
ROS
Read-Out Sub-systems
Read-Out Buffers
ROB
GB/s
Read-Out Links
RoI’s H L T
LVL2
~ 10 ms
L2P
L2SV
L2N
ROIB
RoI
requests
RoI data = 1-2%
~2 GB/s
Event Builder EB
~3 GB/s
LVL2 accept
Event Filter
EFP
~ 1 sec
EFN
~3 GB/s
~ 300 MB/s

26. CMS Event Building CMS perform Event Building after Level-1
Simplifies the architecture, but places much higher demand on technology:
Network traffic ~100 GB/s
1st stage use Myrinet
2nd stage has 8 GbE slices
Time will tell which is better

27. Example for Two electron trigger
e30i +
Signature 
t i m e
LVL1 triggers on two isolated
e/m clusters with pT>20GeV
(possible signature: Z–>ee)
Iso–
lation
STEP 4
e30 +
Signature 
pt>
30GeV
STEP 3
HLT Strategy:
Validate step-by-step
Check intermediate signatures
Reject as early as possible e +
Signature 
track
finding
STEP 2
ecand +
Signature 
Sequential/modular approach facilitates early rejection
Cluster
shape
STEP 1
EM20i +
Level1 seed 

28. Trigger design - Event Filter / Level-3
Event Filter reduce triggers to ~200 Hz
Event Filter budget ~ 4 sec average
Full event detector data is available, but to minimise resources needed:
Only unpack detector data when it is needed
Use information from Level-2 to guide the process
Analysis proceeds in steps with possibility to reject event after each step
Use optimised off-line algorithms

29. Execution of a Trigger Chain
Electromagnetic
clusters
EM ROI
Level1:
Region of Interest is found and position in EM calorimeter is passed to Level 2
Execution of a Trigger Chain
match?
L2 calorim.
L2 tracking
cluster?
track?
Level 2 seeded by Level 1
Fast reconstruction algorithms
Reconstruction within RoI
E.F.calorim.
E.F.tracking
track?
e/ OK?
e/ reconst.
Ev.Filter seeded by Level 2
Offline reconstruction algorithms
Refined alignment and calibration

30. Minimum Bias Trigger Phys.Lett.B 688, Issue 1, 2010 Soft QCD studies
Minbias Trigger Scintillator:
32 sectors on LAr cryostat
Main trigger for initial running
h coverage 2.1 to 3.8
Soft QCD studies
Provide control trigger on p-p collisions; discriminate against beam-related backgrounds (using signal time)
Minimum Bias Scintillators (MBTS) installed in each end-cap;
Example: MBTS_1 – at least 1 hit in MBTS
Also check nr. of hits in Inner Detector in Level-2
LHC collision rate (nb=4)
LHC collision rate (nb=2)
MBTS_1 efficiency = N(MBTS_1 && offline track && mbSpTrack) / N(offline track && mbSpTrack) – systematics from use of control trigger mbSpTrack and from track d0 cut
MBTS_1 rate v. Inst. Lumi for 2 and 4 pairs of colliding bunches in ATLAS - the MBTS rate saturates as it approaches nb times the LHC revolution frequency (nb*fLHC~22 kHz)
rate of colliding bunches - 2 b : 22.5kHz ; 4b : 45 kHz 30

31. e/γ Trigger pT≈3-20 GeV: b/c/tau decays, SUSY
pT≈ GeV: W/Z/top/Higgs
pT>100 GeV: exotics
Level 1: local ET maximum in ΔηxΔφ = 0.2x0.2 with possible isolation cut
Level 2: fast tracking and calorimeter clustering – use shower shape variables plus track-cluster matching
Event Filter: high precision offline algorithms wrapped for online running
L1 EM trigger
pT > 5GeV

32. Discriminate against hadronic showers based on shower shape variables
Use fine granularity of LAr calorimeter
Resolution improved in Event Filter with respect to Level 2

33. Muon Trigger Low PT: J/Y, U and B-physics
High PT: H/Z/W/τ➝μ, SUSY, exotics
Level 1: look for coincidence hits in muon trigger chambers
Resistive Plate Chambers (barrel) and Thin Gap Chambers (endcap)
pT resolved from coincidence hits in look-up table
Level 2: refine Level 1 candidate with precision hits from Muon Drift Tubes (MDT) and combine with inner detector track
Event Filter: use offline algorithms and precision; complementary algorithm does inside-out tracking and muon reconstruction
80% acceptance due to support structures etc.

34. Hadronic Tau Trigger
W/Z ➝ t, SM &MSSM Higgs, SUSY, Exotics
Level 1: start from hadronic cluster – local maximum in ΔηxΔφ = 0.2x0.2 – possible to apply isolation
Level 2: track and calorimeter information are combined – narrow cluster with few matching tracks
Event Filter: 3D cluster reconstruction suppresses noise; offline ID algorithms and calibration used
Typical background rejection factor of ≈5-10 from Level 2+Event Filter
Right: fake rate for loose tau trigger with pT > 12 GeV – aka tau12_loose
MC is Pythia with no LHC-specific tuning

35. Jet Trigger QCD multijet production, top, SUSY, generic BSM searches
Level 1: look for local maximum in ET in calorimeter towers of ΔηxΔφ = 0.4x0.4 to 0.8x0.8
Level 2: simplified cone clustering algorithm (3 iterations max) on calorimeter cells
Event Filter: anti-kT algorithm on calorimeter cells; currently running in transparent mode (no rejection)
Note in preparation

36. Missing ET Trigger SUSY, Higgs
Level 1: ETmiss and ET calculated from all calorimeter towers
Level 2: only muon corrections possible (at present)
Event Filter: re-calculate from calorimeter cells and reconstructed muons
Level 1
5 GeV threshold
Level 1
20 GeV threshold

37. The Trigger Menu Collection of trigger signatures
In LHC GPD’s menus there can be 100’s of algorithm chains – defining which objects, thresholds and algorithms, etc should be used
Selections set to match the current physics priorities and beam conditions within the bandwidth and rates allowed by the TDAQ system
Includes calibration & monitoring chains
Principal mechanisms to adjust the accept rate (and the relative frequency of different triggers)
Adjusting thresholds
Pre-scaling (e.g. only accept every 10th trigger of a particular type) higher rate triggers
Can be used to include a low rate of calibration events

38. Example use of thresholds/prescales at Level-1
L1 trigger items and estimated rates at 10^31 cm−2 s−1 for jets
Jet ET spectrum at 10^31 cm−2 s−1 before (dashed)
and after (solid) pre-scaling at L1

39. Trigger Menu cont’d Basic Menu is defined at the start of a run
Pre-scale factors can be changed during the course of a run
Adjust triggers to match current luminosity
Turn triggers on/off

40. Trigger Commissioning in ATLAS
First Collisions : L1 only
Since June : gradual activation of HLT
Commissioning with cosmics, single-beam 2008 & 2009:
initial timing in of Level-1 signals, ready for first collisions
First Collisions L<2x1029cm-2s-1 : Dec 2009 : 900 GeV; Mar 2010 : 2.36 TeV; April 2010 : 7 TeV
Event selection for first Physics using Level-1
HLT running online in monitoring mode - no HLT rejection (except Min. Bias control trigger)
Allows detailed validation of HLT triggers ready to activate selection when needed
Provides online beam-spot measurement based on Level-2 Tracking
Up to L~2x1027 cm-2s-1 Minimum Bias trigger used to record all collisions
When rate exceeded ~300Hz, Minimum Bias trigger pre-scaled
Level-1 electron/photon, Jet, Tau, Muon and Missing ET triggers un-pre-scaled
7 TeV Collisions 2x1029cm-2s-1 < L < 2x1030cm-2s-1 June/July 2010:
Progressive enabling of HLT
1.5 x 1029 : e & g x1029 : t x1029 : m in end-caps x1030 : Missing ET
Prescale sets pre-generated covering fixed luminosity ranges.
Can be updated during the run to match machine conditions.
Physics menu: being deployed now for running L> 2x1030cm-2s-1:
Shift of emphasis from commissioning to physics

41. Matching problem
Background
Physics channel
Off-line
On-line

42. Matching problem (cont.)
ideally
off-line algorithms select phase space which shrink-wraps the physics channel
trigger algorithms shrink-wrap the off-line selection
in practice, this doesn’t happen
need to match the off-line algorithm selection
For this reason many trigger studies quote trigger efficiency wrt events which pass off-line selection
BUT off-line can change algorithm, re-process and recalibrate at a later stage
So, make sure on-line algorithm selection is well known, controlled and monitored

43. Selection and rejection
as selection criteria are tightened
background rejection improves
BUT event selection efficiency decreases

44. Other issues for the Trigger
Efficiency and Monitoring
In general need high trigger efficiency
Also for many analyses need a well known efficiency
Monitor efficiency by various means
Overlapping triggers
Pre-scaled samples of triggers in tagging mode (pass-through)
Final detector calibration and alignment constants not available immediately - keep as up-to-date as possible and allow for the lower precision in the trigger cuts when defining trigger menus and in subsequent analyses
Code used in trigger needs to be very robust - low memory leaks, low crash rate, fast

45. Other issues for the Trigger – cont’d
Beam conditions and HLT resources will evolve over several years (for both ATLAS and CMS)
In 2010 luminosity low, but also HLT capacity had < 50% of full system
For details of the current ideas on ATLAS Menu evolution see
Gives details of menu since Startup and for 2011
Corresponding information for CMS is at
The expected performance of ATLAS for different physics channels (including the effect of the trigger) is documented in (beware - nearly 2000 pages)

46. Summary
High-level triggers allow complex selection procedures to be applied as the data is taken
Thus allow large samples of rare events to be recorded
The trigger stages - in the ATLAS example
Level 1 uses inclusive signatures (mu’s; em/tau/jet; missing and sum ET)
Level 2 refines Level 1 selection, adds simple topology triggers, vertex reconstruction, etc
Level 3 refines Level 2 adds more refined topology selection
Trigger menus need to be defined, taking into account:
Physics priorities, beam conditions, HLT resources
Include items for monitoring trigger efficiency and calibration
Try to match trigger cuts to off-line selection
Trigger efficiency should be as high as possible and well monitored
Must get it right - events thrown away are lost for ever!
Triggering closely linked to physics analyses – so enjoy!

47. ATLAS works!
Top-pair candidate - e-mu + 2b-tag

48. CMS works!

49. Additional Foils

50. ATLAS HLT Hardware Each rack of HLT (XPU) processors contains
~30 HLT PC’s (PC’s very similar to Tier-0/1 compute nodes)
2 Gigabit Ethernet Switches
a dedicated Local File Server
Final system will contain ~2300 PC’s

51. LFS nodes
UPS for CFS
XPUs
CFS nodes
SDX1|2nd floor|Rows 3 & 2

52. Naming Convention MU 20 I mu 20 i _ passEF
First Level Trigger (LVL1) Signatures in capitals e.g.
LVL1
HLT
type EM e
electron g
photon MU mu
muon HA
tau FJ fj
forward jet JE je
jet energy JT jt
jet TM xe
missing energy
name
threshold
isolated
MU 20 I
HLT in lower case:
name
threshold
isolated
mu 20 i _ passEF
EF in tagging mode
New in :
Threshold is cut value applied
previously was ~95% effic. point.
More details : see :

53. What is a minimum bias event ?
- event accepted with the only requirement being activity in the detector
with minimal pT threshold [100 MeV] (zero bias events have no requirements)
- e.g. Scintillators at L1 + (> 40 SCT S.P. or > 900 Pixel clusters) at L2
- a miminum bias event is most likely to be either:
- a low pT (soft) non-diffractive event
- a soft single-diffractive event
- a soft double diffractive event
(some people do not include the diffractive events in the definition !)
- it is characterised by:
- having no high pT objects : jets; leptons; photons
- being isotropic
- see low pT tracks at all phi in a tracking detector
- see uniform energy deposits in calorimeter as function of rapidity
- these events occur in % of collisions. So if any given crossing
has two interactions and one of them has been triggered due to a high pT
component then the likelihood is that the accompanying event will
be a dull minimum bias event.

54. Example Level-1 Menu for 2x10^33
Level-1 signature
Output Rate (Hz)
EM25i
12000
2EM15i
4000
MU20
800
2MU6
200
J200
3J90
4J65
J60 + XE60
400
TAU25i + XE30
2000
MU10 + EM15i
100
Others (pre-scaled, exclusive, monitor, calibration)
5000
Total
~25000

55. L1 Rates 1031 14.4.0 Multi EM 6400 Multi Object 5500 Single EM
Trigger Group
Rate (Hz)
Multi EM
6400
Multi Object
5500
Single EM
Single Muon
1700
Multi Tau
470
Single Tau
150
Jets 80
Multi Muon 70 XE 50
TOTAL
20000
Removing overlaps between single+multi EM gives 18 kHz
Total estimated L1 rate with all overlaps removed is ~ 12 kHz 55

56. L2 Rates 1031 14.4.0 Electrons 310 Muons 210* Taus+X 180 XE+ 82
Trigger Group
Rate (Hz)
Electrons
310
Muons
210*
Taus+X
180
XE+ 82
Photons 46
B Phys 43
Jets 22
TOTAL
900
X=anything;
+ includesJE,
TE, anything
with MET
except taus;
Bphys includes
Bjet
* Manually prescaled off pass-through triggers mu4_tile, mu4_mu6
Total estimated L2rate with all overlaps removed is 840 Hz 56

57. EF Rates 1031 14.4.0 Muons 80 Electrons 67 Tau+X 56 B Phys 37 Jets 25
Trigger Group
Rate (Hz)
Muons 80
Electrons 67
Tau+X 56
B Phys 37
Jets 25
Photons 18
XE+ 13
Misc
TOTAL
310
91 Hz total is in prescaled triggers;
51 Hz of prescaled triggers is unique rate
Total estimated EF Rate with overlaps removed is 250 Hz 57

58. L1 Rates
Trigger Group
Rate (Hz)
Multi Object
30000
Single Muon
17000
Multi EM
11000
Single EM
8100
Multi Tau
4300
Single Tau
870
Multi Muon
690
Jets
300 XE
TOTAL
73000
Total estimated L1 rate with all overlaps removed is 46 kHz 58

59. L2 Rates
Trigger Group
Rate (Hz)
Tau+X
820
XE+
590
Electrons
390
Muons
280
3 Objects
270
Photons
120
B Phys
110
Jets 33
Misc 28
TOTAL
2600
Total estimated L2 with all overlaps removed is (too high!) 59

60. EF Rates
Trigger Group
Rate (Hz)
Tau+X
187
Electrons 77
Muons 46
Photons
BPhys 45
3 Objects
XE+ 42
Jets 11
Misc
TOTAL
510
Total estimated EF rates with all overlaps removed is 390 Hz
(Fixing L2 will likely come close to fixing EF as well) 60

61. End of pp trigger operations in 2010
Run record peak luminosity 2.1x1032cm-2s-1
Trigger group
Trigger chain
Rate
[Hz]
Single-muon
EF_mu13_tight 24
Di-muon
EF_2mu6 28
Single-electron
EF_e15_medium 38
Di-electron
EF_2e10_loose
2.4
Single-photon
EF_g40_loose 9
Di-photon
EF_2g15_loose
2.1
Single jet
EF_L1J95_NoAlg 11
MET
EF_xe40_noMu 6
Single-tau
EF_tau84_loose
6.8
Di-tau
EF_2tau29_loose1
2.6
L1 output 35kHz, L2 output 5kHz, EF output 400Hz
Trigger evolution in 2010
For a given threshold tighten selection
Loose->medium->tight
Non-isolation->isolation
Go higher in pT
Trigger Report 61
Due to lack of time no physics data collected with 50ns BS

62. Example rates for different objects