1. Michigan State University 11/26/2015 1 Overview of Level 2 James T. Linnemann Michigan State University Level 2 Review Feb 6, 1999

2. Michigan State University 11/26/2015 2 Requirements l 10 KHz input rate 100  sec decision nominal time budget l Reject 90% at acceptable efficiency read out at 1KHz l Deadtime < 5% 16 buffers for events awaiting decision l Flexibility in trigger configuration

3. Michigan State University 11/26/2015 3 L2 Trigger L2: Combines objects into e, , j CAL FPS/CPS CFT SC MDT PDT PS CAL CFT/CPS Muon L1: E T towers, tracks consistent with e, , j Muon Global L2 Track PS Cal e / j / E t (w/o STT)

4. Michigan State University 11/26/2015 4 Architecture l 2 to 3 stage stochastic pipeline 100  sec/stage Preprocessors for individual detectors Global processor to combine detectors –128 trigger conditions (1 to 1 with L1) –each programmable l series of conditions (e, j,  ) and cuts (E T > 20) l 16 buffers in Front end (real events) 16 buffers in front of preprocessors, global –Busy raised by Front Ends l Hardware frame drives readout

5. Michigan State University 11/26/2015 5 Queuing Simulations: Effective Time Budget l 50-75  sec/stage: tails of processing time l RESQ + standalone to check simple cases l Preprocessors: not event synchronous avoid worst of n distribution l Need the buffers in front of each element l Avoid long tails in processing time l Farms feasible only if surrender event order our front ends required order preservation

6. Michigan State University 11/26/2015 6 L3 WorkerWorker AdminAdmin MBTMBT Dec Alpha (Unix) VME Standard Crate JTL, MSU 12/18/97 128 TCC MPMMPM VBDVBD Inputs MBus SCL Outputs to Global (preprocessors only) L2 Answer L2 HWFW (Global only) 7 VME slots minimum up to 5 workers per crate short (non-CDF) MBus

7. Michigan State University 11/26/2015 7 Standard Crate VME Slot Assignments l 1:Bit3 (Crate Controller) no J3 (1 slot) l 2: VBD (2 signals from J3 to Admin) –through hole in blank MBus l 3-6 J3 connector for VTM up to 4 FIC’s, or any non-MBus cards (SLIC/SFO) l 7-21 J3 Magic Bus: 20-21Administrator (all Alphas 2 slots) 19 Pilot MBT (preproc. : 1MBT for 2 Workers) 18 down [Assistant MBT as needed] l [need 1 MBT per 2 Workers for output] 7-8 upup to 5 Workers (or non-MBus cards)

8. Michigan State University 11/26/2015 8 Alphas l Up to 1 GIP Alpha 21164 on VME card small local disk for bootup Enet to Dec Unix Alpha for user.EXE, debugging l Most MBus I/O via MBT card MBus DMA input 80-100 MB/s (Input “for free”) MBus bi-directional programmed I/O 20 MB/s? –preprocessor output to Global –but interprocessor communication w/o MBT l 2 per crate Workerformatting, Output to Global Administratorhousekeeping, L3 R/O

9. Michigan State University 11/26/2015 9 Alphas, continued l VME for L3 readout, monitoring, downloading l 32 bits ECL output scaler gates for monitoring in L1 Scalers available even if alpha crashed to tell states l J2 Inputs miscellaneous communication –e.g. “you have a message from MBT”

10. Michigan State University 11/26/2015 10 MBT Magic Bus Transceiver l VME slave; MBus master and slave Administrator controls card(s) l 7 Cypress Hotlink inputs 16 MB/s each (Gigabit Ethernet UTP)  broadcast to Alphas (Workers & Admin) on MBus normal data Input path l 3 Cypress Outputs 2 Preprocessor outputs to L2 Global 1 Echo of L1 SCL info

11. Michigan State University 11/26/2015 11 MBT, continued l Serial Command Link (SCL) Receiver broadcast L1 to Alphas on MBus –synchronization check –L1 Qualifiers (basic info on handling events) –echo’d on Cypress output for SLIC Queue L2 accept/rej for Administrator MBus reads l Parallel Output (16-128 b) Global uses to send L2 decision to L2 HWFW handy for monitoring/debugging

12. Michigan State University 11/26/2015 12 Other Cards (Not unique to L2) l Bit3 is commercial VME interface multiport for indirect communication with TCC parameter download, monitoring, error logging l VBD is standard DØ VME Readout to L3 tolerable constraints on how Alphas read out Forces interprocessor communication to MBus

13. Michigan State University 11/26/2015 13 Bit3 MPM l Commercial; fiber optic connection l To PCI of a PC; VME master,crate controller l Add Multiport Memory Module l Perform general VME I/O, generate interrupts l Download parameters for run l Run begin/end commands l Collect Monitoring information preferably, already placed in MPM by Administrator Alpha If necessary, can collect from other modules

14. Michigan State University 11/26/2015 14 VBD l Standard DØ card l VME Master to read out to L3 (standard card) l Not interruptible during Readout l Probably 10-20 MB/s effective (more?) l Must read from SAME set of VME addresses every event intent is readout from Worker Alpha move data, or map to actual location some wordcounts may be zero faster if fewer addresses

15. Michigan State University 11/26/2015 15 Standard Crate Uses l Global JUST Standard Crate described so far l Cal: more workers l Standard Crate can also be used with non-Alpha, non-MBus pre-preprocessor Cypress inputs to Worker via MBT –format, massage data for Global handle L2, L3 buffering & I/O, most of monitoring Completely standard data movement software –User code testable once data structure fixed Penalty: extra latency (lose a buffer) –3-stage pipeline as in L2Mu, L2STT

16. Michigan State University 11/26/2015 16 L2 Inputs l Cypress Hot Links 160 Mbit/s UTP well-defined protocol begin, end event special characters compatible with muon (except cable: CIC) l Standard L2 header/trailer defined some header info repeated in trailer –allows more error detection/correction Hardware Longitudinal Parity Check in trailer

17. Michigan State University 11/26/2015 17 L2 Header B0 # objects (NOT IN HEADER)[note 255 max!] B1 Header Length in 4B words (1B) [=3 for default] B2 Object Length in 4B words (1B) [ALL same size!] B3 Header/Trailer Format # ( hi 3 bits) [ONLY changes if new format] Object Format # ( lo 5 bits)[ONLY changes if new format] B4 Data Type # (1B)[unique in all L2 MBT inputs] B5 Bunch # (1B) B6-7 Rotation# (2B)[B6 is LSB of rotation] B8 Algorithm Major Version (1 B)[e.g. 7 from Version 7.1] B9 Algorithm Minor Version (1B)[e.g. 1 from 7.1] or Processor Specific Bits (1B)[esp. if hardware data source] B10 Processor Specific Bits (1B) B11 Status Bits[b7 on means some error] [some standard for L2 Proc]

18. Michigan State University 11/26/2015 18 Standard Status Bits b7, b0 for all; others if L2proc 7error on event (any kind): use at own risk 6no processing attempted (none required) 5object list truncated (any reason) 4 Receiver error on some input physical trailer 3 2 1 more data-type info (processor-specific) other test modes; unbiased-sample data... 0 0 for real data, 1 for MC data

19. Michigan State University 11/26/2015 19 L2 Trailer B0 Bunch # (1B)= B5 of Header B1 Data Type # (1B)= B4 of Header (Swapped even/odd from Header) B2Longitudinal Parity of even Bytes B3 Longitudinal Parity of odd Bytes or--if parity too slow to calculate, Turn # (B6-7 of Header) MBT Out, SLIC, FIC will append physical trailer with 8-bit hardware-generated longitudinal parity Zero padding to 16 B group FOLLOWS trailer, before End of Event

20. Michigan State University 11/26/2015 20 L2 Physical Trailer l FIC, SLIC, MBT Out: add a physical 2B trailer after logical trailer, before End Event –This BREAKS 16B boundary, but handled by MBT B0 8 bit longitudinal parity of received data B1 Status Bits [b7 on if any receive error] –not included longitudinal parity! –b0, b1 are type ID: 0 = FIC, 1 = SLIC, 2 = MBT l MBT inputs place this in B0, B1 of 16B physical trailer adds B14, its own longitudinal parity of everything received B15 its own Error Bits [b7 on if any receive error] reserves 4B for incoming, may give error locations in B4-13 MBT Outs produce 2B physical trailer like FIC

21. Michigan State University 11/26/2015 21 SLIC: Serial Link Input Card l 16 Cypress serial inputs 1-slot VME slave card 4 TI DSP’s, up to 2 GIPs each l more inputs, CPU / slot than Alpha l output via Hotlink to MBT (avoids VBD R/O) l Readout via Worker Alpha via MBT Acts as pre-preprocessor l test registers on all inputs (eg. SCL) l NO MBus! (big simplification)

22. Michigan State University 11/26/2015 22 SFO: SCL Fanout (Really: Cypress Fanout) l Receives L1 SCL information from MBT as Cypress Hotlink l Fans out as Cypress output to 12 SLIC cards event synchronization L1 Qualifiers l purely analog fanout l can be used to fan out any Cypress signal L1HWFW messages to L2 potential use in L2STT?

23. Michigan State University 11/26/2015 23 Standard Crate with SLIC JTL, MSU 12/18/97 L3 WorkerWorker AdminAdmin MBTMBT Dec Alpha (Unix) VME TCC Inputs MBus SCL Outputs to Global 10 VME slots minimum SFOSFO SLICSLIC Inputs MPMMPM VBDVBD

24. Michigan State University 11/26/2015 24 Fiber Input Converter (FIC) l Convert Fiber Input to Cu Cypress 160 Mb/s G-link input 16b data in 20b data frame (24b total) –input thru J3 by standard VTM ( hard G-link engineering done) implement g-link input via VRB card allows passive split for fanout to L3 or STT adds physical trailer with longitudinal parity l Front end to either SLIC or MBT avoids variants of complex card used in L2PS, L2Cal, L2CTT l 4 independent channels per card l VME control, monitoring

25. Michigan State University 11/26/2015 25 FIC: L2CFT from L1 CFT trigger (& L1 Cal) l g-link 1.3Gb/s = 106MB/s 16b=2B data in 24b frame, frames at 53MHz L1CFT: 100B (50 tracks)/fiber to STT in 1  s –standard L2 header –trailer includes 2B longitudinal parity –pad w/ trailing zeros L1Cal: –similar format, fixed-length data –optical split from data for L3 readout

26. Michigan State University 11/26/2015 26 Standard Crate with FIC to MBT JTL, MSU 12/18/97 L3 WorkerWorker AdminAdmin MBTMBT Dec Alpha (Unix) VME TCC MBus SCL Outputs to Global 9 VME slots minimum FICFIC Inputs MPMMPM VBDVBD

27. Michigan State University 11/26/2015 27 Standard Crate with FIC to SLIC JTL, MSU 12/18/97 L3 WorkerWorker AdminAdmin MBTMBT Dec Alpha (Unix) VME TCC Inputs MBus SCL Outputs to Global 11 VME slots minimum SFOSFO SLICSLIC FICFIC Inputs MPMMPM VBDVBD

28. Michigan State University 11/26/2015 28 Trigger Connections Si Trker L2STT (In Design) Cu-AMCC 1.4Gb/s Fi-Glink 1.3Gb/s 20-bit L2G (MBT) Fi-Cypress 160Mb/s L2CTT (FIC/MBT) 12 96 2 6 Undetermined MUON L2  (SLIC/MBT) Cu-Cyp160 Mb/s L1  Cu-AMCC 1.4Gb/s Cu-Cyp 160Mb/s 2 6 L2PS (FIC/MBT) Fi-Glink 1.3Gb/s 20-bit 2 L2CAL (FIC/MBT) Fi-Glink 1.3 Gb/s 20-bit 4 10 FE Broad 288 4 4ax,2st: 6 ~150 ~280 L1  MGR 3 1 3 Cu-Cypress 160 Mb/s CIC ~150 FE Broad Fi-Glink 1. 3Gb/s, 20-bit L1 FPS L1 CFT FEFE Broad L1 CAL

29. Michigan State University 11/26/2015 29 L2 Bandwidth and budget/event @ 10KHz

30. Michigan State University 11/26/2015 30 Loading of Paths Si Trker L2STT (STC/MBT) L2G (MBT) L2CTT (FIC/MBT) 6 96 2 6 MUON L2  (SLIC/MBT) L1  2 6 L2PS (FIC/MBT) 2 L1 CAL L2CAL (FIC/MBT) 4 10 FE Broad 288 4 ~280 L1  MGR 3 1 3 1248/208 (max)600/300 3200/350 240/40 1088/272 (max) 1632/272 (max) FE Broad 4ax,2st: 6 ~150 CIC ~150 5000/100 L1 CFT L1 FPS

31. Michigan State University 11/26/2015 31 % Capacity used

32. Michigan State University 11/26/2015 32 L2 Overview II, and Summary James T. Linnemann Michigan State University Level 2 Review Feb 6, 1999

33. Michigan State University 11/26/2015 33 L2 Maximum Event Sizes (FIFO size choice) l Length = 16B(min) … 4KB (max) X 16 events includes 12B header and 4B trailer source pads to multiples of 16B with zeros after trailer VRB: 32KB or 64KB, but currently no raw data to L2! 5 KHz max (Cypress) is 16B/  s X 200  s =3.2KB –clearly issue of max, not mean! l Actual Max Event FIFO “event” total FIC/CFT/PS272B.5KB 8KB Cal/MBT304B4KB 64KB Mu/SLIC.3 to 3KB.5KB 8KB Global/MBT2.3KB4KB 64KB =255 tracks*8B (255*16B = 4KB = STT?)

34. Michigan State University 11/26/2015 34 SCL INITIALIZE why we avoid it l Needed if event fragments don’t match must clear all buffers EVERYWHERE and restart violent: touches EVERY front end crate l Avoidance: redundancy header to trailer (protect 1-bit errors) try to preserve event format (to find trailer) try to preserve event boundary (else must re-init) –detect missed event boundary (end or begin) –send pads before End Event to reframe if needed

35. Michigan State University 11/26/2015 35 Monitoring (online, data flow) l Every 5 seconds, via TCC/administrators l Some by L1 Scalers, some by VME L1 Scalers available even if alphas crash l buffer occupancy for data flow diagnosis lots of buffers, need to be able to look at them –in all cards owning buffers: FIC, SLIC, MBT, Alpha but DMA: most events into alpha’s buffers l time in state (like L3 in Run I) in all alphas idle, processing, waiting, interrupt... l Global’s pass fraction by bit #; events vs node

36. Michigan State University 11/26/2015 36 L1 Scalers for Alpha States l ECL Gates: sampled every beam crossing l Worker States (5) wait/event, process, wait/admin, interrupt, collecting_status l Admin States (6) wait, reply/worker, manage_L3, interrupt, L2_Acc/Rej, collecting_status multi-workers: wait more complex: –wait/event, wait/worker, wait/L2_Acc/Rej + Processing l Shows time fraction in state thus in state

37. Michigan State University 11/26/2015 37 Alpha Buffer Monitoring (L1 Scalers) l Alpha should be where events stack up l Binned histo: # events in each buffer state l buffer states monitored in Administrator allocated, processing, wait/L2_Acc/Rej, wait/L3 to be allocated to worker n free l Each L2 crate: 22 scalers/admin + 1/worker

38. Michigan State University 11/26/2015 38 VME (“Slow”) Monitoring (Via Bit3, TCC) l Exact event accounting from ALL cards evaluated after marked event (CollectStatus) l All non-Alpha Cards in L2 Crate MBT, SLIC, FIC State and Buffer Occupancy Sampled on board –lower statistics, perhaps 500 Hz l Alphas can monitor distribution of event times Circular buffer of start/stop times (Histogram on host) –1 CPU cycle (2ns) resolution possible same mechanism for any state duration or counts sampled by event (e.g. # tracks)

39. Michigan State University 11/26/2015 39 Monitoring: Event Samples l Histograms of objects found (Examine) few (.2 Hz) Unbiased Sample (No L2 cuts) –mostly fails few more events before L3 cuts (after L2) –passed L2 mostly after passed L3 l Verification: run simulator on these samples check data arrives intact bit by bit compare with online L2 results detects hardware, history bugs (nearly only way)

40. Michigan State University 11/26/2015 40 Test Stand at FNAL l 4 crates: Global simulator (Admin + Worker) 2 preprocessor simulators (A+2W, A+W+Slic) 1 data source (2alphas, MBT’s; own MBus) l Incomplete system-- no L1, L2 not enough parts for full code of any/all crates –except maybe full playback for Global –could reconfigure if need be--painful! l Copy of some real-time inputs? (grounding!?)

41. Michigan State University 11/26/2015 41 Test Stand: What can it do? l (Pre-)Commissioning/debugging alpha-alpha and alpha-MBT issues l Timing, verification of download run in real environment; count clock cycles how good is offline simulator? l Playback drop data into memory testing pre-release after running in simulator l Debugging event dump and restart (else debug = deadtime) hard to write event dump/reload!?

42. Michigan State University 11/26/2015 42 Zvtx? l Zvtx in 6cm bins from L0? actual resolution varies with luminosity IF felt to be worthwhile at L2 resolution –better to know Z better than Z=0 –or avoid making mistakes and possible L dependence studies in progress l L2STT also considering mechanisms candidate vertex Z’s, then algorithm to report one better intrinsic accuracy than L0 – different luminosity dependence than L0

43. Michigan State University 11/26/2015 43 How does L2STT fit in? l Well defined protocol allows it feed into a preprocessor via MBT like SLIC’s do in muon crate l Send data to L2CTT crate pt-ordered list and impact-ordered list to Global –just pt-ordered from L1CFT, lower resolution –extra input already reserved add 1 MBT and alpha to L2CTT crate –allows simultaneous input of L1CFT and L2STT –can build two kinds of lists in parallel –modest cost; $8K (or use spares / cannibalize test stand) –can run in parallel until shaken down

44. Michigan State University 11/26/2015 44 L2 STT impact on L2 performance l Heavier loading on new CTT inputs 16 B per track? Duplicates? l Heavier loading of pt ordered list? More Bytes/track? But can reject tracks, too! l New output: impact parameter ordered list already included in bandwidth estimates l More work for Global? Yes, but controlled by scripts Must limit # tracks used in matches! –STT can only give higher quality

45. Michigan State University 11/26/2015 45 Budget l Detailed Cost Estimate Exists l Latest update is down about 80K$ (SLIC) nearly back to original estimate l L2STT loaner crates, L2CTT upgrades  included in above estimates l engineering costs not over till it’s over l how many extra alphas as insurance? may be hard to do a 2nd run

46. Michigan State University 11/26/2015 46 How Many Alphas? l Roughly X 2 design safety factor (10KHz) more alphas are only lifeboat too slow –but gain is not linear, maybe square root cannibalize test stand (IF it becomes unimportant!) l Other uses for more alphas: Shadow nodes (online test at high statistics) –where? Real crate or test stand? potential use in STT? l Production Alpha order in next few months have requested 2 X for some obsolescent parts

47. Michigan State University 11/26/2015 47 Schedule l Detailed scheduled exists l SLIC now on critical path l Rule: L2STT can’t compromise schedule l Prototypes due March-May 1999 l Production May-Dec 1999

48. Michigan State University 11/26/2015 48 Issues l L2STT design decisions (and money…) l Prototype to production to installation l Transition to software simulation needed (decisions, studies, development) Low level software (“drivers”) finish download path, L3 output path l Manpower crunch coming Monitoring, verification, releases just starting –infrastructure, definitions needed soon, then people... who writes global algorithms? (MSU, probably?) studies of trigger scripts, global algorithms

49. Michigan State University 11/26/2015 49 Conclusions l Solid, modular design for L2 trigger connectivity understood; prototypes in progress clear method for L2STT to integrate l Time budgets understood from simulation l Hardware supports appropriate algorithms l Have the manpower for the hardware l Have excellent core group for the software l Request TDR approval: go-ahead for production