Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine Bosman, Andrea Negri, Serge Sushkov and Sarah Wheeler.

13 October 2005TDAQ workshop - Mainz2 Physics streams 40 MHz x 1.5 MByte Level 1 Level 2 75 kHz x 1.5 MByte Definition and scope. Calibration issues in the EF. –At the moment only EF output. Identify calibration types. –Size, rate, contents. Requirements for the EF. –Data flow. –Processing issues. Implementation scenarios. Design and modifications. –Processing. –Memory management. –Networking/Timing. Plan of work. Processing time: 1 Sec/Evt Number of nodes: 1500 Output:200 Hz 320 MB/s Calibration streams Level 1 Level 2 Other people Event Filter ? 2 kHz x 1.5 MByte

13 October 2005TDAQ workshop - Mainz3 Use cases (identification of calibration streams). Based on the Hawkings/Gianotti document. –All listed calibration types are identified at HLT level (after PESA). Known calibration types in the HLT. –Various duplicates of physics streams: e.g. inclusive high p T electrons and muons (tracking), Z to di-lepton (energy), minimum bias (background). Total: 35 MB/s (10% of physics data). –Liquid Argon Calorimeter. Pulse shape analysis, timing calibration and tuning of filter coefficients. High p T electron sample Electro Magnetic data only. ROI only. Raw data, 5 consecutive samples (i.e. special event type). –Calorimeters and TRT. Hadronic response studies, comparison to test beam data. TRT: e/  separation. High p T isolated hadrons. All subdetectors. ROI only. RAW data.

13 October 2005TDAQ workshop - Mainz4 Use cases – continued… (identification of calibration streams). –MDT small chambers. Hourly realignment. Small muon sample. MDT information only. Overlap regions only. Reprocessing of raw data. –Inner Detector subdetectors (Pixel, SCT and TRT). ROD monitoring (TRT only) and alignment. Generic high p T events. All subdetectors. ROI only. Post-processing of track fit information on HLT level. Other foreseen calibration types. –Liquid Argon Calorimeter might need Z  ee calibration at HLT level. –High statistics (1 kHz) ROI muon sample, containing MDT, CSC and RPC/TGC information. –Your favorite missing calibration stream…

Special events EF Node n Transport time: 19 ms EF Node n Data flow characterization. SFO 320 MB/s Physics Full calibration events SFO 32 MB/s High p T SFO 1.6 MB/s Z di-lepton SFO 1.6 MB/s Min. bias EF processing: 1 second/event SFO 5 kB/s OverlapMu 2.5 MB/s LAr SFO 2 MB/s IsoHad SFO 4 MB/s GenPT Stripping/Collecting Partial calibration events Transport times/rates: LAr OverlapMu IsoHad GenPT 0.5 ms 0.05 ms 5 ms 0.5 ms 50 Hz 5 Hz 100 Hz Output 1 MB/s LVL2Cal Subdetector Fragment 1 ROI info Transport time: LVL2Cal 0.01 ms Partial event SFI? Transport times: 19 ms Subdetector Fragment 1 Subdetector Fragment N Lvl 1/2 info SFI Additional Processing 1 kHz! Sorting EF processing: 1 second/event

13 October 2005TDAQ workshop - Mainz6 EF processing issues. Definitions: –Sorting for calibration: CalID. –Stripping/Collecting for calibration: CalCollect. –Detector calibration: CalDetect or Calibration. Full event streams. –Very similar to physics streams. –Output to multiple SFOs/streams. –Sorting of events for calibration (CalID). Partial event streams. –Processing similar to physics stream. –Stripping and collecting (CalCollect). –Handling of different output event size. –Sometimes requires additional processing (CalDetect/Calibration). Special event streams. –Processing times completely different from physics stream. –Handling of different input and output event sizes. Central issue: Robustness of the EFD Main output stream SFO Diagnostic SFO Node n EFD PT #1 PTIOPTIO SFI Input ExtPTs Output Trash ExtPTs PT cal PTIOPTIO Calibration Stream Output SFO PT #2 PTIOPTIO Event Result SharedHeap

13 October 2005TDAQ workshop - Mainz7 Main output stream SFO EF processing issues. Node n EFD SFI Input ExtPTs Output Event SharedHeap Dataflow application Definitions: –Sorting for calibration: CalID. –Stripping/Collecting for calibration: CalCollect. –Detector calibration: CalDetect or Calibration. Full event streams. –Very similar to physics streams. –Output to multiple SFOs/streams. –Sorting of events for calibration (CalID). Partial event streams. –Processing similar to physics stream. –Stripping and collecting (CalCollect). –Handling of different output event size. –Sometimes requires additional processing (CalDetect/Calibration). Special event streams. –Processing times completely different from physics stream. –Handling of different input and output event sizes. Central issue: Robustness of the EFD

13 October 2005TDAQ workshop - Mainz8 Calibration stream scenarios (1). Additional functionality: CalID algorithm. Parallel output streams. Node n EFD SFI Input ExtPTs Trash Main output stream SFO Output Event Result Physics only events Full calibration events. e.g. Z di-lepton PT #1 PTIOPTIO PESA Main output stream SFO Node n EFD SFI Input ExtPTs Output Trash Event Result PT #1 PESA PTIOPTIO Calibration Stream SFO Output CalID

Calibration Stream 1 Main output stream SFO Node n EFD SFI Input ExtPTs Output Event Result PT #1 PESA CalID PTIOPTIO Calibration stream scenarios (2). Additional functionality: PT for calibration. –Information handling –Stripping/collecting. Memory management. Partial calibration events. e.g. GenPT Node n EFD SFI? Input ExtPTs Event Calibration Stream EF output Output PT #1 CalID PTIOPTIO Calibration Stream 2 Output SFO ExtPTs PT cal PTIOPTIO Stripping Collecting CalResult Special streams. e.g. LVL2Cal Calibration Networking/Timing issues. Sorting

13 October 2005TDAQ workshop - Mainz10 Design and modifications (1). CalID algorithm. –Lightweight algorithm. –Runs after PESA in physics PT – Stability issues. Athena configuration: multiple top algorithms. –Workload: Low – implementation only, thorough testing required. –Impact: High – required for (almost) all calibration streams. –Coordination: Sorting – New PT answers should be discussed. Parallel output streams. –Slight modification of existing algorithm. –Runs in EFD, probably required for PESA as well. –Workload: Low – Modification of standard EFD task. –Impact: High – Required for most (calibration) streams.

13 October 2005TDAQ workshop - Mainz11 Design and modifications (2). PT for calibration. 1.Create stripping/collection algorithm. Requires new eformat (see next slide). Requires modifications in output task. 2.Allow multiple PTs to run consecutively (works already). 3.Transfer information between these PTs. –Should be possible with new eformat for EFResult / CalResult. 4.It might be interesting to transfer “intermediate results”. –Would avoid to Run calibration algorithms in the same PT as PESA. Reanalyze complete event in second PT. –Since EF is a dataflow application, this should be accomplished by writing an extra “Intermediate Result” object in the Shared Heap. This requires ByteStream conversion for complex classes. –Workload: Medium – With exception of item 4 (no use cases yet). –Impact: High – Maybe with exception of item 4.

13 October 2005TDAQ workshop - Mainz12 Memory management and information handling. New eformat. Node n EFD SFI Input ExtPTs SharedHeap Event EFResult PT #1 PESA CalID PTIOPTIO CalResult SFO Output PT cal PTIOPTIO Stripping Collecting x x Integrate with: – Virtual event. – SharedHeap. – Event handling in the EF. – Event modification in EF, i.e. stripping/collecting. Event fragments EFResult Stripping CalResult Virtual event

13 October 2005TDAQ workshop - Mainz13 Open issues Many new developments on a very tight schedule. Memory and performance. –Management: move from open/close backpressure mechanism (barrier) to analog (Nano sleeps). –Timing: revise SFI – EFD – SFO protocol. Coordination… –Investigate common/similar design issues with monitoring –128 bit header word. First discussion yesterday. 32 bits to register appropriate output streams. Investigate usage of this header. Streaming only, since EFResult fragment contains much more detail and is “just around the corner”. –PT answer to EFD. Composite structure. –EFD – SFO sorting, i.e. how is an output stream defined? Load balancing between calibration and physics. Distribution of calibration constants to EF software. –Communication between Athena algorithms and configuration and calibration databases. –Could this be done via e.g. the information service?

13 October 2005TDAQ workshop - Mainz14 Plan of work. Short time scale. –Run calibration algorithm in PT. –Implement parallel output streams. Medium time scale. –Implement CalID algorithm and sorting. –Eliminate dead time in input and output tasks. Medium – Long timescale. –Change memory management. –Implement new eformat. –Implement stripping/collecting. Long timescale. –Transfer of intermediate results between first and second PT

13 October 2005TDAQ workshop - Mainz15 Conclusions. Our understanding of calibration streams in the Event Filter improved a lot. We think we have a realistic overview of the workload involved in modifications of the Event Filter. Implementation has started and this will lead to even better understanding of the topic (and of the work involved). There are still some (many) open issues. Coordination is important, especially because of the tight schedule. More information:

13 October 2005TDAQ workshop - Mainz16 Conclusions. Our understanding of calibration streams in the Event Filter improved a lot. We think we have a realistic overview of the workload involved in modifications of the Event Filter. Implementation has started and this will lead to even better understanding of the topic (and of the work involved). There are still some (many) open issues. Coordination is important, especially because of the tight schedule. More information:

13 October 2005TDAQ workshop - Mainz17 Appendix

13 October 2005TDAQ workshop - Mainz18 A distributed trigger for calibration? Appears to fit with solutions shown in LVL2Mu presentations. Ultralight project (Manuela Cirilli). LVL2Mu calibration stream (Speranza Falciano). Etc. (Enrico Pasqualucci, Alessandro de Salvo). Additional functionality: HLT output Node n EFD SFI? Input ExtPTs Event PT #1 Event Distributor PTIOPTIO Output Moore’s law for networking Gary Stix, Scientific American, January 2001 The Event Filter is CPU dominated. You would like it to be bandwidth dominated… Calibration Stream HLT output

13 October 2005TDAQ workshop - Mainz19 ByteStream conversion Write converters? –No, a lot of work. –Robustness issues. Generic ByteStream conversion? –No support for complicated classes (e.g. multiple inheritance, polymorphism). Something else? Not on the priority list. Under discussion…

13 October 2005TDAQ workshop - Mainz20 Memory management and networking/timing Node n EFD SFI Input ExtPTs PT cal PTIOPTIO SFO SharedHeap Event EFResult Event EFResult Event EFResult CalResult Intermediate Results PT #1 PESA CalID PTIOPTIO Virtual event Output Stripping Collecting 19 milliseconds transport time 0.25 ms dead time 1+ second processing time Barrier

13 October 2005TDAQ workshop - Mainz21 Memory management and networking/timing Node n EFD SFI Input ExtPTs PT cal PTIOPTIO SFO SharedHeap Event EFResult CalResult Event EFResult CalResult Event EFResult CalResult Intermediate Results Intermediate Results Intermediate Results PT #1 PESA CalID PTIOPTIO Virtual event Output Barrier 0.01 ms transport time 25 ms dead time 0 sec. processing time Eliminate dead time. –Redesign of SFI – EFD – SFO communication protocol. –Coordination with networking group. Barrier is insufficient. –Oscillations. –Other memory requests. Stripping Collecting

13 October 2005TDAQ workshop - Mainz22 Memory management and networking/timing Node n EFD SFI Input ExtPTs PT cal PTIOPTIO SFO SharedHeap Event EFResult CalResult Event EFResult CalResult Event EFResult CalResult Intermediate Results Intermediate Results Intermediate Results PT #1 PESA CalID PTIOPTIO Virtual event Output Barrier 0.01 ms transport time 25 ms dead time 0 sec. processing time Eliminate dead time. Barrier is insufficient. –Oscillations. –Other memory requests. Stripping Collecting

13 October 2005TDAQ workshop - Mainz23 Memory management and networking/timing Node n EFD SFI Input ExtPTs PT cal PTIOPTIO SFO SharedHeap Event EFResult CalResult Event EFResult CalResult Event EFResult CalResult Intermediate Results Intermediate Results Intermediate Results PT #1 PESA CalID PTIOPTIO Virtual event Output Eliminate dead time. Barrier is insufficient. –Oscillations. –Other memory requests. Nano sleeps 0.01 ms transport time 25 ms dead time Nano sleeps Stripping Collecting 0 sec. processing time Solution: Nano sleeps, However… – multiple control loops. – Risk of oscillations. – Additional complexity. Workload: High – Especially testing. Impact: ???