1. LEB 2002 @ Colmar CMS DAQ overview Data to Surface (D2S) Requirements Topology Hardware implementation Measurements and simulations Readout kit for FE designers DAQ staging policy Conclusion CMS Data to Surface transportation architecture attila.racz@cern.ch on behalf of the CMS DAQ group

2. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 2 1/8th DAQ Readout Builder: Lv-1 Max. trigger rate 12.5 kHz RU Builder (64x64).125 Terabit/s FB fragment ≈16 kB RU-BU systems 64 Event filter power ≈.7 TeraFlop Event flow control ≈10 5 Mssg/s Local mass storage 10 TByte Data to surface: Average event size ≈1 Mbyte FED S-link64 ports658 DAQ links (5 Gb/s) 512 FED fragment ≈2 kB FED builders (8x8) 64 TTC Timing, Trigger and Control TPDTrigger Primitive Data aTTS asynchronous Trigger Throttle System D2S Data to Surface FRLFrontend Readout Link RU Readout Unit BU Builder Unit FU Filter Unit FFNFilter Farm Network EVM Event Manager RCN Readout Control Network BCN Builder Control Network DCNDetector Control network DSN DAQ Service Network 8-fold DAQ architecture DAQ staging

3. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 3 D2S requirements Reads the data from the FE data sources (~630 sources) Merging of 2 “small” sources (nominal size is 2 KBytes) Leveling effect on un-balanced data sources Staging capability according to the 8-fold DAQ architecture

4. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 4 Data to Surface (D2S) Short link(20m) Long link(200m) Underground area Surface area FED/DCC FRL FED Builder RU FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL

5. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 5 Data sources Detector Number of data sources Number of DAQ links Event size per link KB Fluctuations RMS Nominal event size KB (pp run) Number of trigger partitions Pixel 4036 (merg)1.2 – 2.030%722 Tracker 440272 (merg)0.4 – 1.50.27 KB3004 Preshower 47 2.3?1102 ECAL 52 (DCC)5221 KB~1004 HCAL 24 (DCC)241.7?486 Muons CSC 9 (DCC 4-1)92Huge…16*2 Muons RPC 5 (DCC)5.3?1.54 Muons DT 551.6?82 Glob. trig 442none83 DT track f. 442none82 Total 630458671.531 * Most of the time, the detector is empty: one muon produces 5.5KB These numbers are currently revised/updated and they will change ! Event sizes are strongly dependent on simulated occupancies…

6. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 6 D2S in vivo… From one end of the DAQ building to the end of the control room,the cable path is 187 m long and there are ~90 m of verticality

7. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 7 Data sources location (1)

8. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 8 Data sources location (2)

9. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 9 S-Llink64 to FRL Merger FED Builder Readout Unit Input D2S baseline implementation Myrinet LVDS + Custom FRL+Myrinet

10. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 10 Short distance link LVDS signals, S-link64 compliant –869 MB/sec @ 10m –480 MB/sec @ 17m

11. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 11 Altera ACEX 60MHz FED Altera ACEX 60MHz FIFO 32kB FIFO 32kB FRL S-Link64 -64-bit up to 100MHz -Backpressure -auto-test (Slink spec.) -12 reserved pins LVDS transfer -64-bit @ 60MHz (480MB/s) -cable length 15m max. ALTERA Stratrix -no Slink pinout -64-bit PCI clock -able to read one port at the time FRL merging feature

12. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 12 FRL baseline implementation PCI Myrinet board 160 mm30 mm Standard size

13. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 13 DAQ Opticable: a candidate 200m long 72 fibers, 50/125um multimode, LC-LC connector 32 DAQ links (4 spares) per cable, 16 cables for CMS DAQ Polyester Tape Aramid Yarn PVC Coated Aramid C.S.M. Aramid Yarn Tight Buffer Optical Fiber PVC Sub-unit Jacket * Sub- unit HFFR Jacket * Rip Cord.

14. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 14 FED builder performances (Myrinet) rms=0 (fixed size): puts itself in Barrel Shifter variable size and no traffic shaping: ~55% usage RU 1 8x8 crossbar RU 2 3 4

15. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 15 0 10 20 30 40 50 60 70 80 90 100 00.250.50.7511.25 RMS/average Maximum Utilisation (%) balanced (2.0) unbalance ratio=2 (1.33 2.66) unbalance ratio=3 (1.0 3.0) FED Builder - Unbalanced Inputs Maximum usage: roughly 50% No significant loss due to un-balanced sources

16. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 16 Myrinet Lanai9 (1 link @ 2.5 Gbit/sec), jumbo event (200k) Time Event ID Time Event ID no MTU with MTU “Jumbo” fragment effect…(1)

17. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 17 Myrinet Lanai10 (2 links @ 2.5 Gbit/sec), jumbo event (200k) Time Event ID Time Event ID no MTU with MTU “Jumbo” fragment effect…(2)

18. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 18 Readout Kit for FED designers PCI generic III (see D.Gigi’s talk), Tx-card (PMC form factor) + Rx- card, cable (15m max.) Software libraries and drivers under Linux Supported by X-DAQ (Common DAQ software framework) Documented !!! WEB download: http: //cern.ch/cano/fedkit/http: //cern.ch/cano/fedkit/ Provides an easy transition from local readout to central DAQ readout Price: ~ 2500 FS

19. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 19 DAQ staging policy Staging FED/DCC FRL FED Builder FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL 1 RU builder 64x64, 1MByte @ 12.5KHz RU 2 RU builders 64x64, 1MByte @ 25KHz3 RU builders 64x64, 1MByte @ 37.5KHz RU 8 RU builders 64x64, 1MByte @ 100KHz RU

20. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 20 Conclusion D2S structure and implementation well defined and matches the requirements Minimal custom electronics developments –Can profit from the technological improvements Unexpected situations can be handled simply by adding devices –Event size increase –Number of sources increase D2S pre-building allows a progressive DAQ deployment

21. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 21 FED/DCC DAQ port Specific Detector Electronic S-LINK64 port Link FPGA FED/DCC Detector links Storage area S-LINK64 port is the border between FE and DAQ Setup Control Messages - out-of-sync - failure Monitoring Local readout VME Host Interface Fast signals: (FMM) - busy/ready - overflow warning - out-of-synch

22. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 22 S-LINK64 features Based on S-LINK Specifies 2 connectors but not the physical link in-between S-LINK is 32 bits @ 40 MHz max, flow-control, K/D words To match CMS needs: –64 bit data path, clock speed up to 100 MHz, but link speed is ~500 MB/s –no other changes to keep compatibility with S-LINK products S-LINK64 specs. available at: –http://cmsdoc.cern.ch/cms/TRIDAS/horizontal/

23. 9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 23 16 15 8 7 12 1132 314 3060 596356 55 60 5963 20 194 3 0 032 31 Evt_stat(8)xx$$ BOE_1 KLV1_id(24)BX_id(12) CRC (16) D Sub-detector payload K D FED common data format 32 3160 5963 Evt_ty 56 55 EOE Evt_lgth(24)xxxx Source_id(10+2)FOV xxxx $$Hx 1 BOE_2 K $$ The FED is generating the above encapsulation.