LEB Colmar CMS DAQ overview Data to Surface (D2S) Requirements Topology Hardware implementation Measurements and simulations Readout kit for FE designers DAQ staging policy Conclusion CMS Data to Surface transportation architecture on behalf of the CMS DAQ group

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 2 1/8th DAQ Readout Builder: Lv-1 Max. trigger rate 12.5 kHz RU Builder (64x64).125 Terabit/s FB fragment ≈16 kB RU-BU systems 64 Event filter power ≈.7 TeraFlop Event flow control ≈10 5 Mssg/s Local mass storage 10 TByte Data to surface: Average event size ≈1 Mbyte FED S-link64 ports658 DAQ links (5 Gb/s) 512 FED fragment ≈2 kB FED builders (8x8) 64 TTC Timing, Trigger and Control TPDTrigger Primitive Data aTTS asynchronous Trigger Throttle System D2S Data to Surface FRLFrontend Readout Link RU Readout Unit BU Builder Unit FU Filter Unit FFNFilter Farm Network EVM Event Manager RCN Readout Control Network BCN Builder Control Network DCNDetector Control network DSN DAQ Service Network 8-fold DAQ architecture DAQ staging

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 3 D2S requirements Reads the data from the FE data sources (~630 sources) Merging of 2 “small” sources (nominal size is 2 KBytes) Leveling effect on un-balanced data sources Staging capability according to the 8-fold DAQ architecture

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 4 Data to Surface (D2S) Short link(20m) Long link(200m) Underground area Surface area FED/DCC FRL FED Builder RU FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 5 Data sources Detector Number of data sources Number of DAQ links Event size per link KB Fluctuations RMS Nominal event size KB (pp run) Number of trigger partitions Pixel 4036 (merg)1.2 – 2.030%722 Tracker (merg)0.4 – KB3004 Preshower ?1102 ECAL 52 (DCC)5221 KB~1004 HCAL 24 (DCC)241.7?486 Muons CSC 9 (DCC 4-1)92Huge…16*2 Muons RPC 5 (DCC)5.3?1.54 Muons DT 551.6?82 Glob. trig 442none83 DT track f. 442none82 Total * Most of the time, the detector is empty: one muon produces 5.5KB These numbers are currently revised/updated and they will change ! Event sizes are strongly dependent on simulated occupancies…

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 6 D2S in vivo… From one end of the DAQ building to the end of the control room,the cable path is 187 m long and there are ~90 m of verticality

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 7 Data sources location (1)

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 8 Data sources location (2)

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 9 S-Llink64 to FRL Merger FED Builder Readout Unit Input D2S baseline implementation Myrinet LVDS + Custom FRL+Myrinet

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 10 Short distance link LVDS signals, S-link64 compliant –869 10m –480 17m

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 11 Altera ACEX 60MHz FED Altera ACEX 60MHz FIFO 32kB FIFO 32kB FRL S-Link bit up to 100MHz -Backpressure -auto-test (Slink spec.) -12 reserved pins LVDS transfer 60MHz (480MB/s) -cable length 15m max. ALTERA Stratrix -no Slink pinout -64-bit PCI clock -able to read one port at the time FRL merging feature

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 12 FRL baseline implementation PCI Myrinet board 160 mm30 mm Standard size

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 13 DAQ Opticable: a candidate 200m long 72 fibers, 50/125um multimode, LC-LC connector 32 DAQ links (4 spares) per cable, 16 cables for CMS DAQ Polyester Tape Aramid Yarn PVC Coated Aramid C.S.M. Aramid Yarn Tight Buffer Optical Fiber PVC Sub-unit Jacket * Sub- unit HFFR Jacket * Rip Cord.

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 14 FED builder performances (Myrinet) rms=0 (fixed size): puts itself in Barrel Shifter variable size and no traffic shaping: ~55% usage RU 1 8x8 crossbar RU 2 3 4

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar RMS/average Maximum Utilisation (%) balanced (2.0) unbalance ratio=2 ( ) unbalance ratio=3 ( ) FED Builder - Unbalanced Inputs Maximum usage: roughly 50% No significant loss due to un-balanced sources

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 16 Myrinet Lanai9 (1 2.5 Gbit/sec), jumbo event (200k) Time Event ID Time Event ID no MTU with MTU “Jumbo” fragment effect…(1)

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 17 Myrinet Lanai10 (2 2.5 Gbit/sec), jumbo event (200k) Time Event ID Time Event ID no MTU with MTU “Jumbo” fragment effect…(2)

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 18 Readout Kit for FED designers PCI generic III (see D.Gigi’s talk), Tx-card (PMC form factor) + Rx- card, cable (15m max.) Software libraries and drivers under Linux Supported by X-DAQ (Common DAQ software framework) Documented !!! WEB download: http: //cern.ch/cano/fedkit/http: //cern.ch/cano/fedkit/ Provides an easy transition from local readout to central DAQ readout Price: ~ 2500 FS

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 19 DAQ staging policy Staging FED/DCC FRL FED Builder FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL FED/DCC FRL 1 RU builder 64x64, 12.5KHz RU 2 RU builders 64x64, 25KHz3 RU builders 64x64, 37.5KHz RU 8 RU builders 64x64, 100KHz RU

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 20 Conclusion D2S structure and implementation well defined and matches the requirements Minimal custom electronics developments –Can profit from the technological improvements Unexpected situations can be handled simply by adding devices –Event size increase –Number of sources increase D2S pre-building allows a progressive DAQ deployment

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 21 FED/DCC DAQ port Specific Detector Electronic S-LINK64 port Link FPGA FED/DCC Detector links Storage area S-LINK64 port is the border between FE and DAQ Setup Control Messages - out-of-sync - failure Monitoring Local readout VME Host Interface Fast signals: (FMM) - busy/ready - overflow warning - out-of-synch

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar 22 S-LINK64 features Based on S-LINK Specifies 2 connectors but not the physical link in-between S-LINK is MHz max, flow-control, K/D words To match CMS needs: –64 bit data path, clock speed up to 100 MHz, but link speed is ~500 MB/s –no other changes to keep compatibility with S-LINK products S-LINK64 specs. available at: –

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB Colmar Evt_stat(8)xx$$ BOE_1 KLV1_id(24)BX_id(12) CRC (16) D Sub-detector payload K D FED common data format Evt_ty EOE Evt_lgth(24)xxxx Source_id(10+2)FOV xxxx $$Hx 1 BOE_2 K $$ The FED is generating the above encapsulation.