1. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 20041 Performance of the CMS Silicon Tracker Front-End Driver 10th Workshop on Electronics for LHC Experiments and Future Experiments R. Bainbridge, E. Corrin, C.Foudas, J. Fulcher, G. Hall, G. Iles, J. Leaver, M. Noy, M. Raymond, O. Zorba Imperial College J.A.Coughlan, S.A. Baird, I. Church, C.P.Day, E.J.Freeman, W.J.F.Gannon, R.N.J. Halsall, M. Pearson, G. Rogers, J. Salisbury, S.Taghavirad, I.R.Tomalin CCLRC Rutherford Appleton Laboratory I. Reid Brunel University Presented by Greg Iles: gm.iles@imperial.ac.uk

2. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 20042 Microstrip Tracker readout chain Off Detector (counting room) –Optical links transmit equivalent to 1.75 TB/s @ 100kHz trigger rate –440 Front End Drivers (FEDs) On Detector: –9M silicon strips –73k APV25 readout chips –Analogue readout via 43k optical readout links APV MUX 2:1 APV readout chip 128:1 PLL DCU APV readout chip 128:1 Front End Module Detector FED ADC x12 FPGA ADC x12 FPGA x8 FPGA RAM Transition card S-link card DAQ FMM Throttle signals AOH 9U VME back plane x96

3. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 20043 Front End Driver (FED) 96 optical fibres inputs, each a multiplexed pair of APVs 8 front end blocks each driven by a 12 way optical ribbon cable Raw input data rate (all 96 fibres) = 3.4GB/s. Output rate down slink = 50MB/s /% occupancy VME FPGA Front-End data processing FPGA Power S-Link Back End “System” FPGA FE Unit

4. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 20044 Front End (FE) Unit on FED Opto-RX, 12 way 12 x Buffers 3 x Delay FPGA (ADC clk timing) 6 x Dual 40MHz, 10bit ADCs Virtex II, 2M gate FPGA performs signal processing Optical ribbon cable input Analogue circuitry duplicated on secondary side Signal magnitude Digital header 128 analogue values (one for each microstrip) MIP De-multiplexed fibre channel = APV Data Frame To extract hit need to perform: - Common mode subtraction - Pedestal subtraction - Cluster finding - Sync checking Opto-to-electrical conversionDigitise & sync dataFind hit clusters Time

5. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 20045 FED status At LECC ’03 in Amsterdam –Start ‘03: The first two FEDv1 boards were manufactured –June ‘03: After testing showed there were no major faults a further 3 FEDv1 boards were produced Progress in the last year –Sept. ‘03: Further 6 boards had serious problems –Start ‘04:A further batch of 6 boards manufactured and assembled at different company –Spring ‘04: FEDs distributed to CERN, Pisa, Lyon –June ‘04: Beam test at CERN –Sept. ‘04:FEDv2 should return from manufacture. –End ‘04:Manufacture a further 20 FEDv2 assuming no surprises. Software and DAQ for the CMS Silicon Tracker Front End Driver Poster by Jon Fulcher The Manufacture of the CMS Tracker Front-End Driver Poster by John Coughlan

6. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 20046 Testing in the lab FED –Tested with FED Tester Ensemble FED Tester Ensemble –Drives all 96 FED channels with data similar to that expected in CMS. –100kHz Poisson L1A. –Also provides clock, L1A and throttling from built in Trigger Control System (TCS) –An ensemble is made up of 4 FED Testers. Master FED Tester: Provides clock and control to additional 3 FED Testers FED under test sandwiched between 2 additional FEDs and crate closed to simulate airflow & temperatures in fully populated crate. VME access via SBS 620 PCI-VME bridge

7. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 20047 The FED Tester DACs System FPGA VME FPGA AOHs x-point switches and buffers on back side Power Master & slave I/O Optical outputs Fibre spools Data for 3 channels loaded into FPGA. Converted to analogue form by 3 DACs. Cross-point switch controls distribution of the 3 unique channels to the 24 channels. 8 three channel TOB type AOHs convert the electrical signal to optical signals. Temp of AOHs controlled to +/-1ºC Provides 24 optical channels & Trigger Control System (TCS)

8. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 20048 Laboratory test set-up Slink Rx Generic PCI Card VME crate PC Slink PC (PCI-X slots) Access VME with SBS620 PCI-VME link LVDS cable FED Simulate Local Trigger Control System FT (master) FT (slave) Slink-Tx VME crate Clock & L1As from FT to FED Throttle signals from FED to FT Merge 96 fibres into 8 ribbons of 12 fibres

9. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 20049 Slink transition card Carrier for slink transmitter –Buffers control and data signals –Buffers throttle signals –FED v1 and V2 compatible Status –Returned from manufacture in August. –S-link verified –Throttle signals still under test DAQ – Slink Transmitter FED Slink data and control signals Throttle signals to FMM VME Backplane

10. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 200410 Slink verification Check –Verify data transmitted by the FED via slink is not corrupted.Set-Up –FED sending test patterns (scope mode, sample size 6) –100 kHz repetitive trigger CPU can only verify data transmitted at ~28 Mbytes/sec Hence FED asserts “BUSY” and trigger rate falls to ~17.1 kHzResults –Verified 1 TB of data in 10.8 hrs No errors –No errors observed so far, however it would take 146 days to guarantee no more than 1 error per week in CMS. However.... –In rate test with transition card borrowed from another sub detector (no signal buffering) we did observe occasional errors. –More tests needed. Add CRC check.

11. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 200411 Slink max data throughput FED configuration –Scope mode, repetitive triggers at 100kHz. –Varied scope sample length to vary event size. Results & Conclusions –Maximum transfer rate = 469MB/s –Observed S-link receiver exerting back pressure for events > 4.88kB –PC rather than FED setting upper limit. –Switched to random triggers at 100kHz OK in scope mode & zero suppressed mode Requirements for CMS –200MB/s Average –400MB/s Peak

12. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 200412 Temperature control Fan direction Air deflector Simple air deflector in centre of card lowers temperature in hottest region by ~10°C OptoRx #7 OptoRx #0 Max temp of OptoRx = 70°C

13. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 200413 Where are Tracker buffers ? Buf 1 Buf 12 Front-End FPGA 1 Buf 1 Buf 12 Chan 1 Chan 12 Head Buf Back-End FPGA Mux BE Buf Bus 1 Bus 8 80MB/s 200 MB/s Avg 400 MB/s Peak 640 MB/s Unlimited APV 40 MS/s @ 10bit Chan 1 Chan 12 Laser Driver 40 MS/s @ 10bit APV Buffers, ~10 in decon mode controlled by APVE FE Buffers, 4kB, ~250 ZS events BE Buffers, 2MB, ~1000 ZS events Front-End FPGA 8 Header Buffer

14. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 200414 Buffer tests Check FED buffers can handle high rates & occupancies –Want to test “unconstrained” FED Don’t want slink back pressure limiting results Slink throttle disabled. Data sent to slink oblivion –Ideally would like all FED buffers to assert “Busy” or “Warn” when becoming full. If buffer overflow inevitable -> Detect the event, but ignore data Set flag to indicate data loss and record the number of these events –Not possible yet, although BE buffer can assert throttle signals Use different approach. Not perfect, but still informative. No throttling used Count number of triggers before FED hangs

15. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 200415 Buffer test results Conditions: Data taken in Zero Suppression mode Single strip clusters used to create largest event size possible and thus worst case. APV frame occupancy kept constant. - e.g. 6 strips = 4.7% occupancy - Easier to understand results. - Large buffers -> Valid approx Performed 5000 "tests“. Each comprised 100k triggers, 100kHz Poisson distributedResults: FED handles single strip occupancy up to 6.25% (8 single strip clusters)

16. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 200416 Beam Test June ‘04 CERN X5 Area –4 FEDs –252 fibres –65,000 strips –2 FEDs slink –Provided excellent opportunity for system integration –Large complex system useful for finding system weak points. Beam

17. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 200417 Summary FED v1 –Commissioning tests at RAL, Imperial & CERN continuing well. –Baseline firmware and software operational FED v2 and S-link Transition card –Both back from manufacture in August ’04 –Only preliminary results so far, but everything seems OKFuture –System integration e.g. Calibration of Tracker & Database systems –Production issues e.g. Industrial testing

18. LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 200418 More information... Software and DAQ for the CMS Silicon Tracker Front End Driver Poster by Jon Fulcher The Manufacture of the CMS Tracker Front-End Driver Poster by John Coughlan