J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam The CMS Tracker Front-End Driver 9 th Workshop on Electronics for LHC Experiments Amsterdam J.A.Coughlan, S.A. Baird, I. Church, C.P.Day, E.J.Freeman, W.J.F.Gannon, R.N.J. Halsall, M. Pearson, G. Rogers, J. Salisbury, S.Taghavirad, I.R.Tomalin CCLRC Rutherford Appleton Laboratory E. Corrin, C.Foudas, J. Fulcher, G. Hall, G. Iles, M. Noy, O. Zorba Imperial College I. Reid Brunel University Presented by John Coughlan

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED FEDv1 This Talk Overview of FEDv1 Architecture Board Status & Test RAL & Imperial College Primary Function Data reduction and event building. To extract hit strip information from CMS Silicon Tracker FEnds

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Silicon Strip Tracker Readout Overview ~ 9 million Silicon Strip channels ON Detector: 73K APV25 pipeline L1 Trigger: MUX APV Frame output Analogue Data readout via Optical links (APV Frame: Header + Strip Data) OFF Detector: Front-End Drivers (FED) Digitize / Zero Suppress / DAQ readout 440 x 9U VME64x boards 96 ADC channel boards DAQ Counting Room On Detector FPGA 25 VME 9U FEDs Hybrid Front-End Hybrid Silicon Strips

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED FED Layout VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA Modularity 9U VME64x Form Factor Modularity matches Opto Links 8 x Front-End “units” OptoRx/Digitisation/Cluster Finding Back-End module / Event Builder VME module / Configuration Power module Other Interfaces: TTC : Clk / L1 / BX DAQ : Fast Readout Link TCS : Busy & Throttle VME : Control & Monitoring JTAG : Test & Configuration FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED First Module January - March 2003 “Primary” Side FEDs under test: Ser001: Imperial 2 ORx installed March detailed characterisation of FED with optical inputs Ser002: RAL Firmware development / Electrical / digital tests Ser assembled June Ser 005 with 8 ORx’s Ser PCBs due September To be fully equipped with ORx’s (1 month) OptoRx CFlash VME64x 9U board 34 x FPGAs Analogue TTC FE Unit Power Memories 96 channels JTAG

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED FPGAs & Firmware Delay x 24FE x 8 BE x 1 VME x 1 Delay FPGA: ADC Coarse and Fine Clock Skewing. FE FPGA: Scope and Frame Finding modes. BE FPGA: Event building, buffering and formatting. VME FPGA: Controls and Slow Readout path. 34FPGAs 34 FPGAs on board NB Final Firmware was only implemented after board manufacture. Now ready. Test Firmware used for early results 40k -> 2M gates Xilinx Virtex-II FPGA Configuration System ACE Compact Flash thru JTAG chain

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED First Fully Assembled Board June 2003 Primary Side Secondary Side 9U VME64x PCB (2mm) 14 layers (incl 6 power & ground) 96 ADC channels : AD9218 Dual package MHz ~ 6 K components ; ~ 25 K tracks 1/2 Analogue circuitry on Secondary Side High density at Front-End Units FE Unit BGAs mm pitch JTAG Boundary Scan

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Zoom in on FE Unit Front-End Unit = 12 channels Dual ADCs OpAmps Delay FPGAs Duplicated on Secondary Side “OptoRx” “Primary” Side Test Connector Resistor Packs “OptoRx” modules CERN project Custom Analogue ASIC Commercial Package with PIN Diode + Custom Analogue ASIC TrimDAC

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Cross-point switch test card uses AD x16 analogue X-point switch Electrical testing: ie Pre OptoRx modules 1 card per FE Unit 3 inputs to any of 12 FED channels Chain 8 cards to cover whole FED

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Early Electrical Tests in FEDv1 Chip-Scope Pro Logic Analyser capture 10 bit-raw data on 12 channels in FE FPGA sine input (1 MHz) via Cross-Point Switch test card to 12 channels first 100 (of 4k) 40 MHz ADC count NB Chip-Scope also invaluable as Firmware Debugger! Testing analogue circuits, connections etc

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED First prototype 6U VME Opto-Tester VME interface logic  Analogue section  Sequence control logic Sequence storage DAC Amplification+cm Analogue opto-tx Acknowledgements for following results to Matthew Noy, Imperial College Drives up to 3 Fibres

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Early Analogue Characterisation effects RMS noise of ~ 1.1 counts ≈ 350e - (including ORx) ADC? effect under investigation Ch. 0  23 with ORx Ch. 24  95 without ORx Important that FED meets analogue specifications too detailed characterisation only possible with complete modules and sophisticated tester (built - also needs evaluation and careful control) All preliminary results are positive details under study … e.g. …. Timing scan of 25 nsec pulse

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Other Test Results Require fine & coarse clock skewing independently on all 96 FED channels 32 fine steps of ~ 800 psec Implemented by Virtex-II Digital Clock Managers Independent TrimDAC offsets at ADC inputs on all 96 FED channels ADC Linearity plot

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED FED Large Scale Opto-Tester Relatively sophisticated tester needed to evaluate FEDs fully need to compare analogue input with digital outputs large volume of well defined data must be scrutinised analogue performance should not degrade system noise - characterise complete FED, not just single channels input data should be very stable - challenging given laser T sensitivity Tester system comprises 4 modules, each driving 24 optical channels. Each module has 3 unique analogue channels, each driven by 12 bit, 40MHz DACs. Cross-point switches allow the 24 outputs to select any of these 3 channels or an external one. The FED Tester can act as a Trigger Control System if needed (e.g provide Clk, L1As, BC0, etc) System control is achieved with 2 Virtex II, 1M gate FPGAs

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED FED Large Scale Opto-Tester DACs Fibre reels 8-way MU optical connector s Clks provided via QPLLs Analog ue Optical Hybrids Cross-point switches VME interface System control FPGA Awaiting AOHs to complete first 24 channel module (G. Iles, J. Leaver + C. Foudas, O. Zorba) Refer to talk at these proceedings

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED First FED Readout of 9U Opto-Tester Data Final Firmware Now using Final Firmware in all FPGAs DAQ events Fully formatted DAQ events via VME readout 12th FED channel TrimDAC set to full scale 2nd FED channel disabled Channels 1,3 & 4 delayed with Coarse Skew Tester input on 12 fibres 1 x FE Unit Scope Mode capture (Frame Finding under test) APV25 simulated Frames

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Schedule I) FEDs for Large Scale Assembly (LSA) tests: restricted FED functionality “October ‘03 (1) / end ‘03 (2) / beg ‘04 (2) / mid ‘04 (6)” II) FED Pre-Production Manufacture: Q1-2/04 full FED functionality assumes new design iteration FEDv2 pcbs DesignTest Production & Installation Pre-Pro FEDv3 (500)FEDv2 (20)FEDv1 (20)

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Summary Testing is progressing well…but much more to do! No “show stoppers” for Silicon Large Scale Assembly tests - functions operating. Analogue effects still need to be understood. firmware Expect to deliver first FED to CERN with necessary firmware around end of October Final Firmware with full DAQ Formatted Events readout via VME ready. Issues Major software/firmware effort - but appears to be converging well Availability of important components OptoRx - FED is following schedule closely - no shortage VME crates, controllers, Analogue Optohybrids (late) FPGAs and other parts in hand for ~20 FEDs (including those assembled) Final procurement for Mass Production of 500 FEDs Yield, assembly quality - and consequent testing - will be main concern

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Static Noise Measurements: 2/8 Analogue Opto-Receivers placed Protection caps inserted  absence of input signals D.C. level set by TrimDAC and Opto-rx (if present)  search over parameter space Random triggering from VME Opto-RX (HXR9003) TrimDAC Fully differential op-ampADC To readout

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam DAQ Front-end Readout Link FRL CMS Tracker FED FED-DAQ Interface FRL DAQ links use S-LINK64 standard Implementation: Channel Link 800 MBytes/sec max Average DAQ rate 200 MBytes/sec VME-FPGA TTCrx Back-End FPGA “Event Builder” Buffers FPGA Configuration Hot Swap PSU DC-DC DAQ Link 12 CERN Opto- Rx ADC/Digital 96 Tracker Opto Fibres Front-End FPGA “Cluster Finder” DAQ Mezzanine Card Transition Card TCS EXT CLOCK S-LINK64 VME Interface 9U VME64x Card DAQ TTC Temp Monitor JTAG

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam FED First fully assembled board. Proceed cautiously ~2 months electrical tests OptoRx is new component available March 2003 Limited supply & costly June 2003 First complete board no difficulties to date but not yet equipped to drive all 96 channels simultaneously Primary side... NB Secondary side carries 48 analogue channels...

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam FED performance Important that FED meets analogue specifications too detailed characterisation only possible with complete modules and sophisticated tester (built - also needs evaluation and careful control) All preliminary results are positive details under study … e.g. …. Ch. 0  23 with ORx Ch. 24  95 without ORx RMS noise of ~ 1.1 counts ≈ 350e - (including ORx) Clock phase adjustment (M. Noy)

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam FED Large Scale Assembly Test Requirements 2003 “read out Virgin Raw Data formatted as DAQ events via VME in response to TTC trigger and clock.” Need 96 OptoRx chans. Trigger & Readout rates are not critical. Functionality Does require: Scope Mode and Software Triggers for set up. Controls from VME for run mode, clock source, clock skew, OptoRx offsets (with readback.) VME Event buffer with standard DAQ events. Counters for triggers & errors. System ACE loading, Clock/Trig/Resets on TTC Chan A, Hardware throttle output. FED delivered as a Package including Software Library to drive the Firmware. Does not require: S-LINK readout, Clustering mode, Spy Channel, TTC chan B, TCS (but maybe simple throttle), DAC control, pedestal/threshold data, System ACE interface, VME64x config EPROM…, VME Interrupts, Temp chip control…

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED FED Layout VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA Digital Processing Flexible Digital Logic: Xilinx Virtex-II FPGAs 40K->3M gates* *some in pin compatible packages Features: Dual Ported Block Rams Digital Clock Managers DCM Double Data Rate I/O DDR Digitally Controlled Impedance I/O Various I/O signal standards Debugging: Logic Analyser cores FPGAs programmed in VHDL & VERILOG FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED Firmware Status Clocks Data Serial Controls VME LINK VME Bus VME SystemACE System ACE Clocks EPROM EPROM TTCrx QDR WriteQDR Read Serial Comms Headers TTC chanA VME LinkRegs S-LINKS-LINK Clocks Data Serial Comms Scope Mode Header Mode FIFOs Input Regs Serial Controls Scope Mode Frame-Findng Mode Output Input Regs Opto Rx DAC Opto Rx DAC DELAY FPGA x 3 x 8 FE FPGA x 8 BE FPGA VME FPGA ADC Under Simulation Under Test on FED Only for FEDv2 Controls Data Readout Control Throttle TCS Input Cluster Finding Mode Ed Saeed Ivan Ed Ed, John Saeed, Ivan Chan B I2C Temp “Working” on FED External Devices Temp 14th July 2003 To be Implemented QDR

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Silicon Strip Tracker FED Front-End module 12 Fibre Ribbon PD Array *5* CLK DCM LVDSCLK40 from TTC CERN Opto Rx 6 CLOCK CONTROL DATA DATA 160 MHz 3 N Full Partially Full RESET Cluster Finding FPGA Delay FPGA Delay FPGA Delay FPGA Dual ADC 10-bits 40 MHz OpAmp XC2V1500XC2V40AD9218EL2140 CLK 10 ASIC 4*4* Data Control Digital Processing 12x trim DAC Temp Sensor LM82 * Double Data Rate I/O Each individual ADC clock skew is adjustable in steps ~ 1nsec

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Silicon Strip Tracker FED Front-End FPGA Logic ADC 1 10 sync 11 trig2 Ped sub 11 trig3 Re-order cm sub 8 Hit finding s-data s-addr8 16 hit Packetiser 4 averages 8 header control DPM 16 No hits Sequencer-mux 88 a d a d ADC trig1 sync 11 trig2 Ped sub 11 trig3 Re-order cm sub 8 Hit finding s-data s-addr cycles hit DPM 16 No hits Sequencer-mux 88 a d a d status averages 8 headerstatus nx256x16 trig4 Synch in Synch out Synch emulator in mux Serial I/O Serial Int B’Scan Local IO Config Cluster Finding FPGA VERILOG Firmware Full flags data Global reset Control Sub resets 10 Phase Registers Phase Registers 2 x 256 cycles256 cyclesnx256x16 trig1 Synch error 4x Temp Sensor Delay Line Opto Rx Clock 40 MHz DLL 1x 2x 4x per adc channel phase compensation required to bring data into step + Raw Data mode, Scope mode, Test modes MHz

J. Coughlan et al. 1st October 2003 LECC 2003 Amsterdam CMS Tracker FED FED Data Rates VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA Data Rates 9U VME64x Form Factor Modularity matched Opto Links Analogue: 96 ADC channels ( MHz L1 Trigger : processes 25K MUXed silicon strips / FED FPGA FPGA Digital Processing Raw Input: 3 Gbytes/sec* after Zero Suppression... DAQ Output: ~ 200 MBytes/sec ~440 FEDs required for entire SST Readout System FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x