1. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting The Front-End Driver Card FEDv1 VME crate requirements definition FEDv1 status : hardware / firmware / low-level software FEDv1 testing plans http://www.te.rl.ac.uk/esdg/cms-fed/index.html

2. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Crates Requirements Document Document needed for Tracker Crates ordering Aim to have common VME64x bus crates for FED and FEC List of special needs, connectors, power…etc. E.g. FED may need Transition cards (DAQ links) References to LHC Crate technical specification EP document Released for comments

3. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Crates Requirements Document

4. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Overview VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Compact Flash Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA Modularity 9U VME64x Form Factor Modularity matches Opto Links 8 x Front-End “modules” OptoRx/Digitisation/Cluster Finding Back-End module / Event Builder VME module / Configuration Power module Other Interfaces: TTC : Clk / L1 / BX DAQ : Fast Readout Link TCS : Busy & Throttle VME : Control & Monitoring JTAG : Test & Configuration FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x

5. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Project History VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Compact Flash Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA Design Milestones Analogue Circuit for signal match from OptoRx to ADCs (good working relationship with Francois&Jan) Fit 96 ADC channels on board FPGAs choice of Xilinx Virtex-II family. Fitting FE design into target part. But things (almost always) take longer than you think… S-LINK Transition card / Direct? Power module spent more effort than anticipated FPGA de-coupling schemes. + many minor design decisions. FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x

6. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Board Layout Primary Side Complex board Analogue & Digital & signal integrity Pushing density limits Modular PCB design

7. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FED v1 Board Layout Secondary Side

8. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Overview 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres Xilinx Virtex-II FPGA Modularity 8 x Front-End “modules” FE-FPGA Cluster Finder 9U VME64x

9. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Front-End module 12 Fibre Ribbon PD Array 1 1 1 2 1 2 3 4 3 5 6 2 4 7 8 5 9 10 3 6 11 12 5*5* CLK DCM LVDSCLK40 from TTC CERN Opto Rx 6 CLOCK CONTROL DATA DATA OUT @ 160 MHz 3 N Full Partially Full RESET Cluster Finding FPGA Delay FPGA Delay FPGA Delay FPGA Dual ADC 10-bits 40 MHz OpAmp XC2V1500XC2V40AD9218EL2140 CLK 10 ASIC 4*4* Data Control Digital Processing 12x trim DAC Temp Sensor LM82 * Double Data Rate I/O Each individual ADC clock skew is adjustable in steps ~ 1nsec

10. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Front-End module Primary Routed

11. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Front-End module Secondary Routed

12. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Full-Scale 9U Layout TTCrx BE-FPGA Event Builder Buffers DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres Xilinx Virtex-II FPGA Back-End module For each L1-Trigger: Collects 8 x FE variable length data fragments Formats FED event for DAQ Appends TTC synch information Buffers in External QDR SRAM Sends data via DAQ Front-end Readout Link FRL Signals to TCS : Busy/Throttle FE-FPGA Cluster Finder TTC TCS TCS : Trigger Control System 9U VME64x PinDiode /Amp ASIC

13. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting DAQ Front-end Readout Link FRL DAQ Mezzanine Card Transition Card Busy Throttle S-LINK64 FEDv1 FED-DAQ Interface TTCrx BE-FPGA Event Builder Buffers 12 TTC TCS FRL DAQ links use S-LINK64 standard Implementation: Channel Link 800 MBytes/sec max Average DAQ rate 200 MBytes/sec See talk “CMS Data to surface transportation architecture”: Attila Racz Due to mechanical constraints place DAQ link card on Transition card

14. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting Alternative scheme: “Channel Link” in Virtex-II BE-FPGA DAQ FEDv1 FED-DAQ Interface (alternative) TTCrx BE-FPGA Event Builder Buffers 12 TTC TCS

15. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 VME module VME-FPGA FPGA Configuration 12 VME Interface Xilinx Virtex-II FPGA VME64x Interface: A32/D64 Slave & Master DMA engine Interrupter Plug & Play geographic addressing Live Insertion Transceivers FPGA Configuration: Xilinx System ACE Compact Flash MPU-VME interface for in-situ reprogramming via Crate Controller JTAG for Configuration & Test JTAG 9U VME64x System ACE CF

16. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Power module Power DC-DC 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital Xilinx Virtex-II FPGA Standard LHC spec crate supplies: +3.3V, +5V, +12V Derive: -5V, 1.5V, 2.5V on board Board Estimate ~ 80 W Hot Swap Controllers: Electronic fuses Protection against out of range voltage & current Sequence power Protections: Over temperature shutdown Temp Monitor TCS : Trigger Control System 9U VME64x Warning

17. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Testing VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA JTAG Boundary Scan: Digital connections Chip Scope Integrated Logic Analyser Cores : Capture raw ADC data (without VME) Opto-Tests or Inject electrical signals post-OptoRx Special FPGA loads e.g. Pattern Generators Additional Test Features: Internal/External Clocks External Triggers FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x

18. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Status VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x Board Status

19. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FED Schedule Schedule 2002/Q4 : 2 x FEDv1 @ for UK test Batch 1 2003/Q4 : ~10? x FEDv1 @ CERN Batch 2* 2004/Q4 : ~10? x FEDv2 manufacture 2005/Q2** : 500 x FEDv3 manufacture (**funds permitting) FEDv1 : Full scale Prototype FEDv2 : Pre-production FEDv3 : Final production FEDv1 FEDv2 FEDv3 * Procurement started for critical parts for Batch 2

20. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FED TIB Test Schedule Q. Deliver 2 x FEDv1 @ CERN for TIB test starting July 2003 ?? Very tough. Testing is a big job. Can’t promise to deliver in June? Use Batch1 PCBs. ie No design iteration possible. Risk that design has major fault. Prioritise firmware and testing plan. Require explicit list of functionality required/excluded for the TIB tests. FEDv1 FEDv2 FEDv3 TIB? OptoRx?

21. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Firmware & Low-Level Software

22. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Firmware & Software VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA Digital Processing Flexible Digital Logic: Xilinx Virtex-II FPGAs 40K->3M gates* *some in pin compatible packages Features: Dual Ported Block Rams Digital Clock Managers DCM Double Data Rate I/O DDR Digitally Controlled Impedance I/O Various I/O signal standards Debugging: Logic Analyser cores FPGAs programmed in VHDL & VERILOG FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x

23. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Front-End module 12 Fibre Ribbon PD Array 1 1 1 2 1 2 3 4 3 5 6 2 4 7 8 5 9 10 3 6 11 12 5*5* CLK DCM LVDSCLK40 from TTC CERN Opto Rx 6 CLOCK CONTROL DATA DATA OUT @ 160 MHz 3 N Full Partially Full RESET Cluster Finding FPGA Delay FPGA Delay FPGA Delay FPGA Dual ADC 10-bits 40 MHz OpAmp XC2V1500XC2V40AD9218EL2140 CLK 10 ASIC 4*4* Data Control Digital Processing 12x trim DAC Temp Sensor LM82 * Double Data Rate I/O Each individual ADC clock skew is adjustable in steps ~ 1nsec

24. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting CMS Silicon Strip Tracker FED Front-End FPGA Logic ADC 1 10 sync 11 trig2 Ped sub 11 trig3 Re-order cm sub 8 Hit finding s-data s-addr8 16 hit Packetiser 4 averages 8 header control DPM 16 No hits Sequencer-mux 88 a d a d ADC 12 10 trig1 sync 11 trig2 Ped sub 11 trig3 Re-order cm sub 8 Hit finding s-data s-addr8 16 256 cycles hit DPM 16 No hits Sequencer-mux 88 a d a d status averages 8 headerstatus nx256x16 trig4 Synch in Synch out Synch emulator in mux Serial I/O Serial Int B’Scan Local IO Config Cluster Finding FPGA VERILOG Firmware Full flags data Global reset Control Sub resets 10 Phase Registers Phase Registers 2 x 256 cycles256 cyclesnx256x16 trig1 Synch error 4x Temp Sensor Delay Line Opto Rx Clock 40 MHz DLL 1x 2x 4x per adc channel phase compensation required to bring data into step + Raw Data mode, Scope mode, Test modes... 160 MHz

25. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Full-Scale 9U Layout TTCrx BE-FPGA Event Builder Buffers DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres Xilinx Virtex-II FPGA Back-End module For each L1-Trigger: Collects 8 x FE variable length data fragments Formats FED event for DAQ Appends TTC synch information Buffers in External QDR SRAM Sends data via DAQ Front-end Readout Link FRL Signals to TCS : Busy/Throttle FE-FPGA Cluster Finder TTC TCS TCS : Trigger Control System 9U VME64x PinDiode /Amp ASIC

26. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Back-End FPGA Logic FIFO Circular Buffers Frame_Syncs Readout_Syncs Monitor_Syncs x8 TTC Rx TCS ‘VME’ DECODE CONTROL & MONITOR Data_stream 0 Data_stream 7 6418 Data In 20 Address 18 Data Out 64 FRL to DAQ SLINK64 64 R/W Address Generator APV hdrs Lengths Bx,Ex Em Hdr diagnostics Data 80 Mhz L1 100 kHz 4 4 160 MHz 40 Mhz Clock40 Reset DCM x1 x2 x4 2 x QDR SRAM x2 burst BSCAN 160 MHz 80 MHz Lengths Header FF/PF Flags 2 8 1 1 1 Control 2 MBytes VME copy path to local buffer

27. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Status VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x Board Status

28. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Module Testing Firmware Functionality VME readout A32/D32 only Raw Data only (unpacked) Scope mode for timing TTC interface Individual ADC clock skews Hardware throttle ? S-LINK TCS interface Cluster Finding Auto-Calibration VME64x dynamic addressing Required Not Required Q. What event rate needed for testing? Q. Do we need individual channel DAC offsets ? (or is common OptoRx offset sufficient)...

29. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Low-Level Software XDAQ Hardware Abstraction Layer fedlib class API lowest level access for user programs e.g. fedlib_load_fe_ped(..) fedlib_select_clock(..) Plus additional layer? (see Costas’s talk) Permit VME A32/D32 accesses only. Plus CSR space Possibly D32 BLT No Interrupts fedlib fed code

30. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Address Map Hidden from outside world… Address Map under construction. Implementation depends on Firmware. Need this to do before fedlib API FE FPGA #0 FE FPGA #1 FE FPGA #2 FE FPGA #3 FE FPGA #4 FE FPGA #5 FE FPGA #6 FE FPGA #7 FED local CSR TTCrx Event Memory FPGA confign Pedestals Skew values...etc

31. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Low-Level Software Mechanism of FED setup hidden from User Eg Loading FED parameters not trivial Write constants eg Peds to small local FPGA buffer. Limited FPGA memory. Send command to load to FE FPGA FPGA “engine” handles serial commands Optional readback Repeat for next block Constraints: Block access only NB Can’t access constants during Run! Command & Status Parameter Load block Parameter Readback block

32. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Firmware & Software VME-FPGA TTCrx BE-FPGA Event Builder Buffers FPGA Configuration Power DC-DC DAQ Interface 12 Front-End Modules x 8 Double-sided board CERN Opto- Rx Analogue/Digital 96 Tracker Opto Fibres VME Interface Xilinx Virtex-II FPGA Digital Processing Flexible Digital Logic: Xilinx Virtex-II FPGAs 40K->3M gates* *some in pin compatible packages Features: Dual Ported Block Rams Digital Clock Managers DCM Double Data Rate I/O DDR Digitally Controlled Impedance I/O Various I/O signal standards Debugging: Logic Analyser cores FPGAs programmed in VHDL & VERILOG FE-FPGA Cluster Finder TTC TCS Temp Monitor JTAG TCS : Trigger Control System 9U VME64x

33. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Event Readout DAQ Trailer Formatted FED Data DAQ Header Tracker Header? Readout (via VME) Formatted Events identical? to those sent to S-LINK Poll on Event Counter Get length of event (Request) Readout & Clear Event Repeat… Check local status registers/counters i.e. similar to PMC (sorry) Event Formats: Raw Data unprocessed. Scope Mode. Tracker header to do...

34. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Schedule These are still guesstimates and depend on ease of Firmware implementation… Draft Memory Map (mid October) Release Event Format specs (end November) Release draft fedlib API spec (basic calls/no code) (end December) Release draft operating instructions (end December) Release first version of fedlib (basic calls/code) (end February) Caveat Emptor: It is still possible/likely that changes will be necessary to these releases.

35. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Comments In order to achieve minimum firmware & software on TIB timescales require... Confirmation of test functionality. Close co-operation between UK and Tracker DAQ group. Separate small FEDv1 online meeting. Propose contact from UK to visit CERN for “technology transfer”. Keep systems h/w and s/w as close as possible (within UK) and between UK and CERN.

36. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting FEDv1 Testing

37. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting CMS Silicon Strip Tracker FED Counting Room Layout (illustration) 440 Boards96 ADC/Board 24 Crates 8 Racks 440 Boards96 ADC/Board 24 Crates 8 Racks 40 K ADC Channels10 Bit@40MHz Max Trigger Rate100 kHz Input Rate1.5 T Byte/s Output rate25 Gbyte/s/% 40 K ADC Channels10 Bit@40MHz Max Trigger Rate100 kHz Input Rate1.5 T Byte/s Output rate25 Gbyte/s/% 4 TTC Partitions DAQ FED

38. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting CMS Silicon Strip Tracker FED Crate Layout 1 21 2 FE 1 DAQ TTC FE 2 FE 3 FE 4 FE 5 FE 6 FE 7 Crate Input Data Rate~ 50 Gbyte/s Crate Output Data Rate~ 1 GByte/s per percent hit occupancy B-Scan F-Bus NN Synch 100 KHz FE 8 Throttle

39. Instrumentation DepartmentJ. Coughlan et al.Rutherford Appleton Laboratory23 October 2002Tracker Electronics Meeting 124 E-Net Switch FEC CRATES DCS Tracker Control WS TTC CRATES DAQ VME SBC RTOS CMS Silicon Strip Tracker FED Control & Monitoring D-BASE R/C 50K ADC Channels FED CRATES