1. 2008-08-211 New HCAL DCC aka “DCC2” Eric Hazen, S. X. Wu, A. Heister, P. Lawson, J. Rohlf, J. St John Boston University

2. 2008-08-212 Outline ● DCC Upgrade Motivation-E. Hazen ● Hardware/Firmware Design-S. Wu (presented by E.H.) ● Software Status-P. Lawson ● Testing and plans- E. Hazen

3. 2008-08-213 DCC Upgrade Motivation ● Original DCC design dates to 2001 with some components ca 1999 ● Complex design – firmware maintenance difficult ● Many interconnects – hardware reliability not great ● Internal bandwidth (on HTR inputs) limit to 200MB/sec ● No fast feedback path from HTR input buffers to TTS – Result: loss of sync when trigger rate too high – Very difficult to add to old DCC due to architecture

4. 2008-08-214 Hardware Design ● Xilinx Spartan-3A chosen for all FPGAs – Low cost, easily available – Low technical risk (vs Virtex 5) ● Architecture similar to old DCC – 3 channel groups for HTR inputs – Large buffer at HTR inputs ● Separate VME FPGA – Firmware maintained separately from main FPGA; should not change often ● Separate Event Builder FPGA – E.B. Firmware from old DCC largely ported

5. 2008-08-215 DCC2 Improvements ● Many fewer parts; no back-side components ● Enough bypass capacitors !! ● Supports selective readout ala ECAL: makes better use of limited DAQ bandwidth ● Much simpler design – no complex internal protocols ● Inexpensive to produce – can build lots of spares ● Large and still-buggy PCI-based software support library no longer needed – a simple HAL table will do ● Remote diagnostics/status via Ethernet

6. 2008-08-216 Other New Features ● Fast response to HTR busy/overflow warnings – Avoid current situation where HTR empty events occur ● Long TTC/TTS state history with “trigger” features to capture problems ● Ethernet-based monitoring – Allow remote access for diagnostics independent of current state of DAQ software and computers – Read-only except in the lab!

7. 2008-08-217 Firmware Loading ● Commodity parallel flash stores all firmware ● CPLD loads firmware to all FPGAs ● VME fpga is dual-boot; if primary sector fails, backup sector is used automatically ● Flash may be written over VME ● JTAG connector provided for initial programming or emergency recovery

8. 2008-08-218 DCC2 Block Diagram Xilinx XC3S200 A FPGA RJ-45 HTR inputs (total 15) 32MByte DDR SDRAM Xilinx XC3SD1800A FPGA 32MByte DDR SDRAM TTCrx UMD board LVDS Tx TTS TTC Xilinx XC3S200 A FPGA VME Buffers VME64x S-Link LSC DAQ Data TP Out Readout bits in Buffe rs Out In RJ-45 to/from Readout Processor Future backplane Connector board at rear of crate

9. 2008-08-219 DCC2 Data Paths LVDS Deser. RJ-45 HTR inputs SDRAM Event Builder Runs at 600MB/sec* 9 pairs+ clock TTS TTC VME64x Input: 40MHz * 16 bits (peak) per link = 80MB/sec * 3 links = 240MB/sec 100kHz * ~ 800 bytes per link maximum Max-size non-ZS data Up to 300 MB/sec* 4+ck S-Link 64200 MB/sec DAQ limit (much more possible if FRL can handle it!) Out In TP sent to readout processor (few bits per tower per L1A) TP sends back bit-mask of channels to read Total BW needed ~ 30MB/s * = design speed. Currently operating At 2/3 speed due to Xilinx software issues 1 pair serial 4+ck 100 MB/sec 600 MB/sec* RAM bandwidth 800 MB/sec* RAM bandwidth 150 MB/sec*

10. 2008-08-2110 DCC2 Input Stage Front-end FPGA “LRB” Xilinx XC3S200A RJ-45 HTR inputs 32MByte DDR SDRAM Event Builder FPGA Xilinx XC3SD1800A Event Headers Data request DAQ Data Control, Monitoring Headers (EvN) sent to Event builder as received. Event data buffered in SDRAM Write: 240 MBytes/sec Read: 300 MBytes/sec Event data sent to Event builder on request. To S-Link L1A sent to front-end FPGA Zero-suppression / selective readout algorithm selects channels to readout, sends readout mask back to front-end FPGA. If readout processor used, send TP to RP, receive readout mask, send to front-end DAQ data transferred from SDRAM buffer through FPGAs to S-Link 32MByte DDR SDRAM Event builder temporary storage is for meta-data only (EvN, etc). Use BRAMs. SDRAM is for VME readout Selective Readout Processor

11. 2008-08-2111 Buffering RxRx Inpu t Logi c Header FIFO (EvN, OrN, BcN) Event Builder 256 evt Complete event data 256 evt OFW, BSY, SYN HTR Inputs (15 total) 128 evt VME FIFO SLink TTCrx 1K L1A OFW, BSY, SYN SDRAM partitioned memory Fixed-size buffers FIFO

12. 2008-08-2112 TT C TT S Etherne t HTR s (15) VME6 4 TM Connecto r Input FPGA SDRAM (3 channels per input block) TTCr x Event Builder FPGA and SDRAM Slin k On Rea r

13. 2008-08-2113 New DCC Status -Hardware ● 10 Prototypes completed ● Hardware design verified: – All interfaces tested – HTR/DCC links run for ~ 10e13 bits with no errors – Inter-FPGA links and SDRAMs tested extensively with pseudo-random bit patterns – Event building works on our test stand ● Hardware issue: – Input FPGA is 60-70% full – should we switch to larger? Add'l cost is ~ $7500 (of ~ $40k total prod cost) – (The main FPGA is only 20% full)

14. 2008-08-2114 Software - more details from Phil Lawson - ● As compatible as possible with existing DCC – Minimal changes to DCCManager – Existing cfgScript supported; only enhancements – Compatible monitoring (eventually will enhance) ● New DCC support class – No PCI initialization, simple flat HAL table – Fixed slot-based addressing like HTR – Much simplified initialize(), prepareForRun(), start(), stop() ● Arno and Phil (G.S.) are writing the code

15. 2008-08-2115 DCC2 ─ Proposed Plan ● Test in 904 starting wk 14 (30-Mar) ● Test in next MWGR wk 15 (6-Apr) – One or two modules ● Continue testing with 10 prototypes @ BU, CERN... ● Order production parts (~ 60pc) wk 16 – PCB fabrication and assembly ~ 8 wks – Assembled boards arrive wk 24 (8-Jun) – Test @ BU, ship to CERN ● Test in 904 ● Install in USC

16. 2008-08-2116 Compatibility: DCC1 vs DCC2 ● Goal: plug-replacement (with software updates) ● DCC1:Channel Link Rx DCC2:Spartan 3A FPGA – So far 100% reliable. Timing can be tuned as required ● VME interface is FPGA instead of Universe 2 – Recycled from other project – no anticipated problems ● Event builder VHDL code was ported from DCC1 – Should be relatively bug-free

17. 2008-08-2117 Backup Slides

18. 2008-08-2118 Selective Readout ● Event size strictly limited to 2k bytes (at 100kHz) in current DCC by PCI bandwidth – Total of 60 (TP+TS) per HTR, or 2.5 per channel – If we send 1 non-suppressed TP, that leaves 1.5 – Choosing a workable combination of threshold and (TS,TP) given these constraints is challenging ● We can do a lot with more intelligent readout even without increasing average bandwidth or communicating with other subsystems ● For this to be useful, inter-board communication is a must

19. 2008-08-2119 Selective Readout ● Possible Algorithms - with no ECAL input – Lower threshold around cluster at same depth – Lower threshold downstream (HO) of a cluster in HB ● Possible Algorithms - with ECAL input – Lepton and photon isolation – Other... some MC work is needed ● In addition... – We can run with higher occupancy on some regions (exceeding the 2k average) with a new DCC (trading for lower occupancy elsewhere to keep the DAQ folks happy)

20. 2008-08-2120   DCC Map in  /  HOHO HB/E HF Side +1 Side - 1

21. 2008-08-2121 Selective Readout ● “Load levelling” amongst HTRs possible with no interconnections ● Hardware required for other algorithms – Bi-directional links between neighboring DCCs ● Each DCC would need to talk to 8 neighbors (plus depth!) – Or, send a single point-to-point link to a central processor from each DCC ● This is our preferred solution ● The “processor” would be quite simple (one main FPGA) and would receive a set of TP from each DCC, and return a readout map – Input from ECAL ● Need to discuss this further, but would clearly improve performance