1. ZHULANOV Vladimir Budker Institute of Nuclear Physics Novosibirsk, Russia Beijing 2011.01.25

2. ECL electronics Infrastructure (1) EclShaperDSP EclShaperDSPb PA ~10 m analogue signals EclCollector ~1 m custom digital channel LVDS levels 16 crystals & preAmps connected to each EclShaperDSP 100 Mbit Ethernet JTAG over LVDS Belle2link FAM ~ 1m analogue signals, clock VME backplane 12 EclShaperDSPb connected to one EclConnector TTD LVDS levels

3. ECL electronics Infrastructure (2) TTDTTD EclCollector EclShaperDSPb FAMFAM There are 52 ECL VME crates We don’t plan to install VME controller at the crates The crates have non-standard power supplies! ±7.5 V (instead of ± 5.0V) and ±15 V (instead of ± 12V) Modules: EclShaperDSPb – shaping pulses from CsI preAmps, digitizing and digital processing. EclCollector – merges data from EclShaperDSPb modules and sends the data to the DAQ system TTD – provides timing & trigger information for EclCollector FAM – provides arguments for global trigger from ECL. VME crate

4. EclShaperDSPb (1) Continuous digitizing shaped pulses, fitting with reference waveform and restoration of the amplitude and time of the pulse. Sample frequency is 1.77 MHz = 508.9 MHz/288 and is synchronous with electron bunches. It is important for calculation of pulse time to have trigger time information referenced to the sample frequency and to have the restored peak time at the same clock domain. There are a lot of precomputed coefficients used in DSP and being stored in SDRAM chips The computed data for one channel is packed into one 32-bit integer: 18 bits for amplitude, 12 bits for time, 2 bit for quality flags

5. EclShaperDSPb photo

6. EclShaperDSPb (2) – digital part SDRAM 1SDRAM 2 Xilinx FPGA ADC Two 133MHz 16 bit DDR chips (1 GByte/s) 16 x 18-bit ADCs @ 1.76 MHz SERIALIZER DESERIALIZER LVDS => CMOS STP cable (1m) DAC 16 x 64-taps DCPs (Digitally controlled potentiometer) 66 MHz XTAL SCK=42.33 MHz, TKN ALTERA CPLD CONF

7. EclShaperDSPb (3) – FPGA design DSP ADC DATA PROCESSOR + buffer for 512 samples 16 channel (290 mks) SDRAM 1 PACKAGER Control logic SDRAM 2 ADC data is stored in cyclic buffer for 290 mks and is overwritten with the new data after that!!! Output can be one of three cases: only processed data, only raw ADC data, both ADC and processed data ADC F(clock) =132 MHz=66MHz*2 F(clock)=127 MHz=42.33*3 SCK (42.33 MHz) PLL Control logic ADC control sequence TKN (synchronization signal) Up to 8 trigger events can be processed simultaneously The processing of one trigger can be delayed for upto 290 mks 60 samples used in processing, and 22 of them are already get before trigger signal. 60*567ns = 34 mks = 12mks (before trigger) + 22 mks(after trigger) Data transferred to EclCollector @ 528 Mbps (66MHz *8)  60 ns for 32 bit word

8. EclShaperDSPb(4) – output data packat OffsetHigh 16 bitLow 16 bit 0Magic number = 0x5AA5FF00 1B15..8 = 79 (packet ID) B7..0 = 16 (packet type – Trigger command) 67 (packet length) 3B22..B16 = 60 – number of ADC samples per channel B28..24 = 1 – number of channels with ADC data B7..0 = 14 – (0..143) trigger time B12..8=2 – number of hit channels (length of DSP data) 4B31..16 = 0x00C0 – hit channels mask B15..0 = 1839 Trigger TAG 50B31..16 = 0x0400 – ADC data mask 6B31..30 - quality flag, B29..18 – time, B17..0 – amplitude (ch.7) 7B31..30 - quality flag, B29..18 – time, B17..0 – amplitude (ch.8) 8..67ADC samples (60 samples, ch. 11) 68CRC32 0000000011000000 ch. 1ch. 16 Hit channel mask 0x00C0 0000010000000000 ch. 1ch. 16 ADC data mask 0x0400 No raw ADC data: data packet size = 6…22 32-bit words data latency min = 22.4 mks = 22 mks (ADC data ready) + 0.4 mks (transfer to EclCollector) data latency max = 279.3 mks 278 mks (ADC data ready, processing & transfer inside FPGA) + 1.3 mks (transfer to EclCollector) Raw ADC data present: data packet size = 966…972 32-bit words data latency max = 336 mks 278 mks (ADC data ready, processing & transfer inside FPGA) + 58 mks (transfer to EclCollector) Data latency to EclCollector = 22…336 mks. Expected average latency is 30 mks

9. 9 Tasks of the ECL collector module. 1. Configure Xilinx FPGA on ShaperDSPb modules 2. Synchronize sampling process of ShaperDSPb modules 3. Collect data from 12 connected ShaperDSPb modules 4. Calibration signal generation 5. Interface with remote PC for stand-alone operation 6. Provide interface with Belle II TTD and DAQ

10. EclCollector(1) Block diagram SER/DESER LVDS drivers, SCK, RCK FPGA Xilinx Virtex5 RocketIO DAQ Flash memory 512 MByte DAC LVDS TTD calib FAM Clock Ethernet PC PHY

11. EclCollector(2) clock domains FIFO 8kByte FIFO 8kByte FIFO 8kByte FIFO 8kByte FIFO 8kByte x 12 FIFO 8kByte x 12 EclShaperDSPb data Processing unit (FPGA logic now, Embed. CPU planned) RocketIO Ethernet FLASH memory FLASH memory TTD clock PLL several clocks asynchronous to el. bunches 254 MHz synchronous to el. bunches or on-board OSC OSC 42.33 MHz OSC 42.33 MHz DAC sequence generator DAC SCK (42.33 MHz) & TKN for EclShaperDSPb The requirement for TTD: provide any clock frequency synchronous to el. bunches & trigger signal referenced to this frequency Merge of large data patches can consume a lot of time (next slide) FGPA

12. EclCollector(3) data merge FIFO 8kByte FIFO 8kByte FIFO 8kByte FIFO 8kByte FIFO 8kByte x 12 FIFO 8kByte x 12 Data merger DAQ Belle2link IFC Continuous buffer FIFO 8kByte FIFO 8kByte FIFO 8kByte FIFO 8kByte FIFO 8kByte x 12 FIFO 8kByte x 12 FIFO map DAQ Belle2link IFC Even data locations info Data transfer time is upto 22.2 mks (12 ch. * 976 words/ch. * 3.79 ns/cycle * 0.5) in case the raw ADC data are read out This time can be saved if the second option is selected So, the requirement for Belle2link interface: exclude any data transfers from buffer to buffer – transfer data from FIFO buffers to optical link directly

13. Data latency summary no raw ADC dataraw ADC data present Data latency in EclShaperDSP module 22/30/27829.3/37.3/278 Data transfer to EclCollector0/0.7/1.358/58.7/59.3 Data transfer inside EclCollector0/0.2/0.421.8/22/22.2 Data transfer through optical link (2.5 Gbit/sec for data) 0/1/2.5147.5/148.4/149.9 Total:22/31.9/282.2256.6/266.4/509.4 units are mks (min/avg/max)

14. EclCollector(4) Ethernet interface 2 16 virtual 16-bit registers 2 32 virtual 16-bit memory cells

15. EclCollector(5) Network processor – other FPGA logic interface Network processor reg_num16 reg_wdata16 reg_we1 reg_rdata16 register file control signals, commands, status signals memory map mem_addr32 mem_wdata16 mem_we1 mem_rdata16 buffer 1 buffer 2 buffer 3 Network processor doesn’t know anything about register and memory designations 4 Ethernet actions are enough for the interface : read/write register, read/write memory region I would be happy if Belle2link provide means for control and setup ECL (and maybe other) system in a manner like this

16. FPGA configuration and Startup EclCollector configuration from flash memory EclCollector configuration via JTAG-over- LVDS interface, flash memory update failed EclShaperDSPb configuration and DSP coefficients load from flash memory OK Remote data is used, flash memory update failed Ready for operation OK

17. Status EclShaperDSPb has been developed and the restoration algorithm was checked with it on Belle detector in 2009/2010 season. EclCollector prototype is manufactured and most of its functions have been tested: Ethernet interface, interface with EclShaperDSPb modules, calibration mechanism, RocketIO link. We need to update the RocketIO and TTD connection schematics and check them We need to add JTAG-over-LVDS interface We need to implement embedded processor We need to check flash-memory read/write And finally we need to implement Belle2link and TTD logic design and check the overall system perfomance

18. Than you for your attention!