A. Blas 09 Jan 2009 DSP BC Daughter cards1/21 DSP beam control Overview LEIR BC From M.E. Angoletta

A. Blas 09 Jan 2009 DSP BC Daughter cards2/21 DSP Board 6 unit VME board with: 1 DSP - ADSP-21160M from Analog Devices – 32 bit floating point – Super Harvard Architecture (SHARC), Clocked at 80 MHz (40 MHz external), 600 MFLOPS - with separate data and instruction busses, dedicated I/O processor with separate bus - 64 bit external data bus clocked at 40 MHz (320 MB/s), 6 byte link Ports at 80 MHz (80 MB/s), 2 serial ports at 40 Mbits/s. 1 MB Flash memory, to restore the DSP software. 4 MB DSP RAM, to store external or calculated functions (like voltage and rf program, steering …) 4 daughter board sites, with two 69 pin connectors for each site 1 Timing interface, based on an AlteraTM FPGA 1 VME interface, based on an AlteraTM FPGA 1 Data interface, based on an AlteraTM FPGA that controls the data flux on Board. 4 MB RAM, for acquisition of internal signals (diagnostics) 16 Trigger inputs (via the RTM) 1 Event link, basically a serial link connected to the rear transition module that allows taking into account 256 events with a hundred ns resolution (not used at CERN). 6 link ports, each port allows up to 8 bit data transfer per DSP internal clock cycle (80 MHz)

A. Blas 09 Jan 2009 DSP BC Daughter cards3/21 MDDS Master DDS daughter card AD9858 1GHz clock DDS ADF4106 PLL + CLV1000A VCO (10MHz => 1 GHz) No SRAM Altera Stratix EP1S10F484C5 2 Tagged clock outputs with IEEE1394 mechanical standard Tagging inside the FPGA with external ELMEC 1.2ns delay (needs control of logical cells placement) Designed in November 2004

A. Blas 09 Jan 2009 DSP BC Daughter cards4/21 IEEE 1394b clock distribution 1394b bilingual connector 8 x 5 mm 1.5 ns tag  333 Ms/s equivalent throughput Standard used up to 3.2 Gb/s over 5m cables 9 pin connector LVDS electrical standard

A. Blas 09 Jan 2009 DSP BC Daughter cards5/21 Clock Fan-out VME card 1 input with IEEE1394 mechanical standard 1 Optical input / 1 optical output 9 outputs IEEE1394 Delay error between channels below 390 ps Designed in June 2005

A. Blas 09 Jan 2009 DSP BC Daughter cards6/21 DDC Digital Down Converter daughter card 4 channels 14 bit ADC AD 9245BCP-80 (80 MHz max, 366 mW, pipeline = 7 cycles) 2 MB/10 ns SRAM (1 M x 16 bits) 1 Altera Stratix EP1S20F484C5 1 Tagged clock input with IEEE1394b mechanical standard and de-tagging and divide-by-2 circuit (50% duty cycle and minimum pulse width required by the ADC).

A. Blas 09 Jan 2009 DSP BC Daughter cards7/21 DDC Digital Down Converter daughter card – FPGA content 1.4 down mixers with 4 different LO frequencies 2.Programmable CIC, 0 -> 3 stages, 1 ->15 decimation, 1 -> 255 comb delay, normalization gain 3.CIC, FIR and LO parameters loaded synchronously with a specific double-tag. Allows on-flight MDDS harmonic change 4.I/Q frequency discriminator 5.LO frequency measurement (counter) 6.4 different acquisition lines memorizing 4 of 28 different inner signals. Sampling at a chosen sub- harmonic of the main clock. Up to 1 M points per signal at a maximum sampling of 80 MHz. 7.Channel, CIC, FIR ON/OFF 8.Reset phase accumulator at each tag 1 channel / 4

A. Blas 09 Jan 2009 DSP BC Daughter cards8/21 SDDS Slave DDS daughter card 4 channel 14 bit DAC AD 9754BCP-80 (125 MHz max, 185 mW, no pipeline delay) 2 MB/10 ns SRAM (1 M x 16 bits) Altera Stratix EP1S20F484C5 1 Tagged clock input with IEEE1394 mechanical standard and de-tagging and divide-by-2 circuit (50% duty cycle and minimum pulse width required by the ADC). High/Low DAC gain (0 – 18 dB)  17 bit DAC for lowest amplitude signals

A. Blas 09 Jan 2009 DSP BC Daughter cards9/21 SDDS Slave DDS daughter card 1.4 up-converters with 4 different LO frequencies 2.LO parameters loaded synchronously with a specific double-tag. Allows on-flight MDDS harmonic change 3.LO frequency measurement (counter) 4.4 different acquisition lines memorizing 4 of 28 different inner signals. Sampling at a chosen sub-harmonic of the main clock. Up to 1 M points per signal at a maximum sampling of 80 MHz. 5.Channel ON/OFF 6.Reset phase accumulator at each tag 7.Automatic DAC gain setting with respect to required output amplitude 8.Phase modulation from SRAM

Stratix 1 FPGA filling-up ! DDC with only 2/4 LO’s + only 2 nd order FIR And no monitoring of inner signals Compiled with an old version of Quartus; now it is much faster A. Blas 09 Jan 2009 DSP BC Daughter cards 10/21

A. Blas 09 Jan 2009 DSP BC Daughter cards11/21 Daughter card – FPGA interconnections (Here the DDC)

A. Blas 09 Jan 2009 DSP BC Daughter cards12/21 FPGA market (to be completed, very fast survey!) Xilinx Virtex 2 XC2V2000-6FG676C (Tuner loop) Altera Stratix EP1S20F484C5 (Leir daughter cards) Xilinx Virtex 5 XC5VLX110 1FF676I 1FF676IAltera Stratix III EP3SL150F78 0C4N Altera Cyclone EP3C80F780C6 Logic cells 24,19218,460110,592142,00081,284 Clock frequency < 269 MHz < 450 MHz (-1) < 550 MHz (-3 speed grade < 450 MHz < 600 MHz (C2 version) <340 MHz RAM [kb] 1,0001, ,5002, x 18 multipliers I/O Price [USD] (Avnet) 2184 (Altera) 368 Serial links 177 LVDS channels (840 Mbps each)

A. Blas 09 Jan 2009 DSP BC Daughter cards13/21 FPGA market (to be completed, very fast survey!)

A. Blas 09 Jan 2009 DSP BC Daughter cards14/21 DC to Mother Board data exchange  The DSP on the motherboard sends/receives values to/from registers on the Daughter cards.  Some of these register values are within a servo loop and the acquisition time is important for the stability of this loop.  In the present Leir BC, the data flow within a loop is as follow: 1. The DDC measures I/Q of a variable 2. I/Q is read by the DSP which processes the value taking into account values from other channels, other daughter cards and other DSP boards if required. The DSP process is triggered by a so-called fast clock. 3. A correction value is sent to the MMDS to change the rf clock frequency and possibly parameters of the SDDS

A. Blas 09 Jan 2009 DSP BC Daughter cards15/21 DC to Mother Board data exchange

A. Blas 09 Jan 2009 DSP BC Daughter cards16/21 Loop delay Rule of thumb: The loop will be sufficiently stable if its delay leads to a phase lag <  /4 at the unity loop gain frequency  Loop computation time + Hdw delay< [1/(8.Fmod)] < 21 us for PSB & LEIR (3.Fs=6kHz) < 26 us for PS (3.Fs=4.8 kHz) In LEIR, the in-loop DSP is sampling the data every T S-DSP = 12.5 us (80 kHz). The loop delay within the DSP = import data from DDC ( ≈14us delay within the DSP With a 80 kHz DSP sampling clock, an averaging (CIC) of MHz-samples in the DDC would be adequate. We actually use 256, which means <6.4 us extra delay. This means that we are approaching the reasonable limits required for the LEIR and PSB phase loop. The DSP process time in LEIR is the most time consuming and multiplying by a factor 2 this process speed would almost double the possible bandwidth.

A. Blas 09 Jan 2009 DSP BC Daughter cards17/21 Phase jitter within the Loop The error signal within a loop as described in the previous slide is transferred from the rf clock domain (DDC) to another clock domain (DSP) and again into another clock domain (MDDS or SDDS). From DDC to DSP the uncertainty is 1 rf clock period (<25 ns)+ 1 wait state of the DSP (25 ns) The induced phase jitter is (+/- 360 o x F MOD ) x Δt i = +/- 18 m o / kHz of modulation From DDC to MDDS the uncertainty is just 1 MDDS clock period (8 ns)(the 125 MHz on the frequency word acquisition side) The total phase jitter within the loop is thus around +/- 21 m o / kHz of modulation To this value should be added-up the effect of any change in the DSP computation time from one DSP interrupt clock tic to the following and also all the erratic wait states of the DSP acquisitions that occur in between this DDC acquisition to the MDDS frequency change.

A. Blas 09 Jan 2009 DSP BC Daughter cards18/21 Communication links With the new motherboard, the Daughter cards will communicate with the DSP via a FPGA. This architecture allows the use of many kinds of communication links, serial or parallel. VME BUS FPGA RAM Special Function Board DSP Special Function Board Special Function Board Other DSP Board Other DSP Board Other DSP Board Other DSP Board Do we want the “Daughter card” concept, or do we use detached “special function” boards? Do we use one dedicated link for each daughter card or do we share a single link (all cards receiving the interrupt clock)? Special Function Board Special Function Board Special Function Board Other DSP Board Other DSP Board Other DSP Board Other DSP Board

A. Blas 09 Jan 2009 DSP BC Daughter cards19/21 DC to Mother Board data flow Samtec QSE, 80 (4 small ribons of 20 coax cables on a single connector 160 Gb/s (at 2 GHz) 3.2 Gb/s (at 40 MHz) Used in LHC Gigabit serial link with 8b/10b coding 16bit// 50 MHz -> 1Gb/s serial 800Mb/s Full duplex at the same rate Used in LHC Very common standard DSP bus + mother board to daughter card bus at 40 MHz and 32 bit word: 1.28 Gb/s (to be divided by the number of wait states +2 = 4) 320 Mb/sUsed in Leir DSP DC Link port clocked at 40 MHz (can be used at 80 MHz) 320 Mb/sUsed in Leir DSP Single 32b transfer: < 25ns+25ns = 50 ns with // DSP bus < 32/800MHz + 20 ns + header ? = 60 ns + 10ns? with Gigabit 8b/10b link

A. Blas 09 Jan 2009 DSP BC Daughter cards20/21 Conclusion  The FPGA on the daughter cards needs to be upgraded (except MDDS). It could be the same as the one used on the DSP board (market survey !!)  Finding ADCs and DACs not-requiring 50% duty cycle would allow an increase of the rf sampling clock  Anti-alias filter off-Board or with an easy plug-in / plug-out connector  JTAG connector from VME and Front-Panel  The clock distribution connectors should be more robust, same as for the data serial link (market survey !!)  The data link from DC to FPGA could be serial  Having an ADC (resp. DAC) on a DAC (resp. ADC) board would be convenient and it would allow to use the board for a 1TFB and a TFB (there is a developer available for this job!)

A. Blas 09 Jan 2009 DSP BC Daughter cards21/21 Conclusion Possible architecture