Student Meeting Jose Luis Sirvent PhD. Student 26/05/2014 Beam Secondary Shower Acquisition System: Igloo2 GBT Implementation tests at 2.5GBPS Student Meeting Jose Luis Sirvent PhD. Student 26/05/2014

0. The Set-up 0.1 Material used and system status: The modified version of GBT_FPGA for Igloo2 was finally implemented correctly @ 2.5Gbps. I still have to solve some temporal Issues with the GBT_RX block to run @ 5Gbps (Some constraints are not met). The GBT Firmware was finally organized & commented properly, including an error counter. The Console Application was modified and re-structured to include the error counter. Tullio already offered a very valuable information from his measurements. Needed to verify timing details to check if we can recover the LHC clock on the front-end system. (Study the recovered Clk phase, link latency and ref frequency tolerance TX  RX).

0.2 GBT-FPGA Overview in Igloo2: (Clock Management as in Virtex 6) GBT_BANK (Very simplified view) SERDES_INIT_MASTER APB_BUS (PLL) Data_In (83 bits) @40Mhz Tx_Word (19 bits) @ 240MHz Tx_CLK (240MHz) GBT_TX Scrambler Encoder Gearbox GBT_MGT SERDES_0 Vendor Specific IP TX_Word_CLK (240MHz) TX_Data_P (4.8Gbps) TX_Data_N (4.8Gbps) TX_Frame_CLK (40MHz) RefCLK1_P (120MHz) RefCLK1_N (120MHz) Rx_CLK (240MHz) RX_Data_P (4.8Gbps) RX_Data_N (4.8Gbps) Rx_Word (19 bits) @ 240 MHz Data_Out (83 bits) @ 40Mhz GBT_RX Gearbox Decoder Descrambler RX_Word_CLK (240MHz) RX_Frame_CLK (40MHz) RX_PLL TX_PLL

1. Clock Recovery (Phase) 1 1. Clock Recovery (Phase) 1.1 Word_CLK (SERDES TX_CLK & RX_CLK 125Mhz @ 2.5Gbps) Yellow  EPCS_TX_CLK (Board 1) Blue  EPCS_RX_CLK (Board 2) Test performed resetting every time the boards: Observed random phase difference In theory there should be 20 steps Possible to know RX_CLK phase based on RX_BITSLIP_NUMBER. Possible to adjust phase on LATOP with an extra PLL. Similar results obtained when doing manual RX Reset and CDR Lock steps.

1. Clock Recovery (Phase) 1.2 Frame_CLK (20.8Mhz @ 2.5Gbps) Yellow  FRAME_TX_CLK (Board 1) Blue  FRAME_RX_CLK (Board 2) These are the clocks after the TX & RX PLLs, based on EPCS_ TX & RX _CLK The test were done resetting the boards: As previously the phase variation is random. This would be the clk for acquisition electronics. In theory there should be 20 steps. Possible to adjust PLL CLK phase on LATOP based on RX_BITSLIP_NUMBER (CCC offers 8 possibilities). To be seen if only adjusting Frame_Clk the link provides deterministic latency.

2. Link latency (2.5Gbps) 2.1 Observing the Match_Flag signal Yellow  TX_MATCH_FLAG (Board 1) Blue  RX_MATCH_FLAG (Board 2) These are flags that goes to 1 when certain frame is detected The test were done resetting the boards: As expected the latency is not deterministic, with a pseudorandom behaviour. The Phase differences on the Word & Frame Clocks are the responsible of this delay variation. If we are able to align the clocks properly the link delay would be “more stable”.

3. Ref. Frequency tolerance 3 3. Ref. Frequency tolerance 3.0 How to transmit LHC Clock to the Front-End Electronics LHC Clock 40.079Mhz Surface FPGA SERDES Ref_Clock 120.237Mhz Tunnel FPGA Local SERDES REF_CLK 120Mhz TX_PLL x3 Optial Link GBT @ 4.809GBPS TX_SERDES RX_SERDES LHC Clock 40.079Mhz (RX_Frame_Clock) RX_PLL /6 SERDES EPCS_RX_CLK 240.474Mhz (Word_Clock) It will be possible to work with such difference on the REF_CLKs? Do we recover the correct EPCS_RX_CLK when the REF_CLK are different? Which are the difference limits on REF_CLKs? If the TX REF_CLK varies during transmission the link will suffer errors?

3. Ref. Frequency tolerance 3 3. Ref. Frequency tolerance 3.1 Link performance over Ref_Clk differences on TX & RX LHC operations require a clock tolerance of 50 ppm (the LHC clock varies during operation). We’ll play with Ref_CLK’s around 125Mhz (the local osc of the Dev.Kit) A safe region was found for GBT @ 2.5GBPS Board 1  125.0 Mhz Board 2  125.0 ± 0.5 Mhz Difference  0.8%  8000 ppm >> 50 ppm (we met the specs) The measurements will be repeated for GBT @ 5GBPS Tullio’s measures EPCS_DEMO example @ 3.125Gbps Difference  2.3Mhz  1.8%  18400ppm Board 2 Variable Ref_CLK 123.5 – 126.5 Mhz Board 1 Local Ref_CLK 125Mhz

4. Clocks Frequency Stability 4 4. Clocks Frequency Stability 4.1 Freq Histogram of the TX & RX Word CLKs (125Mhz) Clocks Stability: For Both clocks (TX_WORD_CLK & RX_WORD_CLK), independently of their Ref_CLK source it was observed certain frequency dispersion. EPCS_TX_CLK  STDEV ~ 450kHz (3600ppm) Frame_TX_CLK  STDEV ~ 10kHz (480ppm) EPCS_RX_CLK  STDEV ~ 750kHz (6000ppm) Frame_RX_CLK  STDEV ~ 14kHz (670ppm) Possible Reasons: Not really well understood Part of the SERDES Specifications? More care needed on routing/ Implementation? Some questions to think about: Impact on system performance? Could we tolerate this variations? EPCS_TX_CLK (Board 1) EPCS_RX_CLK (Board 2) EPCS_TX_CLK (Board 2) EPCS_RX_CLK (Board 1)

4. Clocks Frequency Stability 4 4. Clocks Frequency Stability 4.2 Freq Histogram of the TX & RX Frame CLKs ( After PLLs 20.8Mhz) Clocks Stability: For Both clocks (TX_WORD_CLK & RX_WORD_CLK), independently of their Ref_CLK source it was observed certain frequency dispersion. EPCS_TX_CLK  STDEV ~ 450kHz (3600ppm) Frame_TX_CLK  STDEV ~ 10kHz (480ppm) EPCS_RX_CLK  STDEV ~ 750kHz (6000ppm) Frame_RX_CLK  STDEV ~ 14kHz (670ppm) Possible Reasons: Not really well understood Part of the SERDES Specifications? More care needed on routing/ Implementation? Some questions to think about: Impact on system performance? Could we tolerate this variations? Frame_TX_CLK (Board 1) Frame_RX_CLK (Board 2) Frame_TX_CLK(Board 2) Frame_RX_CLK (Board 1)