1. Beam Secondary Shower Acquisition System: Igloo2 GBT Starting with LATOP version Student Meeting Jose Luis Sirvent PhD. Student 16/06/2014 1

2. 0. The Set-up 0.1 Material used and system status: 2 The modified version of GBT_FPGA for Igloo2 was finally implemented correctly @ 2.5Gbps & 5Gbps. Some constraints have to be optimized for correct operation of 5Gbps after every compilation. The GBT Firmware was finally organized & commented properly, including an error counter and Boards auto-detection. The Console Application was modified and re-structured to include the error counter. Tullio already offered a very valuable information from his measurements. Needed to verify timing details to check if we can recover the LHC clock on the front-end system. (Study the recovered Clk phase, link latency and ref frequency tolerance TX  RX).

3. 0.2 GBT-FPGA Overview in Igloo2: (Clock Management as in Virtex 6): Before with STD Version Tx_CLK (240MHz) Rx_CLK (240MHz) TX_Data_P (4.8Gbps) TX_Data_N (4.8Gbps) Rx_Word (20 bits) @ 240 MHz TX_Frame_CLK (40MHz) Data_In (84 bits) @40Mhz TX_Word_CLK (240MHz) RefCLK1_P (120MHz) RefCLK1_N (120MHz) RX_Data_P (4.8Gbps) RX_Data_N (4.8Gbps) GBT_TX Scrambler Encoder Gearbox Tx_Word (20 bits) @ 240MHz GBT_MGT SERDES_0 Vendor Specific IP TX_PLL GBT_RX Gearbox Decoder Descrambler Data_Out (84 bits) @ 40Mhz GBT_BANK (Very simplified view) RX_Frame_CLK (40MHz) RX_Word_CLK (240MHz) RX_PLL SERDES_INIT_MASTER APB_BUS 3

4. 4 0.3 Clock relationships: Understanding some details of the clock recovery

5. 0.2 GBT-FPGA Overview in Igloo2: RX_FRAME_CLK Clock alignment for Igloo2 LATOP Tx_CLK (240MHz) Rx_CLK (240MHz) TX_Data_P (4.8Gbps) TX_Data_N (4.8Gbps) Rx_Word (20 bits) @ 240 MHz TX_Frame_CLK (40MHz) Data_In (84 bits) @40Mhz TX_Word_CLK (240MHz) RefCLK1_P (120MHz) RefCLK1_N (120MHz) RX_Data_P (4.8Gbps) RX_Data_N (4.8Gbps) GBT_TX Scrambler Encoder Gearbox Tx_Word (20 bits) @ 240MHz GBT_MGT SERDES_0 Vendor Specific IP TX_PLL GBT_RX Gearbox Decoder Descrambler Data_Out (84 bits) @ 40Mhz GBT_BANK (Very simplified view) RX_Frame_CLK (40MHz) RX_Word_CLK (240MHz) RX_FRAME_ALIGNER SERDES_INIT_MASTER APB_BUS 5 RX_HEADER_FLAG BITSLIP_NUMBER Aligned RX_FRAME_CLK

6. 6 0.2 GBT-FPGA Overview in Igloo2: Taking a look at Igloo2_gbt_rx_frameclk_phalgnr This module is basically a PLL controlled through the APB Bus: Similar to SERDES Scheme Igloo2_gbt_apb_driver is re-used Only one register is modified: FCCC_PDLY_CR

7. 1. Clock Recovery (Phase) 1.1 Word_CLK (SERDES TX_CLK & RX_CLK 250Mhz @ 5Gbps) 7 Yellow  EPCS_TX_CLK (Board 1) Blue  EPCS_RX_CLK (Board 2) Test performed resetting every time the boards: Observed random phase difference There are 20 possible lock steps Possible to know RX_WORD_CLK phase based on RX_BITSLIP_NUMBER. Study the possibility to tune in the SERDES the EPCS_RX_CLK internally as done in Virtex 6 Similar results obtained when doing manual RX Reset and CDR Lock steps.

8. 8 1. Clock Recovery (Phase) 3.2 Checking that the Word_CLK’s phase difference depends of RX_BITSLIP The test were done resetting the boards: As spected there is a high correlation between RX_BITSLIP_NUMBER and the phase difference between TX_WORD_CLK (Board #1) and RX_WORD_CLK (Board #2) The Phase 0 - 2 pi corresponds to 0 - 19 bit displacement. The measurements were taken using the scope cursors, so there are some measurement error, but the tendency is clear.

9. 9 1. Clock Recovery (Phase) 1.2 Frame_CLK (41.6Mhz @ 5Gbps) Yellow  FRAME_TX_CLK (Board 1) Blue  FRAME_RX_CLK (Board 2) These are the clocks after the TX & RX PLLs, based on EPCS_ TX & RX _CLK The test were done resetting the boards: As previously the phase variation is random. This would be our CLK for acquisition electronics. The PLL can lock on any of the 6 rising edges of RX_WORD_CLK. Possible to optimize phase variations adjusting the PLL CLK phase on based on RX_HEADER_FLAG. To be seen if only adjusting Frame_Clk the link provides deterministic latency.

10. 10 1. Clock Recovery (Phase) 1.2 Frame_CLK (41.6Mhz @ 5Gbps) Yellow  FRAME_TX_CLK (Board 1) Blue  FRAME_RX_CLK (Board 2) PLL Ref_CLK: EPCS_RX_CLK Clock non aligned 6 Possible rising edges to lock 20 possible delays (0-4 ns) Random clock phase relationship  100% uncertainty PLL Ref_CLK: RX_HEADER_FLAG Clock aligned to Header_Flag 20 possible delays (0-4 ns) Phase relationship defined by Bitslip_Number  16% uncertainty PLL Ref_CLK: RX_HEADER_FLAG + Phase alignment Clock aligned to Header_Flag Delay defined by BITSLIP_NUMBER compensated Use of PLL delay lines (steps of 100ps) 20 possible delays (0 - 1.34 ns) Clocks phase more stable  5.58% uncertainty

11. 11 2. Link latency (5Gbps) 2.1 Observing the Match_Flag signal (Memory based Gearbox) Yellow  TX_MATCH_FLAG (Board 1) Blue  RX_MATCH_FLAG (Board 2) These are flags that goes to 1 when certain frame is detected The test were done resetting the boards: As expected the latency is not deterministic, with a pseudorandom behaviour. The Phase differences on the Word & Frame Clocks are the responsible of this delay variation. If we are able to align the clocks properly the link delay would be “more stable”.

12. 12 2. Link Latency (5Gbps) 2.1 Observing the TX & RX Match_Flag signals PLL Ref_CLK: EPCS_RX_CLK Using Gearbox_LATOP Link delay influenced by: FRAME CLK rising edge BITSLIP_Number Pseudo-random latency PLL Ref_CLK: RX_HEADER_FLAG Using Gearbox_STD Link delay influenced by: Elasticity of Gearbox_STD BITSLIP_Number Clear dependency of Bitslip_number (FRAME_CLK delay) Delay Variation  3.8ns PLL Ref_CLK: RX_HEADER_FLAG + Delay compensation Using Gearbox_LATOP Delay Variation  1.4ns Not reduced to 0 – 200ps ?? Clock Frequency variations?

13. 10. Code Availability 10.1 First release on Dropbox/SVN First release STD version available for download (09/06/2014) : – In DropBox: https://www.dropbox.com/sh/5sjvdkp9wwykpz4/AADrIrhM5BKj_zcraAYGxnURa – In SVN (Use Tortoise or other SVN client): https://svn.cern.ch/reps/be-bi-bl/electronics/bwsdev/studies/BWS_pCVD Diamond detector Readout Electronics/GBT_On_Igloo2/Firmware/GBT_FPGA_Igloo2/STD What is provided: – Libero 11.3 Project with GBT_on_Igloo2 code: (2014_06_09_GBT_On_Igloo2_M2GL_EVAL_KIT.rar) Features: GBT Protocol (STD Version) on Igloo2 with UART communication through USB port. Constraints are not always met, so care must be taken when new changes are performed analysing timing reports. All the necessary VHDL files are in : GBT_On_Igloo2_M2GL_EVAL_KIT\hdl Programming file (stp) available in : GBT_On_Igloo2_M2GL_EVAL_KIT\designer\GBT_On_Igloo2_M2GL_EVAL_KIT\export In case of trouble, just let me know!! jsirvent@cern.ch there are many things to improve.jsirvent@cern.ch – Console Application UART_APP_V3.0: (2014_06_09_UART_APP_V3.0.rar) Features: Controls the workflow of the GBT implementation on Igloo2 and checks different signal values and parameters of the link (RX_BITSLIP_NUMBER, Error number…), Boards Auto-Detection. Microsoft Visual Studio 2008 Project: UART_APP_V3.0\UART_APP.sln Executable File: UART_APP_V3.0\Release\UART_APP.exe – Readme File: (2014_06_09_Readme.pdf) Features: Short guide to implement the design on the Dev.Kit and run the application 13