Beam Secondary Shower Acquisition System: Igloo2 GBT Implementation tests at 5Gbps Student Meeting Jose Luis Sirvent PhD. Student 09/06/2014 1

0. The Set-up 0.1 Material used and system status: 2 The modified version of GBT_FPGA for Igloo2 was finally implemented 2.5Gbps & 5Gbps. Some constraints have to be optimized for correct operation of 5Gbps after every compilation. The GBT Firmware was finally organized & commented properly, including an error counter and Boards auto-detection. The Console Application was modified and re-structured to include the error counter. Tullio already offered a very valuable information from his measurements. Needed to verify timing details to check if we can recover the LHC clock on the front-end system. (Study the recovered Clk phase, link latency and ref frequency tolerance TX  RX).

0.2 GBT-FPGA Overview in Igloo2: (Clock Management as in Virtex 6) Tx_CLK (240MHz) Rx_CLK (240MHz) TX_Data_P (4.8Gbps) TX_Data_N (4.8Gbps) Rx_Word ( MHz TX_Frame_CLK (40MHz) Data_In (84 TX_Word_CLK (240MHz) RefCLK1_P (120MHz) RefCLK1_N (120MHz) RX_Data_P (4.8Gbps) RX_Data_N (4.8Gbps) GBT_TX Scrambler Encoder Gearbox Tx_Word (20 240MHz GBT_MGT SERDES_0 Vendor Specific IP TX_PLL GBT_RX Gearbox Decoder Descrambler Data_Out (84 40Mhz GBT_BANK (Very simplified view) RX_Frame_CLK (40MHz) RX_Word_CLK (240MHz) RX_PLL SERDES_INIT_MASTER APB_BUS 3

4 0.3 Clock relationships: Understanding some details of the clock recovery

1. Clock Recovery (Phase) 1.1 Word_CLK (SERDES TX_CLK & RX_CLK 5Gbps) 5 Yellow  EPCS_TX_CLK (Board 1) Blue  EPCS_RX_CLK (Board 2) Test performed resetting every time the boards: Observed random phase difference There are 20 possible lock steps Possible to know RX_WORD_CLK phase based on RX_BITSLIP_NUMBER. Study the possibility to tune in the SERDES the EPCS_RX_CLK internally as done in Virtex 6 Similar results obtained when doing manual RX Reset and CDR Lock steps.

6 1. Clock Recovery (Phase) 3.2 Checking that the Word_CLK’s phase difference depends of RX_BITSLIP The test were done resetting the boards: As spected there is a high correlation between RX_BITSLIP_NUMBER and the phase difference between TX_WORD_CLK (Board #1) and RX_WORD_CLK (Board #2) The Phase pi corresponds to bit displacement. The measurements were taken using the scope cursors, so there are some measurement error, but the tendency is clear.

7 1. Clock Recovery (Phase) 1.2 Frame_CLK 5Gbps) Yellow  FRAME_TX_CLK (Board 1) Blue  FRAME_RX_CLK (Board 2) These are the clocks after the TX & RX PLLs, based on EPCS_ TX & RX _CLK The test were done resetting the boards: As previously the phase variation is random. This would be our CLK for acquisition electronics. The PLL can lock on any of the 6 rising edges of RX_WORD_CLK. Possible to optimize phase variations adjusting the PLL CLK phase on based on RX_HEADER_FLAG. To be seen if only adjusting Frame_Clk the link provides deterministic latency.

8 2. Link latency (5Gbps) 2.1 Observing the Match_Flag signal (Memory based Gearbox) Yellow  TX_MATCH_FLAG (Board 1) Blue  RX_MATCH_FLAG (Board 2) These are flags that goes to 1 when certain frame is detected The test were done resetting the boards: As expected the latency is not deterministic, with a pseudorandom behaviour. The Phase differences on the Word & Frame Clocks are the responsible of this delay variation. If we are able to align the clocks properly the link delay would be “more stable”.

9 2. Link latency (5Gbps) 2.1 Observing the Match_Flag signal (Register based Gearbox) Yellow  TX_MATCH_FLAG (Board 1) Blue  RX_MATCH_FLAG (Board 2) These are flags that goes to 1 when certain frame is detected The test were done resetting the boards: As expected the latency is not deterministic, with a pseudorandom behaviour. The Phase differences on the Word & Frame Clocks are the responsible of this delay variation. If we are able to align the clocks properly the link delay would be “more stable”.

10 3. RX_Header Flag (5Gbps) 3.1 Comparing phase respect Board1 TX_Word_CLK (Register based Gearbox) Yellow  EPCS_TX_CLK (Board 1) Blue  RX_HEADER_FLAG (Board 2) RX_HEADER_FLAG goes to 1 when the word received contains the GBT Header. The test were done resetting the boards: As expected the latency is not deterministic, with a pseudorandom behaviour. The Phase difference on the Word clocks is the responsible of this delay variation (20 possible phases). If we are able to align the word clocks properly the phase would be constant.

11 3. RX_Header Flag (5Gbps) 3.2 Checking that the RX_HEADER phase depends of RX_BITSLIP The test were done resetting the boards: As done previously with the WORD_CLK’s. RX_HEADER_CLK status it’s updated on the RX_WORD_CLK rising edges. RX_HEADER_CLK is used as a sync signal in Virtex 6 version to align RX_FRAME_CLK (in that example the RX_WORD_CLK has been already tunned based on BITSLIP_Number) Measurements taken more carefully, clear tendency.

12 3. Ref. Frequency tolerance 3.0 How to transmit LHC Clock to the Front-End Electronics TX_PLL x3 LHC Clock Mhz TX_SERDES SERDES Ref_Clock Mhz Surface FPGA Tunnel FPGA RX_SERDES RX_PLL /6 Optial Link 4.809Gbps SERDES EPCS_RX_CLK Mhz (Word_Clock) Local SERDES REF_CLK 120Mhz LHC Clock Mhz (RX_Frame_Clock) It will be possible to work with such difference on the REF_CLKs? Do we recover the correct EPCS_RX_CLK when the REF_CLK are different? Which are the difference limits on REF_CLKs? If the TX REF_CLK varies during transmission the link will suffer errors?

LHC operations require a clock tolerance of 50 ppm (the LHC clock varies during operation). We’ll play with Ref_CLK’s around 125Mhz (the local osc of the Dev.Kit) A safe region was found for 2.5 & 5Gbps – Board 1  Mhz – Board 2  ± 0.5 Mhz – Difference  0.8%  8000 ppm >> 50 ppm (we met the specs) Tullio’s measures – EPCS_DEMO 3.125Gbps – Difference  2.3Mhz  1.8%  18400ppm Ref. Frequency tolerance 3.1 Link performance over Ref_Clk differences on TX & RX Board 1 Local Ref_CLK 125Mhz Board 2 Variable Ref_CLK – Mhz

LHC operations require a clock tolerance of 50 ppm (the LHC clock varies during operation). We’ll play with Ref_CLK’s around 125Mhz (the local osc of the Dev.Kit) A safe region was found for 2.5 & 5Gbps – Board 1  Mhz – Board 2  ± 0.5 Mhz – Difference  0.8%  8000 ppm >> 50 ppm (we met the specs) Tullio’s measures – EPCS_DEMO 3.125Gbps – Difference  2.3Mhz  1.8%  18400ppm Ref. Frequency tolerance 3.1 Link performance over Ref_Clk differences on TX & RX Board 1 Local Ref_CLK 125Mhz Board 2 Variable Ref_CLK – Mhz SAME RESULTS FOR 2.5 AND 5Gbps

4. Clocks Frequency Stability 4.1 Freq Histogram of the TX & RX Word CLKs (250Mhz) 15 EPCS_TX_CLK (Board 1) EPCS_RX_CLK (Board 2) EPCS_TX_CLK (Board 2) EPCS_RX_CLK (Board 1) Clocks Stability: – For Both clocks (TX_WORD_CLK & RX_WORD_CLK), independently of their Ref_CLK source it was observed certain frequency dispersion. – EPCS_TX_CLK  STDEV ~ 6.1MHz (24400ppm) – Frame_TX_CLK  STDEV ~ 41kHz (985ppm) – EPCS_RX_CLK  STDEV ~ 3.7MHz (14800ppm) – Frame_RX_CLK  STDEV ~ 45kHz (1081ppm) Possible Reasons: – Not really well understood – Part of the SERDES Specifications? – More care needed on routing/ Implementation? Some questions to think about: – Impact on system performance? – Could we tolerate this variations?

16 4. Clocks Frequency Stability 4.2 Freq Histogram of the TX & RX Frame CLKs ( After PLLs 41.6Mhz) Frame_TX_CLK (Board 1) Frame_RX_CLK (Board 2) Frame_TX_CLK(Board 2) Frame_RX_CLK (Board 1) Clocks Stability: – For Both clocks (TX_WORD_CLK & RX_WORD_CLK), independently of their Ref_CLK source it was observed certain frequency dispersion. – EPCS_TX_CLK  STDEV ~ 6.1MHz (24400ppm) – Frame_TX_CLK  STDEV ~ 41kHz (985ppm) – EPCS_RX_CLK  STDEV ~ 3.7MHz (14800ppm) – Frame_RX_CLK  STDEV ~ 45kHz (1081ppm) Possible Reasons: – Not really well understood – Part of the SERDES Specifications? – More care needed on routing/ Implementation? Some questions to think about: – Impact on system performance? – Could we tolerate this variations?

5.Compilation Resources 5.1 Comparison using Memory & Register Based Gearbox 17 Memory Gearbox Register Gearbox The GBT_FPGA code was compiled successfully on Igloo2 with the following resource usage: No Clock phase alignment has been yet performed for the on Igloo2 version. Implemented using the two gearbox possibilities for clock domain crossing available on Xilinx V6 example. Memory Gearbox : Two port memories Register Gearbox : Through registers. Still TODO  Aligh the FRAME_CLK (We’ll use the information of BITSLIPNUMBER for and RX_HEADER_FLAG for this)

6.Constraining the Design 6.1 On Synplify Pro & on Smart-Time editor. 18 Constraint Clocks on Synplify  Constraint Clocks on Smart-Time  Allow multicycle path on Smart-Time  (Between TX_FIFO & SERDES_TX)

7. Synchronizing SERDES, GBT_TX & GBT_TX 7.1 Maintaining CLK/DATA phase relationships Buffered EPC_TX_CLK & EPCS_RX_CLK: – Helps to reduce the CLK Skew through global resources (and therefore reach the clk contraints for GBT_TX & GBT_RX) – However the buffered VS non buffered versions have different phases. – Synchronization CLK / TX & RX Data needed (done through Registers & Delay lines)  File: Igloo2_gtx_std.vhdl 19 Adjustable Delay

7. Synchronizing SERDES, GBT_TX & GBT_TX 7.1 Maintaining CLK/DATA phase relationships Buffered EPC_TX_CLK & EPCS_RX_CLK: – Helps to reduce the CLK Skew through global resources (and therefore reach the clk contraints for GBT_TX & GBT_RX) – However the buffered VS non buffered versions have different phases. – Synchronization CLK / TX & RX Data needed (done through Registers & Delay lines)  File: Igloo2_gtx_std.vhdl 20 MultiCycle path (2) Adjustable Delay

7. Synchronizing SERDES, GBT_TX & GBT_TX 7.1 Maintaining CLK/DATA phase relationships Buffered EPC_TX_CLK & EPCS_RX_CLK: – Helps to reduce the CLK Skew through global resources (and therefore reach the clk contraints for GBT_TX & GBT_RX) – However the buffered VS non buffered versions have different phases. – Synchronization CLK / TX & RX Data needed (done through Registers & Delay lines)  File: Igloo2_gtx_std.vhdl 21 MultiCycle path (2) Adjustable Delay EPCS_TX_CLK EPCS_TX_CLK_BUFF GBT_TX_DATA GBT_TX_DATA_DELAYED

7. Synchronizing SERDES, GBT_TX & GBT_TX 7.1 Maintaining CLK/DATA phase relationships Buffered EPC_TX_CLK & EPCS_RX_CLK: – Helps to reduce the CLK Skew through global resources (and therefore reach the clk contraints for GBT_TX & GBT_RX) – However the buffered VS non buffered versions have different phases. – Synchronization CLK / TX & RX Data needed (done through Registers & Delay lines)  File: Igloo2_gtx_std.vhdl 22 MultiCycle path (2) Adjustable Delay EPCS_RX_CLK EPCS_RX_CLK_BUFF EPCS_RX_DATA EPCS_RX_DATA_DELAYED MultiCycle path (2)

8.Post-Layout Implementation Report 8.1 Reaching constraints 23 We have to play a bit with the TX & RX delays in order to respect the Max & Min Delay analysis on Smart- Time. NOTE!!: Every time the design is changed and a new compilation (and place&route) is performed the reports change, and under certain circumstances the constraints are not met, therefore more efforts on design constraining have to be done (or we leave this as it is now and we’ll take care about it later when the final design is done). – Currently Experimenting some difficulties to reach the constraints… new ideas are more than welcome!! – Maybe bigger Igloo2 FPGAS with more resources can reach easier the specifications (or at least with bigger margin). – In spite of the limitations on WORD_CLK_RX & WORD_CLK_TX at 241Mhz / 246Mhz, the test were done at 250Mhz with satisfactory result.

9. The beauty is inside: GBT_On_Igloo2 9.1 All the implementation 24 PLLs

9. The beauty is inside: GBT_On_Igloo2 9.2 GBT_BANK 25

9. The beauty is inside: GBT_On_Igloo2 9.3 GBT_BANK/GBT_TX 26

9. The beauty is inside: GBT_On_Igloo2 9.4 GBT_BANK/GBT_RX 27

9. The beauty is inside: GBT_On_Igloo2 9.5 GBT_BANK/MGT 28 Global Clock Network

10. Code Availability 10.1 First release on Dropbox/SVN First release STD version available for download (09/06/2014) : – In DropBox: – In SVN (Use Tortoise or other SVN client): Diamond detector Readout Electronics/GBT_On_Igloo2/Firmware/GBT_FPGA_Igloo2/STD What is provided: – Libero 11.3 Project with GBT_on_Igloo2 code: (2014_06_09_GBT_On_Igloo2_M2GL_EVAL_KIT.rar) Features: GBT Protocol (STD Version) on Igloo2 with UART communication through USB port. Constraints are not always met, so care must be taken when new changes are performed analysing timing reports. All the necessary VHDL files are in : GBT_On_Igloo2_M2GL_EVAL_KIT\hdl Programming file (stp) available in : GBT_On_Igloo2_M2GL_EVAL_KIT\designer\GBT_On_Igloo2_M2GL_EVAL_KIT\export In case of trouble, just let me know!! there are many things to – Console Application UART_APP_V3.0: (2014_06_09_UART_APP_V3.0.rar) Features: Controls the workflow of the GBT implementation on Igloo2 and checks different signal values and parameters of the link (RX_BITSLIP_NUMBER, Error number…), Boards Auto-Detection. Microsoft Visual Studio 2008 Project: UART_APP_V3.0\UART_APP.sln Executable File: UART_APP_V3.0\Release\UART_APP.exe – Readme File: (2014_06_09_Readme.pdf) Features: Short guide to implement the design on the Dev.Kit and run the application 29