1. Student Meeting Jose Luis Sirvent PhD. Student 26/05/2014
Beam Secondary Shower Acquisition System: Igloo2 GBT Implementation tests at 2.5GBPS
Student Meeting
Jose Luis Sirvent
PhD. Student
26/05/2014

2. 0. The Set-up 0.1 Material used and system status:
The modified version of GBT_FPGA for Igloo2 was finally implemented 2.5Gbps.
I still have to solve some temporal Issues with the GBT_RX block to 5Gbps (Some constraints are not met).
The GBT Firmware was finally organized & commented properly, including an error counter.
The Console Application was modified and re-structured to include the error counter.
Tullio already offered a very valuable information from his measurements.
Needed to verify timing details to check if we can recover the LHC clock on the front-end system. (Study the recovered Clk phase, link latency and ref frequency tolerance TX  RX).

3. 0.2 GBT-FPGA Overview in Igloo2: (Clock Management as in Virtex 6)
GBT_BANK
(Very simplified view)
SERDES_INIT_MASTER
APB_BUS (PLL)
Data_In (83 bits)
@40Mhz
Tx_Word (19 bits)
@ 240MHz
Tx_CLK (240MHz)
GBT_TX
Scrambler
Encoder
Gearbox
GBT_MGT
SERDES_0
Vendor Specific IP
TX_Word_CLK (240MHz)
TX_Data_P (4.8Gbps)
TX_Data_N (4.8Gbps)
TX_Frame_CLK (40MHz)
RefCLK1_P (120MHz)
RefCLK1_N (120MHz)
Rx_CLK (240MHz)
RX_Data_P (4.8Gbps)
RX_Data_N (4.8Gbps)
Rx_Word (19 bits)
@ 240 MHz
Data_Out (83 bits)
@ 40Mhz
GBT_RX
Gearbox
Decoder
Descrambler
RX_Word_CLK (240MHz)
RX_Frame_CLK (40MHz)
RX_PLL
TX_PLL

4. 1. Clock Recovery (Phase) 1
1. Clock Recovery (Phase) 1.1 Word_CLK (SERDES TX_CLK & RX_CLK 2.5Gbps)
Yellow  EPCS_TX_CLK (Board 1)
Blue  EPCS_RX_CLK (Board 2)
Test performed resetting every time
the boards:
Observed random phase difference
In theory there should be 20 steps
Possible to know RX_CLK phase based on RX_BITSLIP_NUMBER.
Possible to adjust phase on LATOP with an extra PLL.
Similar results obtained when doing manual RX Reset and CDR Lock steps.

5. 1. Clock Recovery (Phase) 1.2 Frame_CLK (20.8Mhz @ 2.5Gbps)
Yellow  FRAME_TX_CLK (Board 1)
Blue  FRAME_RX_CLK (Board 2)
These are the clocks after the TX & RX
PLLs, based on EPCS_ TX & RX _CLK
The test were done resetting the boards:
As previously the phase variation is random.
This would be the clk for acquisition electronics.
In theory there should be 20 steps.
Possible to adjust PLL CLK phase on LATOP based on RX_BITSLIP_NUMBER (CCC offers 8 possibilities).
To be seen if only adjusting Frame_Clk the link provides deterministic latency.

6. 2. Link latency (2.5Gbps) 2.1 Observing the Match_Flag signal
Yellow  TX_MATCH_FLAG (Board 1)
Blue  RX_MATCH_FLAG (Board 2)
These are flags that goes to 1 when certain frame is detected
The test were done resetting the boards:
As expected the latency is not deterministic, with a pseudorandom behaviour.
The Phase differences on the Word & Frame Clocks are the responsible of this delay variation.
If we are able to align the clocks properly the link delay would be “more stable”.

7. 3. Ref. Frequency tolerance 3
3. Ref. Frequency tolerance 3.0 How to transmit LHC Clock to the Front-End Electronics
LHC Clock
40.079Mhz
Surface FPGA
SERDES Ref_Clock
Mhz
Tunnel FPGA
Local
SERDES REF_CLK
120Mhz
TX_PLL x3
Optial Link
GBPS
TX_SERDES
RX_SERDES
LHC Clock
40.079Mhz
(RX_Frame_Clock)
RX_PLL /6
SERDES EPCS_RX_CLK
Mhz
(Word_Clock)
It will be possible to work with such difference on the REF_CLKs?
Do we recover the correct EPCS_RX_CLK when the REF_CLK are different?
Which are the difference limits on REF_CLKs?
If the TX REF_CLK varies during transmission the link will suffer errors?

8. 3. Ref. Frequency tolerance 3
3. Ref. Frequency tolerance 3.1 Link performance over Ref_Clk differences on TX & RX
LHC operations require a clock tolerance of 50 ppm
(the LHC clock varies during operation).
We’ll play with Ref_CLK’s around 125Mhz (the local osc of the Dev.Kit)
A safe region was found for 2.5GBPS
Board 1  Mhz
Board 2  ± 0.5 Mhz
Difference  0.8%  8000 ppm >> 50 ppm (we met the specs)
The measurements will be repeated for 5GBPS
Tullio’s measures
EPCS_DEMO 3.125Gbps
Difference  2.3Mhz  1.8%  18400ppm
Board 2
Variable Ref_CLK
123.5 – Mhz
Board 1
Local Ref_CLK
125Mhz

9. 4. Clocks Frequency Stability 4
4. Clocks Frequency Stability 4.1 Freq Histogram of the TX & RX Word CLKs (125Mhz)
Clocks Stability:
For Both clocks (TX_WORD_CLK & RX_WORD_CLK), independently of their Ref_CLK source it was observed certain frequency dispersion.
EPCS_TX_CLK  STDEV ~ 450kHz (3600ppm)
Frame_TX_CLK  STDEV ~ 10kHz (480ppm)
EPCS_RX_CLK  STDEV ~ 750kHz (6000ppm)
Frame_RX_CLK  STDEV ~ 14kHz (670ppm)
Possible Reasons:
Not really well understood
Part of the SERDES Specifications?
More care needed on routing/ Implementation?
Some questions to think about:
Impact on system performance?
Could we tolerate this variations?
EPCS_TX_CLK (Board 1)
EPCS_RX_CLK (Board 2)
EPCS_TX_CLK (Board 2)
EPCS_RX_CLK (Board 1)

10. 4. Clocks Frequency Stability 4
4. Clocks Frequency Stability 4.2 Freq Histogram of the TX & RX Frame CLKs ( After PLLs 20.8Mhz)
Clocks Stability:
For Both clocks (TX_WORD_CLK & RX_WORD_CLK), independently of their Ref_CLK source it was observed certain frequency dispersion.
EPCS_TX_CLK  STDEV ~ 450kHz (3600ppm)
Frame_TX_CLK  STDEV ~ 10kHz (480ppm)
EPCS_RX_CLK  STDEV ~ 750kHz (6000ppm)
Frame_RX_CLK  STDEV ~ 14kHz (670ppm)
Possible Reasons:
Not really well understood
Part of the SERDES Specifications?
More care needed on routing/ Implementation?
Some questions to think about:
Impact on system performance?
Could we tolerate this variations?
Frame_TX_CLK (Board 1)
Frame_RX_CLK (Board 2)
Frame_TX_CLK(Board 2)
Frame_RX_CLK (Board 1)