1. Improving IEEE 1588 synchronization accuracy in 1000BASE-T systems
Rodney Greenstreet
Alejandro Zepeda
9th White Rabbit Workshop

2. Data Redundancy using Ring Topology
Ring Topologies are common in industrial and automation applications
Bridged
Device
Often, Ring-topologies are deployed in industrial settings for resiliency
Show GM and slaves in PTP
Show SyncE clock is not aligned and needs to be. Link is taken down to align.
This may be tolerable during configuration.
However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down.
Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

3. Data Redundancy using Ring Topology
Ring Topologies are common in industrial and automation applications
Dataflow is replicated and eliminated for redundancy (IEEE CB)
Source
Bridged
Device
Destination
Often, Ring-topologies are deployed in industrial settings for resiliency
Show GM and slaves in PTP
Show SyncE clock is not aligned and needs to be. Link is taken down to align.
This may be tolerable during configuration.
However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down.
Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

4. Data Redundancy using Ring Topology
Ring Topologies are common in industrial and automation applications
Dataflow is replicated and eliminated for redundancy (IEEE CB)
A broken link does not disrupt dataflow to destination
Source
Bridged
Device
Destination
Often, Ring-topologies are deployed in industrial settings for resiliency
Show GM and slaves in PTP
Show SyncE clock is not aligned and needs to be. Link is taken down to align.
This may be tolerable during configuration.
However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down.
Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

5. 1588 Redundancy using Ring Topology
Best Master Clock Algorithm (BMCA)
Chooses Grand Master (GM)
Creates synchronization spanning tree GM
Master
Slave
Passive
Bridged
Device
Ring-topologies are typical in Industrial use cases
Show GM and slaves in PTP
Show SyncE clock is not aligned and needs to be. Link is taken down to align.
This may be tolerable during configuration.
However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down.
Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

6. 1588 Redundancy using Ring Topology
Best Master Clock Algorithm (BMCA)
Chooses Grand Master (GM)
Creates synchronization spanning tree
Rearranges spanning tree if a break occurs GM
Master
Slave
Passive
Bridged
Device
Ring-topologies are typical in Industrial use cases
Show GM and slaves in PTP
Show SyncE clock is not aligned and needs to be. Link is taken down to align.
This may be tolerable during configuration.
However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down.
Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

7. 1588 Redundancy using Ring Topology
Best Master Clock Algorithm (BMCA)
Chooses Grand Master (GM)
Creates synchronization spanning tree
Rearranges spanning tree if a break occurs GM
Master
Slave
Bridged
Device
Ring-topologies are typical in Industrial use cases
Show GM and slaves in PTP
Show SyncE clock is not aligned and needs to be. Link is taken down to align.
This may be tolerable during configuration.
However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down.
Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

8. 1588 Redundancy W/SyncE using Ring Topology
SyncE clocks must be aligned to synchronization spanning tree GM
Master
Slave
Passive
Bridged
Device
Ring-topologies are typical in Industrial use cases
Show GM and slaves in PTP
Show SyncE clock is not aligned and needs to be. Link is taken down to align.
This may be tolerable during configuration.
However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down.
Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

9. 1588 Redundancy W/SyncE using Ring Topology
SyncE clocks must be aligned to synchronization spanning tree
PHYs must be configured for clock alignment
Relink is required GM
Master
Slave
Passive
Bridged
Device
Ring-topologies are typical in Industrial use cases
Show GM and slaves in PTP
Show SyncE clock is not aligned and needs to be. Link is taken down to align.
This may be tolerable during configuration.
However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down.
Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

10. 1588 Redundancy W/SyncE using Ring Topology
SyncE clocks must be aligned to synchronization spanning tree
PHYs must be configured for clock alignment
Relink is required
If a break occurs
BMCA rearranges spanning tree
SyncE realignment requires links to be reestablished, breaking redundancy GM
Master
Slave
Passive
Bridged
Device
Ring-topologies are typical in Industrial use cases
Show GM and slaves in PTP
Show SyncE clock is not aligned and needs to be. Link is taken down to align.
This may be tolerable during configuration.
However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down.
Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

11. IEEE 1588 Timestamp Model
Reduce error in 1588 event message timestamps
Timestamp is captured when the event message crosses between the node and network
Timestamp point is the first symbol after the start of frame delimiter (SFD)
PHY
Hardware Assist
IEEE 1588
(Application Layer) OS
MAC
Network
Preamble/ SFD DA
Message Timestamp
Point
Time
OSI Model
Media Dependent Interface (MDI)

12. NI’s HW Timestamp Model for 1000BASE-T
Key components of HW model for timestamping event messages
Generic
PHY
Timestamp
Capture
Circuits
GMII
(125 MHz)
MDI
(125 MHz)
OSC
25 MHz
MAC
PHY TS
GTX_CLK
RX DATA
TX DATA
RX_CLK
Network
Timekeeper
(125 MHz)

13. NI’s HW Timestamp Model for 1000BASE-T
Key components of HW model for timestamping event messages
Account for delays between timestamp circuitry and MDI
Fixed delays (e.g. propagation delays)
Link-to-link delays (e.g. link delay skew)
Variable delays (e.g. clock crossings)
Ingress/Egress
Delays
MDI
(125 MHz)
OSC
25 MHz
MAC
PHY TS
GTX_CLK
RX DATA
TX DATA
RX_CLK
Network
Preamble/ SFD DA
Message Timestamp
Point
Time
Bob Noseworthy
Fixed delays 10s of ns

14. 1000BASE-T Link to Link Delay
1000BASE-T media
4 channels of twisted pair copper cabling
Maximum length is 100 meters
Maximum skew between channels is 50 ns
Managing link delay skew
During link training state, PHY transmits IDLE symbols aligned across channels
PHY receiver recovers IDLE symbols and employs deskew FIFOs to realign symbols
Dual-Duplex transmission
Electrically, skew is the same in both directions
Two linked PHYs may choose different deskew parameters, causing asymmetry
Deskew parameters may be accessible through PHY management interface TX
MDI
Length ≤ 100 meters RX
MDI A B C D
TSKEW ≤ 50 ns
Realign to achieve a coherent symbol
Shortest channel longest delay
Deskew
FIFOs

15. 1000BASE-T Clock Architecture
During auto-negotiation, link master and link slave are assigned
Link master generates media dependent TX clock
Link slave recovers clock from incoming data
Link slave transmits using recovered clock
Link master recovers clock from incoming data
Link Master
Link Slave
MAC
PHY TS
GTX_CLK
RX DATA
TX DATA
RX_CLK
OSC
25 MHz
PLL
PLL
Implications for timestamping

16. 1000BASE-T Clock Crossings
Link master generates MDI TX clock
Clock crossing between GTX_CLK and MDI TX clock
MAC
PHY TS
GTX_CLK
RX DATA
TX DATA
RX_CLK
OSC
25 MHz
Link Master
Link Slave
PLL
Clock Crossing
SOF
Sent 𝝓
Fixed Delay
GTX_CLK
MDI TX_CLK
Actual
SOF

17. 1000BASE-T Clock Crossings
Link slave recovers MDI clock and exports as GMII RX_CLK
Clock crossing between RX_CLK and timekeeper clock ( TK_CLK)
Continuously measure phase relationship between RX_CLK and TK_CLK
On detection of SOF, calculate phase offset to previous RX_CLK edge (actual SOF)
MAC
PHY TS
GTX_CLK
RX DATA
TX DATA
RX_CLK
OSC
25 MHz
Link Master
Link Slave
PLL
Clock Crossing
On detection of SOF, adjust timestamp
Measuring phase between two asynchronous clocks key
Actual
SOF 𝝓
RX_CLK
TK_CLK
SOF
Detected

18. 1000BASE-T Clock Crossings
Link slave uses recovered MDI clock as MDI TX clock
Clock crossing between GTX_CLK and MDI TX clock
Continuously measure phase relationship between RX_CLK and TK_CLK
On detection of SOF, calculate phase offset to next RX_CLK edge (actual SOF)
MAC
PHY TS
GTX_CLK
RX DATA
TX DATA
RX_CLK
OSC
25 MHz
Link Master
Link Slave
PLL
Clock Crossing
Actual
SOF 𝝓
RX_CLK
TK_CLK
SOF
Sent

19. 1000BASE-T Clock Crossings
Link master recovers MDI clock and exports as GMII RX_CLK
Clock crossing between RX_CLK and timekeeper clock (TK_CLK)
Continuously measure phase relationship between RX_CLK and TK_CLK
On detection of SOF, calculate phase offset to previous RX_CLK edge (actual SOF)
MAC
PHY TS
GTX_CLK
RX DATA
TX DATA
RX_CLK
OSC
25 MHz
Link Master
Link Slave
PLL
Clock Crossing
Actual
SOF 𝝓
RX_CLK
TK_CLK
Phase offset varies
with cable length
Detected
SOF

20. DDMTD introduction
Signal mixers have been used widely for over a century to shift frequencies around.
Input (ƒi, 𝜙i)
Local oscillator (LO) (ƒLO, 𝜙LO)
(ƒi − ƒLO ,𝜙i −𝜙LO )
(ƒi + ƒLO ,𝜙i +𝜙LO )
Signals can also be mixed digitally, using a common register (flip-flop). Only the low-frequency signal is output.
Input (ƒi, 𝜙i)
Local oscillator (LO) (ƒLO, 𝜙LO)
Deglitcher
(ƒi − ƒLO ,𝜙i −𝜙LO )

21. Phase Measurement using DDMTD
As its name implies, a DDMTD (dual digital mixer time difference) uses two digital mixers to enable better measurements of phase offset
ƒ = ƒ1 – ƒLO
𝜙 = 𝜙1 – 𝜙LO
(ƒ1 , 𝜙1)
Deglitcher
Phase Compare
(ƒ2 , 𝜙2)
Deglitcher
ƒ = ƒ2 – ƒLO
𝜙= 𝜙2 – 𝜙LO
(ƒLO ,𝜙LO)

22. Dual Mixer Time Difference (DMTD)
125 MHz
Timekeeper
2 ns
RX Clk
𝝓 = 90°

23. Dual Mixer Time Difference (DMTD)
(ƒ1 , 𝜙1)
(ƒ2 , 𝜙2)
(ƒLO ,𝜙LO)
Deglitcher
Timekeeper (125 MHz)
RX Clk (125 MHz)
LO Clk (112.5 MHz) ns 8 16 24 32 40 48 56 64 72 80 88
TK_DownMixed
RxClk_DownMixed

24. Dual Mixer Time Difference (DMTD)
(ƒ1 , 𝜙1)
(ƒ2 , 𝜙2)
(ƒLO ,𝜙LO)
Deglitcher ns 8 16 24 32 40 48 56 64 72 80 88
Timekeeper (125 MHz)
RX Clk (125 MHz)
LO Clk (112.5 MHz)
TK_DownMixed
RxClk_DownMixed
~18 ns

25. Dual Mixer Time Difference (DMTD)ns 8 16 24 32 40 48 56 64 72 80 88
Timekeeper (125 MHz)
RX Clk (125 MHz)
LO Clk ( MHz)
LO Clk (112.5 MHz)

26. Dual Mixer Time Difference (DMTD)μs 10 20 30 40 50 60 70 80 90
100
Timekeeper (125 MHz)
RX Clk (125 MHz)
LO Clk ( MHz)
TK_DownMixed
20 μs
RxClk_DownMixed
𝝓 = 90°

27. Phase measurement of non-locked clocks
DDMTD can be used to measure the phase offset even if the clocks are not locked
Link Master
Link Slave
Clock Crossing
PHY
PHY
MAC
MAC
GTX_CLK
RX_CLK TS TS
TX DATA
RX DATA
PLL
RX DATA
TX DATA TS TS
RX_CLK
GTX_CLK
OSC
25 MHz
Actual
SOF 𝝓
RX_CLK
TK_CLK
SOF
Detected

28. Calculate Frequency of Down Mixed Clocks
Select set of timestamps of down-mixed clocks around SOF timestamp Calculate time delta between each successive timestamp within set Average time deltas to reduce jitter error Calculate frequency 𝑓 𝑑𝑜𝑤𝑛𝑀𝑖𝑥𝑒𝑑 = 1 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 Δt
SOF TS
Timekeeper Δt Δt Δt
TK_DownMixed Δt Δt Δt
RxClk_DownMixed

29. Calculate Reference Edge of Down Mixed Clocks
Average same set of timestamps of each down-mixed clock
Reduces jitter error
Identifies reference edge
SOF TS
TK_Clk
TK_DownMixed
RxClk_DownMixed

30. Calculate phase to SOF
Calculate the phase from each down-mixed clock reference edge to the SOF timestamp, based on the calculated frequency for that clock. Calculate the phase offset between the down-mixed clocks
SOF TS
TK_Clk
TK_DownMixed
RxClk_DownMixed
𝝓𝑅𝑥𝐶𝑙𝑘𝐷𝑜𝑤𝑛𝑀𝑖𝑥𝑒𝑑
𝝓𝑇𝐾_𝐶𝑙𝑘𝐷𝑜𝑤𝑛𝑀𝑖𝑥𝑒𝑑
𝝓𝑅𝑥𝐶𝑙𝑘_𝑇𝐾_𝐶𝑙𝑘

31. Calculate phase offset near SOF
Phase offset between MDI clock and timekeeper clock around the SOF is the same as the calculated phase offset between the down-mixed clocks:
SOF @ MDI
SOF TS
TK_CLK
𝝓𝑅𝑥𝐶𝑙𝑘−𝑇𝐾_𝐶𝑙𝑘
RxClk
𝛥𝑡 𝑅𝑋_𝑇𝐾

32. Synchronization performance tests
FPGA with
1000BASE-T MAC +
Timekeeper
Standard
1000BASE-T
PHY
100m
Two PXI-6683 boards synchronizing via IEEE 1588 with 250ms sync interval with a 100m cable, configured to generate a PPS.
Offset between PPSs is recorded with oscilloscope every second for 12 hours (43,200 samples).

33. Test A: Baseline test results
12-hour test with PXI-6683 boards with 8ns timestamp resolution
Test
Peaks from mean
Std. Dev. A
±13.4 ns
2.66 ns

34. Test B: increased timestamping resolution
Test B uses a technique very similar to the one described in ISPCS 2014 paper "Precise clock parameter estimation and ground truth capture for clock error measurements using FPGAs" to increase timestamp resolution to ~0.5ns.
Test
Peaks from mean
Std. Dev. A
±13.4 ns
2.66 ns B
±3.64 ns
0.75 ns

35. Test C: timestamps enhanced with DDMTD
Test C implements the techniques described in this presentation to enhance PTP packet timestamps
Test
Peaks from mean
Std. Dev. A
±13.4 ns
2.66 ns B
±3.64 ns
0.750 ns C
±0.460 ns
0.104 ns

36. Even better results with shorter sync interval
By decreasing the Sync Message Interval to 125 ms the synchronization performance is further improved
Test
Peaks from mean
Std. Dev. A
±13.4 ns
2.66 ns B
±3.64 ns
0.75 ns C
±0.460 ns
0.104 ns D
±0.320 ns
0.077 ns

37. PHY feature requests Provide access to link delay deskew settings
Export RX and TX MDI clocks
Export RX and TX SOF triggers synchronous to their respective MDI clocks
PHY
MAC TS
RX_CLK
GTX_CLK
RX DATA
TX DATA
Link Master
Link Slave
OSC
25 MHz
PLL
RX SOF
TX SOF
MDI_TX_CLK

38. Questions?