Improving IEEE 1588 synchronization accuracy in 1000BASE-T systems Rodney Greenstreet Alejandro Zepeda 9th White Rabbit Workshop

Data Redundancy using Ring Topology Ring Topologies are common in industrial and automation applications Bridged Device Often, Ring-topologies are deployed in industrial settings for resiliency Show GM and slaves in PTP Show SyncE clock is not aligned and needs to be. Link is taken down to align. This may be tolerable during configuration. However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down. Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

Data Redundancy using Ring Topology Ring Topologies are common in industrial and automation applications Dataflow is replicated and eliminated for redundancy (IEEE 802.1CB) Source Bridged Device Destination Often, Ring-topologies are deployed in industrial settings for resiliency Show GM and slaves in PTP Show SyncE clock is not aligned and needs to be. Link is taken down to align. This may be tolerable during configuration. However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down. Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

Data Redundancy using Ring Topology Ring Topologies are common in industrial and automation applications Dataflow is replicated and eliminated for redundancy (IEEE 802.1CB) A broken link does not disrupt dataflow to destination Source Bridged Device Destination Often, Ring-topologies are deployed in industrial settings for resiliency Show GM and slaves in PTP Show SyncE clock is not aligned and needs to be. Link is taken down to align. This may be tolerable during configuration. However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down. Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

1588 Redundancy using Ring Topology Best Master Clock Algorithm (BMCA) Chooses Grand Master (GM) Creates synchronization spanning tree GM Master Slave Passive Bridged Device Ring-topologies are typical in Industrial use cases Show GM and slaves in PTP Show SyncE clock is not aligned and needs to be. Link is taken down to align. This may be tolerable during configuration. However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down. Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

1588 Redundancy using Ring Topology Best Master Clock Algorithm (BMCA) Chooses Grand Master (GM) Creates synchronization spanning tree Rearranges spanning tree if a break occurs GM Master Slave Passive Bridged Device Ring-topologies are typical in Industrial use cases Show GM and slaves in PTP Show SyncE clock is not aligned and needs to be. Link is taken down to align. This may be tolerable during configuration. However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down. Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

1588 Redundancy using Ring Topology Best Master Clock Algorithm (BMCA) Chooses Grand Master (GM) Creates synchronization spanning tree Rearranges spanning tree if a break occurs GM Master Slave Bridged Device Ring-topologies are typical in Industrial use cases Show GM and slaves in PTP Show SyncE clock is not aligned and needs to be. Link is taken down to align. This may be tolerable during configuration. However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down. Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

1588 Redundancy W/SyncE using Ring Topology SyncE clocks must be aligned to synchronization spanning tree GM Master Slave Passive Bridged Device Ring-topologies are typical in Industrial use cases Show GM and slaves in PTP Show SyncE clock is not aligned and needs to be. Link is taken down to align. This may be tolerable during configuration. However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down. Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

1588 Redundancy W/SyncE using Ring Topology SyncE clocks must be aligned to synchronization spanning tree PHYs must be configured for clock alignment Relink is required GM Master Slave Passive Bridged Device Ring-topologies are typical in Industrial use cases Show GM and slaves in PTP Show SyncE clock is not aligned and needs to be. Link is taken down to align. This may be tolerable during configuration. However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down. Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

1588 Redundancy W/SyncE using Ring Topology SyncE clocks must be aligned to synchronization spanning tree PHYs must be configured for clock alignment Relink is required If a break occurs BMCA rearranges spanning tree SyncE realignment requires links to be reestablished, breaking redundancy GM Master Slave Passive Bridged Device Ring-topologies are typical in Industrial use cases Show GM and slaves in PTP Show SyncE clock is not aligned and needs to be. Link is taken down to align. This may be tolerable during configuration. However, during operation, if a link is broken, IEEE 1588 will realign it’s synchronization tree. SyncE does not. To do so, requires links be taken down. Unlike fiber, to align the Ethernet bit clock to IEEE 1588, links may have to be taken down.

IEEE 1588 Timestamp Model Reduce error in 1588 event message timestamps Timestamp is captured when the event message crosses between the node and network Timestamp point is the first symbol after the start of frame delimiter (SFD) PHY Hardware Assist IEEE 1588 (Application Layer) OS MAC Network Preamble/ SFD DA Message Timestamp Point Time OSI Model Media Dependent Interface (MDI)

NI’s HW Timestamp Model for 1000BASE-T Key components of HW model for timestamping event messages Generic PHY Timestamp Capture Circuits GMII (125 MHz) MDI (125 MHz) OSC 25 MHz MAC PHY TS GTX_CLK RX DATA TX DATA RX_CLK Network Timekeeper (125 MHz)

NI’s HW Timestamp Model for 1000BASE-T Key components of HW model for timestamping event messages Account for delays between timestamp circuitry and MDI Fixed delays (e.g. propagation delays) Link-to-link delays (e.g. link delay skew) Variable delays (e.g. clock crossings) Ingress/Egress Delays MDI (125 MHz) OSC 25 MHz MAC PHY TS GTX_CLK RX DATA TX DATA RX_CLK Network Preamble/ SFD DA Message Timestamp Point Time Bob Noseworthy Fixed delays 10s of ns

1000BASE-T Link to Link Delay 1000BASE-T media 4 channels of twisted pair copper cabling Maximum length is 100 meters Maximum skew between channels is 50 ns Managing link delay skew During link training state, PHY transmits IDLE symbols aligned across channels PHY receiver recovers IDLE symbols and employs deskew FIFOs to realign symbols Dual-Duplex transmission Electrically, skew is the same in both directions Two linked PHYs may choose different deskew parameters, causing asymmetry Deskew parameters may be accessible through PHY management interface TX MDI Length ≤ 100 meters RX MDI A B C D TSKEW ≤ 50 ns Realign to achieve a coherent symbol Shortest channel longest delay Deskew FIFOs

1000BASE-T Clock Architecture During auto-negotiation, link master and link slave are assigned Link master generates media dependent TX clock Link slave recovers clock from incoming data Link slave transmits using recovered clock Link master recovers clock from incoming data Link Master Link Slave MAC PHY TS GTX_CLK RX DATA TX DATA RX_CLK OSC 25 MHz PLL PLL Implications for timestamping

1000BASE-T Clock Crossings Link master generates MDI TX clock Clock crossing between GTX_CLK and MDI TX clock MAC PHY TS GTX_CLK RX DATA TX DATA RX_CLK OSC 25 MHz Link Master Link Slave PLL Clock Crossing SOF Sent 𝝓 Fixed Delay GTX_CLK MDI TX_CLK Actual SOF

1000BASE-T Clock Crossings Link slave recovers MDI clock and exports as GMII RX_CLK Clock crossing between RX_CLK and timekeeper clock ( TK_CLK) Continuously measure phase relationship between RX_CLK and TK_CLK On detection of SOF, calculate phase offset to previous RX_CLK edge (actual SOF) MAC PHY TS GTX_CLK RX DATA TX DATA RX_CLK OSC 25 MHz Link Master Link Slave PLL Clock Crossing On detection of SOF, adjust timestamp Measuring phase between two asynchronous clocks key Actual SOF 𝝓 RX_CLK TK_CLK SOF Detected

1000BASE-T Clock Crossings Link slave uses recovered MDI clock as MDI TX clock Clock crossing between GTX_CLK and MDI TX clock Continuously measure phase relationship between RX_CLK and TK_CLK On detection of SOF, calculate phase offset to next RX_CLK edge (actual SOF) MAC PHY TS GTX_CLK RX DATA TX DATA RX_CLK OSC 25 MHz Link Master Link Slave PLL Clock Crossing Actual SOF 𝝓 RX_CLK TK_CLK SOF Sent

1000BASE-T Clock Crossings Link master recovers MDI clock and exports as GMII RX_CLK Clock crossing between RX_CLK and timekeeper clock (TK_CLK) Continuously measure phase relationship between RX_CLK and TK_CLK On detection of SOF, calculate phase offset to previous RX_CLK edge (actual SOF) MAC PHY TS GTX_CLK RX DATA TX DATA RX_CLK OSC 25 MHz Link Master Link Slave PLL Clock Crossing Actual SOF 𝝓 RX_CLK TK_CLK Phase offset varies with cable length Detected SOF

DDMTD introduction Signal mixers have been used widely for over a century to shift frequencies around. Input (ƒi, 𝜙i) Local oscillator (LO) (ƒLO, 𝜙LO) (ƒi − ƒLO ,𝜙i −𝜙LO ) (ƒi + ƒLO ,𝜙i +𝜙LO ) Signals can also be mixed digitally, using a common register (flip-flop). Only the low-frequency signal is output. Input (ƒi, 𝜙i) Local oscillator (LO) (ƒLO, 𝜙LO) Deglitcher (ƒi − ƒLO ,𝜙i −𝜙LO )

Phase Measurement using DDMTD As its name implies, a DDMTD (dual digital mixer time difference) uses two digital mixers to enable better measurements of phase offset ƒ = ƒ1 – ƒLO 𝜙 = 𝜙1 – 𝜙LO (ƒ1 , 𝜙1) Deglitcher Phase Compare (ƒ2 , 𝜙2) Deglitcher ƒ = ƒ2 – ƒLO 𝜙= 𝜙2 – 𝜙LO (ƒLO ,𝜙LO)

Dual Mixer Time Difference (DMTD) 125 MHz Timekeeper 2 ns RX Clk 𝝓 = 90°

Dual Mixer Time Difference (DMTD) (ƒ1 , 𝜙1) (ƒ2 , 𝜙2) (ƒLO ,𝜙LO) Deglitcher Timekeeper (125 MHz) RX Clk (125 MHz) LO Clk (112.5 MHz) ns 8 16 24 32 40 48 56 64 72 80 88 TK_DownMixed RxClk_DownMixed

Dual Mixer Time Difference (DMTD) (ƒ1 , 𝜙1) (ƒ2 , 𝜙2) (ƒLO ,𝜙LO) Deglitcher ns 8 16 24 32 40 48 56 64 72 80 88 Timekeeper (125 MHz) RX Clk (125 MHz) LO Clk (112.5 MHz) TK_DownMixed RxClk_DownMixed ~18 ns

Dual Mixer Time Difference (DMTD) ns 8 16 24 32 40 48 56 64 72 80 88 Timekeeper (125 MHz) RX Clk (125 MHz) LO Clk (124.9875 MHz) LO Clk (112.5 MHz)

Dual Mixer Time Difference (DMTD) μs 10 20 30 40 50 60 70 80 90 100 Timekeeper (125 MHz) RX Clk (125 MHz) LO Clk (124.9875 MHz) TK_DownMixed 20 μs RxClk_DownMixed 𝝓 = 90°

Phase measurement of non-locked clocks DDMTD can be used to measure the phase offset even if the clocks are not locked Link Master Link Slave Clock Crossing PHY PHY MAC MAC GTX_CLK RX_CLK TS TS TX DATA RX DATA PLL RX DATA TX DATA TS TS RX_CLK GTX_CLK OSC 25 MHz Actual SOF 𝝓 RX_CLK TK_CLK SOF Detected

Calculate Frequency of Down Mixed Clocks Select set of timestamps of down-mixed clocks around SOF timestamp Calculate time delta between each successive timestamp within set Average time deltas to reduce jitter error Calculate frequency 𝑓 𝑑𝑜𝑤𝑛𝑀𝑖𝑥𝑒𝑑 = 1 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 Δt SOF TS Timekeeper Δt Δt Δt TK_DownMixed Δt Δt Δt RxClk_DownMixed

Calculate Reference Edge of Down Mixed Clocks Average same set of timestamps of each down-mixed clock Reduces jitter error Identifies reference edge SOF TS TK_Clk TK_DownMixed RxClk_DownMixed

Calculate phase to SOF Calculate the phase from each down-mixed clock reference edge to the SOF timestamp, based on the calculated frequency for that clock. Calculate the phase offset between the down-mixed clocks SOF TS TK_Clk TK_DownMixed RxClk_DownMixed 𝝓𝑅𝑥𝐶𝑙𝑘𝐷𝑜𝑤𝑛𝑀𝑖𝑥𝑒𝑑 𝝓𝑇𝐾_𝐶𝑙𝑘𝐷𝑜𝑤𝑛𝑀𝑖𝑥𝑒𝑑 𝝓𝑅𝑥𝐶𝑙𝑘_𝑇𝐾_𝐶𝑙𝑘

Calculate phase offset near SOF Phase offset between MDI clock and timekeeper clock around the SOF is the same as the calculated phase offset between the down-mixed clocks: SOF @ MDI SOF TS TK_CLK 𝝓𝑅𝑥𝐶𝑙𝑘−𝑇𝐾_𝐶𝑙𝑘 RxClk 𝛥𝑡 𝑅𝑋_𝑇𝐾

Synchronization performance tests FPGA with 1000BASE-T MAC + Timekeeper Standard 1000BASE-T PHY 100m Two PXI-6683 boards synchronizing via IEEE 1588 with 250ms sync interval with a 100m cable, configured to generate a PPS. Offset between PPSs is recorded with oscilloscope every second for 12 hours (43,200 samples).

Test A: Baseline test results 12-hour test with PXI-6683 boards with 8ns timestamp resolution Test Peaks from mean Std. Dev. A ±13.4 ns 2.66 ns

Test B: increased timestamping resolution Test B uses a technique very similar to the one described in ISPCS 2014 paper "Precise clock parameter estimation and ground truth capture for clock error measurements using FPGAs" to increase timestamp resolution to ~0.5ns. Test Peaks from mean Std. Dev. A ±13.4 ns 2.66 ns B ±3.64 ns 0.75 ns

Test C: timestamps enhanced with DDMTD Test C implements the techniques described in this presentation to enhance PTP packet timestamps Test Peaks from mean Std. Dev. A ±13.4 ns 2.66 ns B ±3.64 ns 0.750 ns C ±0.460 ns 0.104 ns

Even better results with shorter sync interval By decreasing the Sync Message Interval to 125 ms the synchronization performance is further improved Test Peaks from mean Std. Dev. A ±13.4 ns 2.66 ns B ±3.64 ns 0.75 ns C ±0.460 ns 0.104 ns D ±0.320 ns 0.077 ns

PHY feature requests Provide access to link delay deskew settings Export RX and TX MDI clocks Export RX and TX SOF triggers synchronous to their respective MDI clocks PHY MAC TS RX_CLK GTX_CLK RX DATA TX DATA Link Master Link Slave OSC 25 MHz PLL RX SOF TX SOF MDI_TX_CLK

Questions?