H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October White Rabbit Sub-Nanosecond timing over Ethernet H.Z. Peek on behalf of the White Rabbit collaboration 1) Alice in Wonderland, Lewis Carroll (1865) “Oh dear! Oh dear! I shall be too late!” 1)

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October 2011 Outline 2 What is White Rabbit Network Building blocks Measurement results Applications Conclusion

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October What is White Rabbit Networking and timing Based on well-known technologies / standards Open Hardware and Open Software International collaboration Main features: Transparent, high-accuracy time distribution, Low-latency, deterministic data delivery, Designed for high reliability, Plug & play.

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October Timing over Ethernet Timing over Ethernet History 1985 NTP (RFC 1305) sub milli-second 2002 PTP (IEEE 1588) sub micro-second 2012 White Rabbit (?)sub nano-second

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October White Rabbit timing distribution Synchronization with sub-ns accuracy over fiber A combination of: Synchronous Ethernet (SyncE) syntonization Digital Dual-Mixer Time Difference (DDMTD) phase detection (  Precision Timing Protocol (PTP) synchronization

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October Timing distribution Averaging all clock edges “mean phase” Syntonization “The adjustment of two electronic circuits or devices in terms of frequency” Reference Clock Master SFP Slave Rx SFP Tx Rx SFP Slave Rx SFP Tx Rx   High Precision Low jitter High Precision Low jitter Ordinary serial data communication channels Ordinary serial data communication channels!

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October 2011 t offset 7  Ref Clk Cnt Master SFP Slave Rx Tx SFP Tx Rx PTP (IEEE1588)  Cnt t1t1 t 2 (= Cnt + SlaveBitSlide) Sync message Follow_Up Message (t 1 ) t 1 t 2 t 3 t 1 t 2 Delay_Req Message Delay_Response Message (t 4 ) t 4 t 3 t 1 t 2 t1t1 t2t2 Master Clock Time Slave Clock Time t3t3 t4t4 t 4 (= Cnt + MasterBitSlide +  ) =0 Data (Packets) Clock Time Stamp t2t2 t3t3 t4t4 t4t4 t1t1 t1t1 1.t 2 –t 1 =offset + MSdelay 2.t 4 –t 3 =-offset + SMdelay 3.MSdelay=SMdelay

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October White Rabbit network

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October White Rabbit Switch V2 Central element of White Rabbit network Fully custom design, designed from scratch 10 SFP ports (1000Base-LX) Capable of driving long distance Single Mode fiber

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October White Rabbit Node Simple PCIe FMC Carrier (SPEC) Currently available

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October White Rabbit Switch V3 2 uplink ports, 16 downlink ports Hardware just assembled. Hardware & Software currently being tested. Production expected Q

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October Measurement test setup Hot-air gun demo:

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October Measurement results

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October 2011 Digital to Time Converter (DTC) = Alarm Clock Time to Digital Converter (TDC) = Time stamping 14 Applications

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October 2011 White Rabbit enables measurement and control applications which are using distributed system technologies. Such applications may be spread over large distances. Data transmission delay changes are continuously measured and compensated. System timing of White Rabbit nodes are synchronized with high precision. 15 Conclusions

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October multi-laboratory, multi-company effort White Rabbit a multi-laboratory, multi-company effort Thank you Coming Soon!

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October e offset between Master and Slave? How do we know the time offset between Master and Slave? Clock & Data coded into one stream DC-Balance Special code-groups / Word Alignment T offset = Total delay (Master>Slave>Master) / 2 Can we measure propagation delay using existing serial communication channels? Serial Communication Coding Properties:

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October 2011 Measure propagation delay using FPGA SerDes technology Start Lattice LFSCM25 SerDes Lattice SC PCI Expressx1 Evaluation Board Xilinx Virtex-5 SerDes Xilinx ML507 Board LEDs Stop 100 Km Fiber 19 Transmitter SFP Receiver SFP 1 3 Regenerate system clock at the receiver Using a barrel shifter for word alignment delay known with bit clock resolution System Clock x 20 System Clock

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October Master Slave Start Stop 10 Km fiber Stop Clock Loopback (DPLL) VCXO DAC Time offset measurement test setup Loopback the recovered clock with a Voltage Controled Xtal Oscillator

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October = = = 3 Resynchronization + Barrel shifting action RxRecClk BitSlide(4:0) Algorithm: Propagation Delay = “Start-Stop” Delay + “LED Value” * UI Start/Stop delay 31 Unit Interval (UI) Details :VLVnT09, October 15, 2009 in Athens Presentation: Paper:

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October  Reference Clk Fine time Count Coarse time Start Stop Master SFP Slave Rx Tx SFP RxUsrClk TxUsrClk RxUsrClk 1.# of system clocks 2.# bit clocks (i.e. barrel shifts) 3.Phase between Master node Tx and Rx clock t offset Tx Rx Measure time offset Bidirectional + Loopback the recovered clock 1.25 Gbps (IEEE BASE-X = Gigabit Ethernet) Time offset is determined by: For details please see Technical Report “ETR ”: Gpbs ps 20 x #bit clocks 800 ps

H.Z. Peek Nikhef Amsterdam Electronics- Technology VLVnT11 Erlangen12-14, October Master time t pd 1 Slave time t offset t disp  mTx  sRx  mRx  sTx 1 2 Time offset and fibre dispersion (t disp = ps over 10.7 km => D( ) = l416 ps/km)