1. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 1 White Rabbit Sub-Nanosecond timing over Ethernet H.Z. Peek...... on behalf of the White Rabbit collaboration 1) Alice in Wonderland, Lewis Carroll (1865) “Oh dear! Oh dear! I shall be too late!” 1)

2. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 Outline 2 What is White Rabbit Network Building blocks Measurement results Applications Conclusion

3. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 3 What is White Rabbit Networking and timing Based on well-known technologies / standards Open Hardware and Open Software International collaboration Main features: Transparent, high-accuracy time distribution, Low-latency, deterministic data delivery, Designed for high reliability, Plug & play.

4. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 4 Timing over Ethernet Timing over Ethernet History 1985 NTP (RFC 1305) sub milli-second 2002 PTP (IEEE 1588) sub micro-second 2012 White Rabbit (?)sub nano-second

5. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 5 White Rabbit timing distribution Synchronization with sub-ns accuracy over fiber A combination of: Synchronous Ethernet (SyncE) syntonization Digital Dual-Mixer Time Difference (DDMTD) phase detection (  Precision Timing Protocol (PTP) synchronization

6. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 6 Timing distribution Averaging all clock edges “mean phase” Syntonization “The adjustment of two electronic circuits or devices in terms of frequency” Reference Clock Master SFP Slave Rx SFP Tx Rx SFP Slave Rx SFP Tx Rx   High Precision Low jitter High Precision Low jitter Ordinary serial data communication channels Ordinary serial data communication channels!

7. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 t offset 7  Ref Clk Cnt Master SFP Slave Rx Tx SFP Tx Rx PTP (IEEE1588)  Cnt t1t1 t 2 (= Cnt + SlaveBitSlide) Sync message Follow_Up Message (t 1 ) t 1 t 2 t 3 t 1 t 2 Delay_Req Message Delay_Response Message (t 4 ) t 4 t 3 t 1 t 2 t1t1 t2t2 Master Clock Time Slave Clock Time t3t3 t4t4 t 4 (= Cnt + MasterBitSlide +  ) =0 Data (Packets) Clock Time Stamp t2t2 t3t3 t4t4 t4t4 t1t1 t1t1 1.t 2 –t 1 =offset + MSdelay 2.t 4 –t 3 =-offset + SMdelay 3.MSdelay=SMdelay

8. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 8 White Rabbit network

9. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 9 White Rabbit Switch V2 Central element of White Rabbit network Fully custom design, designed from scratch 10 SFP ports (1000Base-LX) Capable of driving long distance Single Mode fiber

10. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 10 White Rabbit Node http://www.ohwr.org/projects/spec/wiki Simple PCIe FMC Carrier (SPEC) Currently available

11. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 11 White Rabbit Switch V3 2 uplink ports, 16 downlink ports Hardware just assembled. Hardware & Software currently being tested. Production expected Q1-2012 http://www.ohwr.org/attachments/741/White_Rabbit_Technical_Spec_05.pdf

12. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 12 Measurement test setup http://www.youtube.com/watch?v=ZSRQEExbdq8 Hot-air gun demo:

13. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 13 Measurement results

14. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 Digital to Time Converter (DTC) = Alarm Clock Time to Digital Converter (TDC) = Time stamping 14 Applications

15. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 15 Applications-II

16. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 White Rabbit enables measurement and control applications which are using distributed system technologies. Such applications may be spread over large distances. Data transmission delay changes are continuously measured and compensated. System timing of White Rabbit nodes are synchronized with high precision. 16 Conclusions

17. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 White Rabbit = Networking and timing Prototype hard- soft-ware becomes available this year. Open hard- soft-ware sources available which enables tendering. Standardization effort (IEEE and ITU) is foreseen. 17 Conclusions-II Is ideally suited for

18. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 18 multi-laboratory, multi-company effort White Rabbit a multi-laboratory, multi-company effort Thank you Coming Soon!

19. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 19

20. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 20 e offset between Master and Slave? How do we know the time offset between Master and Slave? Clock & Data coded into one stream DC-Balance Special code-groups / Word Alignment 1 2 3 T offset = Total delay (Master>Slave>Master) / 2 Can we measure propagation delay using existing serial communication channels? Serial Communication Coding Properties:

21. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 Measure propagation delay using FPGA SerDes technology Start Lattice LFSCM25 SerDes Lattice SC PCI Expressx1 Evaluation Board Xilinx Virtex-5 SerDes Xilinx ML507 Board LEDs Stop 100 Km Fiber 21 Transmitter SFP Receiver SFP 1 3 Regenerate system clock at the receiver Using a barrel shifter for word alignment delay known with bit clock resolution System Clock x 20 System Clock

22. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 22 Master Slave Start Stop 10 Km fiber Stop Clock Loopback (DPLL) VCXO DAC Time offset measurement test setup Loopback the recovered clock with a Voltage Controled Xtal Oscillator

23. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 23 0 0001 = 10 0000 = 00 0011 = 3 Resynchronization + Barrel shifting action RxRecClk BitSlide(4:0) 011101011000001010110111010110000010101101110101 0 Algorithm: Propagation Delay = “Start-Stop” Delay + “LED Value” * UI Start/Stop delay 31 Unit Interval (UI) Details :VLVnT09, October 15, 2009 in Athens Presentation:http://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppthttp://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppt Paper:http://dx.doi.org/10.1016/j.nima.2010.04.126http://dx.doi.org/10.1016/j.nima.2010.04.126

24. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 24  Reference Clk Fine time Count Coarse time Start Stop Master SFP Slave Rx Tx SFP RxUsrClk TxUsrClk RxUsrClk 1.# of system clocks 2.# bit clocks (i.e. barrel shifts) 3.Phase between Master node Tx and Rx clock t offset Tx Rx Measure time offset Bidirectional + Loopback the recovered clock 1.25 Gbps (IEEE802.3 1000BASE-X = Gigabit Ethernet) Time offset is determined by: For details please see Technical Report “ETR2010-01”: http://www.nikhef.nl/pub/services/biblio/technicalreports/ETR2010-01.pdf 1.25 Gpbs 10.. 100 ps 20 x #bit clocks 800 ps

25. H.Z. Peek Nikhef Amsterdam Electronics- Technology KM3NeT General Assembly Meeting Catania, 20-23, February 2012 25 Master time t pd 1 Slave time t offset t disp  mTx  sRx  mRx  sTx 1 2 Time offset and fibre dispersion (t disp = 15150 ps over 10.7 km => D( ) = l416 ps/km)