The White Rabbit Fieldbus Fieldbuses WG 26/01/2012 The White Rabbit Fieldbus E.Gousiou (BE/CO/HT) on behalf of the White Rabbit team

Outline Introduction to WR & Technology Overview Implementation & Support Cost Analysis Conclusions

Outline Introduction to WR & Technology Overview Implementation & Support Cost Analysis Conclusions

Real Time Systems Real-time communication systems require the execution of operations with tight time constraints Large distances between nodes give long transmission delays Dynamic changes to the number of nodes is not easy

Real Time Systems Real-time communication systems require the execution of operations with tight time constraints Large distances between nodes give long transmission delays Dynamic changes to the number of nodes is not easy

Real Time Systems Real-time communication systems require the execution of operations with tight time constraints Large distances between nodes give long transmission delays Dynamic changes to the number of nodes is not easy Control Room Trigger at 1298537875

Real Time Systems Real-time communication systems require the execution of operations with tight time constraints Large distances between nodes give long transmission delays Dynamic changes to the number of nodes is not easy Control Room Trigger at 1298537875

Real Time Systems Real-time communication systems require the execution of operations with tight time constraints Large distances between nodes give long transmission delays Dynamic changes to the number of nodes is not easy Control Room Trigger at 1298537875

Real Time Systems Real-time communication systems require the execution of operations with tight time constraints Large distances between nodes give long transmission delays Dynamic changes to the number of nodes is not easy Trigger at 1298537875 Control Room Trigger at 1298537875 Trigger at 1298537875 Trigger at 1298537875

Measures for Aircraft Noise Real Time Systems Real-time communication systems require the execution of operations with tight time constraints Large distances between nodes give long transmission delays Dynamic changes to the number of nodes is not easy Trigger at 12:03698782 Measures for Aircraft Noise Emissions Reduction Trigger at 12:03698782 Trigger at 12:03698782 From NI presentation at …

Measures for Aircraft Noise Real Time Systems Real-time communication systems require the execution of operations with tight time constraints Large distances between nodes give long transmission delays Dynamic changes to the number of nodes is not easy Measures for Aircraft Noise Emissions Reduction 100 m 100 m From NI presentation at …

Real Time Systems 100 m kms 100 m kms Real-time communication systems require the execution of operations with tight time constraints Large distances between nodes give long transmission delays Dynamic changes to the number of nodes is not easy Measures for Aircraft Noise Emissions Reduction EISCAT: The most sophisticated Imaging Radar ever 100 m 100 m kms kms With White Rabbit a farm of sensors extended over several kms is possible From NI presentation at … From NI presentation at …

What is White Rabbit + + White Rabbit is: Ethernet Determinism & Reliability + Synchronization

What is White Rabbit + + White Rabbit is: Ethernet Determinism & Reliability + Synchronization

Ethernet WR switch WR switch WR switch WR switch WR switch WR node WR

Ethernet Up to 2000 nodes WR switch WR switch WR switch WR switch

Ethernet Up to 2000 nodes Copper or fiber medium WR switch WR switch Up to 2000 nodes Copper or fiber medium WR switch WR node WR switch WR node WR node WR node Up to 10 Km fiber links WR node WR node WR node WR node WR node WR node WR node WR node WR switch WR node WR node WR node WR node

Ethernet Up to 2000 nodes Copper or fiber medium WR switch WR switch Up to 2000 nodes Copper or fiber medium WR switch WR node WR switch WR node WR node WR node Up to 10 Km fiber links WR node WR node WR node WR node WR node Bandwidth: 1 Gbps WR node WR node WR node WR switch WR node WR node WR node WR node

Ethernet Up to 2000 nodes Copper or fiber medium WR switch WR switch Up to 2000 nodes Copper or fiber medium WR switch WR node WR switch WR node WR node WR node Up to 10 Km fiber links WR node WR node WR node WR node WR node Bandwidth: 1 Gbps WR node WR node WR node WR switch WR switch: 18 ports WR node WR node WR node WR node

Ethernet Up to 2000 nodes Copper or fiber medium WR switch WR node Up to 2000 nodes Copper or fiber medium WR node WR node WR node WR node Up to 10 Km fiber links WR node WR node WR node WR node Bandwidth: 1 Gbps WR node WR node WR node WR switch: 18 ports WR node WR node Add/ Remove nodes dynamically WR node

Ethernet Up to 2000 nodes Copper or fiber medium WR switch WR node Up to 2000 nodes Copper or fiber medium WR node other GbE switch other node WR node WR node WR node Up to 10 Km fiber links WR node WR node WR node WR node Bandwidth: 1 Gbps WR node WR node WR node WR switch: 18 ports WR node Add/ Remove nodes dynamically WR node Non WR devices

Ethernet Up to 2000 nodes Copper or fiber medium WR switch WR node Up to 2000 nodes Copper or fiber medium WR node Other GbE switch other node WR node WR node WR node Up to 10 Km fiber links WR node WR node WR node WR node Bandwidth: 1 Gbps WR node WR node WR node WR switch: 18 ports WR node Add/ Remove nodes dynamically WR node Non WR devices Ethernet Features (VLAN) & Protocols (SNMP)

What is White Rabbit + + White Rabbit is: Ethernet Determinism & Reliability + Synchronization

Deterministic & Reliable Determinism is the guarantee that packet transmission delay between two stations will never exceed a certain boundary. The upper bound delay latency can be verified by analyzing the publicly available source code. Reliability is the ability of a system to provide its services to clients under both routine and abnormal conditions. White Rabbit ensures reliable services by supporting: Topology Redundancy Data Resilience with Forward Error Correction schemes FEC WR switch WR switch WR switch

What is White Rabbit + + White Rabbit combines: Ethernet Determinism & Reliability + Synchronization

Synchronization Synchronization: common notion of time in the entire network In White Rabbit sub-nanosecond synchronization is achieved in parallel and transparently to the data exchange Two main technologies are used together: Synchronous Ethernet (SyncE) White Rabbit Precision Time Protocol (enhanced PTP)

Synchronization Syntonization with SynchE Timing Master WR Switch WR switch WR node

Synchronization Syntonization with SynchE Timing Master WR Switch The individual clocks of the nodes/ switches can be slightly different 125.00 MHz 124.98 MHZ 125.03 MHz WR switch WR switch WR node WR node WR node WR node WR node WR node WR node devices have independent free running oscillators, PS receiver and uses its output frequency to encode the outgoing data stream. Slave nodes use PLLs to recover the reference clock from the incoming data. The recovered clock is used to encode the data streams propagated to nodes being lower in the system hierarchy and the data stream sent back to the master node. The root node (WR master) takes its clock from an external source (atomic clock, GPS) and uses it to encode the data streams going to its downlink ports. Each layer of the switches senses this data on its uplink ports and extracts their local clock from that data. The phase of the recovered clock is corrected using the precise link delay obtained by the PTP measurement and DMTD phase detector. The recovered clock is then passed to the downlink ports and recovered by the next layer of switches/nodes which propagate it further, and so on, and so forth. Clock encoded in Ethernet carrier Clock recovered by PHY PLL

Synchronization Syntonization with SynchE Timing Master WR Switch The individual clocks of the nodes/ switches can be slightly different All the network devices have the same frequency! The individual clocks of the nodes/ switches can be slightly different 125.00 MHz 124.98 MHZ 125.03 MHz 125.00 MHz 124.98 MHZ 125.03 MHz WR switch WR switch With SyncE all network devices manage to have the same clock, generated by the System Timing Master WR node WR node WR node WR node WR node The clock is encoded in the Ethernet carrier and recovered by the PLL of the PHY; no extra traffic cost! WR node WR node Common notion of frequency

Synchronization Syntonization with SynchE Timing Master WR Switch All the network devices have the same frequency! WR switch WR switch With SyncE all network devices manage to have the same clock, generated by the System Timing Master WR node WR node WR node WR node WR node The clock is encoded in the Ethernet carrier and recovered by the PLL of the PHY; no extra traffic cost! WR node WR node Common notion of frequency However, there are offsets between the clocks!

Synchronization Syntonization with SynchE Timing Master WR Switch All the network devices have the same frequency! WR switch WR switch With SyncE all network devices manage to have the same clock, generated by the System Timing Master WR node WR node WR node WR node WR node The clock is encoded in the Ethernet carrier and recovered by the PLL of the PHY; no extra traffic cost! WR node WR node Common notion of frequency However, there are offsets between the clocks!

Synchronization Offset Adjustment with Enhanced Precision Time Protocol Timing Master WR Switch WR switch WR switch WR node WR node WR node WR node WR node WR node WR node

Synchronization Offset Adjustment with Enhanced Precision Time Protocol Timing Master WR Switch WR switch

Synchronization Offset Adjustment with Enhanced Precision Time Protocol Timing Master WR Switch Timing Master WR Switch WR switch WR switch t1 17:00 17:00 t2 00:02 t3 00:07 00:07 t4 17:12 17:12

Synchronization Offset Adjustment with Enhanced Precision Time Protocol Timing Master WR Switch System Timing Master WR switch Timing Master WR Switch WR switch t1 17:00 t2 00:02 17:00 clock synchronization requires the knowledge of the link latency. In packet networks, such as Ethernet, the latency can be measured by analyzing the packets’ send and receive timestamps. PTP does not require a separate network infrastructure, allowing for mixing precise time information with data transfers. This approach allows for significant cost reduction (no extra cabling, standard hardware) with no compromises in performance. Hardware-based timestamping t3 00:07 t4 17:12 17:12

Synchronization Offset Adjustment with Enhanced Precision Time Protocol System Timing Master Timing Master WR Switch WR switch WR switch 17:00 t2 00:02 t3 00:07 17:12

Synchronization Offset Adjustment with Enhanced Precision Time Protocol Timing Master WR Switch System Timing Master WR switch WR switch 17:00 t2 00:02 t3 00:07 17:12

Synchronization Offset Adjustment with Enhanced Precision Time Protocol WR node WR switch Timing Master WR Switch Timing Master WR Switch System Timing Master WR switch

Synchronization Offset Adjustment with Enhanced Precision Time Protocol WR node WR switch Timing Master WR Switch Timing Master WR Switch System Timing Master PTP synchronizes the local clocks with the master clock by measuring and compensating the delays introduced by the link. WR switch Common notion of time

What is White Rabbit White Rabbit provides: Ethernet Determinism a large scale monitoring and control network with all nodes synchronized (sub-ns) a scalable and modular platform that reconfigures automatically deterministic & robust delivery messages Determinism Synchronization WR switch NI PXI crate management database actuator sensor monitoring

What is White Rabbit White Rabbit provides: Ethernet Determinism a large scale monitoring and control network with all nodes synchronized (sub-ns) a scalable and modular platform that reconfigures automatically deterministic & robust delivery messages Determinism Synchronization WR switch PXI crate management database actuator sensor monitoring

Outline Introduction to WR & Technology Overview Implementation & Support Cost Analysis Conclusions

Implementation & Support White Rabbit technology will be made of commercially available off-the-shelf network gear; the components of a WR network will be offered through the catalogues of companies CO will provide a local knowledge hub for CERN users with reference designs, documentation, linux SW; technical support will also be provided by the companies A White Rabbit network is composed of WR Switches WR switch WR Nodes WR node IP core WR Timing Master Timing Master WR Switch Copper/ Fiber links

Implementation & Support White Rabbit technology will be made of commercially available off-the-shelf network gear; the components of a WR network will be offered through the catalogues of companies CO will provide a local knowledge hub for CERN users with reference designs, documentation, linux SW; technical support will also be provided by the companies A White Rabbit network is composed of WR Switches WR switch WR Nodes WR node IP core WR Timing Master Timing Master WR Switch Copper/ Fiber links

Implementation & Support White Rabbit technology will be made of commercially available off-the-shelf network gear; the components of a WR network will be offered through the catalogues of companies CO will provide a local knowledge hub for CERN users with reference designs, documentation, linux SW; technical support will also be provided by the companies A White Rabbit network is composed of WR Switches WR switch WR Nodes WR node SPEC IP core … ADC DAC TDC WR Timing Master Timing Master WR Switch WR node WR switch Copper/ Fiber links same optical link used for transmission/ reception

An example application SPEC

An example application SPEC Xilinx Spartan 6 ADC user core time data WR PTP core SFP

An example application SPEC Xilinx Spartan 6 ADC user core time data WR PTP core SFP

An example application SPEC Xilinx Spartan 6 ADC user core time WHISBONE EtherBone ETHERNET WR PTP core SFP

An example application SPEC Xilinx Spartan 6 ADC WR PTP core user SFP time WHISBONE EtherBone ETHERNET SPEC ADC

An example application SPEC Data Master Sample@13:01 WR switch … WR switch … SPEC DAC TDC SPEC ADC Sample@13:01 Sample@13:01 Sample@13:01 Sample@13:01 Sample@13:01 Sample@13:01

Outline Introduction to WR & Technology Overview Implementation & Support Cost Analysis Conclusions

Components Costs 3K CHF 0 CHF 1K CHF < 1 CHF/ m SPEC White Rabbit network components costs WR Switch WR switch 3K CHF WR Node WR node 0 CHF SPEC board 1K CHF SPEC Fiber links < 1 CHF/ m

White Rabbit VS standard PTP Technologies Used Accuracy # 1Gb SFP Ports Price (CHF) SyncE PTP HW timestamps CISCO Nexus 5548 32 NO YES NO sub ms 19K Hirschmann MACH1000 16 NO YES YES 30 ns 5K7 Ruggedcom RSG2288 9 NO YES YES 1 us 17K White Rabbit 18 YES YES YES sub ns 3K Media Markt 18 NO NO NO ? 0.2K

Maintenance Costs Network Configuration & Management through standard Ethernet protocols (SNMP, MIB) Use of well-established tools (Wireshark with WR pluggin) Any unexpected behavior can be diagnosed by verifying the source code or simulating the use case; if necessary modifications can be implemented

Outline Introduction to WR & Technology Overview Implementation & Support Cost Analysis Conclusions

“Oh dear! Oh dear! I shall not be too late!” Deterministic and Reliable Ethernet Offering transparently sub-ns synchronization Easy to use; well established standards Support by companies and internally at CERN Guaranteed and risk free solution with publicly available source code

Extras

Backup WR Timing Master White Rabbit network Ref Clock WR Timing Master Backup WR Timing Master other GbE WR switch WR switch WR switch switch other node management WR node controller WR switch WR switch other node management other node monitoring other node database WR node sensor WR node actuator WR node sensor data & timing routes data routes

White Rabbit network Ref. Clock WR timing master WR switch WR switch Backup WR timing master WR switch WR switch WR switch WR switch WR switch WR node WR node WR node data and timing routes