Redundant network topologies for dependable time transfer Increasing Availability & Fault Tolerance Jose Luis Gutierrez, Maciej Lipiński, Javier Diaz jlgutierrez@ugr.es http://www.ugr.es/~jlgutierrez

Redundancy Protocols PRP & HSR

REDUNDANCY PROTOCOLS PRP HSR

HSR PROTOCOL Main Features: What is High-availability Seamless Redundancy (HSR)? HSR (IEC 62439-3 Clause 5) is an Ethernet protocol suited for applications that demand high availability and very short switch over time. It is one of the redundancy protocols selected for substation automation in IEC 61850. It can also be employed in other critical networking applications: industrial automation, motion control and transportation. The nodes are attached by two Ethernet ports to the ring (same MAC address) Main Features: Ring topology Time & data transfer duplication Avoids single point of failure Zero-time recovery Cheaper than PRP

„A“-frame (HSR tagged) HSR PROTOCOL source destinations „C“-frame (untagged) „D“-frame (untagged) “A” frames “B” frames standard frames removal from the ring node CPU DANH DANH „A“-frame (HSR tagged) „B“-frame (HSR tagged) Nodes are arranged as a ring, each node has two identical interfaces, port A and port B. For each frame to send (“C”-frame), the source node sends two copies over port A and B. Each node relays a frame it receives from port A to port B and vice-versa, except if it already forwarded it. The destination nodes consumes the first frame of a pair and discards the duplicate. In case of interruption of the ring, frames still continue to be received over the intact path. B A DANH DANH DANH DANH DANH destinations

HSR PROTOCOL: Node architecture Redundant Ethernet ports Link Redundancy Entity (LRE) Duplicate frames Drop duplicated frames HSR tagger Forwarding frames HSR table (dest, orig, HSR nº) …

HSR PROTOCOL: How it works Timing Rings are formed by Transparent clocks + Ordinary Clocks (TC+OC) PTP are handled in both ports on each HSR node. There can be more than 1 Time reference BUT they never work at the same time.

WR-HSR Implementation Road to WR-HSR

HSR to White-Rabbit HSR Adapting HSR to White-Rabbit-HSR End-to-end mechanism for clock transfer Delay request for delay estimation Tree topology Switchover mechanism in case of failure Boundary Clocks Peer-to-Peer for clock transfer Peer-delay for delay estimation Ring topology Switchover to ring topologies Transparent Clocks New development for White-Rabbit-HSR Development of a HSR tag included in time & data frames Development of gateware/software to duplicate, forward and drop frames Development of HSR management frames Enhance holdover of White- Rabbit devices to ease switchover

WR PPSi-P2P IMPLEMENTATION Master Slave TC CF T1 Sync Tsync_ingress Tsync_egress Follow up RTMT T2 T1 + CF LD LDL LDR LD Correction Field ( - ) + Tsync_egress - Tsync_ingress + Reception Link Delay Slave Offset = T2 T1 + CF LD Residence Time

WRS GATEWARE

WR-HSR Network P2P PTP Behaviour Adds a HSR tag and duplicates the frame In both directions  Sync  follow_up GrandMaster WR slave SyncA  follow_upA  SyncA  follow_upA   SyncB  follow_upB  SyncB  follow_upB  Sync  follow_up Manages PTP frames in both ports independetly BMC (A||B) BMC (A||B) Sync  follow_up  SyncB  follow_upB  SyncB  follow_upB  Removes HSR tag  SyncA  follow_upA  SyncA  follow_upA WR slave BMC (A||B) Sync  Non-HSR Sync frame SyncA HSR-Sync from path A SyncB HSR-Sync from path B Sync  follow_up  WR slave

Single point of failure avoidance High-Availability: Adding Redundancy Features

SWITCHOVER MECHANISM Normal Operation Switch A Failure Detected  PTP  Switch A 1-Pulse per Second output (PPS) switchover Failure Detected Switch B Slave port Backup port 1-Pulse per Second output (PPS) Maciej’s switchover: Slave port: performs the WR-PTP normally Backup port: stores WR-PTP data but doesn’t apply After Switchover When PPS drifts, the WRS starts using the data from the backup port and applies them. 1-Pulse per Second output (PPS)

WR-HSR Development status Presentation title 10.02.2010 WR-HSR Development status PeerDelay mechanism: measures the link delay between two adjacent nodes. (100%) Peer-to-Peer mechanism: Boundary Clock implementation: Middle nodes are syntonized and synchronized with master. (100%) Transparent Clock implementation: Middle nodes are syntonized but not synchronized with master. (100%) HSR Tagging/untagging For timing: 100% For data: 25% (gateware approach ) Switchover mechanism: In case of failure, all nodes switch the path from where they get syntonized/synchronized. (75% adding more WRSs) Holdover enchancement: Probabilistic model to increase current WRS holdover to make possible the switchover mechanism (~100ms). Copyright © Infineon Technologies 2009. All rights reserved.

WR-HSR Development TESTS Presentation title 10.02.2010 WR-HSR Development TESTS Frequency distribution in the ring Number of ring nodes (max?) Holdover < switchover GM GM GM S S S S S S S S S S Mínimum test case S Copyright © Infineon Technologies 2009. All rights reserved.

Generic Substation Events (GSE) TOWARDS SMART GRID Generic Substation Events (GSE) GOOSE (Generic Object Oriented Substation Events) GSSE (Generic Substation State Events). SMV (Sampled Measured Values) The IEC 61850 standard for substation enables the integration of all protection, control, measurement and monitoring functions. These functions require high speed and reliable communication in the substation For mission-critical or time-sensitive applications, even millisecond long network interruptions cannot be tolerated IEC 62439 states that the time that the plant allows for recovery before taking emergency actions (e.g., emergency shut-down, fall-back mode) should be less than 10 ms. IEC 61850 v2 clearly states that the communication redundancy times of GOOSE and SMV protocols in substation automation systems are required to be BUMPLESS (i.e., have a zero switch-over time).

TOWARDS SMART GRID Communication profiles of protocols used in IEC61850

WR-HSR links White-Rabbit HSR project: http://www.ohwr.org/projects/wr-hsr/wiki Ported to Silicon PTP – PPSi (PTPv2) for White-Rabbit: http://www.ohwr.org/projects/ppsi/wiki White Rabbit Switch Software: http://www.ohwr.org/projects/wr-switch-sw/wiki White Rabbit Switch Gateware: http://www.ohwr.org/projects/wr-switch-hdl/wiki White Rabbit Cores: http://www.ohwr.org/projects/wr-cores/wiki IEC 61850: Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems IEC 62439: High Availability Automation Networks

Thanks for your attention