IEC TC57 WG10 New Orleans, 2017-10-23 Hubert Kirrmann, Solutil Time Synchronization Test Interoperability test, New Orleans 2017, October 14-19 Summary report for IEC TC57 WG10 IEC TC57 WG10 New Orleans, 2017-10-23 Hubert Kirrmann, Solutil

Participants Manufacturers Witnesses ABB NIST Belden – Hirschmann Solutil Doble Cisco GE – Alstom Meinberg MOXA NR OMICRON (lead) Siemens - RuggedCom SEL KERI Vizimax

Tests Grandmasters (GM) individual (9) Transparent clocks (TC) individual (10) Transparent clocks chain (16) Boundary clocks (BC) individual (7) Boundary clocks chain (5) Leap second insertion in boundary clocks chain

Hirschmann RSPE35 1 Gbit/s Full-scale TC test GPS emulator Keri 1 Gbit/s Hirschmann RSPE35 1 Gbit/s GE H29 100 Mbit/s Siemens 2288 100 Mbit/s GE H49 100 Mbit/s Siemens 2488 100 Mbit/s Siemens 2488 100 Mbit/s CISCO IE 4010 100 Mbit/s Meinberg 1 Gbit/s GM TC TC TC TC TC TC TC TC Hirschmann RSP20 100 Mbit/s GE 620 100 Mbit/s Hirschmann RSP35 100 Mbit/s Hirschmann EES35 100 Mbit/s Hirschmann EES25 100 Mbit/s Hirschmann RPS35 100 Mbit/s Hirschmann 1040 100 Mbit/s MOXA PTG 100 Mbit/s Omicron 100 Mbit/s TC TC TC TC TC TC TC GM TC OC OC OC OC OC GE Omicron Visimax Opal Hirschmann netgear deviation less than 300ns , jitter up to 500 ns when changing masters with timeout or priority NIST dashboard Wireshark 1 PPS output

Test results Clocks accuracy is about twice better than what IEC/IEEE 61850-9-3 requires. Chain accuracy better than IEC/IEEE 61850-9-3 and substation applications require (± 300ns after 16 transparent clocks, ± 600ns after 7 boundary clocks) Largest excursion occur when switching grandmasters and removing / restoring GPS, but still well below 1 µs (except for one case between two specific clocks) All grandmasters degrade their clockAccuracy in holdover, but some BC do not. BCs introduce longer time constants in the network which lengthens recovery from loss of grandmaster. Users should avoid BCs where TCs can be used, but BCs are useful to keep synchronized islands (e.g. process bus) in case of failure of the grandmaster. Network is plug-and-play, except for 1-step <> 2-step and 100 Mbit/s <> 1 Gbit/s (an implementation issue). Domains worked in parallel without interference. VLAN and priority are useless for PTP traffic, which is always treated as high priority by the bridges (hardware time-stamping ensures precision). VLANs were not used. Incompatibilities were limited to the optional ALTERNATE_TIME_OFFSET TLV (IRIG-B connection).

Implementation issues All Some clocks do not accept indifferently 1-step and 2-step at ingress Grandmasters: Some GMs indicate a high accuracy even in holdover (no GPS) Few GMs adjust the variance (use a fixed value) which makes it useless. GMs adjust “TimeTraceable” and “FrequencyTraceable” differently GMs interpret Alternate_Time_Offset_Indicator TLV differently. GMs should use the port MAC address as source, not the GM identity GM switchover causes temporary excursions of up to 2.5 µs (one case) Transparent clocks: Some TCs do not respond outside of their primary time domain. Boundary clocks: Few BCs in holdover degrade their clockAccuracy. Some BCs do not forward IP traffic. BCs should indicate the time source of their grandmaster, not “PTP” BCs in holdover (loss of GM) clear leap second indicator when not time traceable Slaves: Some slaves do not observe 2-step correction both in Sync and in Follow_Up (?) Media converters: Need calibration and symmetric behavior.

Standard issues IEC/IEEE 61850-9-3 should: Require padding Sync to the same length as Pdelay_Req (Pdelay_Resp) hat to avoid large errors in media converters (1Gbit/s <> 100 Mbit/s) . Define the behavior of Grandmasters at startup / loss of GPS for the TimeTraceable/FrequencyTraceable flags. Define the behavior of Boundary Clocks at startup / loss / restore Grandmaster to avoid that a bad quality BC takes over as master while it is not yet in sync with its GM. Specify if the leap second indication is also valid when TimeTraceable == false (otherwise the slaves miss the leap second). Insist that slaves must select the best master when several are available, which can be the case momentarily. Relate PTP clock quality to 61850-7-2 TimeStamp and SmpSync. Specify lock-in time or at least ask the manufacturer to provide it. General agreement that all technical traffic should use TAI time scale, not UTC.

Testing issues Companies did not prepare sufficiently for the test All clocks brought to the test should have a 1-page datasheet / PICS (only 3 manufacturers did) Manufacturers should implement network management (SNMP / IEC 61850) Poor discipline: e.g. test rigs disrupted because a participant makes changes without telling Future tests need sniffing device capable of logging the exact moment of the leap second. We need better tools (time-triggered Ethernet frame generator as fake grandmaster). Clocks should have a 1 PPS electrical output for substation network commissioning.

What we could not test Exact behavior at leap second insertion Traffic insertion with L3 traffic on the same domain (should never occur) BMCA switchover with a master set at a different time (but result is predictable) High traffic load Behavior in redundant network topology (PRP / HSR) (this was done in the integration test on Substation 1)

IEEE C37.238 issues Few clocks support C37.238-2017, some support only C37.238-2011, most neither. Unclear what makes C37.238 devices compliant: domain number, TLV? Only one clock passed the (simplified) ALTERNATE_TIME_OFFSET TLV test since C37.238-2017 prescribes unnecessary differences from IEEE 1588 §16.3. (the full ATOI test was skipped). No TC updates dynamically the timeInaccuracy field in the IEEE_C37_238_TLV. IEEE_C37_238_TLV (with timeInaccuracy) is useless for the main inaccuracy source, which is master switchover. General agreement that IEEE C37.238:2017 adds no value, but just confusion and development / testing effort.

Utility Communication Architecture