1. Central Research Institute of Electric Power Industry D2-01_24 Prototype and Evaluation of Communication Network for a WAMPAC System Based on International Standards System Engineering Research Laboratory 2013 PS1: Role of ICT in Power System CIGRE SC D2 Colloquium on Smart Grid November 14, 2013 Yoshizumi Serizawa 1

2. Classification of WAMPAC system 2013 2 1 ms100 ms1 s1 min.10 min. Timescale of control Areal range of influence Narrow Wide Rotor angle stability (Transient stability) Overload Frequency stability (Wide area) 10 ms10 s Rotor angle stability Cascades phenomena Sampled value Phasor Rms value Voltage stability (Large disturbance) Frequency stability (Islanding) Voltage stability (Small disturbance) Status data for control

3. A configuration of existing WAMPAC system 2013 3 Central Control Computer Terminal Equipment Transfer Tripping Equipment Processed result (Generator to be shed) System-wide state information Shedding command G G G G G TE TT G TE TT TE Dedicated wide area network Legend Starter

4. Int’l standard-based WAMPAC system 2013 4 CE PMU IED - Measurement - Status - Measurement - Status - Control command - Setting -Control sequence -Control table - Measurement - Status -Control sequence -Control table - Setting WAMPAC-GW: − IEEE C37.244 (Phasor Data Concentrator) PMU: − IEEE C37.118.1 − IEC 60255-118-1 (Synchrophasor measurement) CE – WAMPAC-GW communication: − IEC 61970 (CIM) PDC - PMU/IED Communication : − IEC61850-90-1 − IEC61850-90-5 − IEEE C37.118.2 (Synchrophasor data transfer) PMU - IED Communication : − IEC/TR 61850-90-1 (Inter-substation communication) − IEC/TR 61850-90-5 (Synchrophasor communication) Wide area communication: − IEEE 802.1 series (Internetworking, provider backbone bridge, etc.) − Related IETF RFCs (Routing, IP multicast, etc.) Time synchronization: − IEEE 1588 (Precision Time Protocol) − IEEE C37.238 (IEEE 1588 profile for power system) Cyber security: − IEC/TS 62351-1 to 10 (Data and communication security for power system) − IEC/TR 61850-90-5 (Security profile for synchrophasor communication) WAMPAC-GW (CIM – IEC 61850) - Control command IED CE: Central Equipment CIM: Common Information Model WAMPAC-GW: WAMPAC Gateway PMU: Phasor Measurement Unit IED: Intelligent Electronic Device CT, VT CB status and others CT, VT, CB status and others CB

5. Three types of WAN for WAMPAC system 2013 5 L3/MPLS-basedL2-based L2/L3 combined

6. Generic specifications of communication networks 2013 6 Function Communication port, Bandwidth, VLAN (L2-based), Time synchronization, Communication protocol, Multicast operation for information sharing among devices, Identical bidirectional communication route, Prioritized transmission Performance Transmission delay To meet the required response time, 3 to 5 ms among IEDs and PMUs, 1 s between IED/PMU and WAMPAC-GW/CE Transmission delay variation Less than a half of data sampling or transmission interval for ordinal data transmission. Less than 50 μs for time synchronization control channel, avoiding packet contention at normal communication ports. Time synchronization error Less than 50 μs for most stringent applications Transmission error Error rate less than 1×10 −6 Reliability Unavailability, Route assignment and redundancy, Redundancy of time synchronism Cyber security Security management, Availability, Integrity, Confidentiality, Key management, Access control, Network protection

7. Restrictions of IEEE 1588 internetworking 2013 7 L2 switch network without IEEE 1588 scheme IEEE 1588 grand master clock L3 switch or MPLS network without IEEE 1588 scheme IED with IEEE 1588 scheme (slave) L2 switch network with IEEE 1588 scheme PTP messages Sync Follow_UP Delay_Req Delay_Resp etc. Message delivery schemes Unicast/multicast Routing etc. Combination of networksMessage delivery schemes and PTP clock modes L2 with PTP Ordinary L2Ordinary L3 UnicastMulticast E2E-TCP2P-TCBCE2E-TCP2P-TCBC X −− XX −− XXX −− − XX −− −

8. Performance evaluation of IEEE 1588 internetworking 2013 8 L2 switch L3 switch L2 switch IEEE1588 grand master clock IEEE1588 slave clock IEEE1588 slave clock IEEE1588 slave clock IEEE1588 slave clock Ordinary L3 switch network L2 switch network with IEEE 1588 Connection (a) Connection (b) Time synchronization errors  Connection (a): Tens of nanoseconds regardless of traffic congestions  Connection (b): 10 and 24 μs for background traffic loads of 5 and 95% at the L3 link, respectively, and may be much larger for longer packet traffic

9. Performance evaluation of IEEE 1588 with bidirectional IP multicast and MPLS unicast 2013 9 L2 switch IEEE1588 grand master clock IED/PMU (IEEE1588 slave clock) L3 switch (BIDIR-PIM) L2 switch IED/PMU (IEEE1588 slave clock) L2 switch IED/PMU (IEEE1588 slave clock) L2 switch IED/PMU (IEEE1588 slave clock) Rendezvous point Link failure Time synchronization errors  Bidirectional IP multicast: Temporary increase of errors by more than 30 μs (ordinary errors of 1 to 2 μs) upon a sequence of link failure, switchover and recovery  MPLS unicast: Similar to ordinary L2 switch network

10. Prototype WAMPAC system 2013 10 CE WAMPAC-GW IEEE1588 grand master clock L2 switch L3 switch IED PMU IED PMU IED PMU IED PMU L3 switch Personal computer IED/PMU PDC Communication units AMP Applications RTDS Communication cable Communication specifications Transmission delay between IED and WAMPAC-GW ≤ 10 ms Transmission delay between IEDs≤ 10 ms Communication rate of IED and PMU Twice per electrical cycle Bandwidth 400 kbps per IED/PMU Time synchronization error among IED/PMUs < 50 μs

11. Conclusions  Based on the WAMPAC system architecture, the communication network specifications in terms of function, performance, reliability and cyber security were defined.  The time synchronization characteristics were examined for L2/L3 switches with or without IEEE 1588 schemes implemented as well as multicast/unicast operations in IEDs to show a satisfactory synchronization error of a few to tens of microseconds.  A prototype WAMPAC system comprising four IEDs was established, and the operating time from fault occurrence to tripping measured less than 50 ms together with satisfactory communication and time synchronization performance. 2013 11

12. Special report Q1-20: What are the cases considered for evaluation of the proposed prototype of Wide Area Monitoring, Protection and Control (WAMPAC) system based on IEEE 1588 international standard? A1-20: While WAMPAC systems may utilize various types of WAN such as L2-based, L3/MPLS-based and L2/L3-combined networks, IEEE 1588 was originally L2-based and immature for wide area L3 networks. Therefore, the evaluations were conducted to examine the internetworking of IEEE 1588 L2 and non-IEEE 1588 L2/MPLS/L3 networks with multicast or unicast scheme in terms of time synchronization errors. The results showed the internetworked system mostly fulfilled the WAMPAC time synchronism requirement, 50 μs. 2013 12

13. Cases for evaluation PTP master-slave time synchronism via  PTP-L2 + non-PTP- L3 network with unicast and E2E- TC  PTP-L2 network with multicast and E2E-TC/P2P- TC/BC  Non-PTP-L3/L2 network with bidirectional IP multicast  Non-PTP MPLS network with unicast with respect to traffic congestion with/without priority control, packet losses, network failure/recovery, and master clock switch over (BMC) 2013 13

14. 2013 14 Reserve slides

15. Another configuration of existing WAMPAC system 2013 15 Power plant Substation RPU CPU (Central Processing Unit) (a) Upstream information: Pre/post-fault status data, starter signal (fault detection) (b) Downstream information (command): Generator shedding, load shedding, system separation (a) (b) Starter Unit RPU RPU (Remote Processing Unit) Communication Network Microprocessor-based Control computer-based Pre-calculation Post-calculation Disturbance detection

16. Int’l standard-based WAMPAC system 2013 16 IEDPMU IEC 61850/CIM converter, Phasor Data Concentrator, etc. Power system CT, VT, CB Generator excitation control system -Control command -Control sequence -Control scenarios -Measurement -Status -Measurement -Status -Setting -Measurement -Status -Control sequence -Control scenarios -Setting CE WAMPAC-GW PMUIED Steady-state data flow Data flow in the event of the occurrence of a fault