NERC Published Lessons Learned December 2016 Rev1

NERC Lessons Learned - December 2016 Two NERC Lessons Learned (LL) were published in the December 2016 LL20161201 Loss of ICCP – Local Control Center Notifications LL20161202 SCADA System Software Design Flaw Prevented Processing of Alarms and Events

Loss of ICCP - Details Entity A’s control center experienced multiple ICCP communication failures with external entities Due to the ICCP failures, Entity B’s State Estimator tool failed to solve because it was not receiving ICCP data from Entity A, resulting in a loss of their state estimator monitoring capabilities Entity B was the only entity in the area that experienced a failure of its state estimator during Entity A’s loss of ICCP data

Loss of ICCP – Corrective Actions Entity A returned the firewall to its pre-event state to stabilize the ICCP link Entity B restored the last good values from Entity A to their State Estimator model

Loss of ICCP - Lessons With the increasing reliance on real-time data and accurate state estimator solutions, entities are becoming more dependent on data from other TOPs TOPs and RCs should be encouraged to notify all BAs and TOPS in the area, (not just those adjacent BAs and TOPs) of a loss of ICCP This allows for other’s EMS support teams to make appropriate adjustments to their systems if needed

SCADA System Software Design Flaw - Details SO entered a comment on a SCADA alarm to provide additional information The comment was 179 characters in length (max allowable for this field) Character limit was by design based on the length of the field within the database A defect in the SCADA system only allowed the first 90 characters of the comment to be processed, but allowed 179 characters to be typed

SCADA System Software Design Flaw - Details The state of the alarm changed back to “normal” but the SO could not clear or delete it The database was designed for a 179 character message but it could only handle the first 90 characters due to a coding error; therefore, it hung while trying to evaluate the remaining part of the alarm note Rebooting the alarm process did not resolve the issue as it crashed every time attempting to process the alarm note

SCADA System Software Design Flaw – Details This filled the alarm processor queue, preventing additional alarms from being displayed, thus impacting the SO’s ability to monitor and control transmission facilities Eventually, the support staff, with assistance from the software vendor, were able to identify and truncate the System Operator’s note to less than 90 characters and successfully clear the database table After initial investigation, it was discovered that the vendor had a patch available to resolve this issue

SCADA System Software Design Flaw – Corrective Actions A software patch provided by the vendor was applied to the SCADA system, preventing future notes from inhibiting the system’s ability to process state changes

SCADA System Software Design Flaw - Lessons Software users should consider stress testing all user- enterable fields In this event a non-essential function became a single point of failure Be proactive about reviewing vendor patches, regardless of how insignificant they may appear Engaging vendor support early may help identify and resolve problems faster

SCADA System Software Design Flaw A centralized monitoring of the system’s components and processes may help diagnose these issues sooner SOs should be trained on the importance of situational awareness as it relates to EMS health Early detection of EMS health issues and notification to EMS support personnel leads to quicker resolution

Lessons Learned Survey Link NERC’s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the bulk power system NERC requests that industry provide input on lessons learned by taking the short survey. A link is provided in the PDF version of each Lesson Learned

Hassan Hamdar hhamdar@frcc.com