1. NERC Published Lessons Learned
December 2016
Rev1

2. NERC Lessons Learned - December 2016
Two NERC Lessons Learned (LL) were published in the December 2016
LL Loss of ICCP – Local Control Center Notifications
LL SCADA System Software Design Flaw Prevented Processing of Alarms and Events

3. Loss of ICCP - Details
Entity A’s control center experienced multiple ICCP communication failures with external entities
Due to the ICCP failures, Entity B’s State Estimator tool failed to solve because it was not receiving ICCP data from Entity A, resulting in a loss of their state estimator monitoring capabilities
Entity B was the only entity in the area that experienced a failure of its state estimator during Entity A’s loss of ICCP data

4. Loss of ICCP – Corrective Actions
Entity A returned the firewall to its pre-event state to stabilize the ICCP link
Entity B restored the last good values from Entity A to their State Estimator model

5. Loss of ICCP - Lessons
With the increasing reliance on real-time data and accurate state estimator solutions, entities are becoming more dependent on data from other TOPs
TOPs and RCs should be encouraged to notify all BAs and TOPS in the area, (not just those adjacent BAs and TOPs) of a loss of ICCP
This allows for other’s EMS support teams to make appropriate adjustments to their systems if needed

6. SCADA System Software Design Flaw - Details
SO entered a comment on a SCADA alarm to provide additional information
The comment was 179 characters in length (max allowable for this field)
Character limit was by design based on the length of the field within the database
A defect in the SCADA system only allowed the first 90 characters of the comment to be processed, but allowed 179 characters to be typed

7. SCADA System Software Design Flaw - Details
The state of the alarm changed back to “normal” but the SO could not clear or delete it
The database was designed for a 179 character message but it could only handle the first 90 characters due to a coding error; therefore, it hung while trying to evaluate the remaining part of the alarm note
Rebooting the alarm process did not resolve the issue as it crashed every time attempting to process the alarm note

8. SCADA System Software Design Flaw – Details
This filled the alarm processor queue, preventing additional alarms from being displayed, thus impacting the SO’s ability to monitor and control transmission facilities
Eventually, the support staff, with assistance from the software vendor, were able to identify and truncate the System Operator’s note to less than 90 characters and successfully clear the database table
After initial investigation, it was discovered that the vendor had a patch available to resolve this issue

9. SCADA System Software Design Flaw – Corrective Actions
A software patch provided by the vendor was applied to the SCADA system, preventing future notes from inhibiting the system’s ability to process state changes

10. SCADA System Software Design Flaw - Lessons
Software users should consider stress testing all user- enterable fields
In this event a non-essential function became a single point of failure
Be proactive about reviewing vendor patches, regardless of how insignificant they may appear
Engaging vendor support early may help identify and resolve problems faster

11. SCADA System Software Design Flaw
A centralized monitoring of the system’s components and processes may help diagnose these issues sooner
SOs should be trained on the importance of situational awareness as it relates to EMS health
Early detection of EMS health issues and notification to EMS support personnel leads to quicker resolution

12. Lessons Learned Survey Link
NERC’s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the bulk power system
NERC requests that industry provide input on lessons learned by taking the short survey. A link is provided in the PDF version of each Lesson Learned

13. Hassan Hamdar hhamdar@frcc.com