APS Dual Site ECC Primary Site Shutdown/Restart CIPC Confidentiality: Public Release
The Need for a Power Outage Information Services installing a new UPS for Business Continuity. Two feeder breakers identified as needing replacement. Power outage to entire building during breaker work. ECC Migration to alternate site
EMS/ECC Shutdown Failover primary EMS application functions to alternate site. Failover front-end processors to alternate site. Failover EMS business applications to alternate site. Failover primary EMS network communications to alternate site. Ensure ECC personnel onsite at DV-ECC and ready for site cutover. Prepare and test Avtec communications for transition to alternate site.
EMS/ECC Shutdown Ensure EMS support personnel are at both sites and ready for cutover. Transfer Avtec communications from primary site to secondary site. Shutdown ECC consoles at the primary site. Ensure cutover is complete. Obtain ECC approval and power down all remaining EMS servers and related equipment.
EMS Functional/Logical Distribution
EMS/ECC Restart Ensure EMS support personnel are at both sites and ready for cutover. Restart and validate operation of computer network switches at primary site. Restart and validate operation of the Avtec communication equipment at primary site. Restart and validate operation of EMS Business application services at primary site. Failover primary EMS network communication to primary site.
EMS/ECC Restart Power up and validate EMS application running in backup configuration from primary site. Transfer Avtec communications from secondary site to primary site. Failover front-end processors to primary site and validate proper scanning. EMS servers functionally evaluated in backup configuration at primary site. ECC move from secondary location to primary location.
Learning's and Action Items Personnel at alternate site do not have access to required areas Addressed issue with responsible Security Officer and re-enforced importance of accurate access level assignments. Avtec communications not properly transferring to alternate site. Periodically test T1 cards used for Ring Down assignments at alternate site. Shutting down/restarting network equipment caused temporary EMS console anomaly. EMS and network engineering are still working together to mitigate this occurrence in the future. Non-critical network switches powered up after end user equipment Have more network engineering resources on site to assist with tasks.
Learning's and Action Items Alarm for communications equipment indicating communications fault Document process for correct equipment startup and expectations for restart times. Sonnet equipment configuration not backed up prior to shutdown Ensure procedures are in place to backup Sonnet equipment configuration is properly backed up. Historical data not functional after restart. Modify restart procedure and change chronological order of EMS network site transfer to ensure there are no outstanding dependencies.
Conclusion Breaker replacement was successful. Processes and scripts have demonstrated success APS EMS/ECC functionally split between sites.