1. PORTFOLIO COMMITTEE ON HOME AFFAIRS SITA REPORT ON CENTURION DATA CENTER POWER OUTAGE Dr Setumo Mohapi 14 August 2018

2. Interventions/ mitigations to address the downtime
CONTENT 1
Reasons for downtime 2
Interventions/ mitigations to address the downtime 3
Solutions to prevent future occurrences

3. REASON S FOR DOWNTIME Tshwane Municipality power outage
Faulty cables
Both power grid feeds from Kentron or NIFS were down
Tshwane Municipality power outage
Due to automatic fuel system not keeping the diesel tank full
Fuel pump had burnt out and a new pump was dispatched
Installation and commissioning of replacement pump and fuel system was completed at 12:50
Containerised generator stopped and could not be started
Due to high workload
The voltage on the UPS system was at a low level
Power was restored to data centre at 13:00
Start-up awaited generators to be optimal and UPS’ charged to such a level, before an ENTIRE start-up could commence
Go-ahead for start-up of Mainframe and other infrastructure was given at 13:45
Mainframe operating systems were started by 14:58 and applications available by 15:12
All servers on Open System available at 19:45
Power Grids were restored at 15:21
Service restored

4. MITIGATIONS TO ADDRESS THE DOWNTIMENr
Reason for downtime
Mitigation 1
Tshwane Municipality to contact SITA when there is a power outage; especially as both power grid feeds from Kentron or NIFS were down
Liaise with Tshwane Municipality and update contact details to ensure timely notification for power outages; 2
All stakeholders did not received SMS notifications that the containerised generator was down
The clients were phoned (technical level)
Update the SMS messaging system linked to the infrastructure;
Link the messaging system to the ITSM at SITA SMC and ensure that all lines of business are included;
Acquire a building management system with a messaging system for the data centre infrastructure;
Update the overall Centurion Data Centre disaster and notification processes for all Lines of Business and clients; 3
SMS notifications that didn’t go through to all stakeholders
SMC ITSM Technical team, will assist with the Integration between SITA SMC-ITSM and SMS Messaging system which will automate communication channel in future for all our stakeholders; 4
The containerised generator stopped due to the automatic fuel system not keeping the day tank full
Acquire a dual fuel pumping system and a spare pump for emergency spares.

5. NETWORK DOWNTIME EXPERIENCED ON 13 AUGUST 2018
The network outage that impacted Government departments’ service delivery on Monday 13 August 2018 from 10:05 until 11:40 is related to the first power outage that happened on Friday 10 August 2018 in Centurion; due to a Municipal power failure.
The Broadband Infraco (BBI) transmission equipment; inside the SITA Centurion Switching Centre which had been running on battery power since the power outage on Friday; switched off after the battery was depleted.
Upon investigation it was discovered that the circuit breaker on the main power grid which feeds the Broadband Infraco (BBI) transmission equipment had tripped when the Municipal power was restored on Friday 10 August 2018; causing the equipment to continue running on battery power as it was not receiving any power from the main power feed.
The amperage on the main power grid was adjusted and the circuit breaker restored; and the BBI transmission equipment was powered up at 11:40; all impacted network services were restored.

6. SOLUTIONS TO PREVENT FUTURE OCCURENCES
SITA will fast-track its efforts in the development of a long term data center strategy in order to align with new technologies as to mitigate risk, reduce costs, to be able to use new technologies or look at the possibility to maximize the real estate portfolio.
A data center strategy requires an integrated approach with alignment of the business and technology goals of SITA and the SA Government.
This will allow SITA to migrate to a data centre solution that improves agility, security and business results.
Furthermore SITA will immediately work to improve its incident recovery procedures to ensure the network outage experienced on Monday 13 August 2018 does not re-occur.

7. THANK YOU