Recovery from the earthquake Takashi Sasaki

Disaster recovery “Disaster” comes from human error or hardware failure was considered before We were preparing for such “disasters” – Regular disk backup – System design with high availability – UPS How we could prepare for the big natural disaster?

What happened on 3.11 The scale of the earthquake at Tsukuba was 6- in Japanese scale(0,1,2,3,4,5-,5+,6-,6+,7-,and 7+) at 2:46P.M. on March 11 th, 2011 JST The electric power facility of KEK was damaged and no power was supplied until March 13 – Sudden power cut-off was happened – In city center, only a few minutes interruption of electricity happened All of the employee except some staffs are ordered to stay at home and stand by

KEK

Electric power supplies On March 13rd, electric power supply recovered partly – Only 2MW(727KVA) for all of KEK – 275KW(100KVA) for Computing Research Center From March 22 nd, gradually the breakers were turned on at building by building From March 28 th, all of the employee are ordered to come back to the normal business Because of the nuclear plant accident, usage of electricity was restricted until the end of September – We are expecting to increase the costs of electric power from the next Apri – All of the nuclear plan in Japan had been shut down

Damages on computer systems Super computer – No damage because the system had been shutdown for the system replacement Belle Computing system – Many disks were broken Central computer – GPFS MDS had inconsistency because of sudden power outage and some files were lost – No hardware damage Networking – No hardware trouble

Central computer The racks were not bolted on the floor – Each racks equips skirts to prevent

Telecommunication Infrastructures were not damaged Phones were not useful – Because of congestion Internet works w/o problems – Everybody accessed the information from web sites using their mobile/smart phones s worked fine also

Recovery of services Efforts to recover services begun on March 14 th – The first priority was communication services Internet web – Temporally, with minimum configuration

A natural disaster strikes when people lose their memory of the previous one.

How we prepare for the next one? Off-site backup is important Stand-by servers on a cloud service Web service is rather easier – Always we may hold the link to the stand-by server on the top page Search engines like google will guide the people Mail service – Out sourcing ? Everywhere in Japan have a chance for a big earthquake Mail contents are too sensitive to store off-site

Discussion How we over come the security issues? – Offsite backup should be encrypted Off site backup – We want to backup the VM and disk images outside of KEK for major services