Presentation is loading. Please wait.

Presentation is loading. Please wait.

Disaster Documents The role of documentation in disaster recovery

Similar presentations


Presentation on theme: "Disaster Documents The role of documentation in disaster recovery"— Presentation transcript:

1 Disaster Documents The role of documentation in disaster recovery
by Ray Kim

2 About me... Blog: http://pianorayk.wordpress.com
LinkedIn: Working in technology since 1989: Computer operator , instructor , technical writer , analyst , webmaster , developer Member of CASSUG (Albany SQL user group) and Albany UX/UI user group BS in computer science, Syracuse University (LET’S GO ORANGE!!!) MS in technical communication, Rensselaer Polytechnic Institute (GO RED!!!) Interests: My wife (Lianne) and two cats (Bernard, Nutmeg) Playing music (four different instruments, member of KKΨ band fraternity) Sports: Syracuse football and basketball, RPI ice hockey, NY Yankees, fantasy football, CrossFit

3 About this presentation
This is NOT a technology-specific presentation This presentation is based on personal experience I don’t like to lecture – I prefer to discuss issues and act as a facilitator Please ask questions! Feel free to engage!

4 About this presentation
Here’s what we’ll discuss: The backstory The documents What was missing Where we got lucky After the planes hit Aftermath Lessons learned

5 This presentation… …is based on events that took place on September 11, 2001 …does not discuss disaster recovery technologies or procedures …focuses on the documentation in and of itself

6 Background info… I was an employee of Empire Blue Cross on Sept. 11, 2001, working primarily out of the Albany, NY office Empire had an office in the World Trade Center I worked for a department called Enterprise Server Technologies (a.k.a. the “Server Team”) The Server Team supported several hundred servers, including large data centers in Albany and the World Trade Center

7 A little about the Server Team
Typical team setup Manager Department leader Server jockeys Responsible for building and maintaining the server environment Me – analyst Provide the rest of the Server Team with information or resources Support the support department (me)

8 Some things to bear in mind
“Internet” was not yet a household term Online documentation was still mostly in its infancy Cloud computing? What’s that? (It wasn’t yet widespread) What’s so smart about my phone? (“Smartphones” didn’t exist) Some things that are now automated were manual processes in 2001 We still relied a lot on hardcopy documentation

9 The calm before the storm
I had developed these documents for the Server Team: Server installation checklists Server room maps Instructions for sending and obtaining tape backups to and from Iron Mountain Vendor contact lists Internal telephone contact list At the time, these documents were primarily hardcopy, PDF, Word, or Visio files. Online documentation was still a foreign concept. All of these documents were critical when disaster struck.

10 Server installation checklists
What it was: Checklist for each completed server Why it was important: configuration info for each server Created and maintained in MS Word Completed for every server built Server information: purpose, OS, hardware, software, and configurations Hardcopy checklists were stored in a file cabinet PDFs scanned and stored on a server

11 Server room maps What it was: data center floor plan
Why it was important: inventory and location of all servers Maintained in Visio Provided rack layouts and server locations Multiple hardcopies in all server rooms Updated manually every few weeks

12 Iron Mountain instructions
What it was: Backup instructions and Iron Mountain contact information Why it was important: obtain backups from offsite storage Iron Mountain contact information Instructions on preparing backup tapes for pickup and requesting backups from Iron Mountain Regular weekly backup pickup schedule Obtain backups as needed

13 Vendor contact lists What it was: Vendor contact information Why it was important: supplier contact info Created in MS Word (?) Vendor contacts with whom we maintained a business relationship Hardware, software, even some swag

14 Internal telephone contact list
What it was: Business card-sized list of internal contacts Why it was important: contact info for key employees Created in MS Word in a business card template Included office, home, and mobile phone numbers for important internal contacts Distributed to all Server Team and key staff members

15 What we didn’t have Standard operating procedures
Documented business continuity plan No known document that explained business roles Documented disaster recovery plan Backup departmental intranet/file server What if our intranet/file server had been in the WTC?

16 Where we got lucky Original documents were in Albany, not WTC
Hardcopies of nearly all documents File copies were in multiple locations Documents were readily accessible High-functioning, close-knit team What we didn’t have didn’t kill us

17 DISASTER STRIKES All productivity came to a halt
Nine Empire employees died Personnel convened in Albany Strategy to rebuild critical server infrastructure in Albany First: Domain controllers Second: Critical data servers Additional servers prioritized based on business importance Contact lists were the first document distributed

18 The next two weeks… Round-the-clock shift work
Each shift would pick up where previous left off Server room maps provided WTC inventory Installation checklists provided configurations Vendors supplied us with needed inventory In-house inventory Vendors supplied additional hardware and software Requests were expedited, no questions asked Iron Mountain provided data backups

19 The next two weeks… Maintaining morale was important
Out-of-town workers allowed breaks to travel home Encouraged to maintain normalcy outside of office (home, family, extracurriculars, etc.) We were well-fed – lots of pancakes, eggs, sausage, and bacon

20 After the storm was over
After two weeks… Operational server infrastructure Out-of-town workers returned home Some temporarily assigned to other regional offices Long term… NYC office relocated to Brooklyn (MetroTech) New state-of-the-art technology Documentation went online (SQL Server back-end)

21 What’s changed since 2001 2002 – Department of Homeland Security established 2002 – Sarbanes-Oxley passed 2005 – SQL Server 2005 supports database mirroring 2006 – Amazon Web Services introduces their cloud document storage service 2007 – LG Prada and iPhone introduced; smartphones start becoming mainstream 2016 – GDPR adopted by European Union

22 Lessons learned Documentation is critical for recovery
Documentation must be accessible Have multiple copies and backups Have a recovery plan Communicate with your team Do NOT assume team understands how to recover Conduct disaster drills, if you are able Maintain morale – rest is important!

23

24 Visit my ‘blog at https://pianorayk.wordpress.com


Download ppt "Disaster Documents The role of documentation in disaster recovery"

Similar presentations


Ads by Google