1. Disaster Documents The role of documentation in disaster recovery
by Ray Kim

2. About me... Blog: http://pianorayk.wordpress.com
LinkedIn:
Working in technology since 1989:
Computer operator
, instructor
, technical writer
, analyst
, webmaster
, developer
Member of CASSUG (Albany SQL user group) and Albany UX/UI user group
BS in computer science, Syracuse University (LET’S GO ORANGE!!!)
MS in technical communication, Rensselaer Polytechnic Institute (GO RED!!!)
Interests:
My wife (Lianne) and two cats (Bernard, Nutmeg)
Playing music (four different instruments, member of KKΨ band fraternity)
Sports: Syracuse football and basketball, RPI ice hockey, NY Yankees, fantasy football, CrossFit

3. About this presentation
This is NOT a technology-specific presentation
This presentation is based on personal experience
I don’t like to lecture – I prefer to discuss issues and act as a facilitator
Please ask questions! Feel free to engage!

4. About this presentation
Here’s what we’ll discuss:
The backstory
The documents
What was missing
Where we got lucky
After the planes hit
Aftermath
Lessons learned

5. This presentation…
…is based on events that took place on September 11, 2001
…does not discuss disaster recovery technologies or procedures
…focuses on the documentation in and of itself

6. Background info…
I was an employee of Empire Blue Cross on Sept. 11, 2001, working primarily out of the Albany, NY office
Empire had an office in the World Trade Center
I worked for a department called Enterprise Server Technologies (a.k.a. the “Server Team”)
The Server Team supported several hundred servers, including large data centers in Albany and the World Trade Center

7. A little about the Server Team
Typical team setup
Manager
Department leader
Server jockeys
Responsible for building and maintaining the server environment
Me – analyst
Provide the rest of the Server Team with information or resources
Support the support department
(me)

8. Some things to bear in mind
“Internet” was not yet a household term
Online documentation was still mostly in its infancy
Cloud computing? What’s that? (It wasn’t yet widespread)
What’s so smart about my phone? (“Smartphones” didn’t exist)
Some things that are now automated were manual processes in 2001
We still relied a lot on hardcopy documentation

9. The calm before the storm
I had developed these documents for the Server Team:
Server installation checklists
Server room maps
Instructions for sending and obtaining tape backups to and from Iron Mountain
Vendor contact lists
Internal telephone contact list
At the time, these documents were primarily hardcopy, PDF, Word, or Visio files. Online documentation was still a foreign concept.
All of these documents were critical when disaster struck.

10. Server installation checklists
What it was: Checklist for each completed server
Why it was important: configuration info for each server
Created and maintained in MS Word
Completed for every server built
Server information: purpose, OS, hardware, software, and configurations
Hardcopy checklists were stored in a file cabinet
PDFs scanned and stored on a server

11. Server room maps What it was: data center floor plan
Why it was important: inventory and location of all servers
Maintained in Visio
Provided rack layouts and server locations
Multiple hardcopies in all server rooms
Updated manually every few weeks

12. Iron Mountain instructions
What it was: Backup instructions and Iron Mountain contact information
Why it was important: obtain backups from offsite storage
Iron Mountain contact information
Instructions on preparing backup tapes for pickup and requesting backups from Iron Mountain
Regular weekly backup pickup schedule
Obtain backups as needed

13. Vendor contact lists
What it was: Vendor contact information
Why it was important: supplier contact info
Created in MS Word (?)
Vendor contacts with whom we maintained a business relationship
Hardware, software, even some swag

14. Internal telephone contact list
What it was: Business card-sized list of internal contacts
Why it was important: contact info for key employees
Created in MS Word in a business card template
Included office, home, and mobile phone numbers for important internal contacts
Distributed to all Server Team and key staff members

15. What we didn’t have Standard operating procedures
Documented business continuity plan
No known document that explained business roles
Documented disaster recovery plan
Backup departmental intranet/file server
What if our intranet/file server had been in the WTC?

16. Where we got lucky Original documents were in Albany, not WTC
Hardcopies of nearly all documents
File copies were in multiple locations
Documents were readily accessible
High-functioning, close-knit team
What we didn’t have didn’t kill us

17. DISASTER STRIKES All productivity came to a halt
Nine Empire employees died
Personnel convened in Albany
Strategy to rebuild critical server infrastructure in Albany
First: Domain controllers
Second: Critical data servers
Additional servers prioritized based on business importance
Contact lists were the first document distributed

18. The next two weeks… Round-the-clock shift work
Each shift would pick up where previous left off
Server room maps provided WTC inventory
Installation checklists provided configurations
Vendors supplied us with needed inventory
In-house inventory
Vendors supplied additional hardware and software
Requests were expedited, no questions asked
Iron Mountain provided data backups

19. The next two weeks… Maintaining morale was important
Out-of-town workers allowed breaks to travel home
Encouraged to maintain normalcy outside of office (home, family, extracurriculars, etc.)
We were well-fed – lots of pancakes, eggs, sausage, and bacon

20. After the storm was over
After two weeks…
Operational server infrastructure
Out-of-town workers returned home
Some temporarily assigned to other regional offices
Long term…
NYC office relocated to Brooklyn (MetroTech)
New state-of-the-art technology
Documentation went online (SQL Server back-end)

21. What’s changed since 2001
2002 – Department of Homeland Security established
2002 – Sarbanes-Oxley passed
2005 – SQL Server 2005 supports database mirroring
2006 – Amazon Web Services introduces their cloud document storage service
2007 – LG Prada and iPhone introduced; smartphones start becoming mainstream
2016 – GDPR adopted by European Union

22. Lessons learned Documentation is critical for recovery
Documentation must be accessible
Have multiple copies and backups
Have a recovery plan
Communicate with your team
Do NOT assume team understands how to recover
Conduct disaster drills, if you are able
Maintain morale – rest is important!

24. Visit my ‘blog at https://pianorayk.wordpress.com