9/11 - Respond and Recover: A Case Study Joy Heath-Porter Sidley Austin Brown & Wood LLP LawNet 2002 August 21, 2002
Who are we? Large, multi-office, multi-national firm 14 offices; six countries 1,500 lawyers; 3,000 total personnel Two offices in New York before September 11; 600 person office in the North Tower of the World Trade Center
Status on September 10 Involved in one of the largest law firm mergers ever Two networks with different hubs Two different and document management systems Two different telephone systems
Status on September 10 (contd.) Same financial system, but on different versions with different database engine Still in the process of consolidating technology Had just moved equipment to WTC previous weekend
Brief Chronology 7:50 a.m. Tuesday - how we found out; first steps Mid-morning Tuesday - disaster recovery steering group assembled; began to map out recovery strategy Noon Tuesday - offsite storage provider contacted; arrangements made to drive tapes to Chicago
Brief Chronology (contd.) Throughout Tuesday - made sure phones were manned, so that staff calling in would have a live voice; determined status of our people; identified and arranged for recovery resources; Chicago and New York offices closed, but many staff members were working C.O.B. Tuesday - system recovery priorities established; servers identified for recovery
Brief Chronology (contd.) Tuesday night dial-in information numbers established and published; information made available via the Firms Web site; temporary space identified Wednesday morning - all offices contacted for spare equipment; arrangements made for rental equipment; PC vendor contacted for purchase of equipment; cabling began
Brief Chronology (contd.) Wednesday noon - Tapes arrived; restoration process began Wednesday p.m. - voice mail accounts on alternate voice system began to be created for lawyers Wednesday night- new server built; account creation in process; Internet mail redirected (brownwoodlaw.com)
Brief Chronology (contd.) Thursday a.m. - all lawyers had accounts and mail was flowing; equipment was shipped out from other Firm offices Thursday p.m. - Internet café established in alternate NY sight; image build began for rental equipment Thursday night - secure extranet site established for technology information
Brief Chronology (contd.) Friday a.m. - document management was available to NY users Friday noon - equipment began arriving from other offices and from vendor Friday p.m. - financial systems restoration was complete and consolidation began; image for rentals and new PCs finalized
Brief Chronology (contd.) Saturday a.m. - Rental equipment began arriving; imaging began; teams from Chicago began driving to New York transporting equipment not available in New York (cell phones, card keys, etc.) Throughout weekend - Phones, network equipment, printers, desks, PCs rolled out in temporary space
Six Days Later Monday 10:00 a.m. - all New York personnel assembled in a mid-town hotel for a briefing Monday noon - all New York personnel reported for work in temporary space
Today All servers returned to New York All technology migrated to firm standards New York office moved into new, permanent home over July 4 weekend
Key Aspects of Disaster Recovery Approach Maniacal approach to tape backups - daily offsite storage of tapes; offsite location outside metropolitan area Focus on running our daily operation to perfection Standardized hardware (network, PCs, phones) with standard images
Key Aspects of Disaster Recovery Approach (contd.) Emphasis on documentation and standard operating procedures Commitment to serving our lawyers the way our lawyers serve the Firm's clients
Lessons Learned/Affirmed You cannot plan for a plane hitting your building - accept the fact that you will have to improvise to some degree in virtually any disaster You cannot have too many friends - clients, vendors, co-counsel, even competitors Practice, practice, practice - staff need to be able to perform basic disaster recovery tasks "in their sleep"
Lessons Learned/Affirmed (contd.) If you have multiple offices, leverage them The fact of back-ups is not enough - they must be quality-checked, verified, and viable Efficiencies of centralization must be weighed against risks from a disaster recovery perspective
Lessons Learned/Affirmed (contd.) Recruit for character and creativity, as well as technical expertise Understanding the relative priority of your critical applications cannot be overstated - in a disaster situation, restoring contact information may be more important than restoring documents
Lessons Learned/Affirmed (contd.) Web access to your systems is not a luxury; it is a necessity Stress with principals the importance of using firm-standard repositories - if documents are in three different places, restoring access becomes much more difficult
Lessons Learned/Affirmed (contd.) Think about disaster recovery when making a buy/build decision Smart, dedicated people ultimately make it happen
How does the Story End? We were never down Clients were impressed Our most important assets - our smart and dedicated people - were not lost The loss of people is the only disaster truly to fear - if you are prepared, everything else can be recovered or replaced