Making the best of a bad situation — Dealing with the aftermath of a major server crash Patty Brown & Roger Gant Instructional Design & Development Elon University Copyright Patty Brown & Roger Gant, This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

Alternate title: The crash of ’03 — How we survived without being lynched

Overview Focus is not technical Background Chronology of crash-related events Lessons learned Continuing challenges Suggestions, resources

Background Elon University Private university in NC Piedmont Focus on undergraduate education & liberal arts 252 full-time & 72 part-time faculty 4,584 students in fall 2003

Background Elon Academic buildings & res. halls wired #43 on Intel’s "Most Unwired College Campuses" list Student access to 620 workstations in 19 computer labs 98% of students bring their own computers

Background Course Mgmt Elon Bb campus since CourseInfo in Our role –End-user support & training –Sys admin privileges for managing daily course & account operations Sys Admin unit –Handles hw/sw installation & configuration –Reports to different manager

Instructional Des. Ins. Support. Liaison Assoc. Director Instructional Des. Multimedia Dev. Liaison for Sci/Math n = 53

Background Course Mgmt System – Spring 2003 Running Bb 5.5 ‘Level I’ –Basic, non-enterprise version –1 server –Heavily used, widely accepted Plan to upgrade from Bb 5.5 to Bb 6.0 enterprise level over summer for use in fall

Chronology of crash The best laid plans of management & system administrators often go awry June 2003, shipping mix up & subsequent negotiations cause delay Revised plan: –Postpone rollout until spring 2004 –Test small # of courses winter term

Chronology of crash “Uh oh…” In October, log files cause storage issues Resourceful fix enables business as usual Routine safeguards (backups) normal to all appearances Preparations continue for rollout of Bb 6 Server crashes – hard – just before Thanksgiving Sys admin eventually restores from system backup and log files

Chronology of crash From “Uh oh” to “Oh, no!” Server restored BUT… All transactions after ‘fix’ are lost — appx. 4 weeks of data –Midterm grades and more No better outcome despite all attempts to recover data Some content files accessible from server Alternate solution found for original storage problem – so 5.5 OK through Dec.

Artner’s Law of Intermediate Crisis Management: “The only small problem is one that happens to the other guy.”

Response and recovery Now what? Meetings & communications –No search for guilty, no blame for innocent Lesson: Working relationship is all important Decide to move up Bb 6 rollout to winter We’re still in this together –System Admin readies new servers –IDD deals with Fallout from crash Creating & implementing expedited rollout

Response and recovery Fallout from crash Communicating frequently to faculty –Lesson: An in time saves 9 Responding to calls, s Long hours in December Managers don flak jackets Faculty response: Sympathy! –Lesson: Good service and relationship built over time pays off

Response and recovery Other saving graces Official grades not maintained in Bb Most faculty had source material & grades outside of Bb Made salvaged Bb files available by request Timing; semester ends soon

Response and recovery Getting to Bb 6 sooner, not later Archiving Bb 5.5 sites, restoring in 6 Coordinating creation of accounts, courses –Batch enrollment a plus Training, user support needs for Bb 6 –Workshops –New front end for new log-in page

Response and recovery Getting to Bb 6 sooner, not later Archiving Bb 5.5 sites, restoring in 6 Coordinating creation of accounts, courses Training, user support needs for Bb 6 –Workshops –New front end for new log-in page Shepherding the switch

Crisis management challenges Physical & organizational structure –Reassess? Increased collaboration, joint decision-making Calls for informal relationship-building

Ongoing challenges An ounce of prevention… Differing priorities for System Admin & IDD –Means we have to forecast & articulate needs Shared responsibility –Requires proactive measures Planning for upgrades Communication

Are you ready for a small catastrophe?

Suggestions Identify areas of potential concern Build relationships in advance Make plans but stay flexible Be prepared –To communicate, communicate –To modify plans Don’t sweat the small stuff –To negotiate solutions –To accept less than 100% satisfaction Books, articles, consultants Make your own action plan & test

Keep working at it…