Who Says Servers Can’t Crash? Rocky Mountain PBS Survives Multiple Server Crashes and Lives to tell about it! Presented By Michelle Nesmith Rocky Mountain PBS

Living in a server world Servers control everything we do in one way or another Internet Servers Local Network Servers Traffic Software Servers Video Servers When they crash it can be a disaster to your organization.

How has Rocky Mountain PBS survived server failures? What kinds of failures have we experienced? How have we been impacted by server failures? What have we learn from our experiences? Q & A

After 5 years, what kinds of server failures have we experienced? >Video Server >Automation System >Archive System >Media Manager >Traffic Software Server

Contrary to all the sales pitches Video Servers do crash! >We have experienced the loss of both drives and software on video servers, which means they cannot playback or record anything during that time. >Recognizing this phenomenon, we then implemented new recording procedures so that all daily, weekly and prime time programming is dual recorded so that no matter what fails, we always have the program.

The Automation System that was guaranteed to never fail-did! >Fortunately we only lost one channel’s automation system (and not all 4 at the same time.) >We (the non-believers) had already anticipated this and knew that in this event we would have engineering patch the main RMPBS channel into the feed until the Automation system could be fixed.

The Archive System that was going to run continuously for up to 30 years, died after 2! >The system that we use to house program and interstitial material in computer file form until it’s ready to be played back- has lost its software and had multiple drive failures. Meaning we couldn’t retrieve anything for playback. >We had planned that this would happen eventually, so we implemented the back up plan and built break reels to run until service was restored.

The server that houses the media manager software…(there’s a server for that?) >The software that talks to the video servers, the archive system and the automation system, and moves all the stuff wherever it needs to go…yeah that thing..yes, it lives on a server too, and it can break just like anything else. >When it did, we couldn’t access anything that wasn’t already on a video server waiting to play back. >We built break reels (again.)

Traffic Software Server Failure >The server that houses our Traffic Software system, passed away on a beautiful late summer Friday afternoon. >The backup tape drive died and when it went- it took the hard drive to our server with it. Fortunately we had already completed everything through the following Tuesday so we had a little breathing room.

Traffic Software Server Failure Cont. >We were able to restore the hard drive just long enough for our traffic software vendor to download the information from our server to one in their office. (2500 miles away from us) >Ordered a new server >Reconfigured all users to interface with the remote temporary server. >Were back up and transferring documents to Master Control in under 4 hours. >Two days, 4 consultants and 3 engineers later, we were able to print from the temporary server.

Traffic Software Server Failure Cont. >12 days after the initial failure, the new server arrived. We worked ahead, so that we could be without a server at all for days. It took 2 days to get the new server up and running and 3 days to restore printing.

How have we been impacted by server failures? >Since July of 2000 KRMA has been off the air due to failures(hardware/software/human) >less than 30 minutes (Not including power failures)= % success rate >We have developed backup plans which have been incorporated into our everyday operations which allow us to immediately respond to any type of disaster internal or external.

What have we learned from our experiences? >Eventually all servers will fail in some way. >A minute of prevention is better than an hour of downtime. >Thinking outside the box to find new ways to do our jobs allows us to create an environment in which we can survive and thrive in a server world!

Q & A