Bethel University After Hours/On Call Support Douglas Nguyen Director of User Services Michael Johnson Help Desk Manager

Bethel University Based in St. Paul, Minn., with additional seminary locations on both coasts. Based in St. Paul, Minn., with additional seminary locations on both coasts. Offers undergraduate, graduate, seminary, and adult education programs. Offers undergraduate, graduate, seminary, and adult education programs. 6,200 students from 48 states and 29 countries. 6,200 students from 48 states and 29 countries. 2

How it got started? Weekend outage in 2009 Weekend outage in 2009 Top level cabinet became aware Top level cabinet became aware IT leaders meeting IT leaders meeting Why this now? Why this now? It has been a thorny issue It has been a thorny issue Institution is growing; users needs are growing Institution is growing; users needs are growing They want IT support when they use our service They want IT support when they use our service Everything is increasing => 24/7 Everything is increasing => 24/7 But how are we going to meet demand? But how are we going to meet demand? 3

Issues vs. Outages Issues are an inconvenience and often result in loss of productivity for individual users Issues are an inconvenience and often result in loss of productivity for individual users An outage is a failure or loss of service of a critical institutional resource that affects multiple users An outage is a failure or loss of service of a critical institutional resource that affects multiple users Users often do not understand the difference between “issue” and “outage” Users often do not understand the difference between “issue” and “outage” Recognized early on that we can only hope to address outages, not individual issues with on-call support Recognized early on that we can only hope to address outages, not individual issues with on-call support 4

A better definition of outage “Critical Outage is when a mission critical service is down or severely degraded for the entire campus or a large portion of the campus.” Georgia State University 5

Purpose/Goal “To be an IT conduit for communication, to help solve, and to be a liaison to the extend possible for end users.” Provide first and second tier support for outages. All normal support issues are handled on next business day. 6

Questions? 7

Hours Beyond the normal employee business support hours (M-Th 7:30am-6pm and F 7:30am-5pm), the institution wanted to add: Mon-Thu 7-7:30am & 6-12pm (6.5 hrs) Mon-Thu 7-7:30am & 6-12pm (6.5 hrs) Fri7-7:30am & 5-10pm (5.5 hrs) Fri7-7:30am & 5-10pm (5.5 hrs) Sat 8am-10pm (16 hrs) Sat 8am-10pm (16 hrs) Sun 2pm-12am (10 hrs) Sun 2pm-12am (10 hrs) TOTAL57.5 hours TOTAL57.5 hours 8

Equipment/Tools Internet access – Netbook with broadband for mobility Internet access – Netbook with broadband for mobility Shared PDA Phone – Android phone (Verizon) Shared PDA Phone – Android phone (Verizon) A printed playbook – Support catalog, procedures, SLA, and contacts information A printed playbook – Support catalog, procedures, SLA, and contacts information Network monitoring – Paging for every monitored issue doesn’t work for the on-call staff – We leverage Nagios to look up issues Network monitoring – Paging for every monitored issue doesn’t work for the on-call staff – We leverage Nagios to look up issues 9

Equipment/Tools (cont.) Internal VPN – to get to secure resources Internal VPN – to get to secure resources Remote control tool – bomgard.com Remote control tool – bomgard.com Transcription service – phonetag.com Transcription service – phonetag.com ACD phone tree – added routing to transcription service and the option to leave an emergency outage message during the prescribed hours ACD phone tree – added routing to transcription service and the option to leave an emergency outage message during the prescribed hours 10

Equipment cost Data on laptop - $50 Data on laptop - $50 Data phone - $50 Data phone - $50 PhoneTag - $30 PhoneTag - $30 $130/month $130/month 11

Implementation strategy User call Help Desk extension as normal. User call Help Desk extension as normal. After business hours, they have an option to leave an outage message. After business hours, they have an option to leave an outage message. The call is routed to PhoneTag The call is routed to PhoneTag Notifies on-call phone Notifies on-call phone Creates case from transcription and attaches WAV file Creates case from transcription and attaches WAV file The on-call person determines the best course action based on playbook and escalation rule. The on-call person determines the best course action based on playbook and escalation rule. 12

13

Escalation rule 14 On-call team Acknowledge case 30 min Team Broadcast 60 min Team Director Call mobile phone 15 min VP Call mobile phone

Questions? 15

Communication strategy State expectation up-front, not only in SLA but in the voic greeting. “ You have reached a voic box intended for after-hours reporting of the outage of a critical institutional resource that affects multiple users. Please leave your name, ID number if available, phone number, current location, and a brief description of the outage. Your voic will be transcribed. Please note that we intend to respond to outages within 30 minutes; we will respond to non-outages the next business day. Normally we will communicate via messages generated by Footprints. Thank you for calling.” 16

Employee compensation 6-person rotation 6-person rotation What is fair for staff? What is fair for staff? 1. Monetary 2. Flex-time 3. Perks Team understood that the institution is under no obligation to meet any of the requests Team understood that the institution is under no obligation to meet any of the requests Institution approved 1/10 flex-time but stipulated that it must be used by the next rotation. 17

Results Started 10/1/ weeks trial mode. Started 10/1/ weeks trial mode. ~ 2 a week during first 12 weeks. ~ 2 a week currently. ~ 2 a week during first 12 weeks. ~ 2 a week currently. We have been able to respond on time to almost all calls. We have been able to respond on time to almost all calls. Had one lapse – 4 hours outage that was typically only 15 minutes. We had no ETA to communicate => Escalation rule and quarterly meetings Had one lapse – 4 hours outage that was typically only 15 minutes. We had no ETA to communicate => Escalation rule and quarterly meetings It’s a success. Several prominent faculty members who have been a long time complainers gave us a great review for the work we did for them. Normally, they would have reached no one. It’s a success. Several prominent faculty members who have been a long time complainers gave us a great review for the work we did for them. Normally, they would have reached no one. 18

Thank you Comments and questions? 19 Douglas Nguyen Director of User Services Michael Johnson Help Desk Manager