IT Major Incident Management Response and Communication A Collaborative Approach to IT Major Incident Management Nancy Proctor Chris Wright Kevin Chenoweth October 2012

Collaborative Approach to IT Major Incident Management Objective: Expeditious Return to Normal Operations Managed Measured Deliberate Safe Collaborative effort: IT; Hospital Administration; VUMC Communications; (Inpatient) Systems Support Services; (Outpatient) Operations Systems Engineering, Operational Units; Others as needed

IT Major Incident Definition* High impact, or potentially high operational impact Requires a response that is above and beyond that given to normal incidents. Typically, requires: Cross departmental coordination Management escalation Mobilization of additional resources Increased communications *ITIL Incident Management

IT Major Incident Phased Response and Escalation Points Phase 1 From IT Incident Awareness to activation of IT Technical Conference Call Bridge Phase 2 From Activation of IT Conference Call Bridge to Activation of Administrative Conference Call Bridge Phase 3 From Activation of Administrative Conference Call Bridge to Return to Normal Operations

Early “Heads Up” Advisory to Hospital AC’s (Administrative Coordinators) IT to provide Hospital Administration* with “head ups” alert if IT impact assessment suggests widespread or extended systems outage (analogous to weather “advisory”) Intent is to alert Hosp. AC’s that IT is aware of, and working to resolve, a major incident that has the potential to cause significant impact IT may not have root cause, or estimated time to resolve at time of first alert IT may or may not have initiated IT Conference Call Bridge at this time Step may be skipped if event is significant enough to warrant immediate escalation *Administrative Coordinator. Alert is also sent to Informatics Administrator On Call, and Systems Support (inpatient and/or clinic)

Communication Protocols Early “Heads Up” Advisory to Hospital Administrative Coordinators (via text page) Technical IT Conference Call Bridge (CCB#1) Reserved for IT troubleshooting and internal IT communications only Administrative Conference Call Bridge (CCB#2) Reserved for IT and Hospital Administration communication, and coordination of Operational response Activated by IT on request from Hospital Administration Medical Center Alerts and Communications IT and Operations jointly approve content of alerts and medical center communications

Roles and Responsibilities Incident Manager Helpdesk Manager On Call assumes the role of Incident Manager* Coordinates initial incident response and IT impact assessment Liaise with Helpdesk, workgroups, and Informatics Admin On Call Communicates IT impact assessment and status updates to Informatics Admin On Call, (via DR Admin on Administrative Conference Call Bridge if activated) Maintains list of impacted systems and status Currently Helpdesk Manager but this may change

Roles and Responsibilities Informatics Administrator On Call (IC AOC) Provides IT leadership and direction Communicates impact assessment and status updates to Hospital Administration (ACs and AOC, depending on “phase”) Presence on either Conference Call Bridge dependent on Incident Management Phase Provides input to, and approves content of, enterprise communications IT Counterpart to Hospital Administrator On Call

Roles and Responsibilities Systems Support Services/Operations Systems Engineering Receive initial notification from Helpdesk Receive notification when Technical CCB is opened; SSS Primary On-Call will join the bridge to receive status briefing Escalate to Systems Support Management as necessary Assist with impact assessment(s), workarounds, end user communications, resource requirements, issue verification, and recovery verification Provide advice re need for House-wide downtime Assess need for StarPanel banner message; provide input to IT AOC and Hospital ACs on content Receive notification when Administrative CCB is activated Interact with Informatics Center Admin On Call, Hospital ICs, IT Leadership Liaison, Hospital AOCs on CCB #2 (Admin Bridge Line)

Roles and Responsibilities Hospital AC (Administrative Coordinator) Receives initial “heads up” text alert, notification of IT Conference Call Bridge activation (but does not join call), and IT impact assessment from IC AOC Conducts operational impact assessment Determines need to activate Administrative Conference Call Bridge (CCB#2) Determines need for overhead announcements, alerts and enterprise communications Collaborates with IC AOC on content of announcements, alerts, and enterprise communications Engages other Operational (non-IT) resources as necessary Determines need to escalate within Hospital Administration hierarchy

Roles and Responsibilities Hospital AOC (Administrator On Call) Receives initial notification and/or updates from Hospital Administrative Coordinator Joins Administrative Conference Call Bridge Leads Operational Response Activities Establishes communication with IC AOC via Administrative Conference Call Bridge Determines need to activate Emergency Operations Center (EOC) Determines need for overhead announcements, alerts and enterprise communications Collaborates with IC AOC on content of announcements, alerts and enterprise communications Engages other Operational (non-IT) resources as necessary Optional if addressed during presentation of “Phases” slides