1. IT Major Incident Management Response and Communication
A Collaborative Approach to IT Major Incident Management
Nancy Proctor
Chris Wright
Kevin Chenoweth
October 2012

2. Collaborative Approach to IT Major Incident Management
Objective: Expeditious Return to Normal Operations
Managed
Measured
Deliberate
Safe
Collaborative effort: IT; Hospital Administration; VUMC Communications; (Inpatient) Systems Support Services; (Outpatient) Operations Systems Engineering, Operational Units; Others as needed

3. IT Major Incident Definition*
High impact, or potentially high operational impact
Requires a response that is above and beyond that given to normal incidents. Typically, requires:
Cross departmental coordination
Management escalation
Mobilization of additional resources
Increased communications
*ITIL Incident Management

4. IT Major Incident Phased Response and Escalation Points
Phase 1 From IT Incident Awareness to activation of IT Technical Conference Call Bridge
Phase 2 From Activation of IT Conference Call Bridge to Activation of Administrative Conference Call Bridge
Phase 3 From Activation of Administrative Conference Call Bridge to Return to Normal Operations

5. Early “Heads Up” Advisory to Hospital AC’s (Administrative Coordinators)
IT to provide Hospital Administration* with “head ups” alert if IT impact assessment suggests widespread or extended systems outage (analogous to weather “advisory”)
Intent is to alert Hosp. AC’s that IT is aware of, and working to resolve, a major incident that has the potential to cause significant impact
IT may not have root cause, or estimated time to resolve at time of first alert
IT may or may not have initiated IT Conference Call Bridge at this time
Step may be skipped if event is significant enough to warrant immediate escalation
*Administrative Coordinator. Alert is also sent to Informatics Administrator On Call, and Systems Support (inpatient and/or clinic)

6. Communication Protocols
Early “Heads Up” Advisory to Hospital Administrative Coordinators (via text page)
Technical IT Conference Call Bridge (CCB#1)
Reserved for IT troubleshooting and internal IT communications only
Administrative Conference Call Bridge (CCB#2)
Reserved for IT and Hospital Administration communication, and coordination of Operational response
Activated by IT on request from Hospital Administration
Medical Center Alerts and Communications
IT and Operations jointly approve content of alerts and medical center communications

7. Roles and Responsibilities Incident Manager
Helpdesk Manager On Call assumes the role of Incident Manager*
Coordinates initial incident response and IT impact assessment
Liaise with Helpdesk, workgroups, and Informatics Admin On Call
Communicates IT impact assessment and status updates to Informatics Admin On Call, (via DR Admin on Administrative Conference Call Bridge if activated)
Maintains list of impacted systems and status
Currently Helpdesk Manager but this may change

8. Roles and Responsibilities Informatics Administrator On Call (IC AOC)
Provides IT leadership and direction
Communicates impact assessment and status updates to Hospital Administration (ACs and AOC, depending on “phase”)
Presence on either Conference Call Bridge dependent on Incident Management Phase
Provides input to, and approves content of, enterprise communications
IT Counterpart to Hospital Administrator On Call

9. Roles and Responsibilities Systems Support Services/Operations Systems Engineering
Receive initial notification from Helpdesk
Receive notification when Technical CCB is opened; SSS Primary On-Call will join the bridge to receive status briefing
Escalate to Systems Support Management as necessary
Assist with impact assessment(s), workarounds, end user communications, resource requirements, issue verification, and recovery verification
Provide advice re need for House-wide downtime
Assess need for StarPanel banner message; provide input to IT AOC and Hospital ACs on content
Receive notification when Administrative CCB is activated
Interact with Informatics Center Admin On Call, Hospital ICs, IT Leadership Liaison, Hospital AOCs on CCB #2 (Admin Bridge Line)

10. Roles and Responsibilities Hospital AC (Administrative Coordinator)
Receives initial “heads up” text alert, notification of IT Conference Call Bridge activation (but does not join call), and IT impact assessment from IC AOC
Conducts operational impact assessment
Determines need to activate Administrative Conference Call Bridge (CCB#2)
Determines need for overhead announcements, alerts and enterprise communications
Collaborates with IC AOC on content of announcements, alerts, and enterprise communications
Engages other Operational (non-IT) resources as necessary
Determines need to escalate within Hospital Administration hierarchy

11. Roles and Responsibilities Hospital AOC (Administrator On Call)
Receives initial notification and/or updates from Hospital Administrative Coordinator
Joins Administrative Conference Call Bridge
Leads Operational Response Activities
Establishes communication with IC AOC via Administrative Conference Call Bridge
Determines need to activate Emergency Operations Center (EOC)
Determines need for overhead announcements, alerts and enterprise communications
Collaborates with IC AOC on content of announcements, alerts and enterprise communications
Engages other Operational (non-IT) resources as necessary
Optional if addressed during presentation of “Phases” slides