David Sanneman IT IM Leadership Mayo Clinic Rochester, Mn BCS An Approach to Mitigate Extended SCC Soft Application Outages David Sanneman IT IM Leadership Mayo Clinic Rochester, Mn

BCS – About Me

BCS – Why It Matters You’re in your lab performing your work and all the sudden the building loses all power, everything shuts down, what do you do? Or the maintenance on the hardware for the Laboratory System computers encounters an issue and is estimated to be down for the next 5, 6 or 7 hours, what do you do? More importantly, you’re at the Doctor’s office waiting for your test results to hear those fateful words, if it’s cancer or not, and the Doctor tells you the system is down and to come back tomorrow, what do you do?

You wait! and wait and wait..

BCS - Overview Waiting is never good, especially when you are processing test orders for your patients… Because of extended outages from software and hardware installs and problems, one mitigation strategy is to design and build a Business Continuity System or (BCS) This allows the business to function as expected and provide a consistent and timely episode of care for the patients and prevent the laboratories from performing countless hours of manual specimen processing and manual data entry

BCS Our patient’s trust, is paramount to how they feel about the care they receive.

BCS - What Is It Integral part of an IT disaster recovery plan Two identical Production environments, one Active and one Passive, with facilities geographically separated by a number of miles (Usually 50) Housed in Self-sustaining facilities with multiple power feeds and extended emergency power backup capabilities Theoretically identical software and hardware

BCS - Mayo Clinic

BCS - Mayo Clinic Numbers Overall number of SCC Soft System users ~ 3000 Number of major sites on System 3 Peak number of concurrent users ~ 1200 Weekly transactions for Orders ~ 200,000 Weekly number of tests performed ~ 500,000 Weekly number of generated Results ~1,000,000 Current number of results in system 162,000,000+ Expected UP Time Percent of System 99.99%

BCS - Mayo Clinic (Simplified)

BCS - Why is it Needed Near constant data transmission Better manage code installs Mitigate the affects of extended outages Protect patient’s episodes of care from catastrophic events Maintain business vitality

BCS We are responsible for every aspect of our patient’s lives!

BCS - When is it Needed Hardware failures

BCS - When is it Needed Infrastructure failures

BCS - When is it Needed New Software/Code installs exceeding business downtime requirements

BCS - When is it Needed Catastrophic events

BCS – Should You Implement Need higher level of IT disaster recovery Inability to tolerate extended downtimes efficiently Possibility of a catastrophic event Complexity of the business model

BCS - Challenges Design & Development

BCS - Challenges Leadership Approval and Funding

BCS - Challenges Testing & Implementation/Go-Live

BCS – How We Use It New Code installs lasting > 4 hours Vendor Database Changes Hardware issues with expected downtime > 1 hour Scheduled System Maintenance Infrastructure Issues Catastrophic events

BCS – Our Current Timings JUMP – the act of shutting down the Active PROD Environment and starting up the Passive PROD Environment as Active Required Downtime to JUMP: 1 hour Required Downtime to JUMP with Code Upgrade: 2.5 hours What do you think our future Goal is? 15 Minutes

BCS – Tools Used Data Guard for Straight Database Replication Oracle Golden Gate for Database Transformations File System Synch Scripts Well Defined Implementation Plans and Procedures SCC Soft System Password locking Controls System Monitors One Awesome Vendor!!!

BCS - Mayo Clinic Current

Recap and Summary If your system is/can be affected by any of the issues mentioned previous, you may want to consider investing in a Business Continuity System Keys to a successful implementation Know your limitations Understand the challenges Test, test and test again Realize the effects of a catastrophic event

Questions...