1. David Sanneman IT IM Leadership Mayo Clinic Rochester, Mn
BCS An Approach to Mitigate Extended SCC Soft Application Outages
David Sanneman IT IM Leadership Mayo Clinic Rochester, Mn

2. BCS – About Me

3. BCS – Why It Matters
You’re in your lab performing your work and all the sudden the building loses all power, everything shuts down, what do you do? Or the maintenance on the hardware for the Laboratory System computers encounters an issue and is estimated to be down for the next 5, 6 or 7 hours, what do you do? More importantly, you’re at the Doctor’s office waiting for your test results to hear those fateful words, if it’s cancer or not, and the Doctor tells you the system is down and to come back tomorrow, what do you do?

4. You wait!
and wait
and wait..

5. BCS - Overview
Waiting is never good, especially when you are processing test orders for your patients… Because of extended outages from software and hardware installs and problems, one mitigation strategy is to design and build a Business Continuity System or (BCS) This allows the business to function as expected and provide a consistent and timely episode of care for the patients and prevent the laboratories from performing countless hours of manual specimen processing and manual data entry

6. BCS
Our patient’s trust, is paramount to how they feel about the care they receive.

7. BCS - What Is It Integral part of an IT disaster recovery plan
Two identical Production environments, one Active and one Passive, with facilities geographically separated by a number of miles (Usually 50)
Housed in Self-sustaining facilities with multiple power feeds and extended emergency power backup capabilities
Theoretically identical software and hardware

8. BCS - Mayo Clinic

9. BCS - Mayo Clinic Numbers
Overall number of SCC Soft System users ~ 3000
Number of major sites on System 3
Peak number of concurrent users ~ 1200
Weekly transactions for Orders ~ 200,000
Weekly number of tests performed ~ 500,000
Weekly number of generated Results ~1,000,000
Current number of results in system 162,000,000+
Expected UP Time Percent of System 99.99%

10. BCS - Mayo Clinic (Simplified)

11. BCS - Why is it Needed Near constant data transmission
Better manage code installs
Mitigate the affects of extended outages
Protect patient’s episodes of care from catastrophic events
Maintain business vitality

12. BCS
We are responsible for every aspect of our patient’s lives!

13. BCS - When is it Needed
Hardware failures

14. BCS - When is it Needed
Infrastructure failures

15. BCS - When is it Needed
New Software/Code installs exceeding business downtime requirements

16. BCS - When is it Needed
Catastrophic events

17. BCS – Should You Implement
Need higher level of IT disaster recovery
Inability to tolerate extended downtimes efficiently
Possibility of a catastrophic event
Complexity of the business model

18. BCS - Challenges
Design & Development

19. BCS - Challenges
Leadership Approval and Funding

20. BCS - Challenges
Testing & Implementation/Go-Live

21. BCS – How We Use It New Code installs lasting > 4 hours
Vendor Database Changes
Hardware issues with expected downtime > 1 hour
Scheduled System Maintenance
Infrastructure Issues
Catastrophic events

22. BCS – Our Current Timings
JUMP – the act of shutting down the Active PROD Environment and starting up the Passive PROD Environment as Active
Required Downtime to JUMP: 1 hour
Required Downtime to JUMP with Code Upgrade: 2.5 hours
What do you think our future Goal is?
15 Minutes

23. BCS – Tools Used Data Guard for Straight Database Replication
Oracle Golden Gate for Database Transformations
File System Synch Scripts
Well Defined Implementation Plans and Procedures
SCC Soft System Password locking Controls
System Monitors
One Awesome Vendor!!!

24. BCS - Mayo Clinic Current

25. Recap and Summary
If your system is/can be affected by any of the issues mentioned previous, you may want to consider investing in a Business Continuity System
Keys to a successful implementation
Know your limitations
Understand the challenges
Test, test and test again
Realize the effects of a catastrophic event

26. Questions...