1. Prevention is best … but what if …
One Data Center’s Experience

2. Kern Medical Center (KMC)
County Hospital
222 beds, Level 2 Trauma Center
Teaching Hospital
Minimal funding

3. KMC IT Infrastructure Diverse Construction – 1930 to present
Hardwired LAN, adding wireless
Flat LAN, migrating to segmented LAN
Windows PCs & Citrix Thins
Windows & Linux servers
Primarily McKesson patient care apps
Implementing Open Source EMR

4. Day 1– 7/26 IT Help Desk calls re: “long print jobs”
No noticeable performance issues
No access issues

5. Day 2 – 7/27 AM Continued calls re: “print problems”
Calls re: “slow to boot”
Calls re: “can’t access application”

6. Day 2 – 7/27 Afternoon Severe performance issues Severe access issues
Reports of pornography
Called “Code Triage”
Shutdown all systems and implemented “downtime procedures”
Initial request for AV vendor support

7. Day 2 – 7/27 PM Disconnect KMC from all other County departments
Verify integrity of patient care servers
Cancel downtime procedures for patient care applications

8. Day 3 – 7/28 AM PM Download new .dat file and “stinger” from AV vendor
Deep scan all file servers PM
Implement “super locked down” AV profile
Results of scans don’t match impact reports
Contact alternate AV vendor and secure a 60-day evaluation license

9. Day 4 – 7/29
Staff programmer creates a script to locate and delete all offending .lnk files
Extensive testing
Tighten AV policies
Midnight – success at preventing new .lnk file creation

10. Day 5 – 7/30 Apply latest .dat file
Tighten policies even more and schedule 4:00 a.m. deep scans for all servers and PCs

11. Day 6 – 7/31 Reimage worst PCs
Try to locate any PCs not “talking” to the AV policy administration software
Focus on administrative user PCs

12. Day 7 – 8/1 Reimage worst PCs
Try to locate any PCs not “talking” to the AV policy administration software
Focus on administrative user PCs

13. Day 8 – 8/2 Test latest .dat file Results appear better
Applied Microsoft “patch”

14. Day 9 – 8/3 Apply latest .dat file Continued testing
Disappointing results
Original AV misses 8 of 13 virus
Alternate AV catches all 13
Transmit 13 virus samples and result files to AV vendor – communicate criticality of the situation and threat of no contract renewal

15. Tips from the trenches

16. Remember your business
Patient care is number 1!
Ask for support from your CMO & CNO if necessary
Communicate your priorities to all staff

17. Your AV vendor works for you
Involve your AV vendor
Demand escalation
Have your AV sales representative’s phone number available
Remind your sales representative of the contract renewal date

18. Don’t expect your AV vendor to repair your systems
AV vendors focus on detection and prevention
Repair is a distant second priority for them
Develop a plan for recovery
Identify skilled staff
Identify the recovery effort leader/manager

19. Diversity is good … maybe great
Implement operating system diversity
Linux servers
Windows PCs and Thins
If all Microsoft, then implement antivirus diversity – different AV on servers and PCs
Implement a combination of software and hardware (appliance) prevention

20. Divide and Conquer Servers first Thins next PCs last work inside-out
data integrity is critical
interrupt the transmission path
Thins next
can affect many with least effort
PCs last
most labor intensive

21. Beware of Distractions
Test all manifestations of the virus until you locate the “real” culprit
Save copies of the virus for testing your solution

22. Segment your LAN Can help quarantine the infestation
Allows you to prioritize recovery by functional unit

23. Backups Back up everything – perform an audit
Test restore all backup types

24. Contact Information
Bill Fawns
(661) (office)
Administrative support
Brenda Reed
(661)