Prevention is best … but what if … One Data Center’s Experience

Kern Medical Center (KMC) County Hospital 222 beds, Level 2 Trauma Center Teaching Hospital Minimal funding

KMC IT Infrastructure Diverse Construction – 1930 to present Hardwired LAN, adding wireless Flat LAN, migrating to segmented LAN Windows PCs & Citrix Thins Windows & Linux servers Primarily McKesson patient care apps Implementing Open Source EMR

Day 1– 7/26 IT Help Desk calls re: “long print jobs” No noticeable performance issues No access issues

Day 2 – 7/27 AM Continued calls re: “print problems” Calls re: “slow to boot” Calls re: “can’t access application”

Day 2 – 7/27 Afternoon Severe performance issues Severe access issues Reports of pornography Called “Code Triage” Shutdown all systems and implemented “downtime procedures” Initial request for AV vendor support

Day 2 – 7/27 PM Disconnect KMC from all other County departments Verify integrity of patient care servers Cancel downtime procedures for patient care applications

Day 3 – 7/28 AM PM Download new .dat file and “stinger” from AV vendor Deep scan all file servers PM Implement “super locked down” AV profile Results of scans don’t match impact reports Contact alternate AV vendor and secure a 60-day evaluation license

Day 4 – 7/29 Staff programmer creates a script to locate and delete all offending .lnk files Extensive testing Tighten AV policies Midnight – success at preventing new .lnk file creation

Day 5 – 7/30 Apply latest .dat file Tighten policies even more and schedule 4:00 a.m. deep scans for all servers and PCs

Day 6 – 7/31 Reimage worst PCs Try to locate any PCs not “talking” to the AV policy administration software Focus on administrative user PCs

Day 7 – 8/1 Reimage worst PCs Try to locate any PCs not “talking” to the AV policy administration software Focus on administrative user PCs

Day 8 – 8/2 Test latest .dat file Results appear better Applied Microsoft “patch”

Day 9 – 8/3 Apply latest .dat file Continued testing Disappointing results Original AV misses 8 of 13 virus Alternate AV catches all 13 Transmit 13 virus samples and result files to AV vendor – communicate criticality of the situation and threat of no contract renewal

Tips from the trenches

Remember your business Patient care is number 1! Ask for support from your CMO & CNO if necessary Communicate your priorities to all staff

Your AV vendor works for you Involve your AV vendor Demand escalation Have your AV sales representative’s phone number available Remind your sales representative of the contract renewal date

Don’t expect your AV vendor to repair your systems AV vendors focus on detection and prevention Repair is a distant second priority for them Develop a plan for recovery Identify skilled staff Identify the recovery effort leader/manager

Diversity is good … maybe great Implement operating system diversity Linux servers Windows PCs and Thins If all Microsoft, then implement antivirus diversity – different AV on servers and PCs Implement a combination of software and hardware (appliance) prevention

Divide and Conquer Servers first Thins next PCs last work inside-out data integrity is critical interrupt the transmission path Thins next can affect many with least effort PCs last most labor intensive

Beware of Distractions Test all manifestations of the virus until you locate the “real” culprit Save copies of the virus for testing your solution

Segment your LAN Can help quarantine the infestation Allows you to prioritize recovery by functional unit

Backups Back up everything – perform an audit Test restore all backup types

Contact Information Bill Fawns fawnsb@co.kern.ca.us (661) 326-5480 (office) Administrative support Brenda Reed (661) 326-2424