Responding to a Security Incident Maryland Security Day March 2, 2004 Joy Hughes, CIO
Developing a Response Capacity in Fall 2004 Developing a Response Capacity in Fall 2004 CSIRT-tech IT Security Coordinator formed team with Windows engineer, Unix engineer & Net engineer; sent team to SANS incident response training. CSIRT-exec Deputy CIO formed team with university counsel, VP of UR., FERPA officer, President’s Chief of Staff, and University Safety Officer.
Incident Description January 3 rd, ITU Windows Server Manager noticed his servers being probed by server in ID Card Office. Rushed to ID Card Office and removed server from network. Then called CSIRT-Tech Contained damage, preserved evidence, enabled restoration of service, determined files contained SSNs.
Incident Handling Grade: B network dir CSIRT-Exec decided community had to be notified; president agreed.
Getting/Sharing the Facts Took 3 days to ascertain: - no other servers on LAN were compromised (some had credit card numbers); - no files on original server had other private data; & - get advice from Feds as to how community could protect their identities (but this advice too strong)
Getting/Sharing the Facts Another 2.5 days to get delivered to every class of customer Could not tell if the files had been downloaded or copied and law enforcement forensics teams are too overloaded to work quickly so contracted with forensics firm
Getting/Sharing the Facts Struggled to determine if ID cards were the target. This caused us to contract with police to patrol residence halls and eventually to issue new ID cards to every resident student
Assigning Roles Law Enforcement Coordination – Campus Police (FC, FBI) Communication strategy – University Relations Communication point - CIO Coordination – CIO
More Roles Technical Remediation – Executive Director: Technology Systems Division Customer Web Site – Public Relations Were the files copied? Who did what? Are any other servers in danger? – Forensics Firm Assist Forensics Firm: IT Security Coordinator
Work Involved Engaged and worked with Forensics Firm, every day Worked with law enforcement Interviews with Washington Post, local papers and national.coms Student newspaper (twice) Hundreds of phone calls, hundreds of s
Work Involved Implemented new ID card software Reissued ID cards for resident students Vendor reps, some well connected, persisted in efforts to sell security stuff and identity theft protection services
Work Involved Did a line by line comparison of 36,000 records in ID Card database with corresponding records in student system Surveyed every department to see if they stored private admin data on a server, then worked with company to assess security of every one of these servers
Work Involved Responded to legislative interest, including a bill to turn over all security incident handling to VITA Wrote and rewrote updates to web site Campus police investigated every reported problem
Lessons Learned SSNs are not, by themselves, of interest to criminals looking for a scalable return A percentage of people panic on the issue; more effort needs to be expended to control panic SAs do not know what is on their servers Eliminating SSN as identifier is not sufficient to protect SSNs
Lessons Learned You can not say to the public “it wasn’t my server.” Keep hour by hour records of your response Need to train all SAs in preserving evidence and containing damage
Future Hardware, software & policy changes that will enable: log, log, log Much more ITU involvement in protecting other people’s servers (e.g. MS 2-day) Accelerate intrusion detection implementation
Future Accelerate VPN Accelerate authentication project Develop curriculum, templates, etc. to aid SAs in preserving evidence & containing damage Implement perimeter firewall
Future Insist all data files be removed from web servers Insist all shared drives with sensitive data be specially protected Build security into employee performance plans
Future Ensure CIO has same great relationship with new VP of U.R.