1. Orphaned Servers and Broken Processes 2007 Security Professionals Conference April 12, 2007

2. Agenda Introductions Background Do You Have Orphaned Servers? Vigilance and Awareness Lessons Applied @ CSUN Reflections @ CSUN Lessons Relearned @ CSUN Summary Questions / Discussion

3. Background Moran Technology Consulting (MTC) conducted an Incident Audit for a large research university  Audit focused on two (2) systems that were thought to have been compromised: 1.A server housed in central IT data center containing alumni donor information, including > 100,000 social security numbers 2.A server outside of central IT data center containing campus health center data, including > 50,000 patient records

4. Health Center Server  Incident Detection  Vigilant scanning of system following prior incident  Systems Management  No monitoring or support from central IT (in spite of HIPPA related risks)  Comprises detected:  A Rootkit (Hacker Defender Trojan)  W32/pate.b virus  Server had participated in attack on another university server Alumni Donor Server  Incident Detection  External complaints of brute force attack emanating from server  Systems Management  System was “decommissioned” and powered-off and removed from the network  System was removed and returned to the network several times (as an orphaned server)  Following compromises detected:  A Rootkit  4 GB of unauthorized music files  The Hideout virus Background The Compromises occurred 4 months and possibly 12 months prior to detection, respectively.

5. Background Why did these compromises occur?  Insufficient Operational practices and procedures within central IT  Lack of formalized standards for secure server configuration  No clear roles and lines of responsibility resulted in lack of accountability for systems  Failure of change management procedures to ensure proper system tracking and decommissioning  While 75% of campus servers reside outside of central IT data center little or no effort had gone into securely supporting or monitoring these systems  A “silo” culture existed within central IT  Staff reductions left central IT with too few resources  Previous attempts to establish campus-wide polices failed due to lack of executive support

6. Do You Have Orphaned Servers?  Do you know who owns and is managing EVERY server:  In the central data center?  Across the campus?  How much do you know about the servers on your campus?  What types of data do they house or connect to?  Who is accountable for these systems and data?  Who owns and maintains these systems  Do you have servers on your network that are believed to have been powered OFF but are really powered ON?  What are your current decommissioning policies and procedures  Have these policies and procedures been formalized?  Do they apply and are they in practice campus-wide?

7. Vigilance and Awareness  What formal IT Policies and procedures guide security management roles and responsibilities?  Do the they cover EVERY server on the campus, including both Central Data Center servers AND department servers?  Executive Ownership and Participation is REQUIRED for Security Policies to be effective

8. Lessons Applied @ CSUN 1.Decided to use the incident at another institution to repose the salient questions and to review our security stance!  Have we identified EVERY server in the central data center?  Have we identified EVERY server exposed to the Internet?  Do we have sufficient information to protect these assets?  What additional policies, procedures, and processes are needed? 2.Developed an approach to review our environment.  Reviewed firewall rules for all openings  Reviewed DNS for named servers  Reviewed ARP tables for IP assignments  Performed network scans to identified live servers  Established primary and secondary system owners for each server 3.Formulated a plan to address deficiencies found.

9. Reflections @ CSUN We thought we were in a very secure position … but we were wrong!  We had spent a lot of time protecting the systems and networks … everything you would expect of a professionally run IT organization to do.  We were very surprised by the number of orphans we found on our network. What introduced our orphaned servers?  Well meaning staff (“I’ll get to it.”)  Shifts of responsibilities and new staff. (“I thought they/he owned it”)  Overloaded staff and lack of resources.(refer back to #1) This is a President’s worst nightmare!  The overall cost of an incident includes the damage done to the reputation and credibility of the institution.  The question is not whether but when…  Hackers want two things:  Personal information they can sell and  Access to powerful computing resources

10. Lessons Relearned @ CSUN 1.Always presume that you are less secure than you are! 2.Use external and internal security events to  Increase security awareness at both the staff and executive levels  Perform ad-hoc reviews of your environment 3.Establish periodic reviews of your environment, which includes:  Scans of the network (don’t assume your change management process are fail-proof)  Review the ownership of servers and identify the administrators  Review the associated Firewall and DNS rules for all servers 4.Structure your IT organization/environment with  Clearly defined role and responsibilities for every unit and individual  Predefined separation of duties, with effective working teams.  Established strong change-management procedures that include removing of services from production. (The cleanup process is the killer.) The “one throat to choke” principal does not apply! Everyone must participate to ensure success.

11. Summary 1.Security Incidents happen to everyone.  Incidents that occur at other institutions are excellent opportunities to improve your own security awareness and practices 2.Preparedness and Vigilance are key to minimizing cost  Presume that you are still vulnerable (You are!)  Perform on-going reviews of your environment  Develop a Incident Response Plan 3.Institute policies and procedures to identify, manage and properly decommission every server on campus  Orphan servers provide a window for hackers to exploit your entire IT environment

12. Questions / Discussion