Wolfgang von Rüden, CERN IT, August Computer Security: A permanent and costly battle Update for the CERN Management Board 26 August 2003 Wolfgang von Rüden CERN, IT Division Leader

Wolfgang von Rüden, CERN IT, August The VIRUS trend is going up ! Virus attacks have been with us for a long time and … They seem to get more and more sophisticated and destructive Our latest case is the Blaster Worm

Wolfgang von Rüden, CERN IT, August What happened ? Scandinavia's Nordea bank70 branch offices closed, worm in servers of all 440 offices Stanford University2,500 computers hacked (Blaster) CSX RailwaysCurtailed train service while restoring computer systems during 8 hours New York TimesAsked via public address system to shut off all computers (1/2 day) CFFWeb site problems for users (timetable, ticketing), long waits MarylandMotor vehicle administration affected Federal Reserve Atlanta/GABank affected Air Canada50% of phone reservation system capacity affected plus some check-in operations China2,000 intranet systems stopped OrsayWindows support on holidays

Wolfgang von Rüden, CERN IT, August Events since 16 July 2003 (1) 16 JulyMicrosoft releases a security bulletin warning about a so- called RPC vulnerability (MS03-26) affecting most versions of the Windows operating system 24 JulyIT launches a campaign to protect computers against this vulnerability systems are patched (one command) 1 AugScan tool available: 500 vulnerable systems detected. Administrators contacted using Network DB information 11 AugustLeading antivirus companies warned about an exploit (W32.MSBlaster) rapidly spreading around the world. It is expected to make massive attacks against windowsupdate.com as of Saturday 16 August 13 AugustMail sent to each Division Leader with the list of vulnerable machines

Wolfgang von Rüden, CERN IT, August Events since 16 July 2003 (2) 15 AugustDespite multiple reminders, more than 200 Windows systems are still vulnerable. Site scanning shows suspicious activities, in particular via ACB or VPN, which are blocked for the week-end. Risk that those computers could launch the attacks and thereby potentially bringing down the whole or parts of the network and potentially reducing the ability of the organization to execute its mission. 18 AugustIT management decides to block vulnerable systems at the network level and to continue restrictions on the ACB and VPN service. No time to follow the usual consulting channels. Affected users are informed, provided the entry in the registration DB is up-to-date 18 AugustAn even more severe threat exploiting the vulnerability, "W32.Welchia", appeared and is now causing disruption at several sites 18 AugustTask force in place to help users to get back to normal. 19 AugustIn the afternoon, a mass mailing virus (W32.Sobif.F) started to appear at CERN and affects many users

Wolfgang von Rüden, CERN IT, August Status as of yesterday Better scanning tools in place Network tools added to block bad systems Still 150 systems blocked (half are portables and ~ 40 unregistered) More than 100 systems infected so far Both ACB and VPN are back, but restricted to common facilities (mail, web, file access) Helpdesk got 25% more calls More problems expected as people come back from holidays

Wolfgang von Rüden, CERN IT, August Initial problem analysis More than 500 machines not managed centrally caused the problems Some are CERN owned, locally managed machines and the owner did not follow the instructions (misunderstandings, manual action needed) Some machines are managed by the end-user or belong to visitors. In both cases, we have no means to enforce a patch “Black box” installations by companies Many insecure machines connected from home via ACB or VPN Major worry: Network DB often not updated by users, so they can even not be contacted.

Wolfgang von Rüden, CERN IT, August IT effort involved so far (FTE weeks) ActionPreventiveRepair Apply patch to 5000 machines via NICE 0.1 Security 4.0 Network group 6.0 User Support 3.5 Coordination 0.5 Local support 4.0 Total Does not include effort in other Divisions The hotfix webpage was visited 12’200 times in August The emergency measures page 2600 times since 15 th August

Wolfgang von Rüden, CERN IT, August How can you help ? Insist in your Division to move as many machines as possible to the centrally managed service Nominate a security contact (and alternate) for your division to be contacted in case of alerts Independent machines must be managed by a person competent to apply patches and to ensure virus protection in compliance with OC5. Network DB must be updated whenever a computer moves or the owner changes Fast reaction time needed for security patches! “It’s on my list” is not enough Unmaintained “Black box approach” doesn’t work and should be banned. Secure your computer at home (or don’t connect)

Wolfgang von Rüden, CERN IT, August Proposed actions Enforce hardware address registration for all computers on site using DHCP (portable sockets and wireless) This will also apply to short-time visitors (i.e. FC delegates …) We are ready to start deploying this by the end of September, region by region, to be completed before Christmas Information campaign needed before enforcement ACB is a major security threat We need to move to another solution It also costs 500kCHF/year We propose to move to Internet Service Providers User pays local phone call or uses ADSL Need to understand impact on “poor” visitors Establish a “fire-fighting” procedure with short reaction time

Wolfgang von Rüden, CERN IT, August Conclusion CERN continued to work almost “as usual” while many other sites were knocked out Still, we need to be much more serious about security issues Top management has to buy in to achieve the goal We can’t afford a global break-down Thanks to my colleagues in IT who spent numerous hours overtime to keep things under control. Thanks also for all the help we got from the Divisions.

Wolfgang von Rüden, CERN IT, August Thank you ! Please help us protecting our work place