1. 1 Information Security Vision Network Planning Task Force 9/29/2003 Deke Kassabian and Dave Millar

2. 2 Our Common Problem ■Productivity Loss ■Slammer worm interrupts Internet connectivity campus-wide for several hours and in a few locations on campus for longer. ■1000+ windows machines compromised in four weeks by Blaster worm. ■Managing hundreds of disconnections: run a trace, disable the port, contact the owner, get box fixed, get port re-enabled. ■1,000 infected email attachments a day.

3. 3 Estimated cost of Blaster/Welchia ITEMEST. COST 1200 compromised machines -Manage detection and notification -Format and rebuild machines -Remove Blaster from machines 15-25% 9,000 Vulnerable machines (patched twice) -20 campus-wide scans, 14 mass notifications of vulnerability -4,500 Patch automatically (twice) -4,500 Patch manually (twice) 2-3% 1-2% 30-40% Total $287,000 Lost productivity of faculty/staff machines disconnected ?

4. 4 Security Vision: Defense in Depth ■Prevent ■Patch management tools & services ■Training & education ■Anti-virus filtering on mail servers ■Firewalls/Router filtering ■Virtual Private Network ■Personal firewalls ■Secure out-of-the box ■Campus-wide vulnerability scanning ■Detect ■Intrusion detection ■Respond ■Find a better way to locate compromised and vulnerable DHCP and wireless devices.

5. 5 Patch Management ■Managed vs. Unmanaged ■Managed – LSP runs a Windows Domain Controller; all desktops and workstations are configured to participate in the domain, all users authenticate to the central Domain Controller. LSP has Administrator rights on all machines in the domain, and can manage domain workstations and desktops remotely – i.e. pushing out patches and service packs, applying group policies, etc. ■Unmanaged – Users run their Windows desktop or workstation “stand-alone.” The only way that patches can be applied is if the owner or LSP sits down physically at each desktop/workstation. ■Windows Update ■Window Update client points to Microsoft’s Windows Update site. ■Operates in one of three modes: manual, semi-automatic and completely automatic. ■SUS – Software Update Server ■Allows you to point Windows Update client on your desktops and workstations to your own “mirror” of Microsoft’s Windows Update site. Allows you to test MS patches before deploying them. ■Can run either in managed domain or unmanaged workgroup ■SMS – Systems Management Server ■Administrator downloads patch, creates installation routine, creates query to find machines that need patch, deploys patch to machines from the result of query. ■Commercial products, e.g. HFNetChk Pro, PatchLink, BigFix ■Simplified management ■Handles application hotfixes as well as operating system

6. 6 Four Patch Management Options OptionFor which environment? ProsCons Windows Update Managed or unmanaged Free No messy hardware to manage No provision for testing patches – you’re at Microsoft’s mercy. Too risky for servers, but OK for some desktops and workstations. SUSManaged or unmanaged Free software? Allows you to test patches before deploying Must manage your own SUS server(s) Testing patches takes time Critical updates only today. Service packs are still coming in future version. Won’t support Office Update until SUS 2.0 (February, 2004?) SMSManagedPowerful, remote management, monitoring tool. More demanding and complex to manage Only appropriate for managed machines. Commercial tools, e.g. HFNetChk Pro Managed or unmanaged Support Windows and Office update Some products handle patches for applications like IIS, SQL server, etc,; service packs and feature updates -- not just Windows OS hotfixes. Price

7. 7 Critical Challenge: Patching Student machines ■ Distributing patches through SUS requires either that each desktop/workstation join a Microsoft domain, or at least make a registry change on each “managed” machine. ■ Obviously we don’t own student, machines. We will have an easier time figuring out how to manage patches on staff and faculty machines than students’. # of Windows machines # infected% infected Student-owned296350117% Penn-owned8037700 9%

8. 8 Patch Management Recommendation ■Establish a policy requiring that by 7/1/04, all campus PennNet- connected Windows systems: servers, desktops and workstations, have all critical updates applied within three business days ■ ………………………………………………………………existing staff ■Create a new ISC service: “Patch Management Services” tasked to: ■Work with campus LSPs to identify and share best patch management practices ■Evaluate and license patch management tools ■Create a campus SUS service, testing Microsoft patches against benchmark platforms. ■Support LSPs implementing their own SUS services ■Provide security patch documentation and conduct training for campus LSPs ■Estimated Costs ■Staff…………………………………………………………………….$100,000/yr ■Hardware for campus SUS service…………………...$10,000 every 2-3 yrs. ■Campus license for commercial patch management software ■ Software – 1000 seats……………….............................……..…$6/seat/yr

9. 9 Security: Defense in Depth ■Prevent ■Patch management tools & services ■Training & education ■Virus filtering on mail servers ■Firewalls/Router filtering ■Virtual Private Network ■Personal firewalls ■Secure out-of-the box ■Campus-wide vulnerability scanning ■Detect ■Intrusion detection ■Respond ■Find a better way to locate compromised and vulnerable DHCP and wireless devices.

10. 10 Virus Filtering ■Typical Windows virus spreads via ■Email messages ■Network file transfer ■Network file server shares ■Web traffic ■Other direct attacks over networks ■Removable storage (floppies, CDs, etc) ■Good anti-virus software on Windows desktops can address all of these.

11. 11 Mail Server Virus Filtering ■ Separate from Spam filtering ■ Usually involves checking for virus signatures in email messages ■ Can be implemented on the mail server directly or on a separate server (local or external ASP). ■ Can help to slow virus spread for Windows desktops without adequate virus protection and OS patches

12. 12 POBOX Virus Filtering ■ Proposing use of an outside virus filtering service, separate from our local server- based Spam control tools ■ Mail destined for POBOX users will take a detour through the service provider for filtering of virus messages

13. 13 Campus-Wide Virus Filtering for email? ■Two possible implementations, building on the POBOX approach: ■Replicate the POBOX configuration on other mail servers, but take advantage of the existing business relationship and established pricing ■Create a new user@upenn.edu mail forwarding service, and have mail to users of that service pass through an outside virus filtering service.user@upenn.edu

14. 14 Campus-Wide Virus Filtering for email? Per server virus filtering -- Pros -- Involves no change in email address to take advantage of virus filtering -- Cons -- Distributes complexity Likely will not achieve best pricing Campus-wide virus filtering based on a new user@upenn.edu service user@upenn.edu Probably easier to implement Likely to be much more cost effective Virus filtering only available for messages using the new user@upenn.edu format user@upenn.edu

15. 15 Security: Defense in Depth ■Prevent ■Patch management tools & services ■Training & education ■Anti-virus filtering on mail servers ■Firewalls/Router filtering ■Virtual Private Network ■Personal firewalls ■Secure out-of-the box ■Campus-wide vulnerability scanning ■Detect ■Intrusion detection ■Respond ■Find a better way to locate compromised and vulnerable DHCP and wireless devices.

16. 16 Simple Building Network Router switch Switch switch

17. 17 Simple Building Network, Firewall for all of subnet Router switch Switch switch Pros: ° More coverage from one FW device Cons: ° Blunt instrument, may subject too many things to one set of rules ° Problematic for network management

18. 18 Simple Building Network, with firewall for servers Router switch Switch switch Pros: ° Excellent server- or service- specific protection possible Cons: ° None

19. 19 Simple Building Network, Firewall for one workgroup Router switch Switch switch Pros: ° Group-specific control and protection Cons: ° Can still be a blunt instrument ° Still problematic for network management

20. 20 Simple Building Network,using VLAN Firewall Router switch Switch switch Pros: ° Very flexible in terms of participation ° Addresses net management problem Cons: ° Add complexity and cost

21. 21 Perimeter Firewall: Current Situation Pros: ° Provides limited protection from common attacks Cons: ° Collateral damage ° No provision for legitimate access to risky services.

22. 22 Campus VPN Service Pros: ° Allows us to block the most troublesome services and permit legitimate use. Cons: ° Complexity and cost ° Traffic is not encrypted on PennNet. ° Given the transient nature of PennNet this will at best stave off attacks for a few days

23. 23 Local VPN Service Pros: ° Allows Schools and Centers to implement more restrictive firewall policies. ° Unencrypted traffic need not travel over PennNet. Cons: ° Complexity and cost

24. 24 Where to put a perimeter firewall? Router switch Router Internet Router switch Router switch Router

25. 25 Minimal perimeter filtering in edge routers Router switch Router Internet Router switch Router switch Router

26. 26 Minimal perimeter filtering in internal routers Router switch Router Internet Router switch Router switch Router

27. 27 Campus firewall is not a panacea UniversityDate Netbios ports blocked # Windows machines # infected % infected Penn9/11/200311,0001,10010% Large state university 7/28/200312,0001,50013% Ivy League peer1/2/200218,0003,14617%

28. 28 Personal firewalls Router switch Switch switch

29. 29 Firewalls Recommendations & Estimated Costs Time- frame TargetRecommendations Long- term Servers, desktops and workstations Evaluate a network design and migration strategy that better balances availability against security, and capable of supporting more flexible use of internal and external router filtering/firewall technology…...................................................under evaluation Near- term Servers, desktops and workstations Provide a basic level of network security by implementing router filter rules on external interfaces after campus consultation and on internal interfaces at the request of Schools/Centers Hardware/software for internal filtering…........................$20,000/bldg every 3 years Near- term Servers, desktops and workstations Enable Schools and Centers to implement tighter local security policies: -Publish support for VLANs……………..........…………………….N&T Documentation -Select a campus firewall and VPN standard, avoiding a campus VPN “Tower of Babel”, buying time to patch systems, and enabling Schools and Centers to better implement local firewalls and VPN gateways…………………………under evaluation Near- term Desktops and workstations Evaluate and recommend a personal firewall software product for targeted users and do a pilot evaluation. Software license for 50-100 users…………………..………$2500 - $5000 for 3 years

30. 30 Security: Defense in Depth ■Prevent ■Patch management tools & services ■Training & education ■Anti-virus filtering on mail servers ■Firewalls/Router filtering ■Virtual Private Network ■Personal firewalls ■Secure out-of-the box ■Campus-wide vulnerability scanning ■Detect ■Intrusion detection ■Respond ■Find a better way to locate compromised and vulnerable DHCP and wireless devices.

31. 31 Secure out-of-the box ■ When someone buys a new Dell or IBM machine in the Fall Truckload sale, it is very easy for them to complete the system install without creating an Administrator password. This has led to hundreds of compromised machines within days of going on PennNet. ■ Recommendation: Work with Dell, IBM and Microsoft to create more secure default images for newly purchased Penn machines ………………...…negotiated price < $25/image

32. 32 Security: Defense in Depth ■Prevent ■Patch management tools & services ■Training & education ■Anti-virus filtering on mail servers ■Firewalls/Router filtering ■Virtual Private Network ■Personal firewalls ■Secure out-of-the box ■Campus-wide vulnerability scanning ■Detect ■Intrusion detection ■Respond ■Find a better way to locate compromised and vulnerable DHCP and wireless devices.

33. 33 RPC DCOM Scan results

34. 34 Campus-Wide Vulnerability Scanning ■ Vulnerability scanning has been a very useful tool in helping to manage the patching process University-wide. ■ Focused, campus-wide scans for single vulnerabilities campus- wide can be done quickly (i.e. in 45 minutes or so). This is very helpful to defend against a particular worm. ■ Broader, campus-wide scans for top twenty vulnerabilities take longer – more like 3-4 weeks. This is very helpful for “good hygiene” to ensure that we stay patched as new machines come onto PennNet over time. Goal: 1-3 weeks/campus-wide scan. ■ Finding contact information for the owners of thousands of DHCP (wired and wireless) is difficult and time-consuming. ■ The effectiveness of campus-wide scans is limited to the extent that laptops come and go at different locations on PennNet. We probably miss many vulnerable laptops if they’re “off-line” at the time we scan.

35. 35 Vulnerability Scanning Recommendations ■ Develop more efficient methods of locating the owners of DHCP (wired and wireless) devices. See “Respond” section below.

36. 36 Security: Defense in Depth ■Prevent ■Patch management tools & services ■Training & education ■Anti-virus filtering on mail servers ■Firewalls/Router filtering ■Virtual Private Network ■Personal firewalls ■Secure out-of-the box ■Campus-wide vulnerability scanning ■Detect ■Intrusion detection ■Respond ■Find a better way to locate compromised and vulnerable DHCP and wireless devices.

37. 37 How do worms spread? ■ 60% of the time attack Penn systems ■ 40% of the time: attack external systems

38. 38 How did we learn about Blaster/Welchia infected machines? ■ Outsiders automatically send personal firewall logs to central services like MyNetWatchman, who in turn email the report to us. ■ Penn people have automated extracts from their firewall logs and email us the results. ■ We are automatically scanning our firewall logs and extracting the results every four hours. ■ Strengths: simple approach, inexpensive ■ Weaknesses: limited coverage, relies on Penn people’s willingness to continue to devote the time.

39. 39 How could we improve our detection capability? IDS Box

40. 40 How could we improve our detection capability? OptionsProsCons IDS box connects to local switches ■ Inexpensive ■ Limited visibility IDS box connects to internal routers ■ Broader visibility ■ More expensive equipment – e.g. fiber taps. IDS box connects to edge routers ■ Complete visibility of outbound attacks ■ Technically challenging given our redundant internet connectivity. ■ Most expensive Use edge router flow logs ■ Limited visibility of outbound attacks ■ Less expensive, challenging than IDS on edge routers.

41. 41 Targeted Intrusion Detection Recommendations & Estimated Costs Near-termCreate policy establishing authority of ISC, Schools and Centers to conduct intrusion detection addressing privacy, log retention and related issues…………....................................no incremental cost Near-termDeploy and manage six-ten switch-connected IDS boxes to quickly identify compromised campus systems. Hardware………………………$15,000-$20,000 every 2-3 years Staff to configure, manage, analyze IDS systems and follow up on intrusion reports………………….…$100,000/yr Long-termEvaluate and determine best method to provide router flow logs for intrusion detection……………………………..under evaluation Long-termEvaluate a network design and migration strategy that better balances availability against security, and capable of supporting broader intrusion detection …………............................................................under evaluation

42. 42 Security Vision: Defense in Depth ■Prevent ■Patch management tools & services ■Training & education ■Anti-virus filtering on mail servers ■Firewalls/Router filtering ■Virtual Private Network ■Personal firewalls ■Secure out-of-the box ■Campus-wide vulnerability scanning ■Detect ■Intrusion detection ■Respond ■Find a better way to locate compromised and vulnerable machines as well as targets of copyright complaints.

43. 43 How do we find problem machines? ■ Problem machine is reported either by the RIAA, intrusion detection or campus vulnerability scan. ■ If static IP – look it up in assignments. ■ If DHCP – ask NOC for a port trace which translates the DHCP address to a physical location.

44. 44 Current situation ■ Unable to respond to problems with wireless machines on parts of campus. Can’t very well disable the port :-( ■ Had to just drop 40-50 cases of infected machines because of short DHCP lease lengths. ■ Requested 1149 port traces and 126 disconnects in calendar year 2003 (~$45,000). Trend is up (requested 270 this week alone). ■ Had to hold off requesting some disconnects because it would have been unmanageable.

45. 45 Incident Response Recommendations & Estimated Costs Near-termProvide tools to better support quick lookup of host and DNS contacts………………............…….under evaluation Near-termTargeted deployment of PennKey authenticated network access in, for example, College Houses, GreekNet, Library…............................................$2,000 - $5,000/bldg Long-termFull deployment of PennKey authenticated network access on campus Hardware/Software (one-time)…………..........$1,000,000 Near-termResearch ways of ensuring security of newly connected machines: Vulnerability scan of machines as they connect to PennNet Ability to block infected/vulnerable machines based on MAC address Hardware/Software…………...………..under evaluation Staff………………………………………under evaluation

46. 46 Summary of Recommendations & Estimated Costs Near-termEstablish a policy requiring that by 7/1/04, all campus PennNet-connected Windows systems: servers, desktops and workstations, have all critical updates applied within three business days………………………………………………………………………………………………………….existing staff Near-termCreate a new ISC service: “Patch Management Services” Staff……………………………………………………………..………………………………….$100,000/yr Hardware for campus SUS service………………………………………………...$10,000 every 2-3 yrs. Campus license for commercial patch management software……………………………….…$6/seat/yr Near-termVirus filtering……………………………………………………………………………..………….$5-$6/account/year Long-termEvaluate a network design and migration strategy that better balances availability against security, and capable of supporting more flexible use of internal and external router filtering/firewall technology…................................................................................................................................under evaluation Near-termProvide a basic level of network security by implementing router filter rules on external interfaces after campus consultation and on internal interfaces at the request of Schools/Centers Hardware/software for internal filtering…...............................................................$20,000/bldg every 3 years Near-termEnable Schools and Centers to implement tighter local security policies: -Publish support for VLANs……………..........………………………………..……………….N&T Documentation -Select a campus firewall and VPN standard, avoiding a campus VPN “Tower of Babel”, buying time to patch systems, and enabling Schools and Centers to better implement local firewalls and VPN gateways………………………………………………………………………………….………...…under evaluation Near-termEvaluate and recommend a personal firewall software product for targeted users and do a pilot evaluation. Software license for 50-100 users…………………..……………………………………$2500 - $5000 for 3 years

47. 47 Summary of Recommendations & Estimated Costs Near-termWork with Dell, IBM and Microsoft to create more secure default images for newly purchased Penn machines ………………...…………………………………………………………………………negotiated price < $25/image Near-termCreate policy establishing authority of ISC, Schools and Centers to conduct intrusion detection addressing privacy, log retention and related issues…………..................................................................no incremental cost Near-termDeploy and manage six-ten switch-connected IDS boxes to quickly identify compromised campus systems. Hardware…………………………………………………………………………$15,000-$20,000 every 2-3 years Staff to configure, manage, analyze IDS systems and follow up on intrusion reports…………………………………………………………………………$100,000/yr Long-termEvaluate and determine best method to provide router flow logs for intrusion detection………………………………………………………………………………………………..under evaluation Long-termEvaluate a network design and migration strategy that better balances availability against security, and capable of supporting broader intrusion detection…..................................................................under evaluation Near-termProvide tools to better support quick lookup of Host and DNS contacts………..………….…….under evaluation Near-termTargeted deployment of PennKey authenticated network access in, for example, College Houses, GreekNet, Library…...............................................................................................................................$2,000 - $5,000/bldg. Long-termFull deployment of PennKey authenticated network access on campus Hardware/Software (one-time)………………………………………………………………................$1,000,000 Near-termImplement two additional functions in PennKey network authentication of DHCP connections: Vulnerability scan of machines as they connect to PennNet Ability to block infected/vulnerable machines based on MAC address Hardware/Software…………...………………………………………………..………………..…..under evaluation Staff……………………………………………………………………………………………………under evaluation