University of WashingtonComputing & Communications UW Medicine Networking Update Terry Gray Associate Vice President, IT Infrastructure University of Washington 16 April 2004

University of WashingtonComputing & Communications Key Elements of the Partnership  Changed: C&C now responsible for...  In-building network implementation and operational support for med ctrs, clinics  Med center network design “for real”  Not Changed: C&C still responsible for...  Network backbone, routers  Regional and Internet connectivity  SoM and Health Sciences networking

University of WashingtonComputing & Communications Why the Partnership Makes Sense  Consistency, interoperability, manageability  Leverage C&C networking expertise  Clinical/research hi-performance network needs  24x7 Network Operations Center (NOC)  Advanced network management tools  Avoid design/build organizational conflicts  Beyond the network... hope to share distributed system architecture and network computing expertise

University of WashingtonComputing & Communications Near-term Progress and Plans  Created “Top 10” list --now up to Top 20 :)  Agreement on standard maintenance window  Static addressing work-around (sDHCP)  FDDI, VLAN elimination  Subnet splits/upgrades (1500 computers)  Equipment upgrades  Router consolidation, dedicated subnets, separate med center backbone  Equipment, outlet location database updates  Initial wireless deployment  NetVersant and Cisco external studies

University of WashingtonComputing & Communications The Challenge Create a network computing environment –with excellent security –excellent supportability –that users find reliable and responsive

University of WashingtonComputing & Communications Context: A Perfect Storm  Increased dependency on network apps  Decreased tolerance for outages  Decades of deferred maintenance...  Inadequate infrastructure investment  Some old/unfortunate design decisions  Some fragile applications  Fragmented host management  Increasingly hostile security environment  Increasing legal/regulatory liability  Increasing importance of research/clinical leverage

University of WashingtonComputing & Communications Context: Some Numbers

University of WashingtonComputing & Communications Network Device Growth Note: Most dips reflect lower summer use; last one is a measurement anomaly

University of WashingtonComputing & Communications Network Traffic Growth (linear)

University of WashingtonComputing & Communications Network Traffic Growth (log)

University of WashingtonComputing & Communications System Elements  Environmentals (Power, A/C, Physical Security)  Network  Client Workstations  Servers  Applications  Personnel, Procedures, Policy, and Architecture Failures at one level can trigger problems at another level; need Total System perspective

University of WashingtonComputing & Communications Systemic Network Problems (some of these go back decades)  Old infrastructure (e.g cat 3 wire)  Non-supportable technologies (e.g. FDDI)  Non-supportable (non-geographic) topology  Expensive shortcuts (e.g. cat5 mis-terminated)  Security based on individual IP addresses  Subnets with clients and critical servers  Documentation deficiency  Contact database  Device location database  Critical device registry

University of WashingtonComputing & Communications Systemic General Problems  Ever-increasing system complexity, dependencies  Ever-increasing threats, liabilities  Departmental autonomy  Un-controlled hosts  Un-reliable power and A/C in equipment rooms  No net-oriented application procurement standards  Are HA and DRBR expectations realistic?  Are backup plans workable?

University of WashingtonComputing & Communications Key Operational Objectives simplicity –lower cost –higher MTBF (modulo redundancy) –lower MTTR (quicker diagnosis) consistency –deterministic outlet behavior (Network Utility Model) –connection transparency (open/deterministic Internet) –easier problem diagnosis These objectives conflict with other goals

University of WashingtonComputing & Communications Design Tradeoffs  Networks = Connectivity; Security = Isolation  Fault Zone size vs. Economy/Simplicity  Reliability vs. Complexity  Prevention vs. (Fast) Remediation  Security vs. Supportability vs. Functionality Differences in NetSec approaches relate to:  Balancing priorities (security vs. ops vs. function)  Local technical and institutional feasibility

University of WashingtonComputing & Communications Tradeoff Examples Defense-in-depth conjecture (for N layers) –Security: MTTE (exploit)  N**2 –Functionality: MTTI (innovation)  N**2 –Supportability: MTTR (repair)  N**2 Perimeter Protection Paradox (for D devices) –Firewall value/efficiency  D –Firewall effectiveness  1 / D Border blocking criteria –Threat can’t reasonably be addressed at edge –Won’t harm network (performance, stateless block) –Widespread consensus to do it Security by IP address

University of WashingtonComputing & Communications Network Security Chronology 1990: Five anti-interoperable networks 1994: Nebula shows network utility model viable 1998: Defined border blocking policy 2000: Published Network Security Credo 2000: Added source address spoof filters 2000: Proposed med ctr network zone 2000: Proposed server sanctuaries 2001: Ban clear-text passwords on C&C systems 2001: Proposed pervasive host firewalls 2001: Developed logical firewall solution 2002: Developed Project-172 solution 2003: Slammer, Blaster… death of the Internet 2003: Developed flex-net architecture

University of WashingtonComputing & Communications Next-Gen Network Architecture  Parallel networks; more redundancy  Supportable (geographic) topology  Med center subnets = separate backbone zone  Perimeter, sanctuary, and end-point defense  Higher performance  High-availability strategies  Workstations spread across independent nets  Redundant routers  Dual-homed servers

University of WashingtonComputing & Communications Success Metrics  Tom’s  Nobody gets hurt  Nobody goes to jail  Steve’s  Four Nines or bust!  High ROI (Return On Investment)  Terry’s  Low ROI (Risk Of Interruption)  Low MTTR (Quick to Fix)  High predictability (No surprises)

University of WashingtonComputing & Communications Lessons  Net reliability & host security are inextricably linked  Five 9s is hard (unless we only attach phones?)  $ for $, best security investment is central host management  Nebula existence proof: security in an open network  Watch out for unfair cost shifting  The cost of static IP configuration is very high  Controlling net access is hard --hublets, wireless  Even host firewalls don’t guarantee safety  Perimeter firewalls may increase user confusion, MTTR  It only takes one compromise inside to defeat a firewall  Next-generation threats: firewalls won’t help  Even so… defense-in-depth is a Good Thing

University of WashingtonComputing & Communications Questions? Comments?

University of WashingtonComputing & Communications Network Security Addendum

University of WashingtonComputing & Communications Recent Events attacks –slammer (Jan 2003) –blaster (Aug 2003) –sobig (Sep 2003) –mydoom (Feb 2004) –witty (Mar 2004) impact –demise of the open/transparent/deterministic Internet –demise of the network utility model –demise of the unmanaged/autonomous PC –demise of reliable

University of WashingtonComputing & Communications Seven Security Axioms 1. Network security is maximized when we assume there is no such thing. 2. Large security perimeters mean large vulnerability zones. 3. Firewalls are such a good idea, every computer should have one. Seriously. 4. Remote access is fraught with peril, just like local access. 5. One person's security perimeter is another's broken network. 6. Private networks won't help (Limits of isolation). 7. Network security is about psychology as well as technology.

University of WashingtonComputing & Communications Network Security Credo Focus first on the edge (Perimeter Protection Paradox) Add defense-in-depth as needed Keep it simple (e.g. Network Utility Model) But not too simple (e.g. offer some policy choice) Avoid –one-size-fits-all policies –cost-shifting from “guilty” to “innocent” –confusing users and techs (“broken by design”)

University of WashingtonComputing & Communications Preserving the Net Utility Model What is it? Why important? Incompatible with perimeter security? Too late to save? NUM-preserving perimeter defense –Logical Firewalls –Project 172 Foiled by static IP addressing… –Requires all hosts be reconfigured

University of WashingtonComputing & Communications Conflicting Perspectives System administrator view –some prefer local control/responsibility –some prefer central/big-perimeter defense –some underestimate cost impact on others User view –want just enough openness to run apps –prefer “unlisted numbers”? Network operator view –concerned about increased support costs and repair times due to growing complexity and unpredictability –concerned about loss of network functionality

University of WashingtonComputing & Communications Generic Security Toolkit host choice: truly thin clients; species diversity host configuration management conventional firewalls logical firewalls private addressing (e.g. project 172) IDS, IPS, ADS vulnerability scanning, anti-virus tools QoS (to protect critical traffic types) isolated networks (physical, VLAN, VPN) non-technical: policies, education, staff

University of WashingtonComputing & Communications Lines of Defense network isolation for critical services host integrity (Make the OS net-safe) host perimeter (integral ACLs/firewalling) cluster/lab perimeter (sanctuary, FW, LFW) network zone perimeter (P172, FW) real-time attack detection and containment user education

University of WashingtonComputing & Communications Perimeter Firewalls increase time-to-infection increase time-to-repair provide defense-in-depth may look like a broken network to users are defeated by a single hacked host are defeated by tunneling/encryption often give a false sense of security encourage backdoors may be a performance bottleneck may inhibit legitimate activities, innovation create a vulnerability zone that is hard to protect: –vpns, laptops, wifi, usb drives, social engr attacks –the more you depend on perimeter defense, the more you must invest in defending the perimeter

University of WashingtonComputing & Communications Operational Impact by firewall type host -- best case; user interaction w/FW possible cluster -- no impact on net diagnosis “beyond” logical -- low impact on basic net diagnosis subnet -- impacts almost all diagnosis zone -- impacts inter-zone diagnosis border --impacts inter-enterprise diagnosis NB: cost of maintaining firewall config depends on who is doing it, and how many rules/exceptions there are.

University of WashingtonComputing & Communications Limits of Isolation: attack gateways  hosts connected to two different networks can become attack gateways between the two  example: home PCs with VPN connection to protected network  safer remote access: SSH, SSL, K5, RDP, SSL VPNs

University of WashingtonComputing & Communications Med Center Zone Perimeter purpose –time to defend against zero-day events –protect the otherwise unprotected –defense-in-depth –reduced annoyance/noise traffic –DOS attack mitigation options –conventional inline firewall –private addressing + NAT or proxies –both

University of WashingtonComputing & Communications Protecting Non-fixable Devices  FDA-approved devices, printers, etc  protection options (besides zone perimeter):  private addressing  individual firewall, VPN, or NAT box ($25 - $2500) --depending on performance requirements  cluster/lab perimeter firewalls  logical firewalls

University of WashingtonComputing & Communications NOC view of Firewall Approaches EPFW = End-Point Firewall LFW = Logical Firewall w/masquerading NAT SFW = Subnet Firewall BZFW = Border or Zone Firewall P172 = Project 172-phase III (Private addresses with NAT) IDEAL EPFW LFW P172 SFW BZFW Policy Enforcement Point? Host Host Subnet Zone Subnet Zone Requires host reconfigure? No Yes Yes Yes No No Requires network reconfig? No No No No Yes Yes Destroys E2E transparency? No No No No Yes Yes Assured NOC access to switches? Yes Yes Yes Yes No* No* User sees why app failed? Yes Yes No No No No NOC-Predictable semantics? Yes No No Yes No No Inherent "unlisted number"? - No Yes Yes No No "unlisted number" possible? Yes Yes Yes Yes Yes Yes Adverse impact on internal network troubleshooting: Low Low Med Med High Low Adverse impact on external network troubleshooting: Low Low Med Med High High Size of vulnerability zone: Small Small Med Large Med Large * Can be mitigated by proper access lists and/or OOB connectivity

University of WashingtonComputing & Communications Network Security Trends High Low password guessing self-replicating code password cracking exploiting known vulnerabilities disabling audits back doors hijacking sessions sniffers packet spoofing automated probes/scans denial of service www attacks Attack Sophistication “stealth” / advanced scanning techniques burglaries DDOS attacks Source: Blended attacks

University of WashingtonComputing & Communications Impact of Recent Security Events more perimeter firewalls (demise of open Internet, NUM) more VPNs more tunneling (“firewall friendly” apps) more encryption (thanks to RIAA) more collateral damage (from attacks & remedies) worse MTTR (complexity, broken tools) constrained innovation (e.g. p2p, voip) cost shifted from “guilty” to “innocent” pressure to fix computer security problems in network pressure for private nets pressure to make network topology match org boundaries blaster: triggered more perimeter defense, but showed weakness of conventional perimeter defense