2. Building a Secure and Resilient Network Infrastructure Dan Massey Colorado State University

3. 7 October 042massey@cs.colostate.edu Outline l Changes in the Infrastructure Environment l Using Internet Worm Attacks To Motivate: n Secure and Resilient BGP Communication n Path Vector Algorithm Convergence n Network Fault Identification l New Challenges in Authentication n DNS Security

4. 7 October 043massey@cs.colostate.edu Original Infrastructure Goals l The original designs assumed that: n Hardware is unreliable: servers/routers will fail n Network links are unreliable: connections will fail n Data transport is unreliable: bit errors will occur l The goal was to build protocols that : n Provide functionality despite all of the above n Scale to extremely large size l Tremendously successful in this respect: n BGP routing protocol - 150K+ routes 20K+ systems n DNS naming protocol - 1G of records in 60M zones

5. 7 October 044massey@cs.colostate.edu The Infrastructure Today l Success and growth to large-scale adds: n Implementation and under-specification errors n Configuration errors by diverse administrators n Complex interactions and challenge of scale n Intentional attacks l The Internet works today because: n Robust original design masks many problems. n Clever operational tricks keep the system afloat –Ex: AOL BGP TTL Hack (RFC 3682) to protect routers from DDoS l “For every type of animal there is a most convenient size, and a large change in size inevitably carries with it a change of form.”

6. 7 October 045massey@cs.colostate.edu Changing the Form of the Internet l We need to recognize current design successes. n The Internet generally works today. n Includes millions of already deployed systems. n Provides a laboratory for large-scale system problems. l New challenges require a new approach to design. n Essential to add resilience and security. n But this does not imply we must start from scratch. l New solutions must either be incrementally deployable or must prove the necessity for a fresh start.

7. 7 October 046massey@cs.colostate.edu Slammer Worm After 30 Minutes (graph by CAIDA)

8. 7 October 047massey@cs.colostate.edu BGP Routing Infrastructure l Internet’s Global Routing Protocol n Connects Autonomous Systems (AS) l Path Vector Routing Protocol n Announce the path of AS used to reach destination l Routes adapt to: n Link changes & route polices l Does not adapt to traffic load. n Recent worm attacks should have no impact on BGP. AS 1 AS 2AS 3 Prefix P Path 2,1 Prefix P Path 3,1

9. 7 October 048massey@cs.colostate.edu BGP Updates During Nimda Worm Measurement Artifacts Routing Changes Total Attack

10. 7 October 049massey@cs.colostate.edu BGP Measurement Artifacts l BGP peers establish TCP session and send full route table (120K+ routes) n Updates sent only if routes change. l Our results show frequent session resets between ISP routers and the monitoring point. n Monitoring point sessions cross multiple systems in the Internet. n Each reset adds 120K updates. n But very few ISP-ISP session resets. l Our work in [1] presents rules to remove session reset artifacts. Initial Table (120K+ routes) Route Changes Initial Table (120K+ routes)

11. 7 October 0410massey@cs.colostate.edu What Our Analysis Shows 40.2% A substantial percentage of the BGP messages during the worm attack were not about route changes 37.6% 8.8% 8.3%

12. 7 October 0411massey@cs.colostate.edu FRTR: Improving Peer Communication l BGP Updates Are Not (Topology) Event Driven n Session resets trigger high volume surges –Govindan shows cascade failures can result. l Lifetime of Invalid Routes is Unbounded n Never recover (until reset) if update is somehow lost. –Despite TCP, we found cases of “lost” withdrawals. n Attacker can poison a route with one update. l Soft-state (periodic re-announce) is too costly… l FRTR Uses Periodic Bloom Filter Digests n Digests quickly confirm state after session reset. n Periodic digests bound lifetime of faults (w/ high prob). n Co-Author Keyur Patel (Cisco) is exploring Cisco development.

13. 7 October 0412massey@cs.colostate.edu FRTR Performance l For each route at receiver, check against the digest. n Bloom filter results in no false negatives. l Compare total digests for missing route detection. n False positive possible with known rate. n Add salts to reduce the chance of repeated false positives. l Overhead is a function of digest size and frequency. l Work with Cisco suggests a 1.3% overhead increase. l Complete Details to appear in [2] (DSN 2004)

14. 7 October 0413massey@cs.colostate.edu What Our Analysis Shows (2) 40.2% What about the 60% Not Due to Table Exchange… 37.6% 8.8% 8.3% FRTR Elimanates Bursts

15. 7 October 0414massey@cs.colostate.edu A Closer Look at the Route Changes Actual path changes Updates with no path change This is the U.S. DoD. Explained in [3] (DISCEX 2003)

16. 7 October 0415massey@cs.colostate.edu Improving Path Vector Convergence l Infocom 02 [4] uses consistency to detect invalid paths. n Reject path if r1 is adirect neighbor r1’s path is not n Adjusted to account for policy and implement in BGP l Infocom 03 [Afek, et al] quickly flushes invalid paths. n BGP requires updates be separated by a min interval n Send withdraw (to flush route) if blocked by the interval l Our recent work [5] attaches a new attribute: Root Cause Notification (RCN) n Identifies the failed link and includes a sequence number. n Allows any route relying on the failed link to be rejected.

17. 7 October 0416massey@cs.colostate.edu Simulation Results

18. 7 October 0417massey@cs.colostate.edu What Our Analysis Shows (3) 40.2% Can’t Eliminate the actual topology dynamics… 37.6% 8.8% 8.3% FRTR Elimanates Bursts RCN Improves Convergence

19. 7 October 0418massey@cs.colostate.edu Identifying the Source of Faults l It is believed that worm attacks caused edge instability, but core links remained up. n Can we prove (or disprove) this claim? l The Fault Identification Problem n BGP Monitoring points collect gigabytes of data from an ad hoc selection of monitoring points. n Underlying Internet topology is not known, but data does include path information n What can you conclude regarding faults? l Pursuing Two Parallel Solutions n Enhance protocol to include fault data (RCN). n Design tools and algorithms to automate fault identification

20. 7 October 0419massey@cs.colostate.edu The Link Rank Analysis Toolset l LinkRank [6] developed for analyzing BGP data. l Assigns each AS-AS link a weight based on number of prefixes. l Records aggregate rank changes over time. l Figure shows the graph from AS 6539. n Note all links leaving AS 701 show a route loss.

21. 7 October 0420massey@cs.colostate.edu Combining Multiple Views l Previous snapshots suggested a failure at AS 701. l View from other points shows all BGP monitors saw a shift away from AS 701. l NANOG confirmed a corresponding failure event. l Successfully applied LinkRank to several Internet events.

22. 7 October 0421massey@cs.colostate.edu Formalizing the Results l LinkRank relies heavily on human intuition. n Investigating algorithms to automate detection. l The Fault Identification Problem n Given only path vector routing table snapshots. n Can you find the minimum set of link changes that explain the snapshots? n Can you find a representation of all possible changes the explain the snapshots? l Results for shortest path policies in [7]. n Work in progress on other polices and partial link failures.

23. 7 October 0422massey@cs.colostate.edu Lessons From The Worm Attacks l Worm Shows Complexity of BGP Dynamics n Need to stablize the peer communication (FRTR). n Need to improve path vector convergence (RCN). n Would like to identify real source topology events. l But we must not forget that n Ultimate goal of routing is to delivery packets. –Route updates are only a means to toward this goal. n Worm attack was not directed against routing.

24. 7 October 0423massey@cs.colostate.edu Infrastructure Faults and Attacks Internet c.gtld-servers.net BGP monitor 192.26.92.30 originates route to 192.26.92/24 l BGP and DNS Provide No Authentication n Faults and attacks can mis-direct traffic. n One (of many) examples observed from BGP logs. n Server could have replied with false DNS data. ISPs announced new path for 20 minutes to 3 hours

25. 7 October 0424massey@cs.colostate.edu Cryptography is like magic fairy dust, we just sprinkle it on our protocols and its makes everything secure - See IEEE Security and Privacy Magazine, Jan 2003

26. 7 October 0425massey@cs.colostate.edu Secure DNS Query and Response Caching DNS Server End-user www.darpa.mil www.darpa.mil = 192.5.18.195 Plus (RSA) signature by darpa.mil Attacker can not forge this answer without the darpa.mil private key. Authoritative DNS Servers

27. 7 October 0426massey@cs.colostate.edu There is no magic fairy dust

28. 7 October 0427massey@cs.colostate.edu What To Take Away l A new look at the Internet infrastructure n Scaling up has more profound implications beyond bigger numbers/tables. n Data reveals interesting problems and provides a large- scale systems lab. l Challenges Remain in Improving the System n But we can build backwards compatible changes into the infrastructure (ex: FRTR and RCN) n Need to develop general approaches to resilient design of large-scale systems (Internet, Sensor Nets, Etc.)

29. 7 October 0428massey@cs.colostate.edu Reference Cited 1. Observation and Analysis of BGP Behavior under Stress, L. Wang, X. Zhao, D. Pei, R. Bush, D.Massey, A. Mankin, S. F. Wu, and L. Zhang, Proceedings of the SIGCOMM Internet Measurement Workshop, 2002 2. FRTR: A Scalable Mechanism to Restore Routing Table Consistency, L. Wang, D. Massey, K. Patel, and L. Zhang, To appear in IEEE Dependable Systems and Networks (DSN), July 2004 3. Understanding BGP Behavior Through A Study of DoD Prefixes, X.Zhao, M. Lad, D. Pei, L.Wang, D. Massey, S. F. Wu, and L. Zhang, Proceedings of DISCEX III, April 2003. 4. Improving BGP Convergence with Consistency Assertions, D. Pei, L. Wang, X. Zhao, D. Massey, L. Zhang, A. Mankin, Proceedings of the IEEE INFOCOM 2002. 5. BGP-RCN: Improving BGP Convergence Through Root Cause Notification, D. Pei, M. Azuma, N. Nguyen, J. Chen, D. Massey, and L. Zhang, UCLA Department of Computer Science Technical Report, UCLA CSD TR-030047, October 2003. 6. Link-Rank: A Graphical Tool for Capturing BGP Routing Dynamics, M. Lad, D. Massey, and L. Zhang, To appear in Network Operations and Management Symposium (NOMS), April, 2004. 7. An Algorithmic Approach to Identifying Link Failures, M. Lad, A. Nanavati, D. Massey, and L. Zhang, To appear in 10th Pacific Rim Dependable Computing Symposium (PRDC) March, 2004 8. DNS Security Introduction and Requirements, R. Arends, R. Austein, M. Larson, D. Massey and S. Rose, Work in Progress, IETF DNS EXT Working Group. Feb, 2004

30. 7 October 0429massey@cs.colostate.edu Acknowledgements l Funding Sources n FNIISC Project: August 2000 - May 2004 –DARPA Fault Tolerant Networks –PI: USC/ISI (Dan Massey), UCLA (Lixia Zhang), UC Davis (S. Felix Wu) n Beyond BGP Project: October 2002 - September 2005 –NSF Special Projects in Networking –PI: USC/ISI (Dan Massey) and UCLA (Lixia Zhang) n FMESHD Project: July 2000 - December 2003 –DARPA Fault Tolerant Networks –PI: USC/ISI (Dan Massey); subk to NAI (Russ Mundy) l With Thanks to Collaborators and Graduate Students n Lixia Zhang and Felix Wu n Lan Wang, Dan Pei, and Mohit Lad (UCLA & USC/ISI interns) n Naheed Vora (USC & USC/ISI Intern)

31. 7 October 0430massey@cs.colostate.edu Revised DNS Key Management mil DNS Server darpa.mil DNS Server darpa.mil NS records www.darpa.mil A record www.darpa.mil SIG(A) by key 2 darpa.mil KEY (pub key 1) darpa.mil KEY (pub key 2) darpa.mil SIG(A) by key 1 darpa.mil DS record (hash of pubkey 1) darpa.mil SIG(DS) by mil private key Can Change mil key without notifying darpa.mil Can Change key 2 without notifying.mil

32. 7 October 0431massey@cs.colostate.edu Next Step DNS Security Activities l Co-editor of the IETF specification [8]. n Last call workshop completed last month. n Cleaning up minor issues and nits. l Dept. of Homeland Security DNSSEC Group n Group of 10 advising DHS on DNS security deployment strategies. n Need operational policies for end systems. l Investigating Resilient DNS n Real security is more than authentication. n Joint work with Amir/Terzis and Zhang/Wu. –NSF ITR Proposal just completed (hours ago :)

33. 7 October 0432massey@cs.colostate.edu DNS Key Roll-Over mil DNS Server darpa.mil DNS Server darpa.mil KEY (pub key 1) darpa.mil KEY (pub key 2) darpa.mil SIG(A) by key 1 darpa.mil DS record (hash of pubkey 1) darpa.mil SIG(DS) by mil private key darpa.mil KEY (pub key 3) darpa.mil SIG(A) by key 3 darpa.mil DS record (hash of pubkey 3) darpa.mil SIG(DS) by mil private key Objective: Replace KEY 1 with new KEY 3

34. 7 October 0433massey@cs.colostate.edu Multi-Origin AS Routing Announcement l MOAS exists in current BGP operation n Some due to operational need; some due to faults l Blind acceptance of MOAS dangerous n An open door for traffic hijacking

35. 7 October 0434massey@cs.colostate.edu BGP-based Solution Example router bgp 59 neighbor 1.2.3.4 remote-as 52 neighbor 1.2.3.4 send-community neighbor 1.2.3.4 route-map setcommunity out route-map setcommunity match ip address 18.0.0.0/8 set community 59:MOAS 58:MOAS additive Example configuration: AS58 18/8, PATH, MOAS{4,58,59} AS59 18.0.0.0/8 18/8, PATH, MOAS{58,59} 18/8, PATH, MOAS{52, 58} AS52

36. 7 October 0435massey@cs.colostate.edu (b) Two Origin AS’s(a) One Origin AS BGP false origin detection Simulation Results

37. 7 October 0436massey@cs.colostate.edu BGP Updates During Slammer Worm

38. 7 October 0437massey@cs.colostate.edu Constructing Fault Graphs l Monitor observes a shift from red path to blue path. n (Other monitors reveal node 5) l Convert to a Fault-Graph n Combine all topology data. n Greedy algorithm to select ``core” faults near root. n Recursive search to find alternates for each core fault. n Results in lower fault-graph. l A set of edges is an explanation iff it is cut in the fault-graph. n Min explanation = min cut l Extends to multiple views. n Used to analyze LinkRank Data 1 23 5 7 46 Monitor Desitnation 1 2 5 7 4 Sink Source

39. 7 October 0438massey@cs.colostate.edu Infrastructure Security Enhancements l BGP and DNS lack authentication. n Easy to insert false BGP routes or reply with false DNS data. l S-BGP (BBN) & SoBGP (Cisco) propose adding Public Key Authentication to BGP. n Verify origin is authorized to announce prefix and verify each link in the AS path. –Is this path authentication the right goal? n Requires a heavy-weight PKI structure. l DNSSEC adds authentication to DNS n Further along than the BGP approaches n Provides lessons for BGP authentication.