1. Network Measurement Jennifer Rexford Advanced Computer Networks http://www.cs.princeton.edu/courses/archive/fall08/cos561/ Tuesdays/Thursdays 1:30pm-2:50pm

2. Outline Traffic –SNMP link statistics –Packet and flow monitoring Network topology –IP routers and links –Intradomain route monitoring Interdomain routes –BGP route monitoring and table dumps –Inferring AS-level topology and biz relationships Conclusions

3. Traffic Measurement

4. Why is Traffic Measurement Important? Billing the customer –Measure usage on links to/from customers –Applying billing model to generate a bill Traffic engineering and capacity planning –Measure the traffic matrix (i.e., offered load) –Tune routing protocol or add new capacity Denial-of-service attack detection –Identify anomalies in the traffic –Configure routers to block the offending traffic Analyze application-level issues –Evaluate benefits of deploying a Web caching proxy –Quantify fraction of traffic that is P2P file sharing

5. Collecting Traffic Data: SNMP Simple Network Management Protocol –Standard Management Information Base (MIB) –Protocol for querying the MIBs Advantage: ubiquitous –Supported on all networking equipment –Multiple products for polling and analyzing data Disadvantages: dumb –Coarse granularity of the measurement data E.g., number of byte/packet per interface per 5 minutes –Cannot express complex queries on the data –Unreliable delivery of the data using UDP

6. Collecting Traffic Data: Packet Monitoring Packet monitoring –Passively collecting IP packets on a link –Recording IP, TCP/UDP, or application-layer traces Advantages: details –Fine-grain timing information E.g., can analyze the burstiness of the traffic –Fine-grain packet contents Addresses, port numbers, TCP flags, URLs, etc. Disadvantages: overhead –Hard to keep up with high-speed links –Often requires a separate monitoring device

7. Collecting Traffic Data: Flow Statistics Flow monitoring (e.g., Cisco Netflow) –Statistics about groups of related packets (e.g., same IP/TCP headers and close in time) –Recording header information, counts, and time Advantages: detail with less overhead –Almost as good as packet monitoring, except no fine-grain timing information or packet contents –Often implemented directly on the interface card Disadvantages: trade-off detail and overhead –Less detail than packet monitoring –Less ubiquitous than SNMP statistics

8. Using the Traffic Data in Network Operations SNMP byte/packet counts: everywhere –Tracking link utilizations and detecting anomalies –Generating bills for traffic on customer links –Inference of the offered load (i.e., traffic matrix) Packet monitoring: selected locations –Analyzing the small time-scale behavior of traffic –Troubleshooting specific problems on demand Flow monitoring: selective, e.g,. network edge –Tracking the application mix –Direct computation of the traffic matrix –Input to denial-of-service attack detection

9. Flow Measurement

10. Flow Measurement: Outline Definition –Passively collecting statistics about groups of packets –Group packets based on headers and spacing in time –Essentially a way to aggregate packet measurement data Scope –Medium-grain information about user behavior –Passively monitoring the link or the interface/router –Helpful in characterizing, detecting, diagnosing, and fixing Outline –Definition of an IP “flow” (sequence of related packets) –Flow measurement data and its applications –Mechanics of collecting flow-level measurements –Reducing the overheads of flow-level measurement

11. flow 1flow 2flow 3 flow 4 IP Flows Set of packets that “belong together” –Source/destination IP addresses and port numbers –Same protocol, ToS bits, … –Same input/output interfaces at a router (if known) Packets that are “close” together in time –Maximum spacing between packets (e.g., 15 sec, 30 sec) –Example: flows 2 and 4 are different flows due to time

12. Flow Abstraction A flow is not exactly the same as a “session” –Sequence of related packets may be multiple flows (due to the “close together in time” requirement) –Sequence of related packets may not follow the same links (due to changes in IP routing) –A “session” is difficult to measure from inside the network Motivation for this abstraction –As close to a “session” as possible from inside the network –Flow switching paradigm from IP-over-ATM technology –Router optimization for forwarding/access-control decisions (cache the result after the first packet in a flow) –… might as well throw in a few counters

13. Recording Traffic Statistics (e.g., Netflow) Packet header information (same for every packet) –Source and destination IP addresses –Source and destination TCP/UDP port numbers –Other IP & TCP/UDP header fields (protocol, ToS bits, etc.) Aggregate traffic information (summary of the traffic) –Start and finish time of the flow (time of first & last packet) –Total number of bytes and number of packets in the flow –TCP flags (e.g., logical OR over the sequence of packets) start finish 4 packets 1436 bytes SYN, ACK, & FIN SYN ACK FIN

14. Recording Routing Info (e.g., Netflow) Input and output interfaces –Input interface is where the packets entered the router –Output interface is the “next hop” in the forwarding table Source and destination IP prefix (mask length) –Longest prefix match on the src and dest IP addresses Source and destination autonomous system numbers –Origin AS for src/dest prefix in the BGP routing table Switching Fabric Processor Line card BGP table forwarding table

15. Measuring Traffic as it Flows By input output source AS source prefix source dest AS dest prefix dest intermediate AS Source and destination: IP header Source and dest prefix: forwarding table or BGP table Source and destination AS: BGP table

16. Packet vs. Flow Measurement Basic statistics (available from both techniques) –Traffic mix by IP addresses, port numbers, and protocol –Average packet size Traffic over time –Both: traffic volumes on a medium-to-large time scale –Packet: burstiness of the traffic on a small time scale Statistics per TCP connection –Both: number of packets & bytes transferred over the link –Packet: frequency of lost or out-of-order packets, and the number of application-level bytes delivered Per-packet info (available only from packet traces) –TCP seq/ack #s, receiver window, per-packet flags, … –Probability distribution of packet sizes –Application-level header and body (full packet contents)

17. Collecting Flow Measurements Router A Route CPU that generates flow records …may degrade forwarding performance Router A Line card that generates flow records …more efficient to support measurement in each line card Router ARouter B Monitor Packet monitor that generates flow records …third party CPU

18. Router Collecting Flow Measurement Advantage –No need for separate measurement device(s) –Monitor traffic over all links in/out of router (parallelism) –Ease of providing routing information for each flow Disadvantage –Requirement for support in the router product(s) –Danger of competing with other 1 st -order router features –Possible degradation of the throughput of the router –Difficulty of online analysis/aggregation of data on router Practical application –View from multiple vantage points (e.g., all edge links)

19. Packet Monitor Collecting Flow Records Advantages –No performance impact on packet forwarding –No dependence on support by router vendor –Possibility of customizing the thinning of the data Disadvantages –Overhead/cost of tapping a link & reconstructing packets –Cost of buying, deploying, and managing extra equipment –No access to routing info (input/output link, IP prefix, etc.) Practical application –Selective monitoring of a small number of links –Deployment in front of particular services or sites Packet monitor vendors support flow-level output

20. Mechanics: Flow Cache Maintain a cache of active flows –Storage of byte/packet counts, timestamps, etc. Compute a key per incoming packet –Concatenation of source, destination, port #s, etc. Index into the flow cache based on the key –Creation or updating of an entry in the flow cache #bytes, #packets, start, finish packet key header key

21. Mechanics: Evicting Cache Entries Flow timeout –Remove flows that have not received a packet recently –Periodic sequencing through the cache to time out flows –New packet triggers the creation of a new flow Cache replacement –Remove flow(s) when the flow cache is full –Evict existing flow(s) upon creating a new cache entry –Apply eviction policy (LRU, random flow, etc.) Long-lived flows –Remove flow(s) that persist for a long time (e.g., 30 min) –… otherwise flow statistics don’t become available –… and the byte and packet counters might overflow

22. Sampling: Packet Sampling Packet sampling before flow creation (Sampled Netflow) –1-out-of-m sampling of individual packets (e.g., m=100) –Create of flow records over the sampled packets Reducing overhead –Avoid per-packet overhead on (m-1)/m packets –Avoid creating records for a large number of small flows Increasing overhead (in some cases) –May split some long transfers into multiple flow records –… due to larger time gaps between successive packets time not sampled two flows timeout

23. Conclusions Flow measurement –Medium-grain view of traffic on one or more links Advantages –Lower measurement volume than full packet traces –Available on high-end line cards (Cisco Netflow) –Control over overhead via aggregation and sampling Disadvantages –Computation and memory requirements for the flow cache –Loss of fine-grain timing and per-packet information –Not uniformly supported by router vendors

24. Intradomain Network Topology

25. IP Topology Topology information –Routers –Links, and their capacities Internal links inside the AS Edge links connecting to neighboring domains Ways to learn the topology –Inventory database –SNMP polling/traps –Traceroute –Route monitoring –Router configuration data

26. Below IP Layer-2 paths –ATM virtual circuits –Frame Relay virtual circuits Mapping to lower layers –Specific fibers –Shared optical amplifiers –Shared conduits –Physical length (propagation delay) Information not visible to IP –Stored in an inventory database –Not necessarily generated/updated automatically

27. Intradomain Monitoring: OSPF Protocol Link-state protocol –Routers flood Link State Advertisements (LSAs) –Routers compute shortest paths based on weights –Routers identify next-hop to reach other routers 3 2 2 1 1 3 1 4 5 3

28. Intradomain Route Monitoring Construct continuous view of topology –Detect when equipment goes up or down –Input to traffic-engineering and planning tools Detect routing anomalies –Identify failures, LSA storms, and route flaps –Verify that LSA load matches expectations –Flag strange weight settings as misconfigurations Analyze convergence delay –Monitor LSAs in multiple locations with go –Compare the times when LSAs arrive Detect router implementation mistakes

29. Passive Collection of LSAs OSPF is a flooding protocol –Every LSA sent on every participating link –Very helpful for simplifying the monitor Can participate in the protocol –Shared media (e.g., Ethernet) Join multicast group and listen to LSAs –Point-to-point links Establish an adjacency with a router … or passively monitor packets on a link –Tap a link and capture the OSPF packets

30. Interdomain Route Monitoring

31. Motivation for BGP Monitoring Visibility into external destinations –What neighboring ASes are telling you –How you are reaching external destinations Detecting anomalies –Increases in number of destination prefixes –Lost reachability to some destinations –Route hijacking –Instability of the routes Input to traffic-engineering tools –Knowing the current routes in the network Workload for testing routers –Realistic message traces to play back to routers

32. BGP Monitoring: A Wish List Ideally: knowing what the router knows –All externally-learned routes –Before policy has modified the attributes –Before a single best route is picked How to achieve this –Special monitoring session on routers that tells everything they have learned –Packet monitoring on all links with BGP sessions If you can’t do that, you could always do… –Periodic dumps of routing tables –BGP session to learn best route from router

33. Using Routers to Monitor BGP Talk to operational routers using SNMP or telnet at command line (-) BGP table dumps are expensive (+) Table dumps show all alternate routes (-) Update dynamics lost (-) restricted to interfaces provided by vendors Establish a “passive” BGP session from a workstation running BGP software (+) BGP table dumps do not burden operational routers (-) Receives only best routes from BGP neighbor (+) Update dynamics captured (+) not restricted to interfaces provided by vendors eBGP or iBGP

34. Atlanta St. Louis San Francisco Denver Cambridge Washington, D.C. Orlando Chicago Seattle Los Angeles Detroit Houston New York Phoenix San Diego Austin Philadelphia Dallas 2 Kansas City Collect BGP Data From Many Routers Route Monitor BGP is not a flooding protocol

35. Example: BGP Table (“show ip bgp” at RouteViews) Network Next Hop Metric LocPrf Weight Path * 3.0.0.0 205.215.45.50 0 4006 701 80 i * 167.142.3.6 0 5056 701 80 i * 157.22.9.7 0 715 1 701 80 i * 195.219.96.239 0 8297 6453 701 80 i * 195.211.29.254 0 5409 6667 6427 3356 701 80 i *> 12.127.0.249 0 7018 701 80 i * 213.200.87.254 929 0 3257 701 80 i * 9.184.112.0/20 205.215.45.50 0 4006 6461 3786 i * 195.66.225.254 0 5459 6461 3786 i *> 203.62.248.4 0 1221 3786 i * 167.142.3.6 0 5056 6461 6461 3786 i * 195.219.96.239 0 8297 6461 3786 i * 195.211.29.254 0 5409 6461 3786 i AS 80 is General Electric, AS 701 is UUNET, AS 7018 is AT&T AS 3786 is DACOM (Korea), AS 1221 is Telstra

36. Inferring the AS Topology

37. What is the AS Graph? Node: Autonomous System Edge: Two ASes that speak BGP to each other 1 2 3 4 5 6 7

38. How Do You Know a Node or Edge Exists? Consult the Whois database? –Tells which ASes have been allocated –But, might be out-of-date on who owns it –… and often doesn’t say who the neighbors are See a path that uses the node/edge –Collect measurements of AS paths –Extract all of the nodes and edges –E.g., AS path “7018 1 88” implies Nodes: 7018, 1, and 88 Edges: (7018, 1) and (1, 88)

39. Interdomain Routing Policies Two main decisions –Path selection: which of the paths to use? –Path export: which neighbors to tell? Both driven by business relationships, e.g., –Customer pays provider for Internet access –Peers find it mutually advantageous to cooperate 3 2 1 12.34.158.5 “12.34.158.0/24: path (2,1)” “12.34.158.0/24: path (1)” data traffic

40. Customer-Provider Relationship Customer needs to be reachable from everyone –Provider exports routes learned from customer to everyone Customer does not want to provide transit service –Customer does not export from one provider to another d d provider customer provider Traffic to the customerTraffic from the customer advertisements traffic

41. Peer-Peer Relationship Peers exchange traffic between customers –AS exports only customer routes to a peer –AS exports a peer’s routes only to its customers peer Traffic to/from the peer and its customers d advertisements traffic

42. Paths You Should Never See (“Invalid”) Customer-provider Peer-peer two peer edges transit through a customer

43. Other Kinds of Relationships Siblings –Same company –Mutual transit service –Like one bigger AS –Mergers, acquisitions, … Backup –Used only when failure –Second provider –Backup peering Geography-specific –Customer in U.S. –Peer in Europe A B C D E F G H primary backup

44. AS Relationships Matter Scientific understanding –Understanding Internet structure and evolution –Understanding why certain paths are used for traffic Placement of Web servers –Want to be close to most customer networks Business decisions –Selecting new peer or provider, or renegotiating relations Security policies –Knowing which BGP routes look suspicious Analyzing BGP convergence –Relationships have a big impact here (more later!)

45. Inferring AS Relationships Top down: how routes are selected –AS relationships define routing policy –Routing policy determines the routes you see Bottom up: how policies can be inferred –Routing data are available from public sources –The chosen routes tell you about the policy Example: seeing path “A B C” tells you… –B permits A to transit through B to reach C –(A,B) and (B,C) cannot both be peering links –A and C are not both upstream providers of B

46. Type-of-Relationship Problem Given the inputs –AS graph G(V,E) with vertices V and edges E –Set of paths P on the graph G Find a solution that –Labels each edge with an AS relationship –Minimizes the number of “invalid” paths in P Rich area of research work –http://www-unix.ecs.umass.edu/~lgao/ton.ps –http://www.cs.princeton.edu/~jrex/papers/infocom02.pdf –http://www.caida.org/publications/papers/2006/as_relation ships_inference/

47. Conclusions Passive measurements –Traffic: SNMP, packets, and flows –Routing: intradomain and interdomain Publicly-available measurements –Netflow from Abilene Internet2 –BGP updates and table dumps from RouteViews Constructing AS-level topology –AS graph based on edges in AS paths –Inferring business relationships between ASes