Geneva, 24 March 2011 Cisco experiences of IP traffic flow measurement and billing with NetFlow Benoit Claise, Distinguished Engineer, Cisco ITU-T Workshop on IP Traffic Flow Measurement (Geneva, Switzerland, 24 March 2011)

Geneva, 24 March Abstract The first part of this talk focuses on the latest NetFlow development in Cisco, while the second part will share experience regarding the specific use case of usage based billing with NetFlow.

What is NetFlow? Cache Collector NetFlow Records export Over UDP or SCTP Traffic

What is NetFlow? NetFlow is used for traffic monitoring, security analysis, capacity planning and billing Billing is just a few % of our customers, mainly for charge back within enterprise network (not between service providers) NetFlow = a exporting protocol: NetFlow v5, 7, 8, 9 (RFC3954), and IPFIX (RFC5101/RFC5102) NetFlow v9 and IPFIX work with a template based mechanism Advantage: extensibility, just need to add new Information Element NetFlow = a metering process: Flexible NetFlow Advantages: cache and export content flexibility User selection of flow keys User definition of the records

Flexible NetFlow: Potential Key Fields IPv4 IP (Source or Destination) Payload Size Prefix (Source or Destination) Packet Section (Header) Mask (Source or Destination) Packet Section (Payload) Minimum-Mask (Source or Destination) TTL Protocol Options bitmap Fragmentation Flags Version Fragmentation Offset Precedence IdentificationDSCP Header LengthTOS Total Length Interface Input Output Flow Sampler ID Direction Source MAC address Destination MAC address Dot1q VLAN Source VLAN Layer 2 IPv6 IP (Source or Destination) Payload Size Prefix (Source or Destination) Packet Section (Header) Mask (Source or Destination) Packet Section (Payload) Minimum-Mask (Source or Destination) DSCP ProtocolExtension Headers Traffic ClassHop-Limit Flow LabelLength Option HeaderNext-header Header LengthVersion Payload Length Dest VLAN Dot1q priority

Multicast Replication Factor* RPF Check Drop* Is-Multicast Flexible NetFlow: Potential Key Fields Input VRF Name BGP Next Hop IGP Next Hop src or dest AS Peer AS Traffic Index Forwarding Status Routing Transport Destination PortTCP Flag: ACK Source PortTCP Flag: CWR ICMP CodeTCP Flag: ECE ICMP TypeTCP Flag: FIN IGMP Type*TCP Flag: PSH TCP ACK NumberTCP Flag: RST TCP Header LengthTCP Flag: SYN TCP Sequence NumberTCP Flag: URG TCP Window-SizeUDP Message Length TCP Source PortUDP Source Port TCP Destination Port UDP Destination Port TCP Urgent Pointer Application Application ID* *: IPv4 Flow only

Flexible NetFlow: Potential Non-Key Fields Plus any of the potential key fields: will be the value from the first packet in the flow Counters Bytes Bytes Long Bytes Square Sum Bytes Square Sum Long Packets Packets Long Timestamp sysUpTime First Packet IPv4 Total Length Minimum (*) Total Length Maximum (*) TTL Minimum TTL Maximum (*) IPV4_TOTAL_LEN_MIN, IPV4_TOTAL_LEN_MAX (**)IP_LENGTH_TOTAL_MIN, IP_LENGTH_TOTAL_MAX IPv4 and IPv6 Total Length Minimum (**) Total Length Maximum (**)

Performance Limited Resources in Router Dont enable all flow keys The routers still have to route packets

NetFlow for Billing: Experience

Packet Size Standard Deviation σ f Mean Packet Size µ f #Packets N f Estimation Accuracy (PLT_NZIX1, S24D00, Cisco, f=5% Issue: Can we use Sampled NetFlow for billing? Huge amount of data, must sometimes deal with sampled NetFlow, i.e. 1 out of N packets, depending on the platform Packet Sampling for Flow Accounting: Challenges and LimitationsPacket Sampling for Flow Accounting: Challenges and Limitations, Tanja Zseby, Thomas Hirsch, Benoit Claise, PAM 2008

Issue: Can we use Sampled NetFlow for billing? Square sum of bytes available in Flexible NetFlow Not used in practice, not even by the collectors! Customers afraid of legal issues with sampling along with a billing service

AS=196 E-BGP ISP 1 $5.00 per 100 MB traffic index = 1 Prefix Traffic-index Forwarding Information Base prefix two traffic index = 2 prefix one traffic index = 1 Destination Sensitive Billing Proposal (many years ago) AS=193 Customer E-BGP AS 192 ISP 2 $7.00 per 100 MB 1. BGP routing updates 2. Go through a table-map statement 3. table-map calls a route-map 4. route-maps criteria: if criteria 1 -> traffic-index = 1 if criteria 2 -> traffic-index = 2 prefix one traffic index = 1 Accounting I-BGP

BGP Policy Accounting Principles Allows to classify packets based on IP access lists, BGP community list to characterize the exit points, where each exit point would set an specific community BGP AS paths

The ISP The Customer Issue: What about the Returning Packets? ISP 1 $5.00 per 100 MB ISP 2 $7.00 per 100 MB FTP Request 100 MB back Who should pay for the 100 MB back? Destination Sensitive Billing requires also source lookup (Source Sensitive Billing) Who should pay for the 100 MB back? Destination Sensitive Billing requires also source lookup (Source Sensitive Billing)

The ISP The Customer Issue: What about the Returning Packets? ISP 1 $5.00 per 100 MB ISP 2 $7.00 per 100 MB FTP Request 100 MB back Lookup: On the outgoing packets (on the packets coming back) On the source Same selection criteria Lookup: On the outgoing packets (on the packets coming back) On the source Same selection criteria

The ISP The Customer in Europe Issue: BGP Asymmetry Problem ISP 1 in AsiaISP 2 in US FTP Request 100 MB back Will charge the 10 Meg as if they were directly coming from the US!!!

Issue: BGP Asymmetry Problem The source lookup is based on the route the router would take to reach the source!

Too Many Issues Destination Sensitive Billing requires Source Sensitive Billing BGP asymmetry problem Only the traffic following the BGP routes will be accounted What if local policies outside of BGP? Limited amount of buckets in the Destination Sensitive Billing Doesnt scale: too many entries Performance issues Entire NMS solution to be put in place

Destination Sensitive Billing Conclusion/feedback from customers: too many issues not realistically deployable -> back to some sort of flat rate Benoits concern: If we bill per AS-PATH and each AS get a piece of the pie, people will create new AS and try to attract traffic Bad for the internet performance

Geneva, 24 March 2011 Cisco experiences of IP traffic flow measurement and billing with NetFlow Benoit Claise, Distinguished Engineer, Cisco ITU-T Workshop on IP Traffic Flow Measurement (Geneva, Switzerland, 24 March 2011)