1. 1 Bandwidth management and optimization BCrouter 14-16 March 2006 Dirk Janssens ICTS – K.U.Leuven

2. 2 Introduction into introduction BCrouter is an ongoing network project  Not all features are already implemented or ready for 3th party deployment Constructive feedback  What do you expect from a good solution  Try to fulfill as many expectations as possible

3. 3 Overview 1.Introduction  Problem  Expectations  BCrouter solution 2.BCrouter solution  Components  Example network setups  Integration  Security considerations 3.BCrouter  Introduction  Components  Commands and logging  Routing and Netfilter setup  Quota/Bandwidth exceptions 4.BCpolicer  Introduction  Design principles  Policing alternatives  Complete design 5.Case study: KotNet 6.Development  Current status  Future  Wish list

4. 4 Introduction: problem Bandwidth usage rises rapidly  Increasing Internet population  ‘Richer’ content (HTML,Flash,…)  P2P download applications  Video/music streaming Bandwidth availability is limited  Expensive uplink No alternatives  Expensive hardware

5. 5 Introduction: problem Majority of bandwidth  used by minority of users Minority of users  cause network congestion  cause problems for other users Example: K.U.Leuven KotNet  Student network across region of Leuven  20 000 active students  5% of users caused 50% of used bandwidth

6. 6 Introduction: problem Users are anonymous  Only known by IP address Very easy to change IP address to be anonymous  Everyone can (ab)use the network  What to do if external complaints come in? User awareness is needed  Let the user take responsibility of his own network usage Give the user a ‘personal credit’ he can use (network quota) Notify/block the user if his/her PC acts ‘strange’ and give instructions Answer: User authentication  Makes it possible to map every action on the network to an individual person  Prevents unauthorized access  Makes it possible to use ‘personal’ network settings and actions

7. 7 Introduction: expectations Login system  Each user must authenticate him/herself before using the network  No extra software or configuration needed on client hosts Bandwidth regulation  Works for all protocols and traffic  Prevent that a minority of users take away all the bandwidth for the majority of users  Allow exceptions to certain (educational) sites E.g. OS security updates, e-learning site…  Maximize responsiveness for interactive traffic E.g. Slow down bulk traffic, but don’t touch SSH unless really needed  Every user and/or IP can have its own personal bandwidth settings E.g. Different settings for a lab computer and personal PC  Distribute the individual bandwidth over the individual active network connections

8. 8 Introduction: expectations Volume quota  Every user and/or IP is only allowed to use a certain fixed amount of traffic Learns the user how to manage his Internet behavior  Slow down traffic when a user and/or IP generates too much traffic  Every user and/or group and/or IP can have its own personal quota settings E.g. personal vs. lab PC, limited guest accounts...  A user and/or IP is never blocked from the network (real-time small band)  If a user and/or IP who is on 'small band' stops downloading for a few minutes, the user immediately can use a limited amount of traffic again at normal speed.

9. 9 Introduction: BCrouter solution Why?  Didn’t find another solution that fulfills all the expectations No open source projects Commercial black boxes not really an option It’s interesting, fun and challenging  High performance needed Old quota/login system was maxed out Network usage still increases

10. 10 Introduction: BCrouter solution Features  User login system ‘Unlimited’ number of users Users can login multiple times at different location  Group based routing ‘Unlimited’ number of user groups possible Every group has its own independent routing and policy  Bandwidth regulation and volume quota Individual user/group and IP address based settings with no performance impact Prevent network congestion by dynamically regulating maximum bandwidth Powerful quota and bandwidth exception possibilities  User friendly No user side configuration needed Nice user webpage with information and history information Automatically redirect to login site for login

11. 11 Introduction: BCrouter solution Quota/bandwidth limiting to both user and IP  Example 1: Assign user: –Quota of 1 Gigabyte –Refill the quota at rate of 1 Gigabyte/month –Maximum speed: unlimited Assign IP: –Quota of 10 Mbyte –Refill the quota at rate of 5 Kilobytes/second –Maximum speed: 20 Kilobytes/sec Result: –User settings to determine the maximum volume a user can download each month –IP settings to limit the ‘real-time’ bandwidth usage

12. 12 Introduction: BCrouter solution Quota/bandwidth limiting to both user and IP  Example 2: Assign user: –Unlimited quota –Maximum speed: 50 Kilobytes/second Assign IP: –Quota of 10 Mbyte –Refill the quota at rate of 5 Kilobytes/second –Maximum speed: 20 Kilobytes/sec Result: –If a user logs in multiple times, the sum of all logins cannot exceed the maximum user speed. The speed is divided across the hosts that are logged in.

13. 13 Introduction: BCrouter solution

14. 14 Introduction: BCrouter solution

15. 15 Introduction: BCrouter solution

16. 16 Introduction: BCrouter solution

17. 17 Introduction: BCrouter solution

18. 18 Introduction: BCrouter solution

19. 19 Introduction: BCrouter solution

20. 20 Solution: components Frontend  Login server  Redirect server Backend  User database server  Log/History server  “BCrouter” router

21. 21 Solution: components Login server  Serves secure web pages to the users Login page Statistics page Technical information page …  Contacts the user database server for validating user accounts  Contacts the history server to gather historical information about logins and/or quota  Contacts BCrouter to check current quota and/or login status and performs login/logout

22. 22 Solution: components Redirect server  Redirects HTTP requests to the login page on the Login server  Gets all the traffic that requires a login from non- logged-in hosts  Redirect done by a webpage (not TCP level)  Separate dedicated host because can get DoS  Real time network anomaly detection Detect virus/worm before login… even for 1 st time users Coupled to automatic user blocking system

23. 23 Solution: components User database  Contains all known users  Contacted by the login server  Can be any type of server LDAP Radius Custom type of authentication …

24. 24 Solution: components Log/history server  Receives logs from BCrouter  Parses received log files  Store processed information in a database Historical login information Historical account information  Database contacted by the login server  Possibility to use data mining techniques to detect suspicious user behavior

25. 25 Solution: components BCrouter  Implements the core functionality  Linux based solution  Sends detailed quota reports and issued commands to the log server  Contacted by the login server Get quota information about user and/or IP Get login status of user and/or IP Perform login and logout operations

26. 26 Solution: internet router setup Assumptions  A few 1000’s of users Limit by log/history server  Manage the internet connection  Auto redirect to login website  Minimize the used Internet bandwidth

27. 27 Solution: internet router setup Login server User database Internet Log/History server Internal management network Web cache NAT Firewall BCrouter Internal backbone network Redirect server

28. 28 Solution: main router setup Assumptions  A few 1000’s of users Limit by log/history server  Manage the entire network  Auto redirect to login website  Central DHCP server is used to distribute IP addresses  Minimize the used Internet bandwidth

29. 29 Solution: main router setup Login server User database Internet Log/History server Internal management network Web cache NAT Firewall BCrouter Internal net DHCP DNS Redirect server

30. 30 Solution: setup remarks Webcache and NAT are between BCrouter and Internet  BCrouter needs to ‘see’ the user IP address  Otherwise not possible to make user and IP distinction  Advantage: Transparent web caching is possible  Disadvantage: Cached contents are also accounted and speed limited

31. 31 Solution: integration Suitable for each network? Ethernet based networks BCrouter does not support any routing protocols (RIP,EIGRP…) BCrouter can also play a Cisco Netflow probe High performance –Gigabit speeds with dual CPU system Redundancy (still in development) –Possible to have backup BCrouter in hot standby

32. 32 Solution: integration Scalability  BCrouter server Supports virtual unlimited users –Tested up to 50 000 users (1 Gigabyte RAM) –Handles up to 60 000 login/logout operations per second Supports virtual unlimited IP addresses –Tested up to 200 000 IP’s (1 Gigabyte RAM) Supports up to 300 000 packets/sec (1.5 Gigabit) –Dual Xeon 3.6Ghz  Clustering (Not yet implemented) Possible to use multiple BCrouter servers –Each server handles a part of the given network segments –Inter-BCrouter communication to exchange quota changes

33. 33 Solution: integration Quota/bandwidth exceptions?  Yes… very powerful exception capabilities  Exception flags IP speed limit User speed limit IP accounting User accounting No login required  Exceptions can be made for hosts or even entire networks (both local and/or internet)

34. 34 Solution: integration Quota/bandwidth exceptions examples:  Default: Login required Accounting to both user and local IP Obey both user and local IP speed limits  Local host A does not have to login to access the Internet, but still uses IP quota and speed settings E.g. Embedded device that can’t login and needs network access  Traffic from Internet host B is always possible from any local host and is never accounted, but local host IP speed limits are obeyed E.g. Website with security patches Any combination of exception flags is possible in either direction for any host/network

35. 35 Solution: security considerations Account abuse  Example User A powers off his PC without logging out Malicious user X takes IP of user A X continues to work with credentials of user A  Solution: Auto logout Possibility 1: BCrouter performs logout after X minutes of inactivity Possibility 2: Ping probes Possibility 3: DHCP server –Login server checks if IP that wants to login has been issued by the DHCP server. Refuse login with static IP –Use very short DHCP lease times (e.g. 15 minutes) –Run script every few minutes that logs out inactive DHCP leases  DHCP based auto-logout is preferred

36. 36 BCrouter: introduction Let’s take a look at the core element: BCrouter  Components of BCrouter  Commands and logging  Routing and Netfilter setup  Quota/Bandwidth exceptions

37. 37 BCrouter: introduction Login server User database Internet Log/History server Internal management network Web cache NAT Firewall BCrouter Internal net DHCP DNS Redirect server

38. 38 BCrouter: components ‘open’ black box Linux operating system  User space DHCP forwarder Syslog daemon BCrouter daemon Network configuration script  Kernel space BCpolicer module

39. 39 BCrouter: components BCrouter daemon DHCP forwarder Syslog daemon Kernel space User space Kernel logging Netfilter framework BCpolicer Input interfaces Output interfaces Management interface

40. 40 BCrouter: components DHCP forwarder  Forward broadcast DHCP DISCOVER to a central DHCP server  Dhcp-fwd http://www.nongnu.org/dhcp-fwd/  Very simple application  User space application running in chroot jail  Listens in ‘promiscuous mode’ on specified interfaces

41. 41 BCrouter: components Syslog daemon  Send logs to a remote log server for remote processing  Syslog-ng http://freshmeat.net/redir/syslog-ng/10178/url_homepage/syslog-ng  Very powerful options (filtering, multi logserver…)  Logs both user space as kernel logs

42. 42 BCrouter: components BCrouter daemon  Provides a network-based console to the BCpolicer kernel module  Simple Perl script (Forking TCP server)  Allows simultaneous management access  Listens on a network socket (telnet port 23)  Communicates with the kernel module

43. 43 BCrouter: components Network configuration script  Provides entire interface, routing and Netfilter configuration and setup  Shell script  Executed at boot time

44. 44 BCrouter: components BCpolicer kernel module  Receives login/logout commands and performs accounting and routing decisions  Core element of BCrouter (ipt_bcpolicer)  Works entirely in kernel space  Loadable module which implements an iptables target

45. 45 BCrouter: commands & logging Commands  Login/logout login [username] ip [x.x.x.x] reason [text] logout [username] reason [text] logout [ip] reason [text]  Query information show user ip [x.x.x.x] show ip user [username] show quota ip [x.x.x.x] show quota user [username]  Configuration conf ip … conf user … export all  Miscellaneous show uptime

46. 46 BCrouter: commands & logging Commands example bcrouter1# export all bcrouter1# login user kuleuven/u0022948 ip 10.91.91.1 reason login demo 200 OK: login - 1142031358930300 - kuleuven/u0022948 (1) on 10.91.91.1 (login demo) bcrouter1# show ip user kuleuven/u0022948 204 OK: show ip user - 10.91.91.1 bcrouter1# show user ip 10.91.91.1 203 OK: show user ip - kuleuven/u0022948 bcrouter1# login user kuleuven/u0022948 ip 10.91.91.2 reason 2nd login 200 OK: login - 1142031429848045 - kuleuven/u0022948 (1) on 10.91.91.2 (2nd login) bcrouter1# show ip user kuleuven/u0022948 204 OK: show ip user - 10.91.91.2,10.91.91.1 bcrouter1# export all conf user kuleuven/u0022948 …. conf ip 10.91.91.1 … login user kuleuven/u0022948 ip 10.91.91.1 reason recovering statefull info conf ip 10.91.91.2 …

47. 47 BCrouter: commands & logging network segment  Time of log  Log sequence number  Name of segment  Traffic counters Bytes and packets Download and upload Accounted and not accounted Dropped and accepted  Number of active IP’s host  Time of log  Log sequence number  IP address  Username (–none- if no login)  Traffic counters Bytes and packets Download and upload Accounted and not accounted Dropped and accepted Logging  Log commands and responses  Log network/host statistics

48. 48 BCrouter: routing & Netfilter Routing with BCrouter is done by a BCPOLICER target in the PREROUTING mangle table that alters the fwmark value of the packet and uses this value as selector for policy based routing.

49. 49 BCrouter: routing & Netfilter Use Linux networking capabilities  IEEE 802.1Q support (VLAN technology) Used to limit the number of physical interfaces  Policy based routing (Routing rules) Used for implementing user groups  Netfilter/Iptables framework Used for host exception lists

50. 50 BCrouter: routing & Netfilter IEEE 802.1Q support  Virtual LAN technology (VLAN)  Operates on the data link layer (OSI layer 2) Adds 4 extra bytes to existing ethernet header  Allows multiple LAN’s over 1 physical wire (trunking)  Each VLAN id has its own interface device E.g. eth0.5 indicates VLAN id 5 on physical interface eth0  ‘vconfig’ tool VLAN enabled Device VLAN enabled Device dot1Q ‘trunk’ containing VLAN 1,2,3 VLAN 1 VLAN 2 VLAN 3

51. 51 BCrouter: routing & Netfilter Policy based routing (Routing rules)  Multiple routing tables Routing policy database (RPDB) Each entry in the database: –contains a routing table –has a priority number –has a selector (src addr, dst addr, incoming iface, tos, fwmark) RPDB entries are scanned according priority If the selector of RPDB entry applies, that routing table is used  ‘ip rule’ tool

52. 52 BCrouter: routing & Netfilter Netfilter/Iptables framework  Extensive IP packet filtering rules  There are 3 ‘tables’ Each table has its own purpose  Each table contains several ‘chains’ Each chain operates on a different location in the network packet flow Chains can also be custom-defined  Each chain can contain several ‘rules’ Each rule contains a selector and a target Rules are examined in the order they are listed in the chain If a selector matches, the action is determined by the target  ‘iptables’ tool

53. 53 BCrouter: routing & Netfilter Filter table: Firewall type rules  INPUT: for packets destined for the box itself  FORWARD: for packets being routed through the box  OUTPUT: for locally generated packets NAT: Native Address Translation type rules  PREROUTING: for altering packets as soon as they come in  OUTPUT: for altering locally generated packets before routing  POSTROUTING: for altering packets as they are about to go out Mangle: Packet alteration type rules  PREROUTING: for altering packets before routing  OUTPUT: for altering locally generated packets before routing  INPUT: for altering packets destined for the box itself  FORWARD: for altering packets being routed through the box  POSTROUTING: for altering packets as they are about to go out

54. 54 Output Routing Input Routing BCrouter: routing & Netfilter PREROUTING Netfilter framework FORWARD INPUTOUTPUT POSTROUTING User space IFACEIFACE IFACEIFACE

55. 55 BCrouter: routing & Netfilter Packet filtering and routing are normally 2 different things However:  There is a packet ‘property’ that can be changed by an iptables target and can be used as a routing rule selector: fwmark Fwmark  Netfilter MARK value  Integer value  Associated within the kernel with the packet  Does not alter the packet itself

56. 56 BCrouter: routing & Netfilter Routing with BCrouter is done by a BCPOLICER target in the PREROUTING mangle table that alters the fwmark value of the packet and uses this value as selector for policy based routing. Fwmark value after BCPOLICER selects routing table:  1 … 97 : routing table belonging to the user group of the identified user  98: routing table for non-identified users  99: routing table for redirecting users to the login site  100: routing table that drops all packets

57. 57 BCrouter: routing & Netfilter IFACEIFACE IFACEIFACE PREROUTING mangle table SelectorTarget. Exception1MARK ex.flag1 Exception2MARK ex.flag2 … EverythingBCPOLICER Input routing PrioritySelectorRouting table 1fwmark=1Group1 2fwmark=2Group2 3fwmark=98NoGroup 4fwmark=99Redirect 5fwmark=100Drop Quota/bandwidth exception list changes fwmark before entering BCPOLICER Fwmark changed by BCPOLICER to select correct routing table

58. 58 BCrouter: exceptions Exceptions passed to BCPOLICER via fwmark value Possible to use almost all Iptables selection features List before BCPOLICER target in PREROUTING Fwmark value IP speed limit (src: 0x0001 dst: 0x0002) User speed limit (src: 0x0004 dst: 0x0008) IP accounting (src: 0x0010 dst: 0x0020) User accounting (src: 0x0040 dst: 0x0080) No login required (src: 0x0100 dst:0x0200)  Final value: logical OR of the wanted flags Example  Traffic from Internet host B is always possible from any local host and is never accounted, but local host IP speed limits are obeyed  Src: localnet Dst: hostB mark: 0x0001||0x0100 = 0x0101  Src: hostB Dst: localnet mark: 0x0002||0x0200 = 0x0202

59. 59 BCpolicer: introduction Everything up until now was on macroscopic level Let’s dig (a little bit) into the real core: ipt_bcpolicer The next slides are greatly simplified  Would take couple of hours to completely explain the entire design

60. 60 BCpolicer: introduction TCP/IP  TCP: reliable protocol Every received packet must be acknowledged to Sender by Receiver Sender retransmits when no acknowledgement received within certain time Exponential delay between retransmits When relatively few packets are dropped, transmission speed goes down significantly

61. 61 BCpolicer: introduction Policing vs. Shaping Policing  Accept/drop packets  Don’t delay packets  No queues  No scheduling Shaping  Don’t drop packets  Delay packets  Queuing  Scheduling

62. 62 BCpolicer: design principles Design started by determining the specifications of an ‘ideal’ bandwidth/quota system (with no user login)  Different settings for each individual network segment  For each network segment: Independent up/download settings Distribute available bandwidth across active hosts, favoring low- traffic volume hosts a little Dynamically regulate maximum bandwidth per host to allow maximum bandwidth usage without overloading the segment or uplink  For each host: Distribute host bandwidth equally over active connections Tendency to favor interactive traffic above bulk traffic Enforce traffic quota by dynamic bandwidth speed regulation

63. 63 BCpolicer: design principles Result  General principle: leaky bucket system  Lots of modifications and tweaks required POLICER MeanFillRate TokenBucketMaxSize CurrentRate (0…BurstRate) TokenBucketSize TokenBucket

64. 64 BCpolicer: design principles Example:  Download quota: 4GB/month  Maximum download speed: 50KByte/sec  DownloadTokenBucketMaxSize = 4GTokens  DownloadBurstRate = 50KTokens/sec  DownloadMeanFillRate Must fill up an empty TokenBucket in 30 days =4GTokens/(30days*24hours*60minutes*60seconds) =1657 Tokens/sec

65. 65 BCpolicer: design principles Policer summary  Multi exponential stream policer ‘smooth’ policing Distribute individual host bandwidth equally over its active connections Tendency to favor interactive traffic above bulk traffic

66. 66 BCpolicer: design principles Specifications list  For each host: Distribute host bandwidth equally over active connections –OK : Multi logarithmic stream policer algorithm Tendency to favor interactive traffic above bulk traffic –OK : Multi logarithmic stream policer algorithm Enforce traffic quota by dynamic bandwidth speed regulation –OK : Accurate quota control because there are no Tokens created or destroyed in the bucket system

67. 67 BCpolicer: design principles Specifications list  Different settings for each individual network segment OK : Each IP has its own buckets with its own settings  For each segment: Independent up/download settings –OK : Use separate buckets Distribute available bandwidth across active hosts, favoring low-traffic volume hosts a little –To be done Dynamically regulate maximum bandwidth per host to allow maximum bandwidth usage without overloading the local segment or uplink –To be done

68. 68 BCpolicer: design principles Prevent network congestion:  Implement a ‘valve’ between TokenBucket and the Policer If the ‘valve’ is fully open: –Maximum host speed is BurstRate If the ‘valve’ closes: –Slow down maximum host speed ‘valve’ control –If no network segment congestion: fully open –Network segment congestion: close valves for the local IP’s that cause the congestion until no congestion anymore ‘valve’ is named ‘RateFactor Correction’ The state of the valve is called RateFactor

69. 69 BCpolicer: design principles BCrouter Internet BCpolicer Feedback algorithm Host

70. 70 BCpolicer: design principles POLICER MeanFillRate TokenBucketMaxSize CurrentRate (0…BurstRate) TokenBucketSize TokenBucket RateFactor Correction RateFactor (e.g. <1)

71. 71 BCpolicer: policing alternatives Advantage of policing  Simple to implement No queuing No scheduling  Very fast Drawbacks of policing  Policing overhead problem Relative problem: –Suppose unlimited stream would be 100Kbyte/sec –Limit to 10Kbyte/sec –10 to 15% overhead -> receiving from the Internet 12KByte/sec More overhead on low latency, high bandwidth connections –E.g. Limit to 10KByte/sec; receiving 20KByte/sec

72. 72 BCpolicer: policing alternatives ‘TCP Window Size’ alteration  TCP Window Size Field in TCP packet header Message from Receiver to Sender Indicates to Sender how much data can be sent to Receiver  Should decrease retransmission overhead Avoid fast retransmission on fast connections  Alters packets passing BCrouter Possibly bad and unforeseen consequences?  No need for queuing  Integration in the BCpolicer scheme Use the DelayTime value to decrease the window size

73. 73 BCpolicer: policing alternatives Shaping  Alter the Round Trip Time of a connection ‘simulate’ slow connection TCP will try to adjust by slowing down  Extra load and complexity due to delay queue management  Integration in the BCpolicer scheme BCpolicer already works with delay times Instead of accepting or dropping, put packet into a delay queue for the delaytime

74. 74 BCpolicer: complete design Up until now, we only focused on the bucket design Complete system  Addition of user buckets Same principle as IP buckets  Addition of exception handling Makes program flow complex

75. 75 BCpolicer: main flow Start Src IP kotnet? Yes No Return IPT_CONTINUE srcnlr= 1 Mark translation Src userbucket Verdict (UV1) Src ipbucket Verdict (IV1) Src logge d in? REDIRECT srciac!=0 or srcisl!=0 Dst IP kotnet? Yes No dstnlr= 1 Dst userbucket Verdict (UV2) Dst ipbucket Verdict (IV2) Dst logge d in? DROP Yes (nlr) No (lr) Yes No Yes No Yes No Yes No Yes (nlr) No (lr) Final Verdict & Accounting & Logging srcnua= 1 and srcnus= 1 dstnua= 1 and dstnus= 1 Yes No Yes Src IP Bucket calculation dstiac!=0 or dstisl!=0 Clear src streamupdate flag Clear dst streamupdate flag

76. 76 BCpolicer: bucket verdict Bucket calculations No Respect Speedlimit? TBS- PktSize>0 Verdict=DROPVerdict=ACCEPT No Yes PBS>PTT+PktSize Verdict=ACCEPT Yes No Calculate StreamII and get the correct SAPS and SPLAT for this packet QSAPS=32- ((32*SAPS) / MPS) QDelayF=64- ((64*PBS) / PTT) PDT=DelayMatrix(QSAPS,QDelayF) SPLAT+PDT<PacketTS No Yes Verdict=DROPVerdict=ACCEPT Entrypoint Return Verdict Yes PBS>PktSize Yes No Store StreamII and new SAPS (SAPSnew) Set streamupdate flag for the current ip (used for accounting)

77. 77 BCpolicer: bucket calculation Yes No Yes Entrypoint Endpoint TB_ADD_INTERVAL passed since the last call? Calculate new TBS TBS=max( TBS+TBMFR*TBLAT,TBMS) Calc max CurrentR EffectiveR=max CurrentR PBS+EffectiveR<PBMS Calc max EffectiveR CurrentR = max EffectiveR (CurrentR = = EffectiveR) TBS=TBS-CurrentR FactorR=0 PBS=PBS+EffectiveR

78. 78 BCpolicer: f.verdict&acc&log Entrypoint Src IP kotnet? No Yes SOURCE IP Accounting Calculation DESTINATION IP Accounting Calculation FinalVerdict=DROP Return FinalVerdict FinalVerdict=ACCEPT If PktTime>SITLL+LOGSI Print Src IP logline & clear printed counters If PktTime>SSTLL+LOGSS Print Src Segment logline & clear printed counters REDIR verdict in UV1 || IV1 || UV2 || IV2? No Yes FinalVerdict=REDIR DROP verdict in UV1 || IV1 || UV2 || IV2? No Yes FinalVerdict = REDIR? No Yes Dst IP kotnet? No Yes If PktTime>SITLL+LOGSI Print Dst IP logline & clear printed counters If PktTime>SSTLL+LOGSS Print Dst Segment logline & clear printed counters

79. 79 BCpolicer: accounting FinalVerdict = DROP No Yes Entrypoint Return UsrTBS += PktSize UsrTBS - = PktSizeUsrPBS - = PktSize nua=1 Yes (no uac) No (uac) nus=1 Yes (no usl)No (usl) iac=1 IpTBS += PktSize isl=1 IpTBS - = PktSizeIpPBS - = PktSize No (no iac) Yes (iac) No (no isl)Yes (isl) nua=1 Yes (no uac)No (uac) SIFDB + = PktSize SIFDP++ SINFDB + = PktSize SINFDP++ SINFAB + = PktSize SINFAP++ SIFAB + = PktSize SIFAP++ SSTDB + = PktSize SSTDP++ SSTDI++ if needed SSTAI++ if needed SSTAB + = PktSize SSTAP++ SSTAI++ if needed streamupdate=1 Yes No SAPS at SII=SAPSnew

80. 80 Case study: KotNet Goals  Connect K.U.Leuven association students and personnel to the campus network and Internet from their homes  Enhance possibility of study and research in an academic environment  Low entrance fee and costs KotNet is an enabler for the E-university

81. 81 Case study: KotNet History  1994: Started by students in their student home  1996-97: pilot & test projects  1997-98 en 1998-99 1°, 2° operational phase K.U.Leuven residences access via cable TV network –test project in 1997-98 –operational since Sept. 1998 (2 cable TV operators)  2002 : Wireless KotNet at public places Test project in student restaurants Gradually extended to libraries, meeting rooms, classrooms  2004 : Agreements with associated partners

82. 82 Case study: KotNet Providers  K.U.Leuven – direct fiber to university residences  K.U.Leuven – wifi hotspots in student restaurants, auditoria,...  UPC Belgium – via cable modem  Telenet/Iverlek – via cable modem

83. 83 Case study: KotNet Technology  Layer 2 VLAN's are used to implement city-wide networks –connect remote buildings to centralized KotNet routers spanning-tree for automatic redundancy  Layer 3 use of private address space (10.*.*.*) split data streams –traffic to K.U.Leuven networks is routed through the central firewall –HTTP traffic is redirected to a web cache cluster –other traffic is handled by a Network Address Translator cluster

84. 84 Case study: KotNet Policies  Users will consume mega-amounts of bandwidth if you let them. Not all of their traffic is of an educational nature... implementation of policies and enforcement of these policies is absolutely essential  p2p traffic is not allowed (Gnutella, Kazaa, eDonkey,...) enforced by iptables filters on the NAT servers  individual servers are not allowed enforced by a 200 Mbyte/day upload limit exceptions are possible for registered servers  Internet bandwidth is not free enforced by a 4 Gbyte/month/USER download limit

85. 85 Case study: KotNet Residences  Approx 50 buildings  Approx 4000 connections  Spread geographically across the city of Leuven K.U.Leuven owned fiber network  Each building is a different VLAN  Ethernet network Old : –10Mbit HUB –10Mbit Half-duplex fiber uplink to KotNet backbone New : –100Mbit Switch (Cisco 2950) –100Mbit Full-duplex fiber uplink to KotNet backbone

86. 86 10Mbit HUB Case study: KotNet (residences) 10Mbit FOT KotNet Backbone Network KotNet Backbone Network 100Mbit Switch Old NetworkNew Network Gigabit 802.1Q VLAN trunk Aggregation Switch

87. 87 Case study: KotNet Cable provider Telenet  Coverage across the region of Leuven  Approx 1500 cable modems  Approx 5000 users  3 Cable segments 13 Mbit downstream capacity each 3 Mbit upstream capacity each Single VLAN (flat network)

88. 88 Case study: KotNet (Telenet) KotNet Backbon e Cable segment Cable Router (L3) 100Mbit FOT Telenet location 100Mbit Flat network

89. 89 Case study: KotNet Cable provider UPC  Coverage across the region of Leuven and Brussels  Approx 5100 cable modems  Approx 15000 users  52 Cable segments 50 Mbit downstream capacity each 3 * 10 Mbit upstream capacity each Each cable segment is a different VLAN

90. 90 Case study: KotNet (UPC) KotNet Backbon e Cable Head end (L2) UPC location Gigabit 802.1Q VLAN trunk 52 Cable segments

91. 91 Case study: KotNet KotNet Backbone  Gigabit Ethernet based  Redundant where possible spanning-tree protocol  Separated from K.U.Leuven backbone  Spread geographically across the city of Leuven

92. 92 Case study: KotNet KotNet Backbone -Residences -Telenet -UPC KotNet Backbone -Residences -Telenet -UPC Gigabit 802.1Q VLAN trunk containing 105 KotNet VLAN’s NAT K.U.Leuven Web cache KUL-KotNet Firewall Internet KULeuven network KULeuven network Monitor switch Flow logger BCrouter Redirect NAT Association Gigabit mirror port KUL-INET Firewall Cisco WCCP

93. 93 Case study: KotNet Pre-BCrouter  2 High-end Cisco 7500 routers 30.000 EUR each Couldn’t handle large network segments –Long user access lists requiring lots of CPU power Policy based routing not really flexible Logout operation very expensive (ACL reload) Inflexible shaping –Not possible per ip/user/stream/usergroup and certainly not everything together  Need for more scalable solution!!

94. 94 Case study: KotNet BCrouter  Replaces both Cisco routers  Hardware: Dell Poweredge 2650 Dual Intel(R) Xeon(TM) CPU 3.20GHz with Hyperthreading ServerWorks GC-LE chipset, 400MHz front side bus, 2:1 memory interleaving, 5 PCI buses (three of which are are PCI- X capable) 1 Gig ECC ram Two Intel E1000 PCI-X 133Mhz Network interfaces 3000 EUR  Extremely powerful and flexible configuration

95. 95 Case study: KotNet BCrouter settings  Only 1 simultaneous login allowed by login system  Normally: login required, obey user/IP quota/speed  User buckets Download quota of 4Gbyte/month –Max speed (BurstRate): none Upload quota of 200Mbyte/day –Max speed (BurstRate): none  IP buckets Download quota of 8Gbyte/month –Max speed (BurstRate): 250Kbyte/sec Upload quota of 8Gbyte/month –Max speed (BurstRate): 250Kbyte/sec

96. 96 Case study: KotNet BCrouter exceptions  DHCP, DNS server No login required, no user acc/bw, ip acc/bw  Login website reachable No login required, no user acc/bw, ip acc/bw  Local antivirus download website reachable No login required, no user acc/bw, ip acc/bw  Local FTP server reachable No login required, no user acc/bw, ip acc/bw  Unblock website reachable No login required, no user acc/bw, ip acc/bw  E-learning website reachable Login required, no user acc, user bw, ip acc/bw

97. 97 Case study: KotNet User blocking system (admin view)  Receives information from number of probes Email system Flowlogger Local honeypots  Gets an IP address Looks up the corresponding user Logout user Put user in a blacklist to prevent user login Send email to the user

98. 98 Case study: KotNet User blocking system (user view)  User is logged out  Receives email containing instructions Webmail is always reachable Local antivirus download site also reachable  If the user tries to login: Message his account is blocked More detailed instructions about the blocking reason Link to unblock website  User unblocks himself Only allowed 1 time each day

99. 99 Case study: KotNet Flow Logger  Emits Cisco Netflow data to logging system Very detailed traffic information (for each stream): –Start and stop time –Source and destination IP –Source and destination port –Number of bytes transmitted/received –Number of packets transmitted/received  High performance Netflow probe: nProbe (http://www.ntop.org)  Used to detect network anomalies Host scans and port scans Coupled to automatic user blocking system

100. 100 Case study: KotNet Email  No direct SMTP connections to the internet  All emails must pass the Central Anti Virus cluster Scans for viruses (Mcafee Antivirus) Performs spam classification (Spamassassin) If a virus is detected, user is blocked –User blocking system looks at sending IP

101. 101 Case study: KotNet Transparent caching  No user settings required to use caching  Reduce the traffic on the Internet link  Cisco Content Engine Squid based WCCP –‘transparent’ redundancy Needed a quick solution

102. 102 Case study: KotNet Firewalling  Done on BCrouter Beginning of PREROUTING mangle table  TCP port 135,139,445 are blocked for anyone Prevent virus outbreaks

103. 103 Case study: KotNet P2P blocking  Done on NAT cluster  Do a pattern search in every packet Look for certain P2P strings –Kazaa, Bittorrent, Edonkey Iptables string module Rather CPU expensive Planning to make a better version someday

104. 104 Case study: KotNet Transition to BCrouter  Phased transition Created 2 testing KotNet segments on BCrouter Test connectivity and performance Modify login scripts and backend for BCrouter Convert 2 real KotNet segments to BCrouter Convert all residences to BCrouter Convert all UPC to BCrouter Convert all Telenet to BCrouter Parallel volume accounting on user webpage –Real-time counters from BCrouter –History information by previous KotNet accounting system  No real problems encountered

105. 105 Case study: KotNet Traffic impact (early testing)  User mode test version of BCpolicer  No HTTP traffic  Most popular P2P applications blocked  Only used IP addresses for accounting PolicerBucketMaxSize: 150 Mbyte MeanFillRate: 50 Kbyte/sec BurstRate: 250 Kbyte/sec  Results: 50% reduction in traffic Only 5% of users ‘hit’ BCpolicer  Users noted improvement in network quality

106. 106 Case study: KotNet Some Numbers  Current daily numbers 400 Mbit/sec throughput total (18u – 23u) –150 Mbit Non-http –30 Mbit K.U.Leuven 80 Kpackets/sec Up to 20 000 simultaneous users  BCrouter load 33% CPU at peak

107. 107 Case study: KotNet Management utilities  The next few slides show the web-based information interface

108. 108 Case study: KotNet

109. 109 Case study: KotNet

110. 110 Case study: KotNet

111. 111 Case study: KotNet

112. 112 Case study: KotNet

113. 113 Development: current status BCrouter itself is functioning  Design is done by specifications  Implementation of ipt_bcpolicer is not yet 100% ready Dynamic bandwidth regulation by congestion is not yet implemented (Fixed RateFactor of 1) Login system is not self-contained  Currently uses the previous login system from KotNet

114. 114 Development: future Redundancy  Timeframe: June 2006  Hot-standby backup system which takes over if primary fails  Further development needed Build a stand-alone login/log/history system  Timeframe: September 2006  Easy custom layout  Minimal dependencies  Already have some proof of concept code  Can be done independently of further development  Login system without history information can be done very quickly  Planning: hire a job student for this during the summer

115. 115 Development: future Black Box solution  Timeframe: Jan 2007?  Depends on stand-alone login system  Packaging and generalizing scripts Packaging of BCrouter partly done TCP window size policing  Timeframe: unknown  Still requires research  Planning: make a thesis out of this FactorRate Correction  Timeframe: unknown Not an issue anymore on KotNet  Implementation and testing

116. 116 Development: wish list Transform the policer into a shaper  Probably more work than expected  Planning: make a thesis out of this someday Add the automatic blocking features  Currently uses the previous KotNet solution

117. 117 Conclusion BCrouter concept offers great possibilities  User authentication  Advanced and flexible quota and bandwidth management  High performance and scalable solution BCrouter still requires a lot of implementation  Concepts are already tested in the field

118. 118 THE END Contact information: Dirk Janssens ICTS K.U.Leuven bcrouter@kuleuven.net