Bandwidth management and optimization

1
Bandwidth management and optimization

• BCrouter
• 14-16 March 2006
• Dirk JanssensICTS K.U.Leuven

2
Introduction into introduction

• BCrouter is an ongoing network project
• Not all features are already implemented or ready
  for 3th party deployment
• Constructive feedback
• What do you expect from a good solution
• Try to fulfill as many expectations as possible

3
Overview

• Introduction
• Problem
• Expectations
• BCrouter solution
• BCrouter solution
• Components
• Example network setups
• Integration
• Security considerations
• BCrouter
• Introduction
• Components
• Commands and logging
• Routing and Netfilter setup
• Quota/Bandwidth exceptions

• BCpolicer
• Introduction
• Design principles
• Policing alternatives
• Complete design
• Case study KotNet
• Development
• Current status
• Future
• Wish list

4
Introduction problem

• Bandwidth usage rises rapidly
• Increasing Internet population
• Richer content (HTML,Flash,)
• P2P download applications
• Video/music streaming
• Bandwidth availability is limited
• Expensive uplink
• No alternatives
• Expensive hardware

5
Introduction problem

• Majority of bandwidth
• used by minority of users
• Minority of users
• cause network congestion
• cause problems for other users
• Example K.U.Leuven KotNet
• Student network across region of Leuven
• 20 000 active students
• 5 of users caused 50 of used bandwidth

6
Introduction problem

• Users are anonymous
• Only known by IP address
• Very easy to change IP address to be anonymous
• Everyone can (ab)use the network
• What to do if external complaints come in?
• User awareness is needed
• Let the user take responsibility of his own
  network usage
• Give the user a personal credit he can use
  (network quota)
• Notify/block the user if his/her PC acts
  strange and give instructions
• Answer User authentication
• Makes it possible to map every action on the
  network to an individual person
• Prevents unauthorized access
• Makes it possible to use personal network
  settings and actions

7
Introduction expectations

• Login system
• Each user must authenticate him/herself before
  using the network
• No extra software or configuration needed on
  client hosts
• Bandwidth regulation
• Works for all protocols and traffic
• Prevent that a minority of users take away all
  the bandwidth for the majority of users
• Allow exceptions to certain (educational) sites
• E.g. OS security updates, e-learning site
• Maximize responsiveness for interactive traffic
• E.g. Slow down bulk traffic, but dont touch SSH
  unless really needed
• Every user and/or IP can have its own personal
  bandwidth settings
• E.g. Different settings for a lab computer and
  personal PC
• Distribute the individual bandwidth over the
  individual active network connections

8
Introduction expectations

• Volume quota
• Every user and/or IP is only allowed to use a
  certain fixed amount of traffic
• Learns the user how to manage his Internet
  behavior
• Slow down traffic when a user and/or IP generates
  too much traffic
• Every user and/or group and/or IP can have its
  own personal quota settings
• E.g. personal vs. lab PC, limited guest
  accounts...
• A user and/or IP is never blocked from the
  network (real-time small band)
• If a user and/or IP who is on 'small band' stops
  downloading for a few minutes, the user
  immediately can use a limited amount of traffic
  again at normal speed.

9
Introduction BCrouter solution

• Why?
• Didnt find another solution that fulfills all
  the expectations
• No open source projects
• Commercial black boxes not really an option
• Its interesting, fun and challenging ?
• High performance needed
• Old quota/login system was maxed out
• Network usage still increases

10
Introduction BCrouter solution

• Features
• User login system
• Unlimited number of users
• Users can login multiple times at different
  location
• Group based routing
• Unlimited number of user groups possible
• Every group has its own independent routing and
  policy
• Bandwidth regulation and volume quota
• Individual user/group and IP address based
  settings with no performance impact
• Prevent network congestion by dynamically
  regulating maximum bandwidth
• Powerful quota and bandwidth exception
  possibilities
• User friendly
• No user side configuration needed
• Nice user webpage with information and history
  information
• Automatically redirect to login site for login

11
Introduction BCrouter solution

• Quota/bandwidth limiting to both user and IP
• Example 1
• Assign user
• Quota of 1 Gigabyte
• Refill the quota at rate of 1 Gigabyte/month
• Maximum speed unlimited
• Assign IP
• Quota of 10 Mbyte
• Refill the quota at rate of 5 Kilobytes/second
• Maximum speed 20 Kilobytes/sec
• Result
• User settings to determine the maximum volume a
  user can download each month
• IP settings to limit the real-time bandwidth
  usage

12
Introduction BCrouter solution

• Quota/bandwidth limiting to both user and IP
• Example 2
• Assign user
• Unlimited quota
• Maximum speed 50 Kilobytes/second
• Assign IP
• Quota of 10 Mbyte
• Refill the quota at rate of 5 Kilobytes/second
• Maximum speed 20 Kilobytes/sec
• Result
• If a user logs in multiple times, the sum of all
  logins cannot exceed the maximum user speed. The
  speed is divided across the hosts that are logged
  in.

13
Introduction BCrouter solution
14
Introduction BCrouter solution
15
Introduction BCrouter solution
16
Introduction BCrouter solution
17
Introduction BCrouter solution
18
Introduction BCrouter solution
19
Introduction BCrouter solution
20
Solution components

• Frontend
• Login server
• Redirect server
• Backend
• User database server
• Log/History server
• BCrouter router

21
Solution components

• Login server
• Serves secure web pages to the users
• Login page
• Statistics page
• Technical information page
• Contacts the user database server for validating
  user accounts
• Contacts the history server to gather historical
  information about logins and/or quota
• Contacts BCrouter to check current quota and/or
  login status and performs login/logout

22
Solution components

• Redirect server
• Redirects HTTP requests to the login page on the
  Login server
• Gets all the traffic that requires a login from
  non-logged-in hosts
• Redirect done by a webpage (not TCP level)
• Separate dedicated host because can get DoS
• Real time network anomaly detection
• Detect virus/worm before login even for 1st time
  users
• Coupled to automatic user blocking system

23
Solution components

• User database
• Contains all known users
• Contacted by the login server
• Can be any type of server
• LDAP
• Radius
• Custom type of authentication

24
Solution components

• Log/history server
• Receives logs from BCrouter
• Parses received log files
• Store processed information in a database
• Historical login information
• Historical account information
• Database contacted by the login server
• Possibility to use data mining techniques to
  detect suspicious user behavior

25
Solution components

• BCrouter
• Implements the core functionality
• Linux based solution
• Sends detailed quota reports and issued commands
  to the log server
• Contacted by the login server
• Get quota information about user and/or IP
• Get login status of user and/or IP
• Perform login and logout operations

26
Solution internet router setup

• Assumptions
• A few 1000s of users
• Limit by log/history server
• Manage the internet connection
• Auto redirect to login website
• Minimize the used Internet bandwidth

27
Solution internet router setup
Internal backbone network
User database
BCrouter
Login server
Log/History server
Web cache
NAT Firewall
Redirect server
Internal management network
Internet
28
Solution main router setup

• Assumptions
• A few 1000s of users
• Limit by log/history server
• Manage the entire network
• Auto redirect to login website
• Central DHCP server is used to distribute IP
  addresses
• Minimize the used Internet bandwidth

29
Solution main router setup
Internal net
Internal net
Internal net
Internal net
User database
BCrouter
Login server
Log/History server
Redirect server
Internal management network
Web cache
NAT Firewall
DHCP
Internet
DNS
30
Solution setup remarks

• Webcache and NAT are between BCrouter and
  Internet
• BCrouter needs to see the user IP address
• Otherwise not possible to make user and IP
  distinction
• Advantage
• Transparent web caching is possible
• Disadvantage
• Cached contents are also accounted and speed
  limited

31
Solution integration

• Suitable for each network?
• Ethernet based networks
• BCrouter does not support any routing protocols
  (RIP,EIGRP)
• BCrouter can also play a Cisco Netflow probe
• High performance
• Gigabit speeds with dual CPU system
• Redundancy (still in development)
• Possible to have backup BCrouter in hot standby

32
Solution integration

• Scalability
• BCrouter server
• Supports virtual unlimited users
• Tested up to 50 000 users (1 Gigabyte RAM)
• Handles up to 60 000 login/logout operations per
  second
• Supports virtual unlimited IP addresses
• Tested up to 200 000 IPs (1 Gigabyte RAM)
• Supports up to 300 000 packets/sec (1.5 Gigabit)
• Dual Xeon 3.6Ghz
• Clustering (Not yet implemented)
• Possible to use multiple BCrouter servers
• Each server handles a part of the given network
  segments
• Inter-BCrouter communication to exchange quota
  changes

33
Solution integration

• Quota/bandwidth exceptions?
• Yes very powerful exception capabilities
• Exception flags
• IP speed limit
• User speed limit
• IP accounting
• User accounting
• No login required
• Exceptions can be made for hosts or even entire
  networks (both local and/or internet)

34
Solution integration

• Quota/bandwidth exceptions examples
• Default
• Login required
• Accounting to both user and local IP
• Obey both user and local IP speed limits
• Local host A does not have to login to access the
  Internet, but still uses IP quota and speed
  settings
• E.g. Embedded device that cant login and needs
  network access
• Traffic from Internet host B is always possible
  from any local host and is never accounted, but
  local host IP speed limits are obeyed
• E.g. Website with security patches
• Any combination of exception flags is possible in
  either direction for any host/network

35
Solution security considerations

• Account abuse
• Example
• User A powers off his PC without logging out
• Malicious user X takes IP of user A
• X continues to work with credentials of user A
• Solution Auto logout
• Possibility 1 BCrouter performs logout after X
  minutes of inactivity
• Possibility 2 Ping probes
• Possibility 3 DHCP server
• Login server checks if IP that wants to login has
  been issued by the DHCP server. Refuse login with
  static IP
• Use very short DHCP lease times (e.g. 15 minutes)
• Run script every few minutes that logs out
  inactive DHCP leases
• DHCP based auto-logout is preferred

36
BCrouter introduction

• Lets take a look at the core element BCrouter
• Components of BCrouter
• Commands and logging
• Routing and Netfilter setup
• Quota/Bandwidth exceptions

37
BCrouter introduction
Internal net
Internal net
Internal net
Internal net
User database
BCrouter
Login server
Log/History server
Redirect server
Internal management network
Web cache
NAT Firewall
DHCP
Internet
DNS
38
BCrouter components

• open black box
• Linux operating system
• User space
• DHCP forwarder
• Syslog daemon
• BCrouter daemon
• Network configuration script
• Kernel space
• BCpolicer module

39
BCrouter components
User space
Management interface
BCrouter daemon
DHCP forwarder
Syslog daemon
Kernel space
Kernel logging
Netfilter framework
Input interfaces
Output interfaces
40
BCrouter components

• DHCP forwarder
• Forward broadcast DHCP DISCOVER to a central DHCP
  server
• Dhcp-fwd
• http//www.nongnu.org/dhcp-fwd/
• Very simple application
• User space application running in chroot jail
• Listens in promiscuous mode on specified
  interfaces

41
BCrouter components

• Syslog daemon
• Send logs to a remote log server for remote
  processing
• Syslog-ng
• http//freshmeat.net/redir/syslog-ng/10178/url_hom
  epage/syslog-ng
• Very powerful options (filtering, multi
  logserver)
• Logs both user space as kernel logs

42
BCrouter components

• BCrouter daemon
• Provides a network-based console to the BCpolicer
  kernel module
• Simple Perl script (Forking TCP server)
• Allows simultaneous management access
• Listens on a network socket (telnet port 23)
• Communicates with the kernel module

43
BCrouter components

• Network configuration script
• Provides entire interface, routing and Netfilter
  configuration and setup
• Shell script
• Executed at boot time

44
BCrouter components

• BCpolicer kernel module
• Receives login/logout commands and performs
  accounting and routing decisions
• Core element of BCrouter (ipt_bcpolicer)
• Works entirely in kernel space
• Loadable module which implements an iptables
  target

45
BCrouter commands logging

• Commands
• Login/logout
• login username ip x.x.x.x reason text
• logout username reason text
• logout ip reason text
• Query information
• show user ip x.x.x.x
• show ip user username
• show quota ip x.x.x.x
• show quota user username
• Configuration
• conf ip
• conf user
• export all
• Miscellaneous
• show uptime

46
BCrouter commands logging

• Commands example
• bcrouter1 export all
• bcrouter1 login user kuleuven/u0022948 ip
  10.91.91.1 reason login demo
• 200 OK login - 1142031358930300 -
  kuleuven/u0022948 (1) on 10.91.91.1 (login demo)
• bcrouter1 show ip user kuleuven/u0022948
• 204 OK show ip user - 10.91.91.1
• bcrouter1 show user ip 10.91.91.1
• 203 OK show user ip - kuleuven/u0022948
• bcrouter1 login user kuleuven/u0022948 ip
  10.91.91.2 reason 2nd login
• 200 OK login - 1142031429848045 -
  kuleuven/u0022948 (1) on 10.91.91.2 (2nd login)
• bcrouter1 show ip user kuleuven/u0022948
• 204 OK show ip user - 10.91.91.2,10.91.91.1
• bcrouter1 export all
• conf user kuleuven/u0022948 .
• conf ip 10.91.91.1
• login user kuleuven/u0022948 ip 10.91.91.1 reason
  recovering statefull info
• conf ip 10.91.91.2

47
BCrouter commands logging

• Logging
• Log commands and responses
• Log network/host statistics

• network segment
• Time of log
• Log sequence number
• Name of segment
• Traffic counters
• Bytes and packets
• Download and upload
• Accounted and not accounted
• Dropped and accepted
• Number of active IPs

• host
• Time of log
• Log sequence number
• IP address
• Username (none- if no login)
• Traffic counters
• Bytes and packets
• Download and upload
• Accounted and not accounted
• Dropped and accepted

48
BCrouter routing Netfilter

• Routing with BCrouter is done by a BCPOLICER
  target in the PREROUTING mangle table that alters
  the fwmark value of the packet and uses this
  value as selector for policy based routing.

49
BCrouter routing Netfilter

• Use Linux networking capabilities
• IEEE 802.1Q support (VLAN technology)
• Used to limit the number of physical interfaces
• Policy based routing (Routing rules)
• Used for implementing user groups
• Netfilter/Iptables framework
• Used for host exception lists

50
BCrouter routing Netfilter

• IEEE 802.1Q support
• Virtual LAN technology (VLAN)
• Operates on the data link layer (OSI layer 2)
• Adds 4 extra bytes to existing ethernet header
• Allows multiple LANs over 1 physical wire
  (trunking)
• Each VLAN id has its own interface device
• E.g. eth0.5 indicates VLAN id 5 on physical
  interface eth0
• vconfig tool

51
BCrouter routing Netfilter

• Policy based routing (Routing rules)
• Multiple routing tables
• Routing policy database (RPDB)
• Each entry in the database
• contains a routing table
• has a priority number
• has a selector (src addr, dst addr, incoming
  iface, tos, fwmark)
• RPDB entries are scanned according priority
• If the selector of RPDB entry applies, that
  routing table is used
• ip rule tool

52
BCrouter routing Netfilter

• Netfilter/Iptables framework
• Extensive IP packet filtering rules
• There are 3 tables
• Each table has its own purpose
• Each table contains several chains
• Each chain operates on a different location in
  the network packet flow
• Chains can also be custom-defined
• Each chain can contain several rules
• Each rule contains a selector and a target
• Rules are examined in the order they are listed
  in the chain
• If a selector matches, the action is determined
  by the target
• iptables tool

53
BCrouter routing Netfilter

• Filter table Firewall type rules
• INPUT for packets destined for the box itself
• FORWARD for packets being routed through the box
• OUTPUT for locally generated packets
• NAT Native Address Translation type rules
• PREROUTING for altering packets as soon as they
  come in
• OUTPUT for altering locally generated packets
  before routing
• POSTROUTING for altering packets as they are
  about to go out
• Mangle Packet alteration type rules
• PREROUTING for altering packets before routing
• OUTPUT for altering locally generated packets
  before routing
• INPUT for altering packets destined for the box
  itself
• FORWARD for altering packets being routed
  through the box
• POSTROUTING for altering packets as they are
  about to go out

54
BCrouter routing Netfilter
Netfilter framework
I F A C E
I F A C E
Output Routing
Input Routing
PREROUTING
FORWARD
POSTROUTING
INPUT
OUTPUT
User space
55
BCrouter routing Netfilter

• Packet filtering and routing are normally 2
  different things
• However
• There is a packet property that can be changed
  by an iptables target and can be used as a
  routing rule selector fwmark
• Fwmark
• Netfilter MARK value
• Integer value
• Associated within the kernel with the packet
• Does not alter the packet itself

56
BCrouter routing Netfilter

• Routing with BCrouter is done by a BCPOLICER
  target in the PREROUTING mangle table that alters
  the fwmark value of the packet and uses this
  value as selector for policy based routing.
• Fwmark value after BCPOLICER selects routing
  table
• 1 97 routing table belonging to the user
  group of the identified user
• 98 routing table for non-identified users
• 99 routing table for redirecting users to the
  login site
• 100 routing table that drops all packets

57
BCrouter routing Netfilter
I F A C E
Quota/bandwidth exception list changes fwmark
before entering BCPOLICER
I F A C E
Fwmark changed by BCPOLICER to select correct
routing table
58
BCrouter exceptions

• Exceptions passed to BCPOLICER via fwmark value
• Possible to use almost all Iptables selection
  features
• List before BCPOLICER target in PREROUTING
• Fwmark value
• IP speed limit (src 0x0001 dst 0x0002)
• User speed limit (src 0x0004 dst 0x0008)
• IP accounting (src 0x0010 dst 0x0020)
• User accounting (src 0x0040 dst 0x0080)
• No login required (src 0x0100 dst0x0200)
• Final value logical OR of the wanted flags
• Example
• Traffic from Internet host B is always possible
  from any local host and is never accounted, but
  local host IP speed limits are obeyed
• Src localnet Dst hostB mark
  0x00010x0100 0x0101
• Src hostB Dst localnet mark 0x00020x0200
  0x0202

59
BCpolicer introduction

• Everything up until now was on macroscopic level
• Lets dig (a little bit) into the real core
  ipt_bcpolicer
• The next slides are greatly simplified
• Would take couple of hours to completely explain
  the entire design

60
BCpolicer introduction

• TCP/IP
• TCP reliable protocol
• Every received packet must be acknowledged to
  Sender by Receiver
• Sender retransmits when no acknowledgement
  received within certain time
• Exponential delay between retransmits
• When relatively few packets are dropped,
  transmission speed goes down significantly

61
BCpolicer introduction

• Policing vs. Shaping

Policing Accept/drop packets Dont delay packets No queues No scheduling Shaping Dont drop packets Delay packets Queuing Scheduling
62
BCpolicer design principles

• Design started by determining the specifications
  of an ideal bandwidth/quota system (with no
  user login)
• Different settings for each individual network
  segment
• For each network segment
• Independent up/download settings
• Distribute available bandwidth across active
  hosts, favoring low-traffic volume hosts a little
• Dynamically regulate maximum bandwidth per host
  to allow maximum bandwidth usage without
  overloading the segment or uplink
• For each host
• Distribute host bandwidth equally over active
  connections
• Tendency to favor interactive traffic above bulk
  traffic
• Enforce traffic quota by dynamic bandwidth speed
  regulation

63
BCpolicer design principles

• Result
• General principle leaky bucket system
• Lots of modifications and tweaks required

MeanFillRate
TokenBucket
TokenBucketSize
TokenBucketMaxSize
CurrentRate (0BurstRate)
POLICER
64
BCpolicer design principles

• Example
• Download quota 4GB/month
• Maximum download speed 50KByte/sec
• DownloadTokenBucketMaxSize
• 4GTokens
• DownloadBurstRate
• 50KTokens/sec
• DownloadMeanFillRate
• Must fill up an empty TokenBucket in 30 days
• 4GTokens/(30days24hours60minutes60seconds)
• 1657 Tokens/sec

65
BCpolicer design principles

• Policer summary
• Multi exponential stream policer
• smooth policing
• Distribute individual host bandwidth equally over
  its active connections
• Tendency to favor interactive traffic above bulk
  traffic

66
BCpolicer design principles

• Specifications list
• For each host
• Distribute host bandwidth equally over active
  connections
• OK Multi logarithmic stream policer algorithm
• Tendency to favor interactive traffic above bulk
  traffic
• OK Multi logarithmic stream policer algorithm
• Enforce traffic quota by dynamic bandwidth speed
  regulation
• OK Accurate quota control because there are no
  Tokens created or destroyed in the bucket system

67
BCpolicer design principles

• Specifications list
• Different settings for each individual network
  segment
• OK Each IP has its own buckets with its own
  settings
• For each segment
• Independent up/download settings
• OK Use separate buckets
• Distribute available bandwidth across active
  hosts, favoring low-traffic volume hosts a little
• To be done
• Dynamically regulate maximum bandwidth per host
  to allow maximum bandwidth usage without
  overloading the local segment or uplink
• To be done

68
BCpolicer design principles

• Prevent network congestion
• Implement a valve between TokenBucket and the
  Policer
• If the valve is fully open
• Maximum host speed is BurstRate
• If the valve closes
• Slow down maximum host speed
• valve control
• If no network segment congestion fully open
• Network segment congestion close valves for the
  local IPs that cause the congestion until no
  congestion anymore
• valve is named RateFactor Correction
• The state of the valve is called RateFactor

69
BCpolicer design principles
Host
BCrouter
BCpolicer
Feedback algorithm
Internet
70
BCpolicer design principles
MeanFillRate
TokenBucket
TokenBucketSize
TokenBucketMaxSize
CurrentRate (0BurstRate)
RateFactor Correction
RateFactor (e.g. lt1)
POLICER
71
BCpolicer policing alternatives

• Advantage of policing
• Simple to implement
• No queuing
• No scheduling
• Very fast
• Drawbacks of policing
• Policing overhead problem
• Relative problem
• Suppose unlimited stream would be 100Kbyte/sec
• Limit to 10Kbyte/sec
• 10 to 15 overhead -gt receiving from the Internet
  12KByte/sec
• More overhead on low latency, high bandwidth
  connections
• E.g. Limit to 10KByte/sec receiving 20KByte/sec

72
BCpolicer policing alternatives

• TCP Window Size alteration
• TCP Window Size
• Field in TCP packet header
• Message from Receiver to Sender
• Indicates to Sender how much data can be sent to
  Receiver
• Should decrease retransmission overhead
• Avoid fast retransmission on fast connections
• Alters packets passing BCrouter
• Possibly bad and unforeseen consequences?
• No need for queuing
• Integration in the BCpolicer scheme
• Use the DelayTime value to decrease the window
  size

73
BCpolicer policing alternatives

• Shaping
• Alter the Round Trip Time of a connection
• simulate slow connection
• TCP will try to adjust by slowing down
• Extra load and complexity due to delay queue
  management
• Integration in the BCpolicer scheme
• BCpolicer already works with delay times
• Instead of accepting or dropping, put packet into
  a delay queue for the delaytime

74
BCpolicer complete design

• Up until now, we only focused on the bucket
  design
• Complete system
• Addition of user buckets
• Same principle as IP buckets
• Addition of exception handling
• Makes program flow complex

75
BCpolicer main flow
Start
Src IP kotnet?
Dst IP kotnet?
No
No
Yes
Yes
Clear src streamupdate flag
Clear dst streamupdate flag
Yes (nlr)
Yes (nlr)
srcnlr1
dstnlr1
No (lr)
No (lr)
Src logged in?
Dst logged in?
No
No
Yes
Yes
Yes
srciac!0 or srcisl!0
srcnua1 and srcnus1
dstnua1 and dstnus1
dstiac!0 or dstisl!0
No
No
Yes
Yes
Yes
No
No
Src userbucket Verdict (UV1)
Src ipbucket Verdict (IV1)
Dst userbucket Verdict (UV2)
Dst ipbucket Verdict (IV2)
Src IP Bucket calculation
REDIRECT
DROP
Final Verdict Accounting Logging
Return IPT_CONTINUE
Mark translation
76
BCpolicer bucket verdict
Bucket calculations
Entrypoint
Respect Speedlimit?
No
Yes
Yes
PBSgtPTTPktSize
No
No
PBSgtPktSize
Yes
TBS-PktSizegt0
Calculate StreamII and get the correct SAPS and
SPLAT for this packet
No
Yes
QSAPS32- ((32SAPS) / MPS) QDelayF64- ((64PBS)
/ PTT) PDTDelayMatrix(QSAPS,QDelayF)
VerdictDROP
VerdictACCEPT
No
SPLATPDTltPacketTS
Yes
Store StreamII and new SAPS (SAPSnew) Set
streamupdate flag for the current ip (used for
accounting)
VerdictACCEPT
VerdictDROP
VerdictACCEPT
Return Verdict
77
BCpolicer bucket calculation
Entrypoint
TB_ADD_INTERVAL passed since the last call?
Yes
Calculate new TBS TBSmax( TBSTBMFRTBLAT,TBMS)
No
Calc max CurrentR EffectiveRmax CurrentR
No
PBSEffectiveRltPBMS
Yes
Calc max EffectiveR CurrentR max EffectiveR
(CurrentR EffectiveR) TBSTBS-CurrentR
FactorR0
PBSPBSEffectiveR
Endpoint
78
BCpolicer f.verdictacclog
Entrypoint
FinalVerdictDROP
FinalVerdictACCEPT
FinalVerdictREDIR
If PktTimegtSITLLLOGSI Print Src IP logline
clear printed counters
If PktTimegtSSTLLLOGSS Print Src Segment logline
clear printed counters
SOURCE IP Accounting Calculation
FinalVerdict REDIR?
No
Yes
If PktTimegtSITLLLOGSI Print Dst IP logline
clear printed counters
If PktTimegtSSTLLLOGSS Print Dst Segment logline
clear printed counters
DESTINATION IP Accounting Calculation
Return FinalVerdict
79
BCpolicer accounting
Entrypoint
Yes
SSTAB PktSize SSTAP SSTAI if needed
streamupdate1
No
SAPS at SIISAPSnew
UsrTBS PktSize
SSTDB PktSize SSTDP SSTDI if
needed SSTAI if needed
SINFAB PktSize SINFAP
SIFAB PktSize SIFAP
UsrTBS - PktSize
UsrPBS - PktSize
SIFDB PktSize SIFDP
SINFDB PktSize SINFDP
No (no iac)
iac1
Yes (iac)
IpTBS PktSize
No (no isl)
Yes (isl)
isl1
IpTBS - PktSize
IpPBS - PktSize
Return
80
Case study KotNet

• Goals
• Connect K.U.Leuven association students and
  personnel to the campus network and Internet from
  their homes
• Enhance possibility of study and research in an
  academic environment
• Low entrance fee and costs
• KotNet is an enabler for the E-university

81
Case study KotNet

• History
• 1994 Started by students in their student home
• 1996-97 pilot test projects
• 1997-98 en 1998-99
• 1, 2 operational phase K.U.Leuven residences
• access via cable TV network
• test project in 1997-98
• operational since Sept. 1998 (2 cable TV
  operators)
• 2002 Wireless KotNet at public places
• Test project in student restaurants
• Gradually extended to libraries, meeting rooms,
  classrooms
• 2004 Agreements with associated partners

82
Case study KotNet

• Providers
• K.U.Leuven direct fiber to university
  residences
• K.U.Leuven wifi hotspots in student
  restaurants, auditoria, ...
• UPC Belgium via cable modem
• Telenet/Iverlek via cable modem

83
Case study KotNet

• Technology
• Layer 2
• VLAN's are used to implement city-wide networks
• connect remote buildings to centralized KotNet
  routers
• spanning-tree for automatic redundancy
• Layer 3
• use of private address space (10...)
• split data streams
• traffic to K.U.Leuven networks is routed through
  the central firewall
• HTTP traffic is redirected to a web cache cluster
• other traffic is handled by a Network Address
  Translator cluster

84
Case study KotNet

• Policies
• Users will consume mega-amounts of bandwidth if
  you let them. Not all of their traffic is of an
  educational nature...
• implementation of policies and enforcement of
  these policies is absolutely essential
• p2p traffic is not allowed (Gnutella, Kazaa,
  eDonkey, ...)
• enforced by iptables filters on the NAT servers
• individual servers are not allowed
• enforced by a 200 Mbyte/day upload limit
• exceptions are possible for registered servers
• Internet bandwidth is not free
• enforced by a 4 Gbyte/month/USER download limit

85
Case study KotNet

• Residences
• Approx 50 buildings
• Approx 4000 connections
• Spread geographically across the city of Leuven
• K.U.Leuven owned fiber network
• Each building is a different VLAN
• Ethernet network
• Old
• 10Mbit HUB
• 10Mbit Half-duplex fiber uplink to KotNet
  backbone
• New
• 100Mbit Switch (Cisco 2950)
• 100Mbit Full-duplex fiber uplink to KotNet
  backbone

86
Case study KotNet (residences)
Old Network
New Network
10Mbit HUB
100Mbit Switch
10Mbit FOT
KotNet Backbone Network
10Mbit FOT
Aggregation Switch
Gigabit 802.1Q VLAN trunk
87
Case study KotNet

• Cable provider Telenet
• Coverage across the region of Leuven
• Approx 1500 cable modems
• Approx 5000 users
• 3 Cable segments
• 13 Mbit downstream capacity each
• 3 Mbit upstream capacity each
• Single VLAN (flat network)

88
Case study KotNet (Telenet)
Cable segment
Cable Router (L3)
100Mbit Flat network
KotNet Backbone
100Mbit FOT
100Mbit FOT
Telenet location
89
Case study KotNet

• Cable provider UPC
• Coverage across the region of Leuven and Brussels
• Approx 5100 cable modems
• Approx 15000 users
• 52 Cable segments
• 50 Mbit downstream capacity each
• 3 10 Mbit upstream capacity each
• Each cable segment is a different VLAN

90
Case study KotNet (UPC)
52 Cable segments
Cable Head end (L2)
KotNet Backbone
UPC location
Gigabit 802.1Q VLAN trunk
91
Case study KotNet

• KotNet Backbone
• Gigabit Ethernet based
• Redundant where possible
• spanning-tree protocol
• Separated from K.U.Leuven backbone
• Spread geographically across the city of Leuven

92
Case study KotNet
Gigabit 802.1Q VLAN trunk containing 105 KotNet
VLANs

• KotNet Backbone
• Residences
• Telenet
• UPC

Flow logger
Monitor switch
Gigabit mirror port
Redirect
KUL-KotNet Firewall
NAT K.U.Leuven
Web cache
NAT Association
KULeuven network
Internet
KUL-INET Firewall
93
Case study KotNet

• Pre-BCrouter
• 2 High-end Cisco 7500 routers
• 30.000 EUR each
• Couldnt handle large network segments
• Long user access lists requiring lots of CPU
  power
• Policy based routing not really flexible
• Logout operation very expensive (ACL reload)
• Inflexible shaping
• Not possible per ip/user/stream/usergroup and
  certainly not everything together
• Need for more scalable solution!!

94
Case study KotNet

• BCrouter
• Replaces both Cisco routers
• Hardware
• Dell Poweredge 2650
• Dual Intel(R) Xeon(TM) CPU 3.20GHz with
  Hyperthreading
• ServerWorks GC-LE chipset, 400MHz front side bus,
  21 memory interleaving, 5 PCI buses (three of
  which are are PCI-X capable)
• 1 Gig ECC ram
• Two Intel E1000 PCI-X 133Mhz Network interfaces
• 3000 EUR
• Extremely powerful and flexible configuration

95
Case study KotNet

• BCrouter settings
• Only 1 simultaneous login allowed by login system
  
• Normally login required, obey user/IP
  quota/speed
• User buckets
• Download quota of 4Gbyte/month
• Max speed (BurstRate) none
• Upload quota of 200Mbyte/day
• Max speed (BurstRate) none
• IP buckets
• Download quota of 8Gbyte/month
• Max speed (BurstRate) 250Kbyte/sec
• Upload quota of 8Gbyte/month
• Max speed (BurstRate) 250Kbyte/sec

96
Case study KotNet

• BCrouter exceptions
• DHCP, DNS server
• No login required, no user acc/bw, ip acc/bw
• Login website reachable
• No login required, no user acc/bw, ip acc/bw
• Local antivirus download website reachable
• No login required, no user acc/bw, ip acc/bw
• Local FTP server reachable
• No login required, no user acc/bw, ip acc/bw
• Unblock website reachable
• No login required, no user acc/bw, ip acc/bw
• E-learning website reachable
• Login required, no user acc, user bw, ip acc/bw

97
Case study KotNet

• User blocking system (admin view)
• Receives information from number of probes
• Email system
• Flowlogger
• Local honeypots
• Gets an IP address
• Looks up the corresponding user
• Logout user
• Put user in a blacklist to prevent user login
• Send email to the user

98
Case study KotNet

• User blocking system (user view)
• User is logged out
• Receives email containing instructions
• Webmail is always reachable
• Local antivirus download site also reachable
• If the user tries to login
• Message his account is blocked
• More detailed instructions about the blocking
  reason
• Link to unblock website
• User unblocks himself
• Only allowed 1 time each day

99
Case study KotNet

• Flow Logger
• Emits Cisco Netflow data to logging system
• Very detailed traffic information (for each
  stream)
• Start and stop time
• Source and destination IP
• Source and destination port
• Number of bytes transmitted/received
• Number of packets transmitted/received
• High performance Netflow probe
• nProbe (http//www.ntop.org)
• Used to detect network anomalies
• Host scans and port scans
• Coupled to automatic user blocking system

100
Case study KotNet

• Email
• No direct SMTP connections to the internet
• All emails must pass the Central Anti Virus
  cluster
• Scans for viruses (Mcafee Antivirus)
• Performs spam classification (Spamassassin)
• If a virus is detected, user is blocked
• User blocking system looks at sending IP

101
Case study KotNet

• Transparent caching
• No user settings required to use caching
• Reduce the traffic on the Internet link
• Cisco Content Engine
• Squid based
• WCCP
• transparent redundancy
• Needed a quick solution

102
Case study KotNet

• Firewalling
• Done on BCrouter
• Beginning of PREROUTING mangle table
• TCP port 135,139,445 are blocked for anyone
• Prevent virus outbreaks

103
Case study KotNet

• P2P blocking
• Done on NAT cluster
• Do a pattern search in every packet
• Look for certain P2P strings
• Kazaa, Bittorrent, Edonkey
• Iptables string module
• Rather CPU expensive
• Planning to make a better version someday ?

104
Case study KotNet

• Transition to BCrouter
• Phased transition
• Created 2 testing KotNet segments on BCrouter
• Test connectivity and performance
• Modify login scripts and backend for BCrouter
• Convert 2 real KotNet segments to BCrouter
• Convert all residences to BCrouter
• Convert all UPC to BCrouter
• Convert all Telenet to BCrouter
• Parallel volume accounting on user webpage
• Real-time counters from BCrouter
• History information by previous KotNet accounting
  system
• No real problems encountered

105
Case study KotNet

• Traffic impact (early testing)
• User mode test version of BCpolicer
• No HTTP traffic
• Most popular P2P applications blocked
• Only used IP addresses for accounting
• PolicerBucketMaxSize 150 Mbyte
• MeanFillRate 50 Kbyte/sec
• BurstRate 250 Kbyte/sec
• Results
• 50 reduction in traffic
• Only 5 of users hit BCpolicer
• Users noted improvement in network quality

106
Case study KotNet

• Some Numbers
• Current daily numbers
• 400 Mbit/sec throughput total (18u 23u)
• 150 Mbit Non-http
• 30 Mbit K.U.Leuven
• 80 Kpackets/sec
• Up to 20 000 simultaneous users
• BCrouter load
• 33 CPU at peak

107
Case study KotNet

• Management utilities
• The next few slides show the web-based
  information interface

108
Case study KotNet
109
Case study KotNet
110
Case study KotNet
111
Case study KotNet
112
Case study KotNet
113
Development current status

• BCrouter itself is functioning
• Design is done by specifications
• Implementation of ipt_bcpolicer is not yet 100
  ready
• Dynamic bandwidth regulation by congestion is not
  yet implemented (Fixed RateFactor of 1)
• Login system is not self-contained
• Currently uses the previous login system from
  KotNet

114
Development future

• Redundancy
• Timeframe June 2006
• Hot-standby backup system which takes over if
  primary fails
• Further development needed
• Build a stand-alone login/log/history system
• Timeframe September 2006
• Easy custom layout
• Minimal dependencies
• Already have some proof of concept code
• Can be done independently of further development
• Login system without history information can be
  done very quickly
• Planning hire a job student for this during the
  summer

115
Development future

• Black Box solution
• Timeframe Jan 2007?
• Depends on stand-alone login system
• Packaging and generalizing scripts
• Packaging of BCrouter partly done
• TCP window size policing
• Timeframe unknown
• Still requires research
• Planning make a thesis out of this
• FactorRate Correction
• Timeframe unknown
• Not an issue anymore on KotNet
• Implementation and testing

116
Development wish list

• Transform the policer into a shaper
• Probably more work than expected
• Planning make a thesis out of this someday
• Add the automatic blocking features
• Currently uses the previous KotNet solution

117
Conclusion

• BCrouter concept offers great possibilities
• User authentication
• Advanced and flexible quota and bandwidth
  management
• High performance and scalable solution
• BCrouter still requires a lot of implementation
• Concepts are already tested in the field

118

• THE END
• Contact information
• Dirk Janssens
• ICTS K.U.Leuven
• bcrouter_at_kuleuven.net