Network Quotas for Individuals A better answer to the P2P bandwidth problem

1
Network Quotas for Individuals A better answer
to the P2P bandwidth problem?

• Bruce Curtis
• North Dakota State University

2
Currently Testing New Usage Based Rate Limit
System

• On April 2nd NDSU started testing a new (for us)
  system for limiting bandwidth usage in our
  residence halls.
• Why?
• First a bit of history to explain how we arrived
  at this point

3
First There Came Napster

• Napster was the first P2P app that caused us
  major problems. Downstream from us the GPN
  internet 1 link that we shared was filled with
  outbound traffic. This had a noticeable affect
  on our users web access.

4
Our Response to Napster

• We didnt have funds to increase our bandwidth so
  we tested blocking access to the central Napster
  servers.
• This had the desired affect on network usage but
  we werent real happy with this drastic solution.

5
University of Waterloo

• We heard about a system at the University of
  Waterloo that turned off the port of a student
  after they had used their network allotment.
• In discussions around the office this was the
  most attractive method we had heard of, although
  turning off the port was a bit drastic.

6
University of Waterloo

• But alas, the U of Waterloo solution was based on
  counters on switch ports.
• We only had hand-me-down hubs in our residence
  halls.
• Later I found that wasnt the full U of Waterloo
  story more on that later

7
CAR Commited Access Rate

• Still looking for a better option than blocking
  we learned that our hardware could support CAR.
• We started experimenting with rate limiting the
  new apps by port, although we just left Napster
  blocked.

8
P2P Apps Multiplied

• Our CAR configuration evolved to rate limit every
  new mis-behaved app as we called them.
• And there were a lot of them Kazaa, Gnutella,
  Scour, iMesh, eDonkey, Hotline, Morpheus,
  DirectConnect, AudioGalaxy
• And a few games AfterLife, DirectPlay

9
P2P Apps More Challenging to Rate Limit

• Some of the P2P apps like AudioGalaxy use a range
  of ports instead of a single port and are much
  more challenging to limit without affecting
  things like passive ftp. (Passive ftp is used by
  default when web browsers are used to ftp files).

10
University of Texas

• We heard that the University of Texas was using a
  system similar to the U of Waterloo.
• That was looking more and more attractive after
  tweaking the AudioGalaxy rate limits to not
  affect other apps.
• But alas, the U of Texas system was also based on
  intelligent switches.

11
Students Starting to Catch On

• Our rate limits were fairly draconian, we shared
  internet access with other Universities that
  hadnt implemented rate limits yet and tended to
  compensate by keeping our P2P traffic very low.
• So our students were motivated and a few of them
  were starting to catch on that some P2P apps had
  configurable ports

12
Usage-Pattern-Adaptive Rate Limiting

• At the January NLANR/i2 Joint Techs Charley Kline
  presented the talk Usage-Pattern-Adaptive Rate
  Limiting
• Charleys system was based on Netflow data to
  determine usage and CAR for rate limiting.
• Finally! This looked like a system that would
  work on our network.

13
So Close

• I emailed Charley a couple of times but he had
  multiple jobs and while he worked for the
  University of Illinois at Urbana-Champaign he
  actually developed his code for McLeod.
• McLeod didnt give immediate approval to release
  the code and after their initial reluctance I
  didnt pursue further.

14
Flowscan From Dave Plonka

• In the fall of 2001 I ended up modifying flowscan
  to help us track the usage of the 11
  Universities in North Dakota for potential
  billing purposes.
• So I put in a couple of lines of code to save the
  per IP usage info from the NDSU res halls in a
  separate file.

15
The Pieces Fall in Place

• Later I wrote some perl scripts to take the usage
  info and put CAR rate limits in the res hall
  router and update the access lists every 5
  minutes.
• Charleys system was implemented with MySQL, C,
  perl and php.
• But I just used perl hashes and a couple of flat
  files. Less than 400 lines of perl code written.
  
• Flowscan and the Cisco perl module do the heavy
  lifting.

16
Back to Why?

• Looking back I guess that since day one I was
  motivated by the idea of personal
  accountability. I liked the principle that a
  users actions should affect them more directly,
  rather than have the pain of network slowness be
  spread to the whole ResNet community.

17
More Why?

• With this system we dont have to decide which
  apps are mis-behaved.
• We dont have to keep updating access-lists or
  wait for vendors to update code to recognize the
  latest P2P or other app.
• It may just be a matter of time before P2P apps
  start using random ports and/or encryption.

18
Enough History,Finally Some Details

• OK, here is how we explain this to our students

19
(No Transcript)
20
ResNet Web Page

• New ResNet Network Policy2002-04-01
• ITS and Residence Life are working together to
  help NDSU control its Internet costs. ITS is
  evaluating a new management method which
  allocates 600 MB of the network usage to each
  resident daily, the average ResNet users uses
  about 100MB a day.

21

• This will apply only to traffic going to or
  coming from the Internet. On campus traffic, like
  reading your Mail_at_NDSU e-mail or using
  Blackboard, will not be counted nor will usage
  between 200 AM and 600 AM. The bandwidth usage
  will be reset daily at 600 AM.
• If a resident exceeds their daily allocation,
  they will be placed into a shared, limited pool
  with other users that have exceeded their
  allocation. They will not be disconnected they
  will be sharing 300 kbps, which may be very slow
  for the remainder of the day.

22
Things to Watch Out For

• If you are using an Internet application that
  allows you to search for and download files on
  the Internet, this same application may be
  allowing others on the Internet to download your
  files. The files you download and the files that
  are downloaded from you will be counted against
  your quota.
• If you have a 3MB file and 200 people on the
  Internet download it, this will require 600MB of
  network usage.

23
The Network
24
Information Flows
25
i1 and i2 traffic flows
26
Implementation Details

• Track usage by ethernet number
• Add to CAR access lists every 5 minutes
• Rebuild CAR access lists from scratch hourly to
  avoid limiting someone who inherited via DHCP an
  IP number that is in the access lists.

27
Changing Ethernet or IP

• Ethernet numbers in ResNet must be registered.
  We use the NetReg package from Southwestern U.
  Each ethernet number must be registered before it
  can access the Internet.
• Changing the IP number only helps for 5 minutes
  until the access lists are updated from the usage
  info and arp table.
• Cross checks and countermeasures could be set up
  monitor when these things are tried.

28
Router Impact

• CPU usage about the same as with old system.
• Router has been up for 1.5 years.

29
Two Important Numbers

• Quota or daily allotment
• Bandwidth of limited pools.

30
How to Pick the Numbers

• Our method of choice was successive
  approximation. (Fancy for make a guess and
  adjust using trial and error.)

31
Limiting Issues

• If the quota is too high we use too much
  bandwidth.
• If the quota is too low we rate limit too many
  people and the access lists could grow large.
• One initial goal is to pick target quota that
  affects only 10 of users.

32
Limiting Issues

• Our bandwidth target was 15 Mbps so we chose the
  limited pool size to be 10 of that which is 1.5
  Mbps.
• We have 5 subnets.
• So 1.5 Mbps / 5 pools 300 kbps per pool
• In other words we sacrifice 1.5 Mbps to
  discourteous users taking more than their share
  of bandwidth.
• If successful the quota should affect 10 of the
  users and restrict them to using only 10 of the
  bandwidth. (After reaching their quota)

33
Picking the Numbers

• We postulated a bandwidth hungry but legitimate
  educational activity for an estimated quota.
• A video class offered via H.323 for an hour would
  take 384 Kbps some overhead.
• 500 Kbps 60 s/m 60 m/hr / 8 bits/byte 225
  MB
• We count both inbound and outbound data so 225
  2 450 MB
• Allowing for other daily traffic we picked 600 MB
  as our initial quota for the test.

34
Comparison of Inbound Traffic
Spikes
Spring Break
New rate limit trial started.
35
Comparison of Outbound Traffic
Spike represents demand when no rate limits are
in place.
36
We provided the students more bandwidth and
remained within our limits

Spring Break
Level with out any rate limit.
New rate limit trial started.
37
Percentage is Near Initial Goal of 10
Percent of people who have reached the quota
rises through the day.
38
Before
After
More P2P traffic as expected.
39
Slope of Kazaa traffic declines as more people
reach their quota
Before
After
40
(No Transcript)
41
68
31
42
13
61
43
Before
Graph appears fatter at the bottom due perhaps
to outbound AudioGalaxy traffic (also from scale
change)
There are about 500 AudioGalaxy users
Put in flowscan out graph and mention that about
500 users Are running audiogalaxy. But first
put in before and after outbound graphs.
44
(No Transcript)
45
Data Examples From Others

• From the U Texas resnet web page
• Upon investigating, we found that 2 of ResNet
  users were consuming over 50 of the available
  bandwidth and 0.2 were using 15

46
Data Examples From Others

• Charley Kline
• UIUC dorm network
• Top 2 generates 22
• Top 10 generates 58

47
Data Examples From Others

• Charley Kline
• McLeodUSA Flexabit network
• Top 2 generates 33
• Top 12 generates 95

48
Problem Huge Nightly Outbound Traffic

Should have anticipated high outbound usage at
night with no limits in effect. Inbound is
bounded by the number of students in the
residence halls. Outbound is bounded by the
number of people on the Internet!
49
Problem - Spikes
Tested nightly limits. Nice slope down as more
people reach quota. But large spike as quotas are
reset to 0 at 6 am. Charley Kline defined quota
over the last 24 hours. We defined defined
definite starting points for quota computation.
50
Probation

• Implemented the concept of probation.
• If a student was over their quota yesterday their
  traffic will be limited. But the probation rate
  limits will be 5 times the quota rate limits.
• Probation works well and keeps chronic offenders
  under control which avoids reset spikes.
• But it is complicated to explain to students.

51
Quota reset spike
Smaller spikes
No spike
Transition to yesterdays quota scheme, kazaa
spikes on left, not on right
52
Closeup of Kazaa Spikes
7 Mbps 60 s/m 15 m / 8 bits/byte 787.5
Mbytes Someone reached their quota in 15
minutes. Probation tends to smooth the spikes to
be wider and less tall.
53
Stealth Mode Deployment in Plain Sight

• No announcement to students.
• Provided info to Residence Life and RAs.
• Provided web site with explanation and usage
  meter.
• No calls to help desk in the first week.
• Do you call the cable company if you are getting
  free cable?

54
Earlier Similar Systems

• University of Waterloo
• University of Texas at Austin
• Charley Kline
• Utah folks (mentioned at Jan 02 NLANR)

55
University of Waterloo

• In preparing for this presentation I found that
  the University of Waterloo progressed quickly
  from the method of blocking traffic that I had
  heard of initially back when Napster was young.
• They moved to rate limiting users instead of
  blocking them when their quota was exceeded.
• The timeline is well documented on their web
  site.

56
Tunnels

• The University of Waterloo evolved to a similar
  situation in which on-campus traffic did not
  count against a students quota.
• They had a problem with tunnels. Students would
  set up a tunnel to a computer on the main campus
  and thus access the internet without affecting
  their quota.

57
Tunnels

• I dont believe that we will have the same
  problem with tunnels. Our previous method was
  more restrictive and we did not have a problem
  with tunnels.
• A large bump in P2P traffic from our main campus
  would be easily spotted in our flowscan graphs.

58
Links to Info on the Other Similar Systems

• University of Waterloo
• http//ist.uwaterloo.ca/cn/Residence/history.html
• http//ist.uwaterloo.ca/cn/Residence/rn-excess.htm
  l
• http//ist.uwaterloo.ca/cn/Residence/logic.html
• University of Texas
• http//resnet.utexas.edu/
• Charley Kline
• http//www.ncne.nlanr.net/training/techs/2001/0128
  /presentations/200101-kline1_files/frame.htm

59
Thanks

• Brian Asker Our student who developed the web
  pages with the quota meter on them.
• John Underwood Our Help Desk and ResNet Manager
  who provided valuable input, feedback and I stole
  the free cable joke from him.

60
bruce.curtis_at_ndsu.nodak.edu

• In email Marty Hoag mentioned that we should tell
  the students
• The good news is that we are no longer filtering
  on content. The bad news is that you are
  accountable for your usage. -)