Traffic Engineering for ISP Networks

1
Traffic Engineering for ISP Networks

• Jennifer Rexford
• IP Network Management and Performance
• ATT Labs - Research Florham Park, NJ
• http//www.research.att.com/jrex

2
Outline

• Internet routing protocols
• Traffic engineering using traditional protocols
• Optimizing network configuration to prevailing
  traffic
• Requirements for topology, routing, and traffic
  info
• Traffic demands
• Volume of load between edges of the network
• Technique for measuring the traffic demands
• Route optimization
• Tuning the link weights to the offered traffic
• Incorporating various operational constraints
• Conclusions

3
Autonomous Systems (ASes)

• Internet divided into ASes
• Distinct regions of administrative control
  (14,000)
• Routers and links managed by a single institution
• Service provider, company, university,
• Hierarchy of ASes
• Large, tier-1 provider with a nationwide backbone
• Medium-sized regional provider with smaller
  backbone
• Small network run by a single company or
  university
• Interaction between ASes
• Internal topology is not shared between ASes
• but, neighbors interact to coordinate routing

4
Path Traversing Multiple ASes
Path 6, 5, 4, 3, 2, 1
4
3
5
2
6
7
1
Web server
Client
5
Interdomain Routing Border Gateway Protocol

• ASes exchange info about who they can reach
• IP prefix block of destination IP addresses
• AS path sequence of ASes along the path
• Policies configured by the ASs network operator
• Path selection which of the paths to use?
• Path export which neighbors to tell?

Client (12.34.158.5)
6
Intradomain Routing OSPF or IS-IS

• Shortest path routing based on link weights
• Routers flood the link-state information to each
  other
• Routers compute the next hop to reach other
  routers
• Weights configured by the ASs network operator
• Simple heuristics link capacity or physical
  distance
• Traffic engineering tuning the link weights to
  the traffic

7
Motivating Problem Congested Link

• Detecting that a link is congested
• Utilization statistics reported every five
  minutes
• Sample probe traffic suffers degraded performance
• Customers complain (via the telephone network?)
• Reasons why the link might be congested
• Increase in demand between some set of src-dest
  pairs
• Failed router/link within the AS causes routing
  change
• Failure/reconfiguration in another AS changes
  routes
• Challenges
• Know the cause, not just the manifestations
• Predict the effects of possible changes to link
  weights

8
Traffic Engineering in an ISP Backbone

• Topology of the ISP backbone
• Connectivity and capacity of routers and links
• Traffic demands
• Offered load between points in the network
• Routing configuration
• Link weights for selecting paths
• Performance objective
• Balanced load, low latency,
• Question Given the topology and traffic demands
  in an IP network, what link weights should be
  used?

9
Modeling Traffic Demands

• Volume of traffic V(s,d,t)
• From a particular source s
• To a particular destination d
• Over a particular time period t
• Time period
• Performance debugging -- minutes or tens of
  minutes
• Time-of-day traffic engineering -- hours or days
• Network design -- days to weeks
• Sources and destinations
• Individual hosts -- interesting, but huge!
• Individual prefixes -- still big not seen by any
  one AS!
• Individual edge links in an ISP backbone --
  hmmm.

10
Traffic Matrix
Traffic matrix V(in,out,t) for all pairs (in,out)
in
out
11
Problem Hot Potato Routing

• ISP is in the middle of the Internet
• Multiple connections to multiple other ASes
• Egress point depends on intradomain routing
• Problem with point-to-point models
• Want to predict impact of changing intradomain
  routing
• But, a change in weights may change the egress
  point!

12
Traffic Demand Multiple Egress Points

• Definition V(in, out, t)
• Entry link (in)
• Set of possible egress links (out)
• Time period (t)
• Volume of traffic (V(in,out,t))
• Computing the traffic demands
• Measure the traffic where it enters the ISP
  backbone
• Identify the set of egress links where traffic
  could leave
• Sum over all traffic with same in, out, and t

13
Traffic Mapping Ingress Measurement

• Packet measurement (e.g., Netflow, sampling)
• Ingress point i
• Destination prefix d
• Traffic volume Vid

destination
ingress
d
i
14
Traffic Mapping Egress Point(s)

• Routing data (e.g., forwarding tables)
• Destination prefix d
• Set of egress points ed

destination
d
15
Traffic Mapping Combining the Data

• Combining multiple types of data
• Traffic Vid (ingress i, destination prefix d)
• Routing ed (set ed of egress links toward d)
• Combining sum over Vid with same ed

ingress
egress set
i
16
Application on the ATT Backbone

• Measurement data
• Netflow data (ingress traffic)
• Forwarding tables (sets of egress points)
• Configuration files (topology and link weights)
• Effectiveness
• Ingress traffic could be matched with egress sets
• Simulated flow of traffic consistent with link
  loads
• Challenges
• Loss of Netflow records during delivery (can
  correct for it!)
• Egress set changes between table dumps (not very
  many)
• Topology changes between configuration dumps
  (just one!)

17
Traffic Analysis Results

• Small number of demands contribute most traffic
• Optimize routes for just the heavy hitters
• Measure a small fraction of the traffic
• Must watch out for changes in load and set of
  exit links
• Time-of-day fluctuations
• Reoptimize routes a few times a day (three?)
• Traffic (in)stability
• Select routes that are good for different demand
  sets
• Reoptimize routes after sudden changes in load

18
Three Traffic Demands in San Francisco
19
Underpinnings of the Optimization

• Route prediction engine (what-if tool)
• Model the influence of link weights on traffic
  flow
• Select a closest exit point based on link weights
• Compute shortest path(s) based on link weights
• Capture splitting over multiple shortest paths
• Sum the traffic volume traversing each link
• Objective function
• Rate the goodness of a setting of the link
  weights
• E.g., max link utilization or sum of
  exp(utilization)

20
Weight Optimization

• Local search
• Generate a candidate setting of the weights
• Predict the resulting load on the network links
• Compute the value of the objective function
• Repeat, and select solution with lowest objective
  function
• Efficient computation
• Explore the neighborhood around good solutions
• Exploit efficient incremental graph algorithms
• Performance results on ATTs network
• Much better than simple heuristics (link
  capacity, distance)
• Quite competitive with multi-commodity flow
  solution

21
Incorporating Operational Realities

• Minimize changes to the network
• Changing just one or two link weights is often
  enough
• Tolerate failure of network equipment
• Weights settings usually remain good after
  failure
• or can be fixed by changing one or two weights
• Limit the number of distinct weight values
• Small number of integer values is sufficient
• Limit dependence on accuracy of traffic demands
• Good weights remain good after introducing random
  noise
• Limit frequency of changes to the weights
• Joint optimization for day and night traffic
  matrices

22
Conclusions

• Our approach
• Measure network-wide view of traffic and routing
• Model data representations and what-if tools
• Control intelligent changes to operational
  network
• Application in ATTs network
• Capacity planning
• Customer acquisition
• Preparing for maintenance activities
• Comparing different routing protocols
• Ongoing work interdomain traffic engineering

23
To Learn More

• Overview papers
• Traffic engineering for IP networks
  (http//www.research.att.com/jrex/papers/ieeenet
  00.ps)
• Traffic engineering with traditional IP routing
  protocols(http//www.research.att.com/jrex/pape
  rs/ieeecomm02.ps)
• Traffic measurement
• "Measurement and analysis of IP network usage and
  behavior(http//www.research.att.com/jrex/paper
  s/ieeecomm00.ps)
• Deriving traffic demands for operational IP
  networks(http//www.research.att.com/jrex/paper
  s/ton01.ps)
• Topology and configuration
• IP network configuration for intradomain traffic
  engineering (http//www.research.att.com/jrex/pa
  pers/ieeenet01.ps)
• Intradomain route optimization
• Internet traffic engineering by optimizing OSPF
  weights(http//www.ieee-infocom.org/2000/papers/
  165.ps)
• Optimizing OSPF/IS-IS weights in a changing
  world(http//www.research.att.com/mthorup/PAPER
  S/change_ospf.ps)