Traffic Engineering for ISP Networks

1
Traffic Engineering for ISP Networks

• Jennifer Rexford
• IP Network Management and Performance
• ATT Labs - Research Florham Park, NJ
• http//www.research.att.com/jrex

2
Outline

• Internet routing protocols
• Traffic engineering using traditional protocols
• Optimizing network configuration to prevailing
  traffic
• Requirements for topology, routing, and traffic
  info
• Traffic demands
• Volume of load between edges of the network
• Technique for measuring the traffic demands
• Route optimization
• Tuning the link weights to the offered traffic
• Incorporating various operational constraints
• Other ways of measuring the traffic matrix
• Conclusions

3
Autonomous Systems (ASes)

• Internet divided into ASes
• Distinct regions of administrative control
  (14,000)
• Routers and links managed by a single institution
• Service provider, company, university,
• Hierarchy of ASes
• Large, tier-1 provider with a nationwide backbone
• Medium-sized regional provider with smaller
  backbone
• Small network run by a single company or
  university
• Interaction between ASes
• Internal topology is not shared between ASes
• but, neighbors interact to coordinate routing

4
Path Traversing Multiple ASes
Path 6, 5, 4, 3, 2, 1
4
3
5
2
6
7
1
Web server
Client
5
Interdomain Routing Border Gateway Protocol

• ASes exchange info about who they can reach
• IP prefix block of destination IP addresses
• AS path sequence of ASes along the path
• Policies configured by the ASs network operator
• Path selection which of the paths to use?
• Path export which neighbors to tell?

Client (12.34.158.5)
6
Intradomain Routing OSPF or IS-IS

• Shortest path routing based on link weights
• Routers flood the link-state information to each
  other
• Routers compute the next hop to reach other
  routers
• Weights configured by the ASs network operator
• Simple heuristics link capacity or physical
  distance
• Traffic engineering tuning the link weights to
  the traffic

7
Motivating Problem Congested Link

• Detecting that a link is congested
• Utilization statistics reported every five
  minutes
• Sample probe traffic suffers degraded performance
• Customers complain (via the telephone network?)
• Reasons why the link might be congested
• Increase in demand between some set of src-dest
  pairs
• Failed router/link within the AS causes routing
  change
• Failure/reconfiguration in another AS changes
  routes
• Challenges
• Know the cause, not just the manifestations
• Predict the effects of possible changes to link
  weights

8
Traffic Engineering in an ISP Backbone

• Topology of the ISP backbone
• Connectivity and capacity of routers and links
• Traffic demands
• Offered load between points in the network
• Routing configuration
• Link weights for selecting paths
• Performance objective
• Balanced load, low latency,
• Question Given the topology and traffic demands
  in an IP network, what link weights should be
  used?

9
Modeling Traffic Demands

• Volume of traffic V(s,d,t)
• From a particular source s
• To a particular destination d
• Over a particular time period t
• Time period
• Performance debugging -- minutes or tens of
  minutes
• Time-of-day traffic engineering -- hours or days
• Network design -- days to weeks
• Sources and destinations
• Individual hosts -- interesting, but huge!
• Individual prefixes -- still big not seen by any
  one AS!
• Individual edge links in an ISP backbone --
  hmmm.

10
Traffic Matrix
Traffic matrix V(in,out,t) for all pairs (in,out)
in
out
11
Problem Hot Potato Routing

• ISP is in the middle of the Internet
• Multiple connections to multiple other ASes
• Egress point depends on intradomain routing
• Problem with point-to-point models
• Want to predict impact of changing intradomain
  routing
• But, a change in weights may change the egress
  point!

12
Demand Matrix Motivating Example
Big Internet
User Site
Web Site
13
Coupling of Inter and Intradomain Routing
AS 2
Web Site
User Site
U
AS 3
AS 1
AS 4, AS 3, U
AS 4
14
Intradomain Routing Hot Potato
Zoom in on AS1
OUT 1
25
110
110
300
200
75
300
OUT 2
10
110
110
IN
OUT 3
Hot-potato routing change in internal routing
(link weights) configuration changes flow exit
point!
15
Traffic Demand Multiple Egress Points

• Definition V(in, out, t)
• Entry link (in)
• Set of possible egress links (out)
• Time period (t)
• Volume of traffic (V(in,out,t))
• Computing the traffic demands
• Measure the traffic where it enters the ISP
  backbone
• Identify the set of egress links where traffic
  could leave
• Sum over all traffic with same in, out, and t

16
Traffic Mapping Ingress Measurement

• Packet measurement (e.g., Netflow, sampling)
• Ingress point i
• Destination prefix d
• Traffic volume Vid

destination
ingress
d
i
17
Traffic Mapping Egress Point(s)

• Routing data (e.g., forwarding tables)
• Destination prefix d
• Set of egress points ed

destination
d
18
Traffic Mapping Combining the Data

• Combining multiple types of data
• Traffic Vid (ingress i, destination prefix d)
• Routing ed (set ed of egress links toward d)
• Combining sum over Vid with same ed

ingress
egress set
i
19
Application on the ATT Backbone

• Measurement data
• Netflow data (ingress traffic)
• Forwarding tables (sets of egress points)
• Configuration files (topology and link weights)
• Effectiveness
• Ingress traffic could be matched with egress sets
• Simulated flow of traffic consistent with link
  loads
• Challenges
• Loss of Netflow records during delivery (can
  correct for it!)
• Egress set changes between table dumps (not very
  many)
• Topology changes between configuration dumps
  (just one!)

20
Traffic Analysis Results

• Small number of demands contribute most traffic
• Optimize routes for just the heavy hitters
• Measure a small fraction of the traffic
• Must watch out for changes in load and set of
  exit links
• Time-of-day fluctuations
• Reoptimize routes a few times a day (three?)
• Traffic (in)stability
• Select routes that are good for different demand
  sets
• Reoptimize routes after sudden changes in load

21
Three Traffic Demands in San Francisco
22
Underpinnings of the Optimization

• Route prediction engine (what-if tool)
• Model the influence of link weights on traffic
  flow
• Select a closest exit point based on link weights
• Compute shortest path(s) based on link weights
• Capture splitting over multiple shortest paths
• Sum the traffic volume traversing each link
• Objective function
• Rate the goodness of a setting of the link
  weights
• E.g., max link utilization or sum of
  exp(utilization)

23
Weight Optimization

• Local search
• Generate a candidate setting of the weights
• Predict the resulting load on the network links
• Compute the value of the objective function
• Repeat, and select solution with lowest objective
  function
• Efficient computation
• Explore the neighborhood around good solutions
• Exploit efficient incremental graph algorithms
• Performance results on ATTs network
• Much better than simple heuristics (link
  capacity, distance)
• Quite competitive with multi-commodity flow
  solution

24
Incorporating Operational Realities

• Minimize changes to the network
• Changing just one or two link weights is often
  enough
• Tolerate failure of network equipment
• Weights settings usually remain good after
  failure
• or can be fixed by changing one or two weights
• Limit the number of distinct weight values
• Small number of integer values is sufficient
• Limit dependence on accuracy of traffic demands
• Good weights remain good after introducing random
  noise
• Limit frequency of changes to the weights
• Joint optimization for day and night traffic
  matrices

25
Ways to Populate the Domain-Wide Models

• Mapping assumptions about routing
• Traffic data packet/flow statistics at network
  edge
• Routing data egress point(s) per destination
  prefix
• Inference assumptions about traffic and routing
• Traffic data byte counts per link (over time)
• Routing data path(s) between each pair of nodes
• Direct observation no assumptions
• Traffic data packet samples at every link
• Routing data none

26
Inference Network Tomography
From link counts to the traffic matrix
Sources
3Mbps
5Mbps
4Mbps
4Mbps
Destinations
27
Tomography Formalizing the Problem

• Source-destination pairs
• p is a source-destination pair of nodes
• xp is the (unknown) traffic volume for this pair
• Routing
• Rlp 1 if link l is on the path for src-dest
  pair p
• Or, Rlp is the proportion of ps traffic that
  traverses l
• Links in the network
• l is a unidirectional edge
• yl is the observed traffic volume on this link
• Relationship y Rx (now work back to get x)

28
Tomography Single Observation is Insufficient

• Linear system is underdetermined
• Number of nodes n
• Number of links e is around O(n)
• Number of src-dest pairs c is O(n2)
• Dimension of solution sub-space at least c - e
• Multiple observations are needed
• k independent observations (over time)
• Stochastic model with src-dest counts Poisson
  i.i.d
• Maximum likelihood estimation to infer traffic
  matrix
• Vardi, Network Tomography, JASA, March 1996

29
Tomography Limitations

• Cannot handle packet loss or multicast traffic
• Assumes traffic flows from an entry to an egress
• Statistical assumptions dont match IP traffic\
• Poisson, Gaussian
• Significant error even with large of samples
• Faulty assumptions and traffic changes over time
• High computation overhead for large networks
• Large matrix inversion problem

30
Promising Extension Gravity Models Zhang,
Roughan, Duffield, Greenberg

• Gravitational assumption
• Ingress point a has traffic via
• Egress point b has traffic veb
• Pair (a,b) has traffic proportional to via veb
• Incorporating hot-potato routing
• Combine traffic across egress points to the same
  peer
• Gravity divides as traffic proportional to peer
  loads
• Hot potato identifies single egress point for
  as traffic
• Experimental results on ATT network
• Reasonable accuracy, especially for large (a,b)
  pairs
• Sufficient accuracy for traffic engineering
  applications

31
Conclusions

• Our approach
• Measure network-wide view of traffic and routing
• Model data representations and what-if tools
• Control intelligent changes to operational
  network
• Application in ATTs network
• Capacity planning
• Customer acquisition
• Preparing for maintenance activities
• Comparing different routing protocols
• Ongoing work interdomain traffic engineering

32
To Learn More

• Overview papers
• Traffic engineering for IP networks
  (http//www.research.att.com/jrex/papers/ieeenet
  00.ps)
• Traffic engineering with traditional IP routing
  protocols(http//www.research.att.com/jrex/pape
  rs/ieeecomm02.ps)
• Traffic measurement
• "Measurement and analysis of IP network usage and
  behavior(http//www.research.att.com/jrex/paper
  s/ieeecomm00.ps)
• Deriving traffic demands for operational IP
  networks(http//www.research.att.com/jrex/paper
  s/ton01.ps)
• Topology and configuration
• IP network configuration for intradomain traffic
  engineering (http//www.research.att.com/jrex/pa
  pers/ieeenet01.ps)
• Intradomain route optimization
• Internet traffic engineering by optimizing OSPF
  weights(http//www.ieee-infocom.org/2000/papers/
  165.ps)
• Optimizing OSPF/IS-IS weights in a changing
  world(http//www.research.att.com/mthorup/PAPER
  S/change_ospf.ps)