Freek Dijkstra 1

1
Scalability of Dynamic Transport Networks

• Thesis Overview
• Freek Dijkstra
• SNE group, Universiteit van Amsterdam

2
Overview

• Introduction
• Routing
• Network description
• Path finding
• Fault isolation
• End-node configuration
• Pioneering lightpaths
• Path provisioning
• Transport protocols
• WAN PHY links

3
Context and Question
Context Dynamic transport networks are currently
built, and provisioned by hand. The goal is to
automatically set up dynamic circuits across
domains.
Dynamic transport networks provide switched
circuits for data transport, typically to
e-science applications
Question Do currently proposed protocols and
software solutions for setting up dynamic
circuits scale for multi-domain scenario's? If
not, what solutions scale better?
4
Routing Overview
Question What path finding mechanism can be used
in a scalable multiple domain environment?

• Scalable means
• No forced single technology solution on the
  dynamic part (e.g. SONET, DWDM, Ethernet VLAN)
• No detailed knowledge required for each
  domains(neither detail information about each
  domain, nor about all interrelations).
• This leads to non-trivial routing decisions

5
Routing Example
GE is adapted in STS-24c (SONET channel thru
OC-192)
CAnet Canada
UvA
U. Calgary
GE
OC-192
GE
OC-192
Star Light Chicago
MAN LAN New York
NetherLight Amsterdam
OC-192
OC-192
GE is adapted in STS-3c-7v (SONET channel thru
OC-192)
6
Routing Example
GE is adapted in STS-24c (SONET channel thru
OC-192)
CAnet Canada
UvA
U. Calgary
GE
OC-192
GE is adapted in STS-24c (SONET channel thru
OC-192)
GE
OC-192
Star Light Chicago
MAN LAN New York
NetherLight Amsterdam
OC-192
OC-192
GE is adapted in STS-3c-7v (SONET channel thru
OC-192)
GE is adapted in STS-3c-7v (SONET channel thru
OC-192)
7
Capability Announcement
Does only understand GE
CAnet Canada
UvA
U. Calgary
GE
Can only adapt GE in STS-24c
OC-192
Does only understand GE
GE
OC-192
Star Light Chicago
MAN LAN New York
NetherLight Amsterdam
OC-192
OC-192
Does only understand SONET/SDH
Can only adapt GE in both STS-24c and STS-3c-7v
Can only adapt GE in STS-3c-7v
8
Multi-Layer Graphs
SONET Layer
Ethernet Layer
9
Routing Overview
Question What path finding mechanism can be used
in a scalable multiple domains environment?

• Scalable means
• No forced single technology solution on the
  dynamic part (e.g. SONET, DWDM, Ethernet VLAN)
• No detailed knowledge required for each
  domains(neither detail information about each
  domain, nor about all interrelations).
• This leads to non-trivial routing decisions

Not possible to use simple (one layer) graphs.
10
Network Description
I need network descriptions as a tool for path
finding, but lack of sufficient prior work lead
to research questions.
Question 1 Would the creation of optical
exchanges as the only location for adaptation and
termination make mutli layer transport networks
significantly more simple? Would such a transport
network work in practice?
Question 2 Can G.805 be extended to describe
multi-layer networks?
ITU-T G.805 defines functional elements to
specify multi-layer network connections.
11
Network Description Methods

• Methods
• Define optical exchanges
• build them in collaboration with network
  providers and describe properties at data plane
  and control plane
• compare the properties of what was described with
  what was build
• define a network description language
• extend G.805 in a mathematical model

• Articles
• Optical Exchanges (proceedings, 2004)
• Control Models at Optical and Internet Exchanges
  (not yet published)
• A Network Model Based on ITU G.805 (in writing)
• Using RDF to Describe Networks (journal, 2006)

12
Optical Exchanges
Source Optical Exchanges, 2004

• Defined optical exchanges as peering points, with
  single layer links in between

• Describe properties at data plane and control
  plane (before networks were build)

Source Control Models at Optical and Internet
Exchanges, 2006

• After networks are build Compare the properties
  of what was described with what was build

13
Optical Exchanges
Question 1 Would the creation of optical
exchanges as the only location for adaptation and
termination make transport networks significantly
more simple? Would such a transport network work
in practice?
Results 1 Optical exchanges did emerge.
Consistent with model. Data plane of exchanges
based on SONET rather then optical cross
connects, as predicted (not consistent). Reason
Fault Isolation was too hard with optical cross
connects. Current inter-domain control planes use
the federated control model, as predicted. Not
enough deployment to make this a definitive
result.
14
ITU-T G.805 Extension

• ITU-T describes network connections, our model
  describes networks, with may provide potential
  connections.

• Describe model with each interface having a set
  op potential adaptation functions.

• Map network elements to functional elements of
  G.805. (G.805 is generic).

Source A Network Model Based on ITU-T G.805, in
progress

• Use algebra to define correctness of a
  con-nection, rather then functional elements.

• Our algebra is less strict then ITU-T G.805

15
ITU-T G.805 Extension
Is D?A valid?
AX?Z, DRange(A)?Y
Yes, if ?A1,A2 A1X?Y, A2Y?Z D?A2 IdY
This is less strict then G.805 validity We
require that A2 simply exists. G.805 requires
that it was really there.
We do not take the termination function into
account, only the adaptation.
16
ITU-T G.805 Extension
Question 2 Can G.805 be extended to describe
multi-layer networks?
Results 2 Yes, we could map network elements to
G.805 functional elements, and define an algebra
to check for correctness or validity of a
connection. Our algebra is slightly different
from G.805 validity, but otherwise the algebra
would be very complex. G.805 and our model have
some limitations no broadcast networks (e.g.
wireless), no concept of bandwidth and assumes
all series of links are valid (not true at
physical layer).
17
Path Finding Question
Context Current initiatives for multi-domain
routing in transport networks handle the scale
problem by discriminating between intradomain and
interdomain routing, similar to the regular
Internet. All interdomain routing protocols (e.g.
GMPLS) uses a link state routing protocol
(usually constrained shortest path first). I
claim that this does not scale well.
Questions Does a link state protocol scale for
interdomain routing in transport networks? If so,
what constraints are best to enhance scalability?
If not, would a broadcast protocol work better?
18
Routing Constraints

• network topology constraints (e.g. bandwidth)
• scheduling constraints (i.e. current usage)
• multilayer technology constraints (e.g.
  incompatible technologies like STS-24c vs.
  STS-3c-7v)
• policy-based constraints (i.e. access control)

19
Broadcast path discovery

• Based on ideas in SS7 (telephony networks)
• Transport network consist of scarce resources.
  This means that links may not be assumed to be
  available at all time. As a consequence, the
  state of the network must be taken into account
  for each path setup request.
• If there are enough constraints, broadcasts may
  scale better then constraint based routing.

Ethernet
Ethernet in STS-24c
Ethernet in STS-24c
Ethernet in STS-24c
Ethernet
Ethernet in STS-24c or In STS-3c-7v
Ethernet in STS-24c
Ethernet in STS-3c-7v
20
Path Finding Results

• Methods
• describe a network with NDL
• Implement a new and an existing path finding
  algorithm
• Test how well this performs with descriptions of
  current networks (e.g. GLIF) or artificially
  large networks.

Results Work in progress Proposed solution
broadcast instead of constraint based routing
21
Fault Isolation Question
Context Fault isolation for international
switched circuits is cur-rently done manually by
network engineers. It would be an advantage to
make this an automated process, which ideally
could be initiated or verified by the end-user.
Questions What possible mechanisms exist to
detect and isolate faults in multi layer network
connections, and which of these methods is the
best in a multi-domain environment?
22
Fault Isolation Method
Method Describe and categorize different methods
for fault isolation, preferably based on existing
ideas or existing implementa-tions. Determine the
two best mechanisms. Implement these methods in a
small scale network and compare the results with
the expectations.
Data We defined five methods for fault isolation
single technology (e.g. LMP, SS7, SDH), record
route (out-of-band traceroute), loopback tool,
measurement points (at any layer, or at Ethernet
layer only), and expert system. The layer 2
measurement point have been implemented by
colleagues in Dragon project, and we learned from
their results. We implemented the expert system
(in progress).
23
Fault Isolation Methods

• Overview of different methods

Tool Property Ping / Traceroute SS7 / LMP / SDH Record Route Loopback tool Measure-ment point Measurem. Point L2 Expert system
Detection time Path established Early detection Path established During path provisioning Early detection Path established Data flows
Support required Backward compatible All devices All devices Backward compatible Backward compatible Backward compatible Depends on layer
Software complexity ICMP SS7 / GMPLS GMPLS Loopback provisioning Regular provisioning Regular provisioning Webservice or SNMP
Multi-plexing yes yes yes no No yes yes
Keeps connection keeps keeps keeps breaks breaks (or multicast) keeps keeps
Intrusive-ness Active Non-intrusive Active Non-intrusive Passive Intrusive Intrusive (or multicast) Active Non-intrusive Passive
In/out band In-band In-band Out-of-band In-band In-band In-band Out-of-band
Layer Layer 3 only Any SS7 / Any GMPLS Any GMPLS Any layer Any layer Layer 2 only Any Layer
Source Fault Isolation in Multi-layer Transport
Networks Exchanges, In writing
24
Expert System

• Idea gather as much data from devices along the
  path, and predict if the connection will work or
  will fails, and if so where it has failed.
• Expert system
• software
• Architecture

Source Fault Isolation in Multi-layer Transport
Networks Exchanges, In writing
25
Fault Isolation Results
Results The measurement points only work well for
layer 2. At lower layers, the circuit must be
broken for measurements. It was expected that the
expert system would cope badly with incomplete
information. Preliminary results show this can be
dealt with by debugging per layer at a time.
A discussion in the GLIF community resulted that
some people believed that everyone should use
SONET, others did not think that would be
viable. For further research, an idea taken from
SS7 on early detection systems was coined and
well received in the community.

• Article
• Fault Isolation in Multi-layer Transport Networks
  Exchanges (In writing)

26
Node configuration Question
Context After creating a circuit, end-users
struggle with defining the correct set-up for
their machines at IP layer and up (e.g. deciding
on IP ranges, and adjusting TCP parameters).
Questions Is it possible to automatically have
end-hosts configure themselves to use a switched
circuit, without relying on a central
configuration mechanism? Can zero-configuration
technology be used for this?
27
Zero Configuration

• DHCP will not work out-of-the-box, since it is
  not clear which domain should run it.
• Use Zero Configuration protocols
• Automatic configuration of IP addresses
• RFC3927 for IPv4 or RFC1971 for IPv6
• Name lookup of hosts (to discover IP address of
  target host)
• Multicast DNS (mDNS) or Link-Local Multicast Name
  Resolution (LLMNR)
• Discovery of services
• DNS Service Discovery (DNS-SD), or Simple Service
  Discovery Protocol (SSDP, in UPnP), or Service
  Location Protocol (SLP) (or even UDDI, SDP,
  Salutation, or Jini)
• Three software suites, used multiple
  implementations
• RFC3927 ZCIP and autoip for Linux, native in OS
  X and Windows
• mDNS mDNSResponder, tmdns, and Porchdog mDNS
• hooking gethostby() to use mDNS tmdns and
  libnss_mdns

28
iGrid Demonstration

• Demonstrated auto-matic IP configuration
• Used broadcast ping to discover all hosts
• Used multicast DNS and gethostbyaddr() hook to
  discover hostnames
• Tested IP collisions
• Also demonstrated service discovery through DNS

Source Using Zero Configuration Technology for
IP addressing in Optical Networks, 2006
29
Node configuration Results
Method Implemented an existing technology,
link-local IP addressing, and apply to another
application (transport networks rather then home
networks).
Results Successfully demonstrated concept at
iGrid 2005. Also shown which part of applications
(by taking iGrid as the sample) can benefit from
this technique. Pointed to existing research for
TCP tuning.

• Article
• Using Zero Configuration Technology for IP
  addressing in Optical Networks (journal, 2006)

30
Pioneering Lightpaths

• In 2003, there was no international transport
  network for e-Science applications
• Now, hybrid networks (e.g. SURFnet 6) and optical
  private networks (e.g. for LOFAR) have been
  build.GLIF consists of 11 optical exchanges, and
  many links.
• Three research topics I worked on before summer
  2004
• Path provisioning
• Transport protocols
• WAN PHY links

31
Path Provisioning Question
Context Connections through transport networks
typically cross multiple domains. Since resources
are allocated to a single user, authorization
mechanisms must be in place.
Question Does path provisioning with separate
intra- and interdomain control planes work in
practice?Is it possible to integrate generic AAA
with the path setup decision?
32
Path Provisioning Results
Method Shown three demonstrations at
supercomputing with resp. PIN/PDC software, AAA
software and AAA/DRAC software
Results Successfully set up connections with
interworking between our AAA software, PIN
software and DRAC software.

• Articles
• Applications Drive Secure Lightpath Creation
  across Heterogeneous Domains (journal, 2006, main
  author Leon Gommans)
• Dynamic paths in multi-domain optical networks
  for grids (journal, 2005, main author Bas van
  Oudenaarde)

33
Transport Protocols Question
Context Standards (new Reno) TCP does not scale
well for high bandwidth high delay connections.
Alternative protocols have been developed to
circumvent this problem. However, those protocols
may be so aggressive that they disrupt other
traffic on the same link.
Question Are alternative transport mechanisms
fast, fair (to other streams) and friendly (fair
to TCP streams)?
34
UDT versus TCP flows
Earlier variant of UDT with 10 TCP streams.
Later variant of UDT with 121 TCP streams.
35
Transport Protocols Results
Method Use a transatlantic testbed with one
specific protocol, UDT, and combine it with other
UDT and other TCP streams.
Results UDT seems to be fast, friendly and fair.

• Article
• Teraflows over Gigabit WANs with UDT (journal,
  2005, main author Robert Grossman)

36
WAN PHY Question
Context Most international connections are based
on SONET/SDH technology, while local connections
are based on Ethernet. There are different ways
to pack Gigabit Ethernet in SONET/SDH channels,
and only one way to pack 10 Gb/s Ethernet in
SONET/SDH OC-192 circuits.
Question Is it possible to use WAN PHY over long
distance links, even though the specs of
SDH/SONET are more strict then those of WAN PHY
Ethernet?
37
WAN PHY Results
Method Connect Amsterdam to Geneva by directly
connecting the link of a carrier to a Ethernet
WAN PHY interface. Measure the flow, and check
for bit errors, and other transport problems.
Results Remarkably, the WAN PHY interfaces can
power up a OC-192 circuit, and no packet loss was
measured while doing throughput test for a longer
period of time.

• Article
• Native 10 Gigabit Ethernet experiments over long
  distances (journal, 2005, main author Catalin
  Meirosu)

38
Context and Question
Context Dynamic transport networks are currently
built, and provisioned by hand. The goal is to
automatically set up dynamic circuits across
domains.
Dynamic transport networks provide switched
circuits for data transport, typically to
e-science applications
Question Do currently proposed protocols and
software solutions for setting up dynamic
circuits scale for multi-domain scenario's? If
not, what solutions scale better?
39
Method Overview
I covered most steps of setting up a lightpath
for e-Science applications

• WAN PHY (building transport networks)
• Network description (for routing)
• Path finding (routing)
• Path provisioning
• Fault detection and fault isolation
• End-node configuration
• Transport protocols (can apps benefit?)

Early work, I did not pursue further because I
believed that topic was more or less finished, or
covered by other groups.
40
Final Conclusions

• Path provisioning works fine by using a separate
  intradomain and interdomain protocol.
• Currently proposed solutions for interdomain
  routing (e.g. RSVP-TE in GMPLS) do not scale.
  This thesis proposes a broadcast solution for
  path finding, in favour over constraint based
  routing.
• A good approach for fault isolation in
  multi-domain networks for layer 2 is to use
  measurement points. For lower layers, expert
  system is a better approach.
• Zero-configuration technology is a solution for
  end-node addressing that scales for non-routed
  paths.

41
Questions?
42
Articles (Recent Work)

• Routing (network description)
• Optical Exchanges (proceedings, 2004)
• Control Models at Optical and Internet Exchanges
  (not yet published)
• A Network Model Based on ITU G.805 (in writing)
• Using RDF to Describe Networks (journal, 2006)
• Routing (path finding)
• In progress
• Fault Isolation
• Fault Isolation in Multi-layer Transport Networks
  Exchanges (In writing)
• Node Configuration
• Using Zero Configuration Technology for IP
  addressing in Optical Networks (journal, 2006)

43
Articles (Earlier Work)

• Path Provisioning
• Applications Drive Secure Lightpath Creation
  across Heterogeneous Domains (journal, 2006, main
  author Leon Gommans)
• Dynamic paths in multi-domain optical networks
  for grids (journal, 2005, main author Bas van
  Oudenaarde)
• Transport Protocols
• Teraflows over Gigabit WANs with UDT (journal,
  2005, main author Robert Grossman)
• WAN PHY
• Native 10 Gigabit Ethernet experiments over long
  distances (journal, 2005, main author Catalin
  Meirosu)