Experiences with Lambda Networking in the USA

1
Experiences with Lambda Networking in the USA
CANS 2007, XiAn, China, August 25th, 2007

• John Silvester
• Special Advisor to CIO for High Performance
  Networking, Professor of EE, University of
  Southern California
• International Peering Advisor, NLR

2
What is Lambda Networking?

• Originally meant effective utilization of WDM
  derived waves (40 waves at OC-192 or 10G on a
  fiber, for example)
• Led to the term lightpath to mean a end to end
  set of waves
• This term also used for dedicated end to end
  paths of lower capacity (1G or 100M, for example)
• Static versus dynamic allocation. Today mostly
  static but much interest in dynamic and even
  usercontrolled (UCLP).

3
Lambda Networking in the USA

• Originally lambda based networks were developed
  at some Universities and then spread to a small
  number of State networks (I-Wire, CENIC CalREN,
  operational 2004)
• Major national scale initiative was National
  LambdaRail - WDM facility over leased dark fiber.
  Implemented with 32-40 10G or OC-192 waves.
  (operational 2005-6)
• Led to the development of many more state or
  regional owned optical fiber based networks.
  (operational 2004-2006)
• And Internet2s new network a national multiple
  wave backbone (initially 10 x 10G expandable as
  needed) (operational 2007)

4
CALIFORNIA
5
CENIC Infrastructure

• WDM capability for 32-40 10G waves
• Originally used to support two IP networks for
  California (CalREN-DC and CalREN-HPR)
• Extended to provided dedicated waves CalREN-XD
  and to transport the California piece of NLR and
  PacificWave

6
NLR
7
NLR Infrastructure

• Over 11,000 route miles of fiber
• 20-yr IRUs on dark fiber
• Layer 1 Optics Cisco 15808s and 15454s
  supporting up to 40 10 Gbps lambdas LANPHY
• Layer 2 Cisco 6509 switches for Gigabit Ethernet
• Layer 3 Cisco CRS-1 routers for IP
• Wavelengths available for experimental or
  production networks

8
Regional Optical Networks - RONs

• California (CALREN)
• PNWGP
• Colorado/Utah (FRGP)
• Illinois (I-WIRE)
• Michigan (MLR)
• Indiana (I-LIGHT)
• New York (NYSERnet)
• Maryland, D.C. Virginia (MAX/MATP)

• N. Carolina (NCREN)
• Georgia (Southern Light Rail)
• Florida (Florida LambdaRail)
• Louisiana (LONI)
• Texas (LEARN)
• Oklahoma (OneNet)
• Ohio (TFN)

Courtesy of Steve Corbato
9
Internet2 New Network
10
Internet2 Infrastructure

• Close partnership with an optical carrier to
  provide a dedicated multi-wave system, based on
  Infinera equipment
• Partnership with other US National RE Network
  interests (ESNET) to share costs
• Building out a dynamic point-point circuit
  switched capability DCS based on CIENA equipment

11
NLR-I2 Network Planning Team

• Over the last several months there have been
  extensive discussions to merge the two
  organizations (NLR, Internet2). This process is
  still underway.
• Part of the process involved the creation of a
  Network Planning Team to discuss how the two
  national networks might be consolidated.
• Their recommendation was to continue to operate
  both backbones for the next several years since
  they represent different approaches, provide
  greater capacity, allow for more experimentation
  and potentially allow for more rapid adoption of
  new technologies and ideas.

12
Exchange Points

• Major International RE Exchange points in the US
  (Starlight, PacifcWave, MANLAN, Atlantic Wave)
  are also providing lightpath connectivity
• Much discussion and experimentation with how to
  provide dynamic lightpaths across these exchange
  points

13
End to End?

• Many regional networks have not yet deployed
  capabilities for multi-wavelength or lightpath
  networking. Most are just now defining the
  requirements and determining how to connect to
  the national networks.
• Also most campus have not implemented the ability
  to deliver lightpaths to the researcher. At best
  they can deliver the service to a few labs or end
  points. Policy issues (security) are often a
  major challenge here.

14
Suymmary of NLR Usage

• Three main network services offered
• Wavenet (layer 1 or 1.5) dedicated capacity
  true Lighptaths
• Framenet (layer 2) VLAN over Ethernet structure.
  Also called lightpaths. (can be with or without
  bandwidth management)
• Packetnet (layer 3) Traditional IP service

15
WaveNet Overview

• ETE circuits consisting of wave segments
• Dedicated waves purchased by regionals or
  research projects
• Research waves, supported by CISCO, made
  available in support of approved research
  projects
• Quickstart waves (if available and preemptable
  for fail-over)

16
Infrastructure Update - WaveNet

• Steady demand for WaveNet services
• NLR has allocated 199 WaveNet Circuits to date
• 126 Active Circuits
• 75 Customer Circuits
• 51 Backbone Circuits (L2, L3, Optical Switch)
• 32 Misc Circuits - spare, unallocated, reserved,
  SC07
• 41 Decommissioned circuits
• includes SC related circuits, circuits no long
  in use by customers, etc.

17
Projects with Dedicated 10G waves

• TeraGrid NSF, U. of Chicago
• TeraGrid NSF, PSC
• TeraGrid NSF, TACC
• OptIPuter NSF, UCSD/UIC
• UltraScience DOE, ORNL
• ESNet DOE, LBNL
• Ames/Goddard NASA
• HOPI Internet2
• Pacific Wave CENIC/PNWGP
• Cheetah NSF Project (U of VA)
• FLR Backhaul FLR
• LEARN Backhaul LEARN
• Cornell Backhaul Cornell

18
(No Transcript)
19
FrameNet Overview

• National Exchange Fabric Service
• NEF is a single VLAN and broadcast domain that
  extends to every member
• Allows members to arrange bi-lateral peerings
• Every member gets a single 1GigE interface on
  their local NLR FrameNet node for use of the NEF
  service
• Point-to-point, Dedicated Bandwidth Service
• Members may order private VLANs to connect 2
  different locations together, with dedicated
  bandwidth, for a circuit-like service
• Bandwidth available from 100Mbps to 10Gbps
• Point-to-multipoint, Best Effort Service
• Members may order private VLANs to multiple
  locations for a circuit-like service
• Traffic on these VLANs carried through the
  network on a best effort basis

20
FrameNet

• Demand continues to grow, 10 GE interfaces and
  non-dedicated bandwidth are popular
• 86 Active VLANs, both dedicated and non-dedicated
• Currently primary use of dedicated service is for
  Layer 3 backup connections
• Interest by research groups
• Collaborative multi-point projects
• Experiment on non-dedicated service, understand
  utilization, then migrate to dedicated service.
• New Services being Developed, in particular
  on-demand dedicated services
• Adding FrameNet connection to MANLAN switch to
  support layer2 connectivity to international
  networks

21
FrameNet on-demand

• Scheduled dedicated bandwidth
• Not real-time, but advanced reservations
• Phased implementation - completed by year end
• Beta testing with the ETF
• Initially fixed endpoint, configured VLAN
• ETF uses as needed, billed according to use
• Phase 2
• Web interface to implement established VLAN
• No access to additional VLAN parameters
• Phase 3
• Web access to VLAN parameter

22
FrameNet - Dedicated Bandwidth
23
FrameNet - VLAN Count
24
FrameNet Traffic Map
25
Lightpath/Hybrid Networks - Challenges

• Network diagnosis
• Much harder to diagnose a problem
• What layer
• Who is responsible for the infrastructure in
  question.
• Network performance monitoring
• Similar to network diagnosis
• More components to tune.
• No longer just an IP performance issue
• NLR is implementing more network monitoring
  capability to help in network management and
  diagnosis

26
Example International Lightpath Projects

• EnLIGHTened / PHOSPHORUS interoperation
• GLAMBDA
• CINEGRID
• Optical Multicast - HPDMnet (High Performance
  Digital Media Network)
• Thomas Sterling HPC Class in HD
• eVLBI
• HD Video-conferencing

27
Summary

• Fairly extensive deployment of nationwide
  infrastructure in place
• Reasonable amount of experimentation with
  dedicated static lightpaths to date
• Some experimentation with dynamic lightpaths
• Need for extension of capabilities through the
  regional networks and onto campuses
• More development of dynamic lightpath capable
  exchange points (GOLES)
• Experimentation with Internet2 DCS services

28
Contact Information
silvester_at_usc.edu