CANARIE CA*net 4 Planning

1
CANARIECAnet 4 Planning

• http//www.canarie.ca http//www.canet3.net

Bill.St.Arnaud_at_canarie.ca Tel [phone]
2
CAnet 3 National Optical Internet
Consortium Partners Bell Nexxia Nortel Cisco JDS
Uniphase Alcatel
CAnet 3 Primary Route
CAnet 3 Diverse Route
GigaPOP
ORAN
Deployed a 4 channel CWDM Gigabit Ethernet
network 400 km
Deploying a 4 channel Gigabit Ethernet
transparent optical DWDM 1500 km
Condo Fiber Network linking all universities and
hospital
Multiple Customer Owned Dark Fiber Networks
connecting universities and schools - 3500km
Netera
MRnet
SRnet
ACORN
St. Johns
BCnet
Calgary
Regina
Winnipeg
Charlottetown
RISQ
ONet
Fredericton
Montreal
Vancouver
Halifax
Ottawa
Seattle
STAR TAP
Toronto
Chicago
New York
3
CAnet 3 Status

• All GigaPOPs and regional networks connected and
  operational
• ORANs up and running in Nfld, PEI, Quebec,
  Alberta
• Announcements expected shortly for ORANs in BC,
  Manitoba, Ontario and Nova Scotia, New Brunswick
• 360network connection to Europe and NY through
  Dalhousie coming soon
• Will make Canada Dalhousie global hub for most
  international research traffic
• Several US networks and universities looking to
  connect directly to CAnet 3
• NYSERnet to be completed shortly
• UoMinnesota in discussions
• Traffic volumes exceeding 500 Mbps peak on all
  links
• Network expected to be reach peak capacity by
  September with on-going WDD trials and HDTV
  sessions between UoCalgary and McGill
• Numerous countries have built or plan to build
  networks modeled on CAnet 3 and proposed CAnet
  4
• Chile, Brazil, Poland, SURFnet, Greece Czech
  Republic, Australia, CENIC, NYSERnet, Quilt
• Over 200 university research institutions and
  2000 school connected
• Over 2000 schools connected
• Over 50 International peers and over 20
  International transit networks

4
Peer Networks

• USA 6 networks
• Abilene (Internet 2), ANL (Argonne), vBNS (NSF),
  Esnet (Energy), NISN (NASA), NREN (NASA)
• NEW Nysernet July 2001
• STARTAP in Chicago 20 national networks
• CERN, IUCC, APAN/ TRANSPAC (Korea, Malaysia,
  Australia, Philipines), RENATER2 (France), SINET
  (Japan), SingAREN (Singapore), SURFnet
  (Netherlands), NORDUnet (Iceland, Norway, Sweden,
  Finland, Denmark), TANet2 (Taiwan)
• New Reuna Chile, RTP Brazil, MIRnet Russia,
  Renater2 (France), KRnet (Korea)
• TEN-155 in New York 26 European networks
• ACOnet (Austria), ARNES (Slovenia), BELnet
  (Belgium), CESNET (Czech Republic), DFN
  (Germany), GARR (Italy), GRNET (Greece), HEAnet
  (Ireland), HUNGERNET (Hungary), JANET (U.K.),
  POL34 (Poland), RCCN (Portugal), RedIRIS (Spain),
  RENATER2 (France), RESTENA (Luxembourg), SWITCH
  (Switzerland), Nordunet (Iceland, Norway, Sweden,
  Finland, Denmark)
• Seattle 4 networks
• NTON (DARPA), Supernet (DARPA), PCNW (Washington
  State)
• New Australia (AARnet), ESnet

5
Internet Transit Service

• Internet 2 and CANARIE offering research transit,
  e.g.
• Europe (DANTE) or Asia to US agency networks
  NASA, NREN, Esnet
• Australia to Europe (DANTE)
• Korea to STAR TAP
• All European traffic to US research networks
  (except Internet 2) goes through Canada
• Will promote international collaboration in
  advanced research
• New 360network donation through Dalhousie will
  significantly enhance Canadas reputation as a
  global hub for advanced networks
• If history is any kind commercial networks will
  soon follow research networks

6
Applications

• Distributed caching to remote schools
• NFB video server
• multi-channel virtual studio
• digital mammography (UoT)
• Pinkas Zukerman interactive violin lessons
• global brain scan database (McGill)
• NRC bio-informatics genome sequence
• national repository strategy for learning objects
  
• HDTV over IP McGill
• Virtual Vet courses

7
International Developments

• Distributed Terabit Facility
• 40 Gbps network with wavelengths and dark fiber
  from Qwest
• Will interconnect over 1000 Linux processors
• Internet 3 and Quilt
• Internet 2s next generation network expected to
  be announced this fall
• Quilt based on many concepts proposed for CAnet
  4
• OMNInet in Chicago
• STAR LIGHT
• SURFnet (Holland), CERN, NODRUnet, DTF will all
  bringing wavelengths to Chicago this fall
• Maybe CAnet 4?
• PIONIER (Poland), GRnet (Greece), etc
• DANTE will be issuing RFP for wavelengths across
  the Atlantic

8
CAnet 3 Lessons

• Community Condominium dark fiber networks
  exploding
• Non traditional carriers building dark fiber
  networks
• Construction companies, university networks, etc
• LAN architectures, technologies and most
  importantly LAN economics are invading the WAN
• Low cost optical technology -CWDM, Gigabit
  Ethernet
• Control and management of the optics and
  wavelengths will increasingly be under the domain
  of the LAN customer at the edge, as opposed to
  the traditional carrier in the center
• Over time the current hierarchical connection
  oriented telecom environment will look more like
  the Internet which is made up of autonomous
  peering networks.
• These new concepts in customer empowered
  networking are starting in the same place as the
  Internet started the university and research
  community.
• The Internet model of networking is moving into
  traditional telecoms

9
Customer Empowered Networks

• School boards and municipalities throughout North
  America are working with next generation carriers
  in building condo open access, dark fiber
  networks and/or purchasing virtual fiber
• Spokane
• Chicago CivicNet
• Manhattan project
• Illinois Iwire
• Indiana Gwire
• California CENIC
• Polish Pionoier
• Czech municipal networks
• Coming soon BC, Ontario, Nova Scotia, New
  Brunswick
• Carrier are selling dim wavelengths or virtual
  fiber managed by customer to interconnect dark
  fiber networks
• Williams, Level 3, Qwest

10
Research Network Philosophy

• Research and Education networks must be at
  forefront of new network architecture and
  technologies
• Should be undertaking network technology
  development that is well ahead of or orthogonal
  to any commercial interest
• But any network architecture can only be
  validated by connecting real users with real
  applications and must solve real world problems
• Test networks per se are not sufficient
• There is a growing trend for many schools,
  universities and businesses to control and manage
  their own dark fiber
• Can we extend this concept so that they can also
  own and manage their own wavelengths?

11
The Concept for CAnet 4

• Conventional optical networks are built on the
  paradigm that a central entity has control and
  management of the wavelengths
• It therefore must have control of the edge device
  for the setup and tear down of the wavelengths
• Customer empowered optical networks are built on
  the paradigm that customer owns and controls the
  wavelengths (Virtual Dark Fiber) and dark fiber
• Customer controls the setup, tear down and
  routing of the wavelength between itself and
  other customers
• Customer may trade and swap wavelengths with
  other like minded customers ultimately leading to
  wavelengths as market commodity
• Network is now an asset, rather than a service
• Analogy to time sharing computing in the early
  1970s versus customer owned mini-computers or
  client-server computing
• Will empowering customers to control and manage
  their own networks result in new applications and
  services similar to how the mini-computer and PC
  empowered users to develop new computing
  applications?

12
Current View of Optical Internets
ISP
AS 1
AS 4
Carrier controls and manages edge devices
Optical VLAN
NNI
AS 1
Customer
AS 5
AS 3
UNI
Big Carrier Optical Cloud using MP?S or ASON for
management of wavelengths for provisioning,
restoral and protection
AS 2
13
Customer Empowered Network
City C
City A
Carrier Neutral IX
Carrier Neutral IX
Condo Wavelengths
City B
Condo Dark Fiber
Condo Wavelengths
14
Future Optical Networks
Massive peering at the edge
Customer D
Customer A
Condo Wavelength
Customr A elects to cross connect with Customer C
rather than D
Customer C
Customer B
Condo Fiber
15
CAnet 4 Research Areas

• New optical technologies that support customer
  empower networking
• OBGP, CWDM, hybrid optics and HWDM, customer
  controlled optical switches
• BGP scaling issues
• Object Oriented Networking
• Wavelengths and optical switch treated as an
  object and method to be incorporate into
  middleware
• Or treated as fungible product
• Distributed Computing Applications and Grids
• Wavelength Disk Drives (WDD)
• eScience
• Grids for weather forecasting, forestry
  management, education, health, etc

16
Object Oriented Networking

• Combines concepts of Active Networks and Grids
• See DARPA
• See Globus
• Customer owns sets of wavelengths and cross
  connects on an optical switch
• Network elements can be treated as a set of
  objects in software applications or grids
• Complete with inheritances and classes, etc
• Rather than distributed network objects ( e.g.
  Java or Corba) distributed object networks
• In future researchers will purchase networks just
  like super computers, telescopes or other big
  science equipment
• Networks will be an asset not a service
• Will be able to trade swap and sell wavelengths
  and optical cross connects on commodity markets

17
Example OON

• Earthquake Visualization Grid
• Globus Middleware Begin
• Establish connection to other grid participants
• Network Object wavelength to STAR LIGHT
  Chicago
• Network Object wavelength to Research center
  Amsterdam
• Network Object wavelength to SDSC Visualization
  Computer
• Network Object wavelength to Seismology Center
  Calgary
• Link objects and create grid
• Run Visualization
• Release Network objects
• Globus Middleware End
• Earthquake Visulization End

18
CAnet 4 Community Networks

• CAnet 4 will be a national resource for K-12
  networks and supporting community NBTF
  initiatives
• eScience grids, learning grids and health grids
• Researchers educator may want to use computing
  resources of schools and homes as part of large
  distributed computing projects
• CAnet 4 will interconnect environmental and
  health grids with students and researchers
• New grid projects in bio-informatics,
  pharmaceutical research, particle physics need
  access to millions of computers

19
What is eScience?

• The ultimate goal of e-science is to allow
  students and eventually members of the general
  public to be full participants in basic research.
• Using advanced high speed networks like CAnet 4
  and novel new concepts in distributed peer to
  peer computing, called Grids many research
  experiments that used to require high end super
  computers can now use the computer capabilities
  of thousands of PCs located at our schools and in
  our homes.
• High performance computers that are part of C3.ca
  can be seamlessly integrated with eScience
  distributed computers using CANARIE Wavelength
  Disk Drive over CAnet 4
• Allows researcher access to the significant
  computational capabilities of all these
  distributed computers at our schools and homes
• With e-science it might be possible that the next
  big scientific discovery could be by a student at
  your local school.

20
FightAIDS_at_Home

• Scientists at The Scripps Research Institute
  (TSRI) are using computational methods to
  identify drugs that have the right shape and
  interaction characteristics to fight diseases
  such as AIDS.
• Once such candidates are identified, they can be
  synthesized in a laboratory, tested according to
  FDA guidelines, and released as prescription
  drugs to benefit the public.
• Such computations require a vast number of trial
  dockings, testing variations in the target
  protein and the trial drug molecules

21
Philanthropic Peer to Peer

• The Intel Philanthropic Peer-to-Peer Program
  helps to combat life-threatening illnesses by
  linking millions of PCs to be the largest and
  fastest computing resource in history.
• This "virtual supercomputer" uses peer-to-peer
  technology to make unprecedented amounts of
  processing power available to medical researchers
  to accelerate the development of improved
  treatments and drugs that could potentially cure
  diseases.

22
ALTA Cosmic Ray eScience

• The earth is constantly bombarded by subatomic
  particles from space, with an energy spectrum
  that reaches far higher than any terrestrial
  accelerator could hope to probe. 
• At the highest energies such showers can be
  detected at the Earths surface over areas on the
  order of 100 square kilometers. 
• It is believe some of these cosmic rays were
  created at the creation of the universe
• Will allow researchers to gainer a deeper
  understanding of deepest reaches of space and
  time 

23
ALTA Cosmic Ray eScience

• The ALTA project is a collaborative scientific
  research project involving the University of
  Alberta Center for Subatomic Research and over 50
  high schools across Canada in the area of cosmic
  ray detection.
• Teachers and students actively contribute to the
  physics research while learning about an exciting
  area of modern science. 
• Distributed computing at schools will be required
  to analysize data from sensors in near real time
• Program has now expanded into USA and soon
  countries around the world
• CHICOS (California HIgh school Cosmic ray
  ObServatory), Caltech, UC/Irvine and Cal
  State/Northridge, California, USA.
• CROP (the Cosmic Ray Observatory Project),
  University of Nebraska, Lincoln, NE, USA.
• WALTA (WAshington Large area Time coincidence
  Array), University of Washington, Seattle, WA,
  USA.
• SALTA Roaring Fork Valley area of Colorado

24
Neptune Undersea Grid
25
Wavelength Disk Drives

• CAnet 4 will be nation wide virtual disk drive
  for grid applications
• Big challenges with grids or distributed
  computers is performance of sending data over the
  Internet
• TCP performance problems
• Congestion
• Rather than networks being used for
  communications they will be a temporary storage
  device
• Ideal for processor stealing transaction
  intensive applications where you dont know where
  the next available processor is located
• CFD
• Visualization

26
Wavelength Disk Drives
St. Johns
Regina
CAnet 3/4
Calgary
Winnipeg
Charlottetown
Montreal
Halifax
Fredericton
Vancouver
Ottawa
WDD Node
Toronto
Computer data continuously circulates around the
WDD
27
WDD Architecture
WDD Partners CANARIE, Can-Sol, Viagenie CRC,
Carleton U, MACI C3.Ca, Memorial,
Dalhousie UdeMontreal, UoToronto, SFU,
UoAlberta, BCnet

• 3. The SGI writes back the task onto the ring
  where it is received by Forest Fire Raster Engine
  and results displayed on X-Window terminal at CRC

UdeMontreal
UoToronto
Dalhousie
UoAlberta
Memorial
SFU
CRC
WDD Node
WDD Node
Vancouver
WDD Node
Calgary
Halifax
WDD Ring on CAnet 3
Forest Fire Modeling Raster Engine

• 2. Tasks circulate in WDD ring and first
  available SGI processor removes next task out of
  the ring and completes computation

1. Forest Fire Modeling Raster Engine injects 64K
x 64K raster computational tasks into WDD ring
28
Forest Fire Modeling eScience

• Emergency officials and civic defense officials
  need to model forest fires in real time
• But each forest fire model may take hours to
  compute
• By utilizing thousands of distributed computers
  at our schools and Wavelength Disk Drive on
  CAnet 4 network forest fire models in near real
  time
• First prototype to be demonstrated on CAnet 3 in
  May using 256 SGI processors across the country
  on WDD

29
WDD Process

• Forest Fire Modeling Raster Engine injects 64K x
  64K raster computational tasks into WDD ring at
  BCnet node in Vancouver
• Tasks circulate in WDD ring and first available
  SGI processor removes next task out of the ring
  and completes computation
• The SGI writes back the task onto the ring where
  it is received by Forest Fire Raster Engine and
  results displayed on X-Window terminal at CRC

30
CAnet 4 Possible Architecture
Layer 3 aggregation service Optional Service
Available to any GigaPOP
St. Johns
Regina
Calgary
Winnipeg
Large channel WDM system
Charlottetown
Europe
Vancouver
Montreal
Customer controlled optical switches
Fredericton
Halifax
Seattle
Ottawa
Chicago
New York
Toronto
31
STAR LIGHT Interconnection?

• We see STAR LIGHT, CAnet 4, DTF and Vancouver
  Transit exchange facing same design issues
• How do we signal interconnect wavelengths
  (SDH/SONET subchannels) between STAR LIGHT
  participants?
• Like STAR TAP we will probably need a mix of
  Layer 1-3 solutions
• Layer 1 cross connect ATM plus
• Layer 3 router and/or route server
• Current ATM approaches
• Full mesh ATM like current STAR TAP
• Not possible with wavelengths or SDH/SONET
  channels
• PVC created on demand
• E.g Peer maker at MAEs

32
STAR LIGHT Options

• Layer 0 - Patch panel or optical switch
• Needs common wavelength and protocol
• Not easily subject to change and will not allow
  multiple peers
• Layer 1 - SDH/SONET cross connect switch
• Issues related to how identify and address
  SDH/SONET channels
• Layer 2 - GMPLS using IP and SONET/optical switch
• Main thrust of industry see Juniper/Nortel,
  Accelight, Cisco, NTT
• Requires significant centralized management
• Layer 2 -Map SDH/SONET channels to GbE channels
  use GbE switch
• Layer 3 - Each network terminates on its own
  router routers meshed together
• N squared meshing
• Layer 3 - BIG ROUTER
• Will it scale and needs central management and AS
• Layer 4 OBGP with CWDM with optical switch
• Each CWDM wavelength mapped to SDH/SONET channel
• Control of switch is by research networks

33
OBGP Status Report

• OBGP first draft submitted to IETF
• Prototype working at Carleton U
• We want input on next steps for OBGP and see if
  it will fit within STAR LIGHT plans
• Key features
• SDH/SONET Optical cross connects controlled by
  attached networks
• SDH/SONET Optical cross connects identified by
  IP addresses AS
• RPSL with OON extensions is database used to
  query who is connected at switch and at what port
• BGP OPEN message is used like Peer maker to
  request optical peering across the switch
• BGP UPDATE message and community Tags ( and maybe
  GMPLS) will be used to setup multihop wavelengths

34
OBGP

• Proposed new protocol to support control and
  management of wavelengths and optical switch
  ports
• Control of optical routing and switches across an
  optical cloud is by the customer not the
  carrier true peer to peer optical networking
• Use establishment of BGP neighbors or peers at
  network configuration stage for process to
  establish light path cross connects
• Customers control of portions of OXC which
  becomes part of their AS
• Optical cross connects look like BGP speaking
  peers serves as a proxy for link connection,
  loopback address, etc
• Traditional BGP gives no indication of route
  congestion or QoS, but with DWDM wave lengths
  edge router will have a simple QoS path of
  guaranteed bandwidth
• Wavelengths will become new instrument for
  settlement and exchange eventually leading to
  futures market in wavelengths
• May allow smaller ISPs and RE networks to route
  around large ISPs that dominate the Internet by
  massive direct peerings with like minded networks

35
Wavelength Scenarios
Workstation to Workstation Wavelength
University to University Wavelength
Campus OBGP switch
St. Johns
CWDM
GigaPOP to GigaPOP Wavelength
Regina
Winnipeg
RISQ
Halifax
Calgary
BCnet
Vancouver
Montreal
Seattle
Toronto
36
Wavelength Setup
AS 2- AS 5 Peer
AS 3
12
10
University
Regional Network
3
13
AS 1
2
15
4
AS 5
14
AS 1- AS 6 Peer
AS 2
5
7
9
1
AS 4
AS 6
Regional Network
6
8
University
Dark Fiber
ISP router
Wavelength Object owned by primary customer
Wavelength Subcontracted by primary customer to
a third party
37
Wavelength Logical Mapping
AS 2- AS 5 Peer
AS 3
12
10
University
Regional Network
3
13
AS 1
2
15
4
AS 5
14
AS 1- AS 6 Peer
AS 2
5
7
9
1
AS 4
AS 6
Regional Network
6
8
University
Primary Route
ISP router
Backup Route

38
Resultant Network Topologies
BGP Peering on switches at the edge Packet
Forwarding in the core
39
OBGP Variations

• OBGP Cut Thru
• OBGP router controls the switch ports in order to
  establishes an optical cut through path in
  response to an external request from another
  router or to carry out local optimization in
  order to move high traffic flows to the OXC
• OBGP Optical Peering
• External router controls one or more switch ports
  so that it can establish direct light path
  connections with other devices in support peering
  etc
• OBGP Optical Transit or QoS
• To support end to end setup and tear down of
  optical wavelengths in support of QoS
  applications or peer to peer network applications
• OBGP Large Scale
• To prototype the technology and management issues
  of scaling large Internet networks where the
  network cloud is broken into customer empowered
  BGP regions and treated as independent customers

40
OBGP Optical Peering

• Primary intent is to automate BGP peering process
  and patch panel process
• Operator initiates process by click and point to
  potential peer
• Original St. Arnaud concept
• Uses only option field in OPEN messages
• Requires initial BGP OPEN message for discovery
  of OBGP neighbors
• Virtual BGP routers are established for every OXC
  and new peering relationships are established
  with new BGP OPEN message
• Full routing tables are not required for each
  virtual router
• No changes to UPDATE messages
• No optical transit as all wavelengths are owned
  by peer
• Uses ARP proxy for routers on different subnets
• Wade Hong Objects concept
• Uses an external box (or process) to setup
  optical cross connects
• SSH is used to query source router of AS path to
  destination router
• Each optical cross connect is treated as an
  object with names given by AS path
• Recursive queries are made to objects to discover
  optical path, reserve and setup
• NEXT_HOP at source router is modified through
  SSH
• End result is a direct peer and intermediate ASs
  disappear
• Requires all devices to be on same subnet

41
Target Market for OBGP

• University research and community networks who
  are deploying condominium fiber networks who want
  to exchange traffic between members of the
  community but who want to maintain customer
  control of the network at the edge and avoid
  recreating the need for aggregating traffic via
  traditional mechanisms
• E.g. Ottawa fiber build, Peel County, I-wire,
  SURAnet, G-Wire, CENIC DCP, SURFnet, etc etc
• Next generation fiber companies who are building
  condominium fiber networks for communities and
  school boards and who want to offer value added
  fiber services but not traditional
  telcommunications service
• E.g. C2C, Universe2u, PF.net, Williams,
  QuebecTel, Videotron, etc
• Next generation collocation facilities to offer
  no-cost peering and wavelength routing
• Metromedia, Equinix, LINX, PF.net, LayerOne,
  Westin, PAIX, Above.com, Colo.com, etc etc
• Over 500 Ixs and carrier hotels worldwide