ICFA Standing Committee on Interregional Connectivity SCIC

1

• ICFA Standing Committee on
  Interregional Connectivity (SCIC)

Harvey B. Newman California
Institute of TechnologyICFA Seminar,
CERNOctober 9, 2002
2
ICFA and Global Networks for HENP

• National and International Networks, with
  sufficient capacity and capability, are essential
  for
• The daily conduct of collaborative work in both
  experiment and theory
• Detector development, construction on a global
  scale Data analysis involving physicists from
  all world regions
• The conception, design and implementation of
  next generation facilities as global networks
• The formation of worldwide collaborations
• Collaborations on this scale would never have
  been attempted, if they could not rely on
  excellent networks

3
ICFA and International Networking

• ICFA Statement on Communications in Intl
  HEPCollaborations of October 17, 1996
  See http//www.fnal.gov/directorate/icfa/icfa_comm
  unicaes.html
• ICFA urges that all countries and institutions
  wishing to participate even more effectively and
  fully in international HEP Collaborations should
• Review their operating methods to ensure they
  are fully adapted to remote participation
• Strive to provide the necessary communications
  facilities and adequate international bandwidth

4
ICFA Network Task Force 1998 Bandwidth
Requirements Projection (Mbps)
NTF
1001000 X Bandwidth Increase Foreseen for
1998-2005 See the ICFA-NTF Requirements
Report http//l3www.cern.ch/newman/icfareq98.htm
l
5
ICFA Standing Committee on Interregional
Connectivity (SCIC)

• Created by ICFA in July 1998 in Vancouver
  Following ICFA-NTF
• CHARGE
• Make recommendations to ICFA concerning the
  connectivity between the Americas, Asia and
  Europe
• As part of the process of developing
  theserecommendations, the committee should
• Monitor traffic
• Keep track of technology developments
• Periodically review forecasts of future
  bandwidth needs, and
• Provide early warning of potential problems
• Create subcommittees when necessary to meet the
  charge
• The chair of the committee should report to ICFA
  once peryear, at its joint meeting with
  laboratory directors (Feb. 2003)
• Representatives Major labs, ECFA, ACFA, NA
  Users, S. America

6
ICFA-SCIC Core Membership

• Representatives from major HEP laboratories
• Manuel Delfino (CERN)
• to W. Von Rueden Michael Ernst (DESY)
  Matthias Kasemann (FNAL) Yukio Karita (KEK)
  Richard Mount (SLAC)
• User Representatives Richard Hughes-Jones
  (UK) Harvey Newman (USA)
• Dean Karlen (Canada)
• For Russia Slava Ilyin (MSU)

• ECFA representatives
• Frederico Ruggieri (INFN Frascati),Denis Linglin
  (IN2P3, Lyon)
• ACFA representatives
• Rongsheng Xu (IHEP Beijing)
• HwanBae Park (Kyungpook Natl University)
• For South America Sergio F. Novaes
  (University of Sao Paulo)

7
LHC Data Grid Hierarchy
CERN/Outside Resource Ratio 12Tier0/(?
Tier1)/(? Tier2) 111
PByte/sec
100-400 MBytes/sec
Online System
Experiment
CERN 700k SI95 1 PB Disk Tape Robot
Tier 0 1
HPSS
Tier 1
2.5 Gbits/sec
FNAL 200k SI95 600 TB
IN2P3 Center
INFN Center
RAL Center
2.5 Gbps
Tier 2
2.5 Gbps
Tier 3
Institute 0.25TIPS
Institute
Institute
Institute
Tens of Petabytes by 2007-8.An Exabyte within 5
Years later.
100 - 1000 Mbits/sec
Physics data cache
Tier 4
Workstations
8
Transatlantic Net WG (HN, L. Price)
Bandwidth Requirements (2001)

Installed BW. Maximum Link Occupancy 50
Assumed See http//gate.hep.anl.gov/lprice/TAN
9
History One large Research Site
Much of the TrafficSLAC ? IN2P3/RAL/INFNvia
ESnetFranceAbileneCERN
Projections 0.5 to 24 Tbps by 2012
10
ICFA SCIC Meetings and Topics

• Focus on the Digital Divide This Year
• Identification of problem areas work on ways to
  improve
• Network Status and Upgrade Plans in Each Country
• Performance (Throughput) Evolution in Each
  Country, and Transatlantic
• Performance Monitoring World-Overview (Les
  Cottrell, IEPM Project)
• Specific Technical Topics (Examples)
• Bulk transfer, QoS, Collaborative Systems,
  Security, VOIP
• Preparation of Reports to ICFA (Lab Directors
  Meetings)
• Last Report World Network Status and Outlook -
  Feb. 2002
• Next Report Digital Divide, Monitoring,
  Advanced Technologies, and Requirements Evolution
  Feb. 2003
• Six Meetings in 2002 Next at KEK In December
  13.

11
SCIC Sub-Committees

• Web Page http//cern.ch/ICFA-SCIC/
• Monitoring Les Cottrell (http//www.slac.stanfor
  d.edu/xorg/icfa/scic-netmon) With Richard
  Hughes-Jones (Manchester), Sergio Novaes (Sao
  Paolo) Sergei Berezhnev (RUHEP), Fukuko Yuasa
  (KEK), DanielDavids (CERN), Sylvain Ravot
  (Caltech), Shawn McKee (Michigan)
• Advanced Technologies Richard Hughes-Jones with
  Vladimir Korenkov (JINR, Dubna), Olivier
  Martin(CERN), Harvey Newman
• End-to-end Connectivity Richard Mount (SLAC)
• With Michael Ernst, Denis Linglin, Alexandre
  Sztajnberg (Rio, Brazil)
• The Digital Divide Alberto Santoro (Rio, Brazil)
  
• With Slava Ilyin, Yukio Karita, David O. Williams
• Also Dongchul Son (Korea), Hafeez Hoorani
  (Pakistan), Sunanda Banerjee (India), Vicky
  White (FNAL)
• Key Requirements Harvey Newman
• Also Charlie Young (SLAC)

12
Network Progress andIssues for Major Experiments

• Major RE Networks have weathered the economic
  storm
• Backbones major links advancing rapidly to 10
  Gbps range
• Gbps end-to-end throughput data flows have
  been tested will be in production soon (in 1-2
  years)
• Network advances are changing the view of the
  nets roles
• Progress to and beyond 10 Gbps within next few
  years
• Likely to have a profound impact on the
  experiments Computing Models, and bandwidth
  requirements
• More dynamic view dynamic path provisioningGB
  to TB data transactions
• Net RD Driven by Advanced integrated
  applications, such as Data Grids, that rely on
  seamless LAN and WAN operation
• With reliable, quantifiable (monitored), high
  performance
• All of the above will further open the Digital
  Divide chasm. We need to take action

13
Signs of the Times UncertaintyBut No Change in
Outlook

• While Key Providers are in Bankruptcy
• KPNQwest, Teleglobe, Global Crossing, FLAG
  Worldcom
• AR Nets and Major Links Have Continued Their
  Advance Switched Vendors where needed
• E.g. T-Systems (Deutsche Telecom) for US-CERN
• Telecom Market Outlook
• Good pricing continues, in open competition
  situations
• MCI/Worldcom networks continue to operate20 M
  customers in US UK academic research network
• Level3 market share advance perhaps also Tyco,
  GX
• Link Donations ! NY-Amsterdam OC192 (Tyco to
  IEEAF)
• Overall Strong Outlook in RE Nets
• Regional Progress, and Visions E.g. IWIRE
  CALREN 1 Gbps to Every Californian by 2010

14
ICFA SCIC RE Backbone and International Link
Progress

• GEANT Pan-European Backbone (http//www.dante.net/
  geant)
• Now interconnects gt31 countries many trunks 2.5
  and 10 Gbps
• UK SuperJANET Core at 10 Gbps
• 2.5 Gbps NY-London, with 622 Mbps to ESnet and
  Abilene
• France (IN2P3) 2.5 Gbps RENATER backbone by
  mid-October
• Lyon-CERN Link being Upgraded to 1 Gbps Ethernet
• Proposal for dark fiber to CERN by end 2003
• SuperSINET (Japan) 10 Gbps IP and 10 Gbps
  Wavelength Core
• Tokyo to NY Links Now 2 X 2.5 Gbps Need to get
  to Starlight
• CAnet4 (Canada) Interconnect customer-owned
  dark fiber nets across Canada at 10 Gbps,
  started July 2002
• Lambda-Grids by 2004-5
• GWIN (Germany) 2.5 Gbps Core Connect to US at 2
  X 2.5 GbpsSupport for SILK Project Satellite
  links to FSU Republics
• Russia Now 155 Mbps links to Moscow
• Moscow-Starlight Link to 155 Mbps (US NSF
  Russia Support)
• Moscow-Stockholm Link connect to GEANT at 155
  Mbps

15
RE Backbone and Intl Link Progress

• Abilene (Internet2) Upgrade from 2.5 to 10 Gbps
  Underway
• Encourage high throughput use for targeted
  applications
• ESNET Upgrade to 10 Gbps Now
• US-CERN
• to 622 Mbps in August Move to STARLIGHT
• 2.5G Research Triangle from August 2002
  STARLIGHT-CERN-NL to 10G in 2003
• SLAC IN2P3 (BaBar)
• To 100 Mbps throughput over US-CERN and Renater
  links
• 600 Mbps Throughput is BaBar Target for this Year
  (with ESnet and IN2P3 Link Upgrades)
• FNAL
• ESnet Link Upgrade to 622 Mbps
• Plans for dark fiber to STARLIGHT, planned for
  this Fall
• NY-Amsterdam Donation from Tyco, September 2002
  Arranged by IEEAF 622 Gbps10 Gbps Research
  Wavelength

16
National Research Networks in Japan

• SuperSINET
• Started operation January 4, 2002
• Support for 5 important areas
• HEP, Genetics, Nano-Technology,Space/Astronomy,
  GRIDs
• Provides 10 ?s
• 10 Gbps IP connection
• Direct intersite GbE links
• 7 Univ. Connected Add 3 More this Month
• HEPnet-J
• Reconstructed with MPLS-VPN in SuperSINET
• Proposal Two TransPacific 2.5 Gbps
  Wavelengths, and Japan-CERN Grid Testbed by
  2003

NIFS
IP
Nagoya U
NIG
WDM path
IP router
Nagoya
Osaka
Osaka U
Tokyo
Kyoto U
NII Hitot.
ICR
Kyoto-U
U Tokyo
ISAS
Internet
IMS
NAO
U-Tokyo
17
National RE Network ExampleGermany DFN
Transatlantic Connectivity 2002

• 2 X OC48 NY-Hamburg and NY-Frankfurt
• Direct Peering to Abilene (US) and Canarie
  (Canada)
• UCAID will adding another 2 OC48s in a Proposed
  Global Terabit Research Network (GTRN)

• FSU Connections via satelliteYerevan, Minsk,
  Almaty, Baikal
• Speeds of 32 - 512 kbps
• SILK Project (2002) NATO funding
• Links to Caucasus and Central Asia (8
  Countries)
• In 2001-2 64-512 kbps
• Proposed VSAT for 10-50 X BW NATO State
  Funding

18
Max. Sustained TCP Throughput on Transatlantic
Links

• 8/10/01 105 Mbps reached with 30 Streams
  SLAC-IN2P3
• 9/1/01 102 Mbps in One Stream CIT-CERN
• 11/5/01 125 Mbps in One Stream (modified
  kernel) CIT-CERN
• 1/09/02 190 Mbps for One stream shared on 2
  155 Mbps links
• 3/11/02 120 Mbps Disk-to-Disk with One Stream
  on 155 Mbps link (Chicago-CERN)
• 5/20/02 450-600 Mbps SLAC-Manchester on OC12
  with 100 Streams
• 6/1/02 290 Mbps Chicago-CERN One Stream on
  OC12 (mod. Kernel)
• 9/02 850 Mbps Chicago-CERN One Stream on
  OC48 (mod. Kernet) 1350 (1900)
  Mbps with 2 (3) Streams on 2 (3) GbE Ports

Also see http//www-iepm.slac.stanford.edu/monitor
ing/bulk/ and the Internet2 E2E Initiative
http//www.internet2.edu/e2e
19
US-CERN DataTAG Link Tests with Grid TCP
(S.Ravot et al.) 9/02
TOECPUProtocol
3 Streams3 GbE Ports
3
2
1
1 to 3 Streams1-3 GbE Ports
Next Tests to 10 Gbps 1 GB/sec disk to disk
20
IEPM PingER Deployment
Monitoring Sites

• Measurements from
• 34 monitors in 14 countries
• Over 600 remote hosts
• Over 72 countries
• Over 3300 monitor-remote site pairs
• Measurements go back to Jan-95
• Reports on RTT, loss, reachability, jitter,
  reorders, duplicates
• Countries monitored
• Contain 78 of world population
• 99 of online users of the Internet
• Mainly AR sites

Remote Sites
21
History - Throughput Quality
Improvements from US
80 annual improvement Factor 100/8 yr
Bandwidth of TCP lt MSS/(RTTSqrt(Loss)) (1)
Progress but Digital Divide is Maintained
(1) Macroscopic Behavior of the TCP Congestion
Avoidance Algorithm, Matthis, Semke, Mahdavi,
Ott, Computer Communication Review 27(3), July
1997
22
Work on the Digital DivideSeveral Perspectives

• Identify Help Solve Technical Problems
  Natl, Regional, Last 10/1/0.1 km
• Inter-Regional Proposals (Example Brazil)
• US NSF Proposal (10/2002) possible EU LIS
  Proposal
• Work on Policies and/or Pricing pk, in, br, cn,
  SE Europe,
• E.g. RoEduNet (2-6 Mbps) Pricing not so
  differentfrom US-CERN price in 2002 for a few
  Gbps
• Find Ways to work with vendors, NRENs, and/or
  Govts
• Use Model Cases Installation of new advanced
  fiber infrastructures Convince Neighboring
  Countries
• Poland (to 5k km Fiber) Hungary Slovakia
  Ireland
• Exploit One-off Solutions E.g. extend the SILK
  Project (DESY/FSU satellite links) to a SE
  European site
• Work with other organizations Terena, Internet2,
  AMPATH, IEEAF, UN, etc. to help with technical
  and/or political solns

23
GEANT 155Mbps
Romania 155 Mbps to GEANT and BucharestInter-Ci
ty Links of 2-6 Mbps
Annual Cost gt 1 MEuro
24
Digital Divide Committee
25
(No Transcript)
26
HUNGARY
27
NREN Core Network Size (Mbps-km)http//www.teren
a.nl/compendium/2002
100M
Logarithmic Scale
Leading
Nl
10M
Fi
Cz
Advanced
Hu
Es
1M
Ch
In Transition
It
Pl
Gr
100k
Ir
Lagging
10k
Ro
1k
Ukr
100
28
RNP Brazil (to 20 Mbps)
FIU Miami from So. America (to 80 Mbps)
Note Auger (AG), ALMA (Chile), CMS-Tier1 (Brazil)
29
Digital Divide Activities

• Questionnaire Being Distributed (Discuss At ICFA
  Meeting)
• CERN/IT to Assist with Web Form Online
  Submission
• Plan on Project to Build HENP World Network
  Map Updated and Maintained on a Web Site,
  Backed by Database
• Systematize and Track Needs and Status
• Information Link Bandwidths, Utilization,
  Quality, Pricing, Local Infrastructure, Last
  Mile Problems, Vendors, etc.
• Identify Urgent Cases Focus on Opportunities to
  Help
• First ICFA SCIC Workshop Focus on the Digital
  Divide
• Target Date February 2004 in Rio de Janeiro
  (LISHEP)
• Organization Meeting July 2003
• Plan Statement at the WSIS, Geneva (December
  2003)
• Install and Leave Behind a Good Network
• Then 1 (to 2) Workshops Per Year, at Sites that
  Need Help

30
Digital Divide Sub-Committee Questionnaire
Response Extract
31
Cultivate and promote practical solutions to
delivering scalable, universally available and
equitable access to suitable bandwidth and
necessary network resources in support of
research and education collaborations.
Groningen Carrier Hotel March 2002
http//www.ieeaf.org
32
CA-Tokyo by 1/03
NY-AMS 9/02
(Research)
33
(No Transcript)
34
Global Medical Research Exchange
Initiative Bio-Medicine and Health
Sciences
Global Quilt Initiative GMRE Initiative - 001
Propose Global Research and Education Network for
Physics
35
Networking for HENP

• Networks are vital for global collaboration, and
  the present and future of Our Field But
  Digital Divide
• What ICFA and the HENP Community Can Do
• Spread the message ICFA SCIC is there to help
• Help identify and highlight specific needs (to
  Work On)
• Policy problems Last Mile problems etc.
• Encourage Joint programs such as in DESYs Silk
  project Japanese links to SE Asia and China
  AMPATH to So. America
• NSF LIS Proposals US and EU to South America
• Make direct contacts, arrange discussions with
  govt officials
• ICFA SCIC is prepared to participate
• Help Start, or Get Support for Workshops on
  Networks ( Grids)
• Discuss Create opportunities
• Encourage, help form funded programs
• Help form Regional support training groups
  (requires funding)

36
Some Extra Slides Follow
37
Networks, Grids and HENP

• Next generation 10 Gbps network backbones are
  almost here in the US, Europe and Japan
• First stages arriving, starting now
• Major transoceanic links at 2.5 - 10 Gbps in
  2002-3
• Getting high (reliable Grid) application
  performance across networks means!
• End-to-end monitoring a coherent approach
• Getting high performance (TCP) toolkits in
  users hands
• Working in concert with AMPATH, Internet2,
  Terena DataTAG, the Grid projects and the
  Global Grid Forum
• Network improvements are especially needed in SE
  Europe, So. America SE Asia, and Africa
• Key Examples India, Pakistan, China Brazil
  Romania
• Removing regional, Last mile bottlenecks and
  compromises in network quality are now (in all
  world regions) On the critical path

38
Baseline BW for the US-CERN Link HENP
Transatlantic WG (DOENSF)
Transoceanic NetworkingIntegrated with the
Abilene, TeraGrid, Regional Netsand Continental
NetworkInfrastructuresin US, Europe, Asia,
South America
Baseline evolution typicalof major HENPlinks
2001-2006

• DataTAG 2.5 Gbps Research Link since Summer 2002
• 10 Gbps Research Link by Approx. Mid-2003

39
History Loss Quality
(Cottrell)

• Fewer sites have very poor to dreadful
  performance
• More have good performance (lt 1 Loss)

40
History one Research ISP
100 growth in traffic/yr for last 12
years Continuous upgrades Increase packet size
(bulk throughput apps)
41
Key Network Issues Challenges

• Net Infrastructure Requirements for High
  Throughput
• Packet Loss must be Zero (at and below 10-6)
• I.e. No Commodity networks
• Need to track down uncongested packet loss
• No Local infrastructure bottlenecks
• Multiple Gigabit Ethernet clear paths between
  selected host pairs are needed now
• To 10 Gbps Ethernet paths by 2003 or 2004
• TCP/IP stack configuration and tuning Absolutely
  Required
• Large Windows Possibly Multiple Streams
• New Concepts of Fair Use Must then be Developed
• Careful Router, Server, Client, Interface
  configuration
• Sufficient CPU, I/O and NIC throughput
  sufficient
• End-to-end monitoring and tracking of performance
• Close collaboration with local and regional
  network staffs
• TCP Does Not Scale to the 1-10 Gbps Range

42
iGrid2002 OC192OC48 SetupSeptember 2002
Argonne
Amsterdam
Short Term Donation from Level 3
Starlight-Chicago
CERN
Assisted by Loans Level3 (OC192) and Cisco
(10GbE and 16X1GbE)
43
HENP Major Links BW Roadmap in Gbps (HBN)
Shown at ICHEP2002
44
LHCnet Network Late 2002
GEANT
Switch
IN2P3
WHO
CERN -Geneva
Alcatel 7770 DataTAG (CERN)
Cisco 7606DataTAG (CERN)
Juniper M10 DataTAG(CERN)
Linux PC for Performance tests Monitoring
Cisco 7609 CERN
Optical Mux/Dmux Alcatel 1670
622 Mbps (Prod.)
2.5 Gbps (RD)
Linux PC for Performance tests Monitoring
Optical Mux/DmuxAlcatel 1670
Cisco 7609 Caltech(DoE)
Cisco 7606Caltech(DoE)
Juniper M10 Caltech (DoE)
Alcatel 7770 DataTAG (CERN)
Caltech/DoE PoP StarLight Chicago
Abilene
MREN
ESnet
STARTAP
NASA
Development and tests
45
DataTAG Project
NewYork
ABILENE
STARLIGHT
ESNET
GENEVA
Wave Triangle
CALREN
STAR-TAP

• EU-Solicited Project. CERN, PPARC (UK), Amsterdam
  (NL), and INFN (IT)and US (DOE/NSF UIC, NWU
  and Caltech) partners
• Main Aims
• Ensure maximum interoperability between US and EU
  Grid Projects
• Transatlantic Testbed for advanced network
  research
• 2.5 Gbps Wavelength Triangle 7/02 (10 Gbps
  Triangle in 2003)

46
The Rapid Pace of Network Technology Advances
Continues

• Within the Next One to Two Years
• 10 Gbps Ethernet on Switches and Servers
  LAN/WAN integration at 10 Gbps
• Intel 10 GbE server NICs in Beta Test
• 40 Gbps Wavelengths Being Shown
• NextGen Routers 100 Mpps forwarding engines, 4
  and More 10 Gbps ports per Slot Terabit/sec
  backplanes etc.
• Broadband Wireless Multiple 3G/4G
  alternatives the drive to defeat the last mile
  problem
• 802.11 ab, UWB Mobile CDMA2000/1xEV-DO

47
HENP Lambda GridsFibers for Physics

• Problem Extract Small Data Subsets of 1 to 100
  Terabytes from 1 to 1000 Petabyte Data Stores
• Survivability of the HENP Global Grid System,
  with hundreds of such transactions per day
  (circa 2007)requires that each transaction be
  completed in a relatively short time.
• Example Take 800 secs to complete the
  transaction. Then
• Transaction Size (TB) Net Throughput
  (Gbps) 1
  10
• 10
  100
• 100
  1000 (Capacity of
  Fiber
  Today)
• Summary Providing Switching of 10 Gbps
  wavelengthswithin 3 years and Terabit
  Switching within 5-7 yearswould enable
  Petascale Grids with Terabyte transactions,as
  required to fully realize the discovery potential
  of major HENP programs, as well as other
  data-intensive fields.

48
WSIS-RO Bucharest Nov. 7-9WSIS is 12/03 05
(Geneva Tunis)

• The Regional Ministerial Conference, based on the
  UN/ECE membership, is in preparation of the World
  Summit on the Information Society - Bucharest,
  7-9 November 2002
• The UN General Assembly adopted in 2001 a
  resolution endorsing the organization of the
  World Summit on the Information Society (WSIS),
  to be convened under the patronage of the United
  Nations Secretary-General, Kofi Annan, with the
  International Telecommunication Union taking the
  lead role in its preparation along with
  interested UN organizations and the host
  countries. The Summit will offer a unique
  opportunity for the world community to discuss
  and give shape to the Information Society by
  bringing together governments, international
  organizations, private sector and civil society.
  In preparation for this Summit regional
  conferences are organized to develop specific
  proposals.

49
WSIS-RO Role of New Technologies in Developing
the Information Society

• Tony Hey, UK Core E-Science Program
• Peter Zangli, European Commission
• Hans Hoffmann, CERN Scientific and TT Director
• Fabrizio Gagliardi EDG Project Leader
• William Johnston, Lawrence Berkeley Lab
• Dave Reese or Tom West, CENIC, Pacific and Natl
  Light Rail
• Don Riley or Ed Fontegrossi, IEEAF
• Harvey Newman, Caltech

50
Building Petascale Global GridsImplications for
Society

• Meeting the challenges of Petabyte-to-Exabyte
  Grids, and Gigabit-to-Terabit Networks, will
  transform research in science and engineering
• These developments will create the first truly
  global virtual organizations (GVO)
• If these developments are successful this could
  lead to profound advances in industry, commerce
  and society at large
• By changing the relationship between people
  and persistent information in their daily
  lives
• Within the next five to ten years