HENP Networks and Grids for Global VOs: from Vision to Reality

1

• HENP Networks and Grids for Global
  VOs from Vision to Reality

Harvey B. Newman California Institute
of TechnologyGNEW2004March 15, 2004
2
ICFA and Global Networks for HENP

• National and International Networks, with
  sufficient (rapidly increasing) capacity and
  seamless end-to-end capability, are essential for
• The daily conduct of collaborative work in both
  experiment and theory
• Detector development construction on a
  global scale
• Grid systems supporting analysis involving
  physicists in all world regions
• The conception, design and implementation of
  next generation facilities as global networks
• Collaborations on this scale would never have
  been attempted, if they could not rely on
  excellent networks

3
The Challenges of Next Generation Science in the
Information Age
Petabytes of complex data explored and analyzed
by 1000s of globally dispersed scientists, in
hundreds of teams

• Flagship Applications
• High Energy Nuclear Physics, AstroPhysics Sky
  Surveys TByte to PByte block transfers at
  1-10 Gbps
• eVLBI Many real time data streams at 1-10 Gbps
• BioInformatics, Clinical Imaging GByte images
  on demand
• HEP Data Example
• From Petabytes in 2004, 100 Petabytes by 2008,
  to 1 Exabyte by 2013-5.
• Provide results with rapid turnaround,
  coordinating large but limited computing and
  data handling resources,over networks of varying
  capability in different world regions
• Advanced integrated applications, such as Data
  Grids, rely on seamless operation of our LANs
  and WANs
• With reliable, quantifiable high performance

4
Four LHC Experiments The
Petabyte to Exabyte Challenge

• ATLAS, CMS, ALICE, LHCBHiggs New particles
  Quark-Gluon Plasma CP Violation

6000 Physicists Engineers 60 Countries
250 Institutions
Tens of PB 2008 To 1 EB by 2015 Hundreds
of TFlops To PetaFlops
5
Large Hadron Collider (LHC) CERN, Geneva 2007
Start

• pp ?s 14 TeV L1034 cm-2 s-1
• 27 km Tunnel in Switzerland France

CMS
TOTEM
pp, general purpose HI
First Beams April 2007 Physics Runs from
Summer 2007
ALICE HI
LHCb B-physics
ATLAS
Atlas
6
LHC Higgs Decay into 4 muons (Tracker only)
1000X LEP Data Rate
109 events/sec, selectivity 1 in 1013 (1 person
in a thousand world populations)
7
LHC Data Grid Hierarchy
CERN/Outside Resource Ratio 12Tier0/(?
Tier1)/(? Tier2) 111
PByte/sec
100-1500 MBytes/sec
Online System
Experiment
CERN Center PBs of Disk Tape Robot
Tier 0 1
Tier 1
10 Gbps
FNAL Center
IN2P3 Center
INFN Center
RAL Center
2.5-10 Gbps
Tier 2
2.5-10 Gbps
Tier 3
Institute
Institute
Institute
Institute
Tens of Petabytes by 2007-8.An Exabyte 5-7
Years later.
Physics data cache
0.1 to 10 Gbps
Tier 4
Workstations
Emerging Vision A Richly Structured, Global
Dynamic System
8
ICFA Standing Committee on Interregional
Connectivity (SCIC)

• Created by ICFA in July 1998 in Vancouver
• CHARGE
• Make recommendations to ICFA concerning the
  connectivity between the Americas, Asia and
  Europe
• As part of the process of developing
  theserecommendations, the committee should
• Monitor traffic
• Keep track of technology developments
• Periodically review forecasts of future
  bandwidth needs, and
• Provide early warning of potential problems
• Representatives Major labs, ECFA, ACFA, North
  and South American Physics Community

9
SCIC in 2003-2004http//cern.ch/icfa-scic

• WGs formed in March 2002 Monitoring, Digital
  Divide, Advanced Technologies, Requirements
• Strong Focus on the Digital Divide Continues
• Progress in Monitoring
• A World Survey of Natl and Intl Networks
  Optical Net Initiatives
• Presentations Demos at gt 40 Meetings and
  Workshops
• E.g., Internet2, TERENA, AMPATH, APAN, CHEP2003,
  SC2003, Trieste, Telecom World 2003, WSIS/RSIS,
  GLORIAD LaunchDigital Divide and HEPGrid
  Workshop February 16-20 in Rio
• HENP increasingly visible to governments
• Through Network advances (records), Grid
  developments, Work on the Digital Divide and
  issues of Global Collaboration (Rio Workshop
  the WSIS Process)
• A Striking Picture is Emerging of Remarkable
  Progress, and a Deepening Digital Divide Among
  Nations

10
SCIC in 2003-2004 http//cern.ch/icfa-scic

• Strong Focus on the Digital Divide Since
  2002Three 2004 Reports Presented to ICFA Feb.
  13
• Main Report Networking for HENP H. Newman
  et al.
• Includes Brief Updates on Monitoring, the Digital
  Divide and Advanced Technologies
• A World Network Overview (with 27 Appendices)
  Status and Plans for the Next Few Years
  of National and Regional Networks, and
  Optical Network Initiatives
• Monitoring Working Group Report L.
  Cottrell
• Digital Divide in Russia V.
  Ilyin
• Also See the 2003 SCIC Reports of the
  Advanced Technologies and Digital Divide Working
  Groups

11
ICFA Report Networks for HENPGeneral
Conclusions (1)

• Bandwidth Usage Continues to Grow by 80-100 Per
  Year
• Current generation of 2.5-10 Gbps backbones and
  major Intl links used by HENP arrived in the
  last 2 Years USEuropeJapanKorea
• Capability Increased from 4 to several hundred
  times, i.e. much faster than Moores Law
• This is a direct result of the continued
  precipitous fall of network prices for 2.5 or
  10 Gbps links in these regions
• Technological progress may drive BW higher, unit
  price lower
• More wavelengths on a fiber Cheap, widespread
  Gbit Ethernet
• Grids may accelerate this growth, and the demand
  for seamless high performance
• Some regions are moving to owned or leased dark
  fiber
• The rapid rate of progress is confined mostly to
  the US, Europe, Japan , Korea, and the major
  TransAtlantic and Pacific routes
• This may worsen the problem of the Digital Divide

12
History of Bandwidth Usage One Large Network
One Large Research Site
ESnet Accepted Traffic 1/90 1/04Exponential
Growth Since 92Annual Rate Increased from 1.7
to 2.0X Per Year In the Last 5 Years
SLAC Traffic 300 Mbps ESnet LimitGrowth in
Steps 10X/4 YearsProjected 2 Terabits/s by
2014
13
Internet Growth in the World At Large
Amsterdam Internet Exchange Point Example 75-100
Growth Per Year
5 MinuteMax
20 G
Avg
10 G
Some Growth SpurtsTypically In Summer-Fall
The Rate of HENP Network Usage Growth (100 Per
Year) is Similar to the World at Large
14
Bandwidth Growth of Intl HENP Networks (US-CERN
Example)

• Rate of Progress gtgt Moores Law. (US-CERN
  Example)
• 9.6 kbps Analog (1985)
• 64-256 kbps Digital (1989 - 1994)
  X 7 27
• 1.5 Mbps Shared (1990-3 IBM)
  X 160
• 2 -4 Mbps (1996-1998) X
  200-400
• 12-20 Mbps (1999-2000)
  X 1.2k-2k
• 155-310 Mbps (2001-2)
  X 16k 32k
• 622 Mbps (2002-3)
  X 65k
• 2.5 Gbps ? (2003-4)
  X 250k
• 10 Gbps ? (2005)
  X 1M
• A factor of 1M over a period of 1985-2005 (a
  factor of 5k during 1995-2005)
• HENP has become a leading applications driver,
  and also a co-developer of global networks

15
Pan-European Multi-Gigabit Backbone (33
Countries)February 2004
Note 10 Gbps Connections to Poland, Czech
Republic, Hungary
Planning Underway for GEANT2 (GN2),
Multi-Lambda Backbone, to Start In 2005
16
Core Capacity on Western European NRENs 2001-2003
10G
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
1G
100M
Log Scale
TERENA Compendiumwww.terena.nl
15 European NRENs have made a step up to 1, 2.5
or 10 Gbps core capacity in the last 3 years
17
SuperSINET in JAPAN Updated Map Oct. 2003

SuperSINET 10 Gbps
Intl Circuit 5-10 Gbps

Domestic Circuit 30 100 Mbps

• SuperSINET
• 10 Gbps IP Tagged VPNs
• Additional 1 GbE Inter-University
  Waves For HEP
• 4 X 2.5 Gb to NY 10 GbE Peerings to
  Abilene, ESnet and GEANT

18
SURFNet5 in the Netherlands Fully Optical 10G IP
Network

• Fully dual stack IPv4 IPv6
• 65 of Customer Base Connects with Gigabit
  Ethernet

19
Germany 2003, 2004, 2005

• GWIN Connects 550 Universities, Labs, Other
  Institutions

GWIN Q4/04 Plan
XWIN Q4/05(Dark Fiber Option)
GWIN Q4/03
20

ESnet in 2003 10 Gbps and 2.5 Gbps Backbone
Coming Into ServiceLinks to US HENP Labs 622 at
Mbps Evolution Not Sufficient
21
Abilene - Upgrade Completed!
22
PROGRESS in SE Europe (Sk, Pl, Cz, Hu, )
1660 km of Dark Fiber CWDM Links, up to 112 km.
1 to 4 Gbps (GbE) August 2002 First NREN in
Europe to establish Intl GbE Dark Fiber Link, to
AustriaApril 2003 to Czech Republic. Planning
10 Gbps Backbone dark fiber link to Poland this
year.
23
(No Transcript)
24
Romania Improved to 34 to 155 Mbps in 2003
GEANT-Bucharest Link Improved 155 to 622 Mbps.
Note Inter-City Links Were Only 2 to 6 Mbps in
2002

GEANT connection
Timisoara
RoEduNet2004
Plans for IntercityDark Fiber Backbone
25
Australia (AARnet) SXTransport Project in 2004

• Connect Major Australian Universities to 10 Gbps
  Backbone
• Two 10 Gbps Research Links to the US
• Aarnet/USLIC Collaboration on Net RD Starting
  Soon

Connect Telescopesin Australia, Hawaii (Muana
Kea)
26
GLORIAD Global Optical Ring (US-Russia-China)

• Little Gloriad (OC3) Launched January 12
  to OC192

ITER Distributed Ops.Fusion-HEP Cooperation
Also Important for Intra-Russia
ConnectivityEducation and Outreach
27
National Lambda Rail (NLR)
Transition beginning now to optical,
multi-wavelength Community owned or leased fiber
networks for RE

• NLR
• Coming Up Now Initially 4 10G Wavelengths
• Full Footprint by 4Q04
• Internet2 HOPI Initiative (w/HEP)
• To 40 10G Waves in Future

28
18 State Dark Fiber InitiativesIn the U.S. (As
of 3/04) California (CALREN), Colorado
(FRGP/BRAN)Connecticut Educ. Network,Florida
Lambda Rail, Indiana (I-LIGHT), Illinois
(I-WIRE), Md./DC/No. Virginia (MAX),Michigan,
Minnesota, NY New England (NEREN), N.
Carolina (NC LambdaRail), Ohio (Third Frontier
Net) Oregon, Rhode Island (OSHEAN), SURA
Crossroads (SE U.S.), Texas,Utah, Wisconsin
The Move to Dark Fiber is Spreading
FiberCO
29
CAnet4 (Canada)Two 10G Waves Vancouver- Halifax

• Interconnects Regional Nets at 10G
• Acts as Parallel Discipline-Oriented Nets 650
  Sites
• Connects to US Nets at Seattle, Chicago and NYC
• Third Natl Lambda Later in 2004
• User Controlled Light Path Software(UCLP) for
  LambdaGrids

30

?
?
31
SURFNet6 in the Netherlands 3000 km of Owned
Dark Fiber
40M Euro ProjectScheduled Start
Mid-2005Support Hybrid Grids
32
HEP is Learning How to Use Gbps Networks Fully
Factor of 500 Gain in Max. Sustained TCP
Thruput in 4 Years, On Some USTransoceanic
Routes

• 9/01 105 Mbps 30 Streams SLAC-IN2P3 102
  Mbps 1 Stream CIT-CERN
• 5/20/02 450-600 Mbps SLAC-Manchester on OC12
  with 100 Streams
• 6/1/02 290 Mbps Chicago-CERN One Stream on
  OC12
• 9/02 850, 1350, 1900 Mbps Chicago-CERN
  1,2,3 GbE Streams, 2.5G Link
• 11/02 LSR 930 Mbps in 1 Stream
  California-CERN, and California-AMS
• ? FAST TCP 9.4 Gbps in 10 Flows
  California-Chicago
• 2/03 LSR 2.38 Gbps in 1 Stream
  California-Geneva (99 Link Use)
• 5/03 LSR 0.94 Gbps IPv6 in 1 Stream
  Chicago- Geneva
• TW SC2003 LSR 5.65 Gbps (IPv4), 4.0 Gbps
  (IPv6) GVA-PHX (11 kkm)
• 3/04 LSR 6.25 Gbps (IPv4) in 8 Streams
  LA-CERN

33
FAST TCP Baltimore/Sunnyvale
88

• Fast convergence to equilibrium
• RTT estimation fine-grain timer
• Delay monitoring in equilibrium
• Pacing reducing burstiness

10G
90
9G

• Measurements 11/02
• Std Packet Size
• Utilization averaged over gt 1hr
• 4000 km Path

90
Average utilization
92
8.6 Gbps 21.6 TB in 6 Hours
95
Fair SharingFast Recovery
1 flow 2 flows 7 flows
9 flows 10 flows
34
Transatlantic Ultraspeed TCP TransfersThroughput
Achieved X50 in 2 years

• Terabyte Transfers by the Caltech-CERN Team
• Across Abilene (Internet2) Chicago-LA, Sharing
  with normal network traffic
• Oct 15 5.64 Gbps IPv4 Palexpo-L.A. (10.9 kkm)
• Peaceful Coexistence with a Joint
  Internet2- Telecom World VRVS Videoconference
• Nov 18 4.00 Gbps IPv6 Geneva-Phoenix (11.5
  kkm)
• March 2004 6.25 Gbps in 8 Streams (S2IO
  Interfaces)

Juniper, HPLevel(3)Telehouse
Nov 19 23 Gbps TCP Caltech, SLAC, CERN, LANL,
UvA, Manchester
35
HENP Major Links Bandwidth Roadmap in Gbps
Continuing the Trend 1000 Times Bandwidth
Growth Per DecadeA new DOE Science Network
Roadmap Compatible
36
HENP Lambda GridsFibers for Physics

• Problem Extract Small Data Subsets of 1 to 100
  Terabytes from 1 to 1000 Petabyte Data Stores
• Survivability of the HENP Global Grid System,
  with hundreds of such transactions per day
  (circa 2007-8)requires that each transaction be
  completed in a relatively short time.
• Example Take 800 secs to complete the
  transaction. Then
• Transaction Size (TB) Net
  Throughput (Gbps)
• 1
  10
• 10
  100
• 100
  1000 (Capacity of
  Fiber
  Today)
• Summary Providing Switching of 10 Gbps
  wavelengthswithin 2-4 years and Terabit
  Switching within 5-8 years would enable 10G
  Lambda Grids with Terabyte transactions,to
  fully realize the discovery potential of major
  HENP programs, as well as other data-intensive
  research.

37
ICFA Report Networks for HENPGeneral
Conclusions (2)

• Reliable high End-to-end Performance of networked
  applications such as large file transfers and
  Data Grids is required. Achieving this requires
• Removing local, last mile, and natl and intl
  bottlenecks end-to-end, whether technical or
  political in origin.While National and
  International backbones have reached 2.5 to 10
  Gbps speeds in many countries, the bandwidths
  across borders, the countrysideor the city may
  be much less. This problem is very widespread in
  our community, with examples stretching from
  China to South America to the Northeastern U.S.
  Root causes for this vary, from lack of local
  infrastructure to unfavorable pricing policies.
  
• Upgrading campus infrastructures. These are
  still not designed to support Gbps data transfers
  in most of HEP centers. One reason for the
  under-utilization of National and International
  backbones, is the lack of bandwidth to groups of
  end-users inside the campus.
• End-to-end monitoring extending to all regions
  serving our community. A coherent approach to
  monitoring that allows physicists throughout our
  community to extract clear, unambiguous and
  inclusive information is a prerequisite for
  this.

38
ICFA Report Networks for HENPGeneral
Conclusions (3)

• We must Remove Firewall Bottlenecks Also at
  some Major HEP Labs
• Firewall systems are so far behind the needs that
  they wont match the data flow of Grid
  applications. The maximum throughput measured
  across available products is limited to a few X
  100 Mbps !
• It is urgent to address this issue by designing
  new architectures that eliminate/alleviate the
  need for conventional firewalls. For example,
  Point-to-point provisioned high-speed circuits as
  proposed by emerging Light Path technologies
  could remove the bottleneck.
• With endpoint authentication as in Grid AAA
  systems, for example, the point-to-point paths
  are private, intrusion resistant circuits, so
  they should be able to bypass firewalls if the
  endpoint sites trust each other.

39
HENP Data Grids, and Now Services-Oriented Grids

• The original Computational and Data Grid concepts
  are largely stateless, open systems known to
  be scalable
• Analogous to the Web
• The classical Grid architecture had a number of
  implicit assumptions
• The ability to locate and schedule suitable
  resources, within a tolerably short time (i.e.
  resource richness)
• Short transactions with relatively simple
  failure modes
• HENP Grids are Data Intensive
  Resource-Constrained
• 1000s of users competing for resources at dozens
  of sites
• Resource usage governed by local and global
  policies
• Long transactions some long queues
• HENP ?Stateful, End-to-end Monitored and Tracked
  Paradigm
• Adopted in OGSA Now WS Resource Framework

40
The Move to OGSA and then Managed Integration
Systems
App-specific Services
Integrated Systems
Stateful Managed
Open Grid Services Arch
Web ServicesResrc Framwk
Web services
Increased functionality, standardization
GGF OGSI, ( OASIS, W3C) Multiple
implementations, including Globus Toolkit
Globus Toolkit
X.509, LDAP, FTP,
Defacto standards GGF GridFTP, GSI
Custom solutions
Time
41
Managing Global Systems Dynamic Scalable
Services Architecture
MonALISA http//monalisa.cacr.caltech.edu

• Station Server Services-engines at sites host
  manyDynamic Services
• Scales to thousands of service-Instances
• Servers autodiscover and interconnect dynamically
  to form a robust fabric
• Autonomous agents

42
Grid Analysis Environment
CLARENS Web Services Architecture

• Analysis Clients talk standard protocols to the
  Grid Services Web Server, a.k.a. the Clarens
  data/services portal, with a simple Web service
  API
• The secure Clarens portal hides the complexity of
  the Grid Services from the client
• Key features Global Scheduler, Catalogs,
  Monitoring, and Grid-wide Execution
  serviceClarens servers forma Global Peer to
  peer Network

43
UltraLight Collaborationhttp//ultralight.caltec
h.edu

• Caltech, UF, FIU, UMich, SLAC,FNAL,MIT/Haysta
  ck,CERN, UERJ(Rio), NLR, CENIC,
  UCAID,Translight, UKLight, Netherlight, UvA,
  UCLondon, KEK, Taiwan
• Cisco, Level(3)

• Integrated hybrid experimental network,
  leveraging Transatlantic RD network
  partnerships packet-switched dynamic optical
  paths
• 10 GbE across US and the Atlantic NLR, DataTAG,
  TransLight, NetherLight, UKLight, etc.
  Extensions to Japan, Taiwan, Brazil
• End-to-end monitoring Realtime tracking and
  optimization Dynamic bandwidth provisioning
• Agent-based services spanning all layers of the
  system, from the optical cross-connects to the
  applications.

44
GLIF Global Lambda Integrated Facility
GLIF is a World Scale Lambda based Lab for
Application Middleware development, where Grid
applications ride on dynamically configured
networks based on optical wavelengths ... GLIF
will use the Lambda network to support data
transport for the most demanding e-Science
applications, concurrent with the normal best
effort Internet for commodity traffic.
10 Gbps Wavelengths For RE Network Development
Are Proliferating, Across Continents and Oceans
45
SCIC Report 2004 The Digital Divide

• As the pace of network advances continues to
  accelerate, the gap between the economically
  favored regions and the rest of the world is in
  danger of widening.
• We must therefore work to Close the Digital
  Divide
• To make Physicists from All World Regions Full
  Partners in Their Experiments and in the Process
  of Discovery
• This is essential for the health of our global
  experimental collaborations, our plans for
  future projects, and our field.

46
SCIC Monitoring WG PingER (Also
IEPM-BW)
Monitoring Sites

• Measurements from
• 33 monitors in 12 countries
• 850 remote hosts in 100 Countries 3700
  monitor-remote site pairs
• Measurements go back to 95
• Reports on link reliability, quality
• Aggregation in affinity groups
• Countries monitored
• Contain 78 of world population
• 99 of Internet users

Affinity Groups (Countries)
Anglo America (2), Latin America (14), Europe
(24), S.E. Europe (9), Africa (21), Mid East
(7), Caucasus (3), Central Asia (8), Russia
includes Belarus Ukraine (3), S. Asia (7),
China (1)  and Australasia (2).
47
SCIC Monitoring WG -
Throughput Improvements
1995-2004
Bandwidth of TCP lt MSS/(RTTSqrt(Loss)) (1)
60 annual improvement Factor 100/10 yr
Some Regions 5-10 Years Behind
SE Europe, Russia, Central Asia May be Catching
Up (Slowly) India Ever-Farther Behind
Progress but Digital Divide is Mostly Maintained
(1) Matthis et al., Computer Communication Review
27(3), July 1997
48
?
?
?
?
?
?
?
?
?
?
?
?
49
Inhomogeneous Bandwidth Distributioin Latin
America. CAESAR Report (6/02)



Intl Links0.071 Gbps Used 4,236 Gbps Capacity
to Latin America
Need to Pay Attentionto End-point connections
50
DAI State of the World
51
DAI State of the World
52
Digital Access Index Top Ten
? Pakistan 0.03 0.54
0.41 0.2 0.01 0.24
53
DAI State of the World
54
Work on the Digital Dividefrom Several
Perspectives

• Work on Policies and/or Pricing pk, in, br, cn,
  SE Europe,
• Share Information Comparative Performance and BW
  Pricing
• Exploit Model Cases e.g. Poland, Slovakia, Czech
  Republic
• Find Ways to work with vendors, NRENs, and/or
  Govts
• Inter-Regional Projects
• GLORIAD, Russia-China-US Optical Ring
• South America CHEPREO (US-Brazil) EU ALICE
  Project
• Virtual SILK Highway Project (DESY) FSU
  satellite links
• Help with Modernizing the Infrastructure
• Design, Commissioning, Development
• Provide Tools for Effective Use Monitoring,
  Collaboration
• Participate in Standards Development Open Tools
• Advanced TCP stacks Grid systems
• Workshops and Tutorials/Training Sessions
• Example ICFA Digital Divide and HEPGrid Workshop
  in Rio
• Raise General Awareness of the Problem
  Approaches to Solutions
• WSIS/RSIS Trieste Workshop

55
Dai Davies SERENATE Workshop Feb. 2003
www.serenate.org
Ratio to 114 If Include Turkey, MaltaCorrelated
with the Number of Competing Vendors
56
Virtual Silk Highway
The SILK Highway Countriesin Central Asia
the Caucasus

• Hub Earth Station at DESY with access to the
  European NRENs and the Internet via GEANT
• Providing International Internet access directly
• National Earth Station at each Partner site
• Operated by DESY, providing international access
• Individual uplinks, common down-link, using DVB
• Currently 4 Mbps Up 12 Down for 350k/Year

Note Satellite Links are a Boon to the Region,
but Unit Costs are Very High compared to
Fiber.There is a Continued Need for Fiber
Infrastructure
57
A Series of Strategic Studies into the Future of
Research and Education Networking in Europe
From Summary and Conclusions by D.O. Williams,
CERN

• A significant divide exists in Europe the
  worst countries Macedonia, B-H, Albania, etc.
  are 1000s of times worse off than the best.
• Also many of the 10 new EU members are 5X worse
  off than the 15 present members.
• If there is one single technical lesson from
  SERENATE it is that transmission is moving from
  the electrical domain to optical.
• When theres good competition users can still
  lease traditional communications services
  (bandwidth) on an annual basis.
• But Without enough competition prices go through
  the roof.
• The more you look at underlying costs the more
  you see the need for users to get access to
  fibre.
• Our best advice has to be If youre in a mess,
  you must get access to fibre.

See www. serenate.org
58
Dark Fiber in Eastern Europe Poland PIONIER
Network
2650 km Fiber Connecting16 MANs 5200 km and
21 MANs by 2005

• Support
• Computational GridsDomain-Specific Grids
• Digital Libraries
• Interactive TV
• Addl Fibers for e-Regional Initiatives

59
CESNET Dark Fiber Case Study (Czech Republic)
Within Reach
2513 km Leased Fibers (Since 1999)
Case Study ResultWavelength ServiceVs. Fiber
Lease Cost Savings of 50-70 Over 4 Yearsfor
Long 2.5G or 10G LinksFor Example 4 X 10G
Over 300 km for 14k Euro/Month
60
Romania Future Prospect of 1 to 2.5 G
Wavelength Services or Dark Fiber from RDS
Romania Data System(RDS) January 2004
61
HEPGRID and Digital Divide Workshop UERJ, Rio de
Janeiro, Feb. 16-20 2004
Theme Global Collaborations, Grids and Their
Relationship to the Digital Divide For the past
three years the SCIC has focused on understanding
and seeking the means of reducing or eliminating
the Digital Divide, and proposed to ICFA that
these issues, as they affect our field of High
Energy Physics, be brought to our community for
discussion. This led to ICFAs approval, in July
2003, of the Digital Divide and HEP Grid
Workshop.  http//www.uerj.br/lishep2004
NEWS Bulletin ONE TWOWELCOME BULLETIN
General InformationRegistrationTravel
Information Hotel Registration Participant List
How to Get UERJ/Hotel Computer Accounts Useful
Phone Numbers Program Contact us Secretariat
Chairmen

• Tutorials
• C
• Grid Technologies
• Grid-Enabled Analysis
• Networks
• Collaborative Systems

SPONSORS
All Sessions and Tutorials AvailableLive Via VRVS
 CLAF  CNPQ FAPERJ       
UERJ
62
World Summit on the Information Society(WSIS)
Geneva 12/2003 and Tunis in 2005

• The UN General Assembly adopted in 2001 a
  resolution endorsing the organization of the
  World Summit on the Information Society (WSIS),
  under UN Secretary-General, Kofi Annan, with the
  ITU and host governments taking the lead role in
  its preparation.
• GOAL To Create an Information Society A Common
  Definition was adoptedin the Tokyo Declaration
  of January 2003 One in which highly developed
  ICT networks, equitable
• and ubiquitous access to information,
  appropriate content in accessible formats and
  effective communication can help people achieve
  their potential
• Kofi Annan Challenged the Scientific Community to
  Help (3/03)
• CERN and ICFA SCIC have been quite active in the
  WSIS in Geneva (12/2003)

63
Role of Science in the Information Society.
Palexpo, Geneva 2004

• CERN SIS Forum, and
• CERN/Caltech Online Stand
• Visitors
• Kofi Annan, UN Secy General
• John H. Marburger, Science Adviser to US
  President
• Ion Iliescu, President of Romania and Dan Nica,
  Minister of ICT
• Jean-Paul Hubert, Ambassador of Canada in
  Switzerland
• Carlo Lamprecht, Pres. of Economic Dept. of
  Canton de Geneva

64
Role of Sciences in Information Society.
Palexpo, Geneva 2003

• Demos at the CERN/Caltech RSIS Online Stand
• Distance diagnosis and surgery using Robots with
  haptic feedback (Geneva-Canada)
• World Scale multisite multi-protocol
  videoconference with VRVS (Europe-US-Asia-Sout
  h America)
• Music Grid live performance with bands at St.
  Johns, Canada and the Music Conservatory of
  Geneva on stage
• Advanced network and Grid-enabled analysis
  demonstrations
• Monitoring very large scale Grid farms with
  MonALISA

65
VRVS on Windows
KEK (JP)
VRVS (Version 3) Meeting in 8 Time Zones
Caltech (US)
RAL (UK)
Brazil
CERN (CH)
AMPATH (US)
Pakistan
SLAC (US)
Canada
28k registered hosts Users in 103 Countries 2-3X
Growth/Year
AMPATH (US)
66
Networks, Grids and HENP

• Network backbones and major links used by HENP
  experiments are advancing rapidly
• To the 10 G range in lt 2 years much faster than
  Moores Law
• Continuing a trend a factor 1000 improvement
  per decade a new HENP (and DOE) Roadmap
• HENP is learning to use long distance 10 Gbps
  networks effectively
• 2003-2004 Developments to 6 Gbps flows over 11
  kkm
• Transition to community-owned and operated RE
  networks is beginning (ca, nl, us, pl, cz,
  sk ) or considered (de, ro, )
• Removing Regional, Last Mile, Local Bottlenecks
  and Compromises in Network Quality are now
  On the critical path, in all world regions
• Digital Divide Network improvements are
  especially neededin SE Europe, Much of Asia,
  So. America and Africa
• Work on These Issues in Concert with Internet2,
  Terena, APAN, AMPATH DataTAG, the Grid projects
  the GGF

67
Some Extra Slides Follow
68
ICFA and International Networking

• ICFA Statement on Communications in Intl
  HEPCollaborations of October 17, 1996
  See http//www.fnal.gov/directorate/icfa/icfa_comm
  unicaes.html
• ICFA urges that all countries and institutions
  wishing to participate even more effectively and
  fully in international HEP Collaborations should
• Review their operating methods to ensure they
  are fully adapted to remote participation
• Strive to provide the necessary communications
  facilities and adequate international bandwidth

69
NREN Core Network Size (Mbps-km)http//www.teren
a.nl/compendium/2002
100M
Logarithmic Scale
Leading
Nl
10M
Fi
Cz
Advanced
Hu
Es
1M
Ch
In Transition
It
Pl
Gr
100k
Ir
Lagging
10k
Ro
1k
Ukr
100
70
Network Readiness IndexHow Ready to Use Modern
ICTs ?
Market
(US)
Environment
Political/Regulatory
(SG)
Infrastructure
(IC)
(US)
Individual Readiness
(FI)
NetworkReadinessIndex
Readiness
Business Readiness
(US)
Govt Readiness
(SG)
(SG)
(FI)
Individual Usage
(KR)
Usage
Business Usage
(DE)
( ) Which Country is First
(FI)
Govt Usage
(FI)
From the 2002-2003 Global Information Technology
Report. See http//www.weforum.org
71
Throughput vs Net Readiness Index

• NRI from Center for Intl Development, Harvard
  http//www.cid.harvard.edu/cr/pdf/gitrr2002_ch02.p
  df

NRI Tops Finland 5.92 US 5.79 Singapore 5.74 Swe
den 5.58 Iceland 5.51 Canada
5.44 UK 5.35 Denmark 5.33 Taiwan 5.31 Germany 5.
29 Netherlnd 5.28 Israel
5.22 Switzland 5.18 Korea 5.10
AR focus
Internet for all focus
Improved correlation (0.21 ?0.41) by Using
derived throughput MSS / (RTT sqrt(loss))
fit an exponential Interesting Outliers
Slovakia, Hungary, Portugal. Lithuania
72
UltraLight
http//ultralight.caltech.edu

• Serving the major LHC experiments developments
  broadly applicable to other data-intensive
  programs
• Hybrid packet-switched and circuit-switched,
  dynamically managed optical network
• Global services for system management
• Trans-US wavelength riding on NLR
  LA-SNV-CHI-JAX
• Leveraging advanced research production
  networks
• USLIC/DataTAG, SURFnet/NLlight, UKLight,
  Abilene, CAnet4
• Dark fiber to CIT, SLAC, FNAL, UMich Florida
  Light Rail
• Intercontl extensions Rio de Janeiro, Tokyo,
  Taiwan
• Flagship Applications with a diverse traffic mix
• HENP TByte to PByte block data transfers at
  1-10 Gbps
• eVLBI Real time data streams at 1 to several
  Gbps

73
User Requirements

• In ALL countries and in ALL disciplines
  researchers are eagerly anticipating improved
  networking tools. There is no divide on the
  demand-side. Sciences, such as particle physics,
  which make heavy use of advanced networking, must
  help to break down any divide on the supply-side,
  or else declare themselves elitist and irrelevant
  to researchers in essentially all developing
  countries.

74
Connectivity pricing and competition

• In some locations the price of connectivity is
  (really) unreasonably high
• Linked (obviously) to how competitive the market
  is
• Strong competition on routes between various key
  European cities, and between major national
  centres
• Less competition ? effectively none as you move
  to countries with de facto monopoly or simply to
  parts of countries where operators see little
  reason to invest.
• While some expensive routes are where you would
  expect, others are much more surprising (at first
  sight), like Canterbury and Lancaster (UK) and
  parts of Brittany (F)

75
Understanding transmission costs and DIY solutions

• Own trenching only makes sense in very special
  cases. Say 1-30 km. Even then look for partners.
• Maybe useful (as a threat) over longer distances
  in countries with crazy pricing
• Now possible to lease (short- or long-term)
  fibres on many routes in Europe 0.5 to 2
  KEuro/km Typ.
• Transmission costs jump at 200 km below which
  you can operate with Nothing In Line (NIL),
  above which you need amplifiers and 800 km
  above which you need signal regenerators
• Possibly leading to some new approaches in
  GEANT-2 implementation

76
Jensen, ICTP
Typ. 0-7 bpsPer Person
77
Limited by many external systemic
factors Electricity Import
Duties Education Trade restrictions
Progress in Africa ?
Jensen, ICTP
78
(No Transcript)
79
(No Transcript)
80
DAI State of the World
81
DAI State of the World
82
RENATER3 in France2.5 G Backbone (Since 10/2002)

• 650 Sites, Most Connect Through MANs
• lt 50 Direct Connected
• GEANT Connection to 10G in March 2004
• IN2P3 InvestigatingDark Fiber to CERN

83
APAN
84
AMPATH Miami-RNP Brazil
AMPATH Miami-REUNA (Chile)
Note CMS-Tier1 (Brazil) ALMA (Chile)
85
Brazil RNP in Early 2004
86
NREN Core Network Size (Mbps-km)Last
Yearhttp//www.terena.nl/compendium/2002
87
Relative Cost of Connectivity Compared with
Number of Suppliers
88
Multipliers for Differing Circuit Speeds
89
PingER derived throughput vs. the ITU Digital
Access Index (DAI) for PingER countries monitored
from the U.S.
90
NRNs Bandwidth in Latin America
Source CAESAR - Review of Develop-ments in Latin
America
91
Undersea Optical Infrastructure
The total aggregate bandwidth capacity Latin
America and Caribbean region is estimated at
4,236 GB
92
It is generally accepted that once a technology
is perceived as having broad utilitarian value,
price as a of per capita income, is the main
driver of penetration
C. Casasus, CUDI (Mexico) CANARIE
93
Penetration of telecommunications in low income
countries is further inhibited by at least 3
factors

• Low income per capita
• Less competition. Higher prices from monopolies
• Fewer applications. No broad utilitarian value

C. Casasus, CUDI (Mexico) CANARIE
94
Income vs. penetration given the price of a
technology
Per capita Income
High
Low income countries
Penetration
Low
Low
High
C. Casasus, CUDI (Mexico) CANARIE
95
Telecom monopolies have even higher prices in low
income countries

• Fewer entrants. Less competition
• No unbundling
• Price cap regulation creates cross subsidies
  between costumer groups.
• Large customers (inelastic) subsidize small
  costumers (elastic). High bandwidth services are
  very expensive
• Inefficient ROW regulation
• Inefficient spectrum policies

C. Casasus, CUDI (Mexico) CANARIE
96
Virtual Silk Highway Project Managed by
DESY and Partners

• Virtual SILK Highway Project (from 11/01) NATO
  ( 2.5 M) and Partners ( 1.1M)
• Satellite Links to South Caucasus and
  Central Asia (8 Countries)
• In 2001-2 (pre-SILK) BW 64-512 kbps
• Proposed VSAT to get 10-50 X BW for same cost
• See www.silkproject.org
• Partners CISCO, DESY. GEANT, UNDP, US
  State Dept., Worldbank, UC London, Univ.
  Groenigen

Note NATO Science for Peace Program
97
KREONet in Korea2.5-5G Backbone (from 2003)

• 230 Organizations Linked to 11 Regional Centers
• SuperSIReN 10G Net for Grid RD and Applications
• Link to KOREN with 47 Institutions, 2.5-10G
  Four-City Ring

98
Dynamic Distributed Services Architecture (DDSA)

• Station Server Services-engines at sites host
  Dynamic Services
• Auto-discovering, Collaborative
• Scalable to thousands of service-Instances
• Servers interconnect dynamically form a robust
  fabric
• Service Agents Goal-Oriented, Autonomous,
  Adaptive
• Adaptable to Web services many platforms
  working environments (also mobile)

See http//monalisa.cacr.caltech.edu
http//diamonds.cacr.caltech.edu
Caltech/UPB (Romania)/NUST (Pakistan)
Collaboration