CA*net 4 International Grid Testbed

1
CAnet 4 International Grid Testbed

• Wade Hong
• Carleton University
• May 27, 2003

2
The Proposal

• To develop and deploy a persistent experimental
  infrastructure network testbed, in collaboration
  with international participants, to advance the
  realization and utilization of increasingly high
  bandwidth-delay product networks, specifically as
  offered bandwith approaches 10 Gbps and beyond.

3
Motivation

• primary motivation was to provide an experimental
  infrastructure for the developers of the user
  control interfaces and frameworks for the CAnet
  4 Lightpath Cross-Connect devices to test
  end-to-end lightpath provisioning
• an equally important and complementary purpose
  for the International Grid Testbed is to provide
  a sandbox for investigating the effective
  utilization of the established end-to-end long
  distance high bandwidth lightpaths

4
Background

• ATLAS Canada Lightpath bulk data transfer trial
  over high speed long distance networks at the
  iGrid 2002 conference
• Yotta Yotta high bandwidth utilization trial at
  CANARIE 8th Advanced Networks Workshop
• became evident that to continue with the research
  initiated, persistent network resources and
  infrastructure was necessary.

5
RE Networks for Next Generation Science

• some computing challenges
• data intensiveness of experiments
• complexity of the data
• global extent of the collaborations
• emerging technologies
• grid computing
• experimental infrastructure networks
• circuit switched, hybrid networks, e2e lightpaths

6
The ATLAS Experiment

• largest collaboration ever undertaken in the
  physical sciences, with 2000 physicists
  participating from more than 150 institutions and
  laboratories in 34 countries
• main physics goals Higgs Boson discovery
• data volumes rising to the order of Exabytes
  (1018 Bytes) within a decade

7
The ATLAS Experiment
Canada
Canada
8
Atlas Canada Lightpath Data Transfer Trial

• Canadian HEP participated at iGrid 2002 in
  Amsterdam last fall
• Core Team
• Wade Hong, Corrie Kost, Steve McDonald, Bryan
  Caron
• Demonstrated a manually provisioned e2e
  lightpath
• longest single hop network
• Transfered 1TB of ATLAS MC data generated in
  Canada from TRIUMF to CERN
• Tested alpha 10GbE technology and channel bonding
• Established a new benchmark for high performance
  disk to disk throughput over a high bandwidth
  delay product network

9
e2e Lightpaths

• core design principle of CAnet 4
• dedicated point to point pipes
• ultimately give control of lightpath creation,
  teardown and routing to the end user
• hence, Customer Empowered Networks
• provide a flexible infrastructure for emerging
  intensive grid applications
• inherent QoS, security, and high performance

10
e2e Lightpath

• e2e lightpaths manually provisioned today
• frameworks to enable provisioning of e2e
  lightpaths being developed under CAnet 4
  Directed Research Program

11
High Bandwidth Delay Product e2e lightpaths

• can create high bandwidth long distance e2e
  lightpaths
• problem is how to effectively utilize bandwidth
• TCP is not well suited
• bulk data transfers
• OS kernels, disk subsystems, network interfaces,
  TCP/IP stack must be optimized in order to
  achieve extremely high throughput

12
The Network
13
TRIUMF Local Loop
only one SM fiber pair available between TRIUMF
and the CAnet 4 ONS 15454 at the BCnet POP
14
CERN Local Loop

• receive host located at the CERN IXP

15
The End Points
TRIUMF
CERN
16
We are live continent to continent!

• e2e lightpath up and running Sept 20 2145 CET
• iGrid 2002 starts Sept 23

traceroute to cern-10g (192.168.2.2), 30 hops
max, 38 byte packets 1 cern-10g (192.168.2.2)
161.780 ms 161.760 ms 161.754 ms
17
Results

• transfer rates from TRIUMF to CERN using the
  normal routed path yielded 3.4 Mbps with ftp
• clearly not high performance over the routed path
  through UBC campus
• transferred 1TB in under 3 hrs using bbftp
  (IN2P3) and Tsunami protocol (Indiana University)
  over e2e lightpath
• sustained transfers of over 1.2 Gbps disk to
  memory
• sustained transfers of 825 Mbps disk to disk
• created longest known single hop network (16K
  kms)

18
Sunday Nite Summaries
19
Comparative Results
Tool Transferred Average Max Avg
wuftp 100 MbE 600 MB 3.4 Mbps
wuftp 10 GbE 6442 MB 71 Mbps
iperf 275 MB 940 Mbps 1136 Mbps
pftp (single stream) 600MB 532 Mbps
bbftp (10 streams) 1.4 TB 666 Mbps 737 Mbps (13)
Tsunami - disk to disk 0.5 TB 700 Mbps 825 Mbps
Tsunami - disk to memory 12 GB gt 1GBps
Sept 26 2002
20
What Next?

• continue testing 10GbE
• the next version of the Intel 10GbE cards and
  other emerging 10GbE NICs
• emerging 10GbE switches
• bond multiple GbEs across CAnet 4
• aggregate multiple sources to fill a 10 Gbps pipe
  across CAnet 4
• drive up to 10 Gbps from a single host

21
CAnet 4 International Grid Testbed

• Initially, a few Canadian sites
• UVic, TRIUMF, UofA, UofT, Carleton U, CANARIE,
  UdeM
• CERN
• future sites
• UIC/OMNInet, Ultralight

22
Participants

• principal participants
• Carleton U
• UofA
• TRIUMF
• UVic
• collaboratorative participants
• CERN
• UIC
• SURFnet
• Indiana University
• ORANs - Bcnet, Netera, ORION, RISQ

23
CAnet 4 International Grid Testbed
24
General Objectives

• create a facility where the user controlled
  interfaces and frameworks for the CAnet 4
  Lightpath Cross-Connect Devices can be tested and
  demonstrated safely in a real world environment,
  following thorough in-vitro'' testing.
• develop and adapt grid applications which are
  designed to interact with a LightPath Grid
  Service which treats networks and network
  elements as grid resources which can be reserved,
  concatenated, consumed and released.
• in parallel with grid enabling applications, tune
  applications for performance over a single hop
  end-to-end lightpath or a routed multi-hop
  end-to-end lightpath.
• characterize the performance of bulk data
  transfer over an end-to-end lightpath.

25
General Objectives

• investigate and test protocols for high speed
  long distance optical networks to effectively
  utilize the available bandwidth
• investigate and test emerging technologies and
  its impact - 10 GbE, RDMA/IP, FC/IP, serial SCSI,
  HyperSCSI over long distance ethernet, etc.
• collaborate with the EU ESTA project which is
  developing 10 GbE equipment with CERN, industrial
  and academic partners
• collaborate with other international efforts such
  as DataTag, EU DataGrid, CERN OpenLab,
  Ultralight, etc
• test interoperability of emerging 10 GbE
  networking equipment

26
Specific Projects

• projects that have emerged and planned for the
  testbed include
• Bulk Data Transfer
• Grid Canada Testbed
• ATLAS Remote Real Time Processing Farms
• ATLAS Forward Calorimeter TestBeam Streaming
• s2io collaboration
• RDMA Trials
• more projects will emerge over the lifetime of
  the testbed

27
Bulk Data Transfer

• continue work initiated with the iGrid 2002 trial
• attempt to understand bottlenecks limiting
  effective use of high bandwidth delay networks
  for data intensive transfers
• with the newer beta Intel 10GbE NICs, it is
  expected that we would obtain 4 times the
  peformance of the alpha NICs
• continue working with hybrid protocols such as
  Tsunami
• investigate emerging bulk transfer protocols
• explore Web100 and Net100 to further understand
  TCP and UDP performance by instrumenting the IP
  stack
• investigate zerocopy and IPv6

28
Grid Canada Testbed Over End-to-End Lightpaths

• Grid Canada testbed has been established across
  Canada primarily running ATLAS and Babar Monte
  Carlo generation using the Globus Toolkit
• performance issues have been identified with
  internal campus problems - it is proposed to
  investigate performance of the testbed with sites
  meshed with end-to-end lightpaths

29
ATLAS Remote Real Time Processing Farms

• collaborative effort to investigate and exploit
  long distance, high bandwidth, Trans-Atlantic
  links between CERN and Canada with particular
  emphasis on remote real time processing farms for
  the ATLAS experiment
• similar efforts are underway between CERN and
  European sites using a traditional packet routed
  IP network, however, as part of the IGT, an
  end-to-end lightpath would be established between
  CERN and Canadian sites
• measurement station developed at CERN installed
  in Canada, at the University of Alberta
• measurement and optimization of data transfers
  using TCP and other protocols, data sources and
  sinks would be high end PCs supporting GE and 10
  GE NICs

30
ATLAS Remote Real Time Processing Farms

• testing real applications, for example, remote
  processing of data from the ATLAS test beam
• investigate passively monitoring 10 GbE e2e
  lightpath with OC192mon
• initial planned activity period os 2 years

31
ATLAS Forward Calorimeter TestBeam Data Streaming

• ATLAS Canada will be involved in an FCAL
  testbeam, generating a large amount of data to be
  sent back to participating sites - Carleton U,
  UofT
• estimated data rate of 30 Gbytes/day
• explore scheduling an e2e lightpath to stream the
  data back to Canada

32
s2io Collaboration

• test s2ios new TCP assist 10GbE NICs over a high
  speed long distance e2e lightpath
• s2io will expose all tunable parameters through
  APIs - it will be very interesting to experiment
  with and characterize these parameters to
  understand performance and fully utilize the
  offered bandwidth
• further experiments to be developed
  collaboratively

33
RDMA Trials

• s2io next generation 10 GbE NICs will support
  RDMA/IP by mid 2004, an interesting demo for
  iGrid 2004 would be data intensive transfers over
  a 10Gbps e2e lightpath
• many other vendors like Extreme Networks looking
  at RDMA/IP

34
A private railroad car is not an acquired taste.
One takes to it immediately.