Masaki Hirabaru and Yasuhiro Koyama masaki,koyamanict'go'jp

1
e-VLBI Science over High-Performance Networks
APEC-TEL APGrid Workshop September 6, 2005

• Masaki Hirabaru and Yasuhiro Koyamamasaki,koyama
  _at_nict.go.jp

2
Radio Telescopes
Shanghai25m
NICT Kashima Space Center 34m
Perks 64m (right)Australia Telescope National
Facility
Onsala Space Observatory 20m (left)
MIT Haystack 18m
3
VLBI Applications

• Geophysics and Plate Tectonics

Kashima-Kauai Baseline Length
-63.5 ? 0.5 mm/year
Fairbanks
Fairbanks-Kauai Baseline Length
-46.1 ? 0.3 mm/year
Kashima
Kauai
Kashima-Fairbanks Baseline Length
1.3 ? 0.5 mm/year
4
VLBI Applications (2)

• Radio Astronomy High Resolution Imaging,
  Astro-dynamics
• Reference Frame Celestial / Terrestrial
  Reference Frame
• Earth Orientation Parameters, Dynamics of Earths
  Inner Core

Halca(Muses-B)
Radio Telescope Satellite Halca and its images
Earth Orientation Parameters
NGC4261
5
VLBI (Very Long Baseline Interferometry)

• e-VLBI geographically distributed observation,
  interconnecting radio antennas over the world

ASTRONOMY
GEODESY

• Gigabit / real-time VLBI multi-gigabit rate
  sampling

6
VLBI System Transitions
2nd Generation
3rd Generation
1st Generation
2002 PC-based System Hard-disk Storage Software
Correlator e-VLBI over Internet
1983 Open-Reel Tape Hardware Correlator
1990 Cassette Tape Hardware Correlator e-VLBI
over ATM
7
Recent e-VLBI System Developments K5 by NICT
ADS1000 (1024Msample/sec 1ch 1bit or 2bits)
PC-VSI Board (Supports VSI-H specifications)
Correlator other DAS
VSI
VSI
Internet
VSI-ERTP/RTCP
IP-VLBI Board (16Msample/chsec, 4ch, 8bits)
ADS2000 (64Msample/chsec, 16ch, 1bit or 2bits)
PC Data Acquisition Correlation
8
Motivations

• MIT Haystack NICT Kashima e-VLBI Experiment on
  August 27, 2003 to measure UT1-UTC in 24 hours
• 41.54 GB NICT ? MIT 107 Mbps (50 mins) 41.54 GB
  MIT ? NICT 44.6 Mbps (120 mins)
• RTT 220 ms, UDP throughput 300-400 MbpsHowever
  TCP 6-8 Mbps (per session, tuned)
• BBFTP with 5 x 10 TCP sessions to gain performance

• HUT NICT Kashima Gigabit VLBI Experiment
• - RTT 325 ms, UDP throughput 70 MbpsHowever
  TCP 2 Mbps (as is), 10 Mbps (tuned)
• - Netants (5 TCP sessions with ftp stream restart
  extension)

There was bandwidth available but we could not
utilize.
9
VLBI - Characteristics

• Observing Bandwidth ? Data rate
• ? (Precision of Time Delay)-1
• ? (SNR)1/2
• Wave Length / Baseline Length ? Angular
  Resolution
• Baseline Length ? (EOP Precision)-1

Faster Data Rate Higher Sensitivity
Longer Distance Better Resolution
10
Long Distant Rover Control
(at least) 7 minutes one way delay
Image
Command
Earth
Mars
When operator saw collision, it was too late.
11
Long-Distance End-to-End Congestion Control
Overflow
B
A
C
Sender(JP)
Receiver(US)
Merge (Bottleneck) AB gt C
Feedback
200ms round trip delay
BWDP Amount of data sent but not yet
acknowledged 64Kbps x 200ms 1600B 1 Packet
1Gbps x 200ms 25MB 16700 Packets
12
ExampleHow much speed can we get?
High-Speed Backbone
100M
1G
1G
Receiver
Sender
L2/L3SW
Delay at light speed 100ms
13
Analyzing Advanced TCP Dynamic Behavior in a Real
Network(Example From Tokyo to Indianapolis at
1G bps with HighSpeed TCP)
Throughput
RTT
Window Sizes
Packet Losses
The data was obtained during e-VLBI demonstration
at Internet2 Member Meeting in October 2003.
14
Performance Measurement Platformfor
High-Performance Scientific Data Transfer
Seoul XP
10G
Korea
Kashima
100km
Daejon
bwctl server
JGNII
KOREN
perf server
1G (10G)
Taegu
U of Tokyo
SWITCH
2.5G
Kwangju
e-vlbi server
Busan
Koganei
Tokyo XP / JGN II I-NOC
1G
1G(10G)
250km
GEANT
2.5G SONET
JGN II
10G
APII/JGN II
7,000km
10G
10G
Kitakyushu
1,000km
9,000km
Chicago
MIT Haystack
TransPAC
1G (10G)
1G
Fukuoka
Abilene
2.4G
Pittsburgh
Genkai XP
Fukuoka Japan
10G
4,000km
U. Hawaii
Washington DC
Indianapolis
Los Angeles
Performance Measurement Point Directory
http//e2epi.internet2.edu/pipes/pmp/pmp-dir.html
Locate the problemInternational collaboration to
support for science applications
15
Solutions by Advanced TCPs
How can wee foresee collision (queue overflow)?

• Loss-Based ? AQM (Advanced Queue
  Management)Reno, Scalable, High-Speed, BIC,
• Delay-BasedVegas, FAST
• Explicit Router NotificationECN, XCP, Quick
  Start, SIRENS, MaxNet

16
TCP Performance with Different Queue Sizes
17
Measuring Bottleneck Queue Sizes
Capacity C
Receiver
Sender
lost packet
packet train
measured packet
Queue Size C x (Delaymax Delaymin)
Switch / Router Queue Size Measurement Result
set to 100M for measurement
cross traffic injectedfor measurement
18
Typical Bottleneck Cases
b-1)
Queue100
Queue1000
a)
Router
Switch
VLANs
1Gbps (10G)
Router
Switch
100Mbps(1G)
b-2)
Switch/Router
10G LAN-PHY Ethernet Untag
9.5G WAN-PHY 802.1q Tag
19
e-VLBI Demonstration in JGN II Osaka (Jan. 2005)
Dr. Koyama
10G
Osaka
Abilene(10G)
Tokyo
1G
WAS
1G
CPU x8
CHI
1G/2.5G
1G x4
1G x4
MITHaystack
1G/10G
1,2
3,4
NICTKashima
JGN II Intl (10G)
Raid Disks
5,6
4 Apple G5 Server machines (8 CPUs in Total)
7,8
TCP parameters were tuned for the path.
Raid Disks

• Software Cross Correlation 240Mbps per
  station

• e-VLBI data transfer achieved 700Mbps from
  Haystack to Osaka 900Mbps from Kashima to
  Osaka

20
VLBI Antenna Locations in North-East Asia
Shintotsukawa 3.8m
Nanshan (Urumqi) 25m70km from Urumqi
Tomakomai 11m, FTTH (100M)70km from Sapporo
Miyun (Beijing) 50m50km from Beijing
Nobeyama 45mOC48/ATM Galaxy
Mizusawa 10m 20m118km from Sendai
Seoul 20mYonsei U
Usuda 64m, OC48/ATM Galaxy
Daejon 14mTaeduk
Tsukuba 32m, OC48/ATMx2 SuperSINET
2Mbps
Ulsan 20mU Ulsan
Kashima 34m, 1Gx2 JGN II, OC48/ATM Galaxy
Jeju 20mTamna U
Koganei 34m, 1Gx2 JGN II, OC48/ATM Galaxy
Observatory is on CSTNET at 100M
Gifu 11m 3m, OC48/ATMx2 SuperSINET
Iriki 20m
Kagoshima 6mAira 10m
Yamaguchi 32m1G, 75M SINET
Sheshan (Shanghai) 25m30km from Shanghai
2Mbps
Yunnan (Kunming) 3m (40m)10km from Kunming
Ogasawara 20m
Chichijima 10m
Ishigaki 20m
Legend
connected
not yet connected
antenna under construction
21
e-VLBI Data Transfer
Common e-VLBI file transfer
Carry the disk to the nearest stationto put
on-line
Real-time e-VLBI flat-rate live data transfer
Internet
Correlation
Synchronize
22
Summary

• High-performance scientific data transfer faces
  on network issues we need to work out.
• Big science applications like e-VLBI and
  High-Energy Physics need cooperation with network
  and Grid researchers.
• Deployment of performance measurement
  Infrastructure over research networks is on-going
  on world-wide basis.