GS40 Earth Simulator Project: Background, Scientific Opportunities, and Risks

1
GS40 Earth Simulator Project Background,
Scientific Opportunities, and Risks

• J. J. Hack
• Head, Climate Modeling Section National Center
  for Atmospheric Research
• Boulder, Colorado USA

2
Outline

• Background on the Earth Simulator Project
• put NCAR connection in context
• Provide some firsthand evidence the machine is
  real
• based on visit to Earth Simulator Facility on 11
  April 2002
• based on benchmark tests conducted over the last
  few months
• real applications, not toy kernels
• Scientific opportunities in context of machine
  capabilities
• machine capabilities dwarf anything available in
  the United States
• Dilemma associated with participation
• technical and bureaucratic challenges of
  long-distance partnership
• risks associated with ignoring the scientific
  opportunities

3
Some Important Reality Checks

• I am not a spokesperson for the Earth Simulator
  Project
• NCAR has had no direct involvement with the Earth
  Simulator Project
• most everything I will share is public
  information
• Earth Simulator public presentation materials
  will be used where possible
• see http//www.es.jamstec.go.jp/esrdc/eng/public/p
  ublicconts.html
• all Earth Simulator source material will be
  appropriately identified (ESRDC)
• NCAR not presently partnered with Earth Simulator
  Project
• proposed GS40 partnership exists
• with the Central Research Institute for the
  Electric Power Industry (CRIEPI)
• proposed partnership basis for comments in 20
  April 2002 New York Times
• CRIEPI-NCAR relationship is a longstanding
  collaboration
• formal scientific and technical collaboration for
  more than 10 years
• proposed GS40 partnership a natural outgrowth of
  ongoing collaboration

Scientists from the National Center for
Atmospheric Research in Boulder, Colo., said they
were planning to work with the Japanese earth
simulation center to convert United States
weather modeling codes to work with the new
computer.
4
(No Transcript)
5
The Earth Simulator (ES) is the largest parallel
vector processor in the world that is mainly
dedicated to large-scale simulation studies for
global change. Keiji Tani, JAERI
ICS02 Keynote Address Abstract
6
(No Transcript)
7
(No Transcript)
8
However, some researchers fear that bureaucratic
and budgetary squabbles could handicap their
work. The centers budget is split between
several agencies that have different priorities.
Science, 1 March 2002
9
ES Performance Target is a minimum of 1000 times
numbers above
10
16 GB/s bisection bandwidth

• Other noteworthy facility features
• 700 Tbytes Disk Space 1.6 Pbytes Mass Store

11
Technology
12
Technology
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
11 April 2002
20
11 April 2002
21
11 April 2002
22
11 April 2002
23
11 April 2002
24
11 April 2002
25
System Performance

• Linpack benchmark
• 35.6 TF
• 87.2 efficiency
• problem size n 1,041,216 8.7 Tbytes of memory
• 5.8 hours execution time
• In the context of the Top500
• Earth Simulator Performance gt ? (top 20
  computers in United States)
• Earth Simulator Performance gt ? (all DOE and DOD
  computers)
• all DOE and DOD computers in Top500 27.6 TF

Performance data courtesy of Jack Dongarra
26
System Performance (reported)

• AFES (Atmospheric General Circulation Model For
  Earth Simulator)
• Original Developers
• CCSR (Center for Climate System Research, Univ.
  of Tokyo)
• NIES (National Institute for Environmental
  Studies)
• Redesign and optimization for the Earth Simulator
  funded by NASDA
• optimized at the Earth Simulator Research and
  Development Center
• Example of target application for the Earth
  Simulator
• global spectral model with full physics at
  T1279L96 wavenumber truncation
• F90 compatible using MPI/microtasking for
  parallelism
• implementation allows for coupling w/ oceanic and
  other component models
• Sustained Execution Rate
• 14.5 TF using 320 nodes (1/2 of machine 70.8 of
  peak)
• 7.6 TF using 160 nodes (1/4 of machine 74.2
  of peak)
• Dr. Tetsuya Sato, Director-General, The Earth
  Simulator Center
• This achievement is surprising even to us, the
  persons involved in the project.
• ICS02 Keynote Address Abstract

27
System Performance (estimated)

• NCAR CCM3
• global spectral atmospheric model no
  optimization
• T341 spectral truncation (40 km transform grid)
  using Earth Simulator
• 72 seconds/simulated day (1200 x real time) on
  64 nodes (512 PEs)
• ? 351012 floating point operations ? sustained
  execution rate ? 500GF
• T42 spectral truncation (300 km transform grid)
  using IBM SP (375 Mhz)
• 17 seconds/simulated day (5000 x real time) on 16
  nodes (64 PEs)
• ? 72109 floating point operations ? sustained
  execution rate ? 4.2GF
• projected to be 10GF on new NCAR Power 4 (1350
  Mhz)
• LANL/NCAR POP
• global ocean model, 0.1 eddy-resolving
  resolution
• projected turnaround of ?10 simulated yrs/day on
  128 nodes (1000PEs)
• 3600 x real time
• current turnaround 0.12 simulated yrs/day on 128
  IBM SP nodes (512 PEs)
• 45 x real time

Preliminary evaluation based on CRIEPI GS40
measurements
28
Science Opportunities

• GS40 enables fundamentally new simulation
  capabilities
• efficient exploration of high-resolution
  parameter space
• address science issues pacing systematic biases
  and internal variability
• opportunity to do multi-decadal eddy-resolving
  global ocean modeling
• investigate skill on regional scales (climate
  impacts)
• extensions to the physical climate system
• bio-geochemistry (carbon cycle)
• interactive stratospheric and tropospheric
  chemistry
• exploration and use of ensemble techniques
• enable sophisticated treatments of processes with
  large uncertainties
• Dedicated facility, focused computational
  resource
• Earth Simulator capabilities dwarf anything
  available in US
• CSL will consist of 500 Power 4 processors
  100GF sustained
• ES5000 SX6 processors w/ high-performance
  interconnect 5000GF sustained
• access through Japanese collaborative partnerships

29
IPCC 1995 Climate Model Projection Uncertainty
30
Some Sources of Uncertainty
IPCC Working Group I (2001)
31
Uncertainty due to treatment of clouds
Cess et al. (1990)
32
Signal to Noise Problem Detection and Attribution
IPCC Working Group I (2001)
33
Conceptual Illustration of Natural Variability
Hansen et al. (1993)
34
Natural Variability Ensemble Methodologies
Hansen et al. (1993)
35
How can ensemble techniques be useful?

• Consider dominant modes of variability in the
  climate system
• provides the opportunity to evaluate climate
  sensitivity
• response of the climate system to a specific
  forcing factor
• evaluate modeled response on a hierarchy of time
  scales
• exploit natural forcing factors to test model
  response
• diurnal and seasonal cycles
• El Niño Southern Oscillation (ENSO)
• intraseasonal variability e.g., MJO
• solar variability
• volcanic aerosol loading

36
Testing AGCM Sensitivity
Pacific SST Anomalies and ENSO
Hack (1998)
37
Testing AGCM Sensitivity
OLR Anomalies and ENSO
Hack (1998)
38
Costs and Risks of Participation

• Software engineering challenges
• maintenance of large evolving software
  infrastructure on multiple architectures
• ready access to vector architecture
• Project management
• coordination of experimental design, setup, and
  execution
• data migration
• Enhancement of data storage and data analysis
  capabilities
• online data access
• dedicated data analysis platform
• Access to Earth Simulator Facility
• closed facility on-site presence required (for
  now)
• science challenge of component model development
  on remote facility
• Navigating Japanese funding and management
  infrastructure
• complex governance many administrative
  uncertainties
• unknown allocation, scheduling, and access
  policies/restrictions
• basic management process poorly communicated,
  even in Japan

39
Summary

• GS40 Earth Simulator is the real thing
• shouldnt be a surprise went almost exactly
  according to plan
• clearly the most powerful general-purpose HPC
  system in the world
• may not be a one-of-a-kind system in the longer
  term
• real HPC effort, not repackaging or another
  demonstration project
• example of real vision along with a sense of
  accountability
• dedicated high-performance simulation capability
• production-quality resource focused on a specific
  scientific problem
• paradigm for other areas of socioeconomic or
  scientific importance
• provision of focused resources to accelerate
  progress

40
The End