Global and Regional Climate Modeling

1
Global (and Regional) Climate Modeling
John Drake Computational Climate Dynamics Group
Computer Science and Mathematics Division Oak
Ridge National Laboratory

• Overview
• What is the climate system?
• How is it modeled?
• What are the computational issues?

For Fall Creek Falls Workshop, October 27, 2003
2
Why is DOE Interested in Climate?
3
(No Transcript)
4
Climate Change Doesnt Just Happen

• A Key figure

5
PCM Ensembles of 21st Century Business as Usual

• Results show that the global warming trend since
  the late 1970s is likely to continue through
  much of the twenty-first century.
• Model includes a full range of greenhouse gas
  changes together with sulfate aerosol effects and
  no artificial flux adjustments.
• The pattern of surface warming from 1961-90 to
  2070-99 under BAU scenario shows that warming
  ranges between 1 and 2 C over oceans and is above
  2 C over many land areas, especially in northern
  high latitudes during winter where the warming is
  above 5 C.
• The meridional overturning in the North Atlantic
  is reduced by 20 from 1961-90 to 2070-99
  reduced local vertical mixing causes cooling over
  the mid-latitude North Atlantic.
• Ensemble-averaged precipitation in BAU shows a
  20-40 increase at high latitudes during winter
  and a 10-30 decrease over subtropical dry areas.

6
Simulation of Future Climates
Computing for this simulation was done at DOE's
National Energy Research Scientific Computing
Center (NERSC) at Lawrence Berkeley National
Laboratory, NCAR and Oak Ridge National
Laboratory Center for Computational Sciences
(CCS).   Credits  Animation and data
management Michael Wehner/Lawrence Berkeley
National Laboratory   Parallel Climate
Model Warren Washington, Jerry Meehl, Julie
Arblaster, Tom Bettge and Gary Strand/National
Center for Atmospheric Research   Visualization
software Dean Williams/Lawrence Livermore
National Laboratory
7
PCM Business as Usual and Stabilization Scenarios

• The ensemble-mean temperatures under the BAU and
  STA550 scenarios start to diverge in 2040 but
  become significantly different only after the mid
  2060s.
• Stabilized atmospheric CO2 at 550ppm by 2150 will
  only slow down the warming moderately during the
  21st century but could be large (1.5 C globally
  and 12 C in DJF at northern high-latitudes) by
  the later part of the 22nd century.

Dai, Meehl, Washington, Wigley, Arblaster,
Ensemble Simulation of Twenty-First Century
Climate Changes, BAMS Vol 82, No. 11, November
2001 Dai, Wigley, Meehl and Washington, Effects
of Stabilizing Atmospheric CO2 on Global Climate
in the Next Two Centuries, GRL Vol. 28, No. 23,
December 2001
8
(No Transcript)
9
Why a Community Model?

• NCAR adopted to encourage climate research in
  universities using state of the art models. 1980
• New components of a coupled system. - 1990
• Individual projects unable to adapt to parallel
  architectures -2000
• Pool resources and complete mission critical
  simulations for NSF, DOE and NASA
• CCSM has become the national model
• Annual Breckenridge CCSM Workshop grown to over
  270 researchers

10
DOE SciDAC Workshop on Porting CCSM to the CRAY X1

• Goals
• Identify individuals and organizations engaged in
  porting one or more of the CCSM component models
• Report progress and problems in current CCSM
  vectorization activities.
• Identify gaps or issues in the current efforts.
• Establish lines of communication between the
  different efforts and NCAR software engineers to
  encourage sharing of results and code.
• Begin defining requirements and procedures for
  the adoption ofvector-friendly code in future
  released versions of CCSM.
• Represented NCAR, NASA-Goddard, ORNL, LANL,
  LBNL, Cray, NEC, Fujitsu, CRIEPI
• Held Feb. 2003 in Boulder, also June in
  Breckenridge 2003

11
Coupler ArchitectureTony Craig, Rob Jacob, Brian
Kaufman, Jay Larson, E. Ong

• Issues
• sequencing
• frequency
• distribution
• parallelism
• single or multiple
• executables
• stand alone execution

• MPH3 (multi-processor handshaking) library for
  coupling component models
• CPL6 -- Implemented, Tested, Deployed

Version 1.0 Released November 2002
12
Evolution of Performance of the Community
Atmospheric ModelPat Worley, John Drake

• Hybrid MPI/OpenMP programming paradigm
• Cache friendly chunks, load balance, improved
  algorithms

13
Land Surface Model and River Transport
Model Forrest Hoffman, Mariana Verenstein,
Marcia Branstetter

• Community Land Model
• SciDAC software engineering is focused on the
  interface and reduction of gather/scatters
  communications bottleneck removed
• Rewrite for vectorization and CLM2.2 now complete
• RTM is currently single processor -- designing
  parallel implementation and data structures
• Analysis of runoff in CCSM control simulation.
  Effect on July ocean salinity.

14
POP Ocean ModelP. Jones, J. Dukowicz, J.
Baumgardner, W. Lipscomb

• Software Engineering for POP and CICE
• Design and implementation for the new ocean model
  (HYPOP) and CICE in progress
• Ocean Model Performance
• POP2 new design involves a decomposition of the
  computational domain into blocks that can be
  sized to fit into cache
• On 1/10 degree, SGI (2x), IBM (1.25x), long
  vector gets 50 peak on Fujitsu
• HYPOP Model Development
• Treat purely Lagrangian dynamics of constant-mass
  layers as they inflate and deflate in regions
  intersecting bottom topography
• Pressure gradient is split into a 'baroclinic'
  part that vanishes and a 'barotropic' part that
  does not vanish when the density is uniform
• Comparison of surface height in Lagrangian and
  Eulerian vertical after 400 baroclinic steps

15
Sea Ice ModelJ. Schramm, P. Jones, W. Lipscomb

• Incremental Remapping for Sea Ice and Ocean
  Transport
• Incremental remapping scheme that proved to be
  three times faster than MPDATA, total model
  speedup of about 30 --added to CCSM/CSIM
• Cache and vector optimizations
• CICE3.0 restructered for vector Community Sea
  Ice Model
• Sensitivity analysis and parameter tuning test of
  the CICE code
• Automatic Differentiation (AD)-generated
  derivative code
• Major modeling parameters that control the sea
  ice thickness computation were the ice-albedo
  constants, densities and emissivities of ice and
  snow, and salinity constant
• Parameter tuning experiment with gradient
  information

16
Atmospheric ChemistryP. Cameron-Smith, J.
Taylor, D. Erickson, J.F. Lamarque, S. Walters,
D. Rotman

• Gas-phase chemistry with emissions, deposition,
  transport and photo-chemical reactions for 89
  species.
• Experiments performed with 4x5 degree Fvcore
  ozone concentration at 800hPa for selected
  stations (ppmv)
• Mechanism development with IMPACT
• A)    Small mechanism (TS4), using the ozone
  field it generates for photolysis rates.
• B)     Small mechanism (TS4), using an ozone
  climatology for photolysis rates.
• C)    Full mechanism (TS2), using the ozone field
  it generates for photolysis rates.

Zonal mean Ozone, Ratio A/C
Zonal mean Ozone, Ratio B/C
17
Ocean BiogeochemistryS. Elliot, S. Chu, M.
Maltrud

• Iron Enrichment in the Parallel Ocean Program
• Surface chlorophyll distributions in POP
• for 1996 La Niña and 1997 El Niño

18
Global DMS Flux from the Ocean using POPD.
Erickson, J. Hernandez, M. Maltrud, S. Chu
The global flux of DMS from the ocean to the
atmosphere is shown as an annual mean. The
globally integrated flux of DMS from the ocean to
the atmosphere is 23.8 Tg S yr-1 .
19
Toward Regional Impacts

• Snow pack in Western U.S. reduced by greater than
  50 by mid-century
• Higher likelihood of wintertime flooding along
  Cascades and Sierras
• Increasing model resolution
• Cluster analysis of PCM results

20
Resolution and PrecipitationPhil Duffy
(DJF) precipitation in the California region in 5
simulations, plus observations. The 5 simulations
are CCM3 at T42 (300 km), CCM3 at T85 (150 km) ,
CCM3 at T170 (75 km), CCM3 at T239 (50 km), and
CAM2 with FV dycore at 0.4 x 0.5 deg.
CCM3 extreme precipitation events depend on model
resolution. Here we are using as a measure of
extreme precipitation events the 99th percentile
daily precipitation amount. Increasing
resolution helps the CCM3 reproduce this measure
of extreme daily precipitation events.
21
Subgrid Orography SchemeSteve Ghan, Tim Shippert

• Reproduces orographic signature without
  increasing dynamic resolution
• Realisitic precipitation, snowcover, runoff
• Month of March simulated with CCSM

22
Why Climate Prediction is Compute Limited

• Long time integrations
• Historical validation 1870-2000
• Future scenarios 2000-2200
• Comprehensive, coupled processes
• Models still under development
• Nonlinear feedbacks and sensitivities
• Multi-scale interactions
• Need for ensemble forecasts
• Decision support scenarios

23
Leadership Class Computing

• Will enable
• Additional atmospheric chemistry
• Tropospheric
• Stratospheric
• Interactive land and biogeochemistry
• Comprehensive carbon cycle models
• Increased resolution
• Atm 30 km
• Ocn 1/10 degree
• Lnd 1 km
• Better throughput for coupled models

24
Effect of Earth Simulator onHardware and
Software Issues

• Challenges assumptions
• Capability computing versus capacity computing
• software is the issue
• Any code can be made to run fast on any machine.
  If not, change the algorithm.
• Special purpose processors and vector
  supercomputers have run out of steam
• Price - performance ratio
• Mass market business model.

• Assertions
• Vector versus cache is not the issue
• Effective bandwidth and latency of memory
  subsystem and interconnect are key
• Performance portability among platforms is
  possible
• High percentage of peak indicates a balanced
  system

25
Modeling Resource Requirements

• 2003
• Global Coupled Models Current
• Model years/day 8
• Storage (TB/century) 1

At current scientific complexity a century
requires 12.5 days to simulate. Single researcher
transfers 80Gb/day. Requires 30TB storage for
year.
26
How Many Terabytes of Output?

• PCMDI Archive contains gt3600 years of coupled
  simulation
• IPCC simulations beginning

27
Infrastructure Challenges

• Integration with existing tools
• Extension of existing tools for multiple
  component climate system models
• Speed of transfers
• Parallelism I/O(NetCDF) and analysis
• Metadata and Grid databases
• Analysis of ensembles
• Advanced statistical methods for analysis of
  coupled systems.

UKMO Hadley(2100)
Current (VEMAP)
CCC(2100)
28
Teams Using Multiple Centers

• DOE Supports Analysis and Archiving of Climate
  Data at PCMDI.

GFDL
LANL
LANL
PCMDI
NCAR
ORNL-CCS
NCDC
NASA DAO
NERSC
Core Universities
29
(No Transcript)
30
The End
31
Goals of the Consortium

• Performance portability for the CCSM
• Open software design process
• Layered software architecture to insulate
  modeling
• Readiness for global to regional climate change
  simulations
• High fidelity ocean and ice models
• Extension of atmospheric chemistry capability
• Development of biogeochemical, terrestrial and
  hydrological aspects of the CCSM
• Towards comprehensive coupled climate simulations
  for study of decadal to century climate change.

32
Tools