M' Satoh

1
Development of the Nonhydrostatic Icosahedral
Atmospheric Model at Frontier Research System
for Global Change
M. Satoh H. Tomita K. Goto The Integrated
Modeling Research Program Frontier Research
System for Global Change
Meeting of the Swedish Researchers and the FRSGC
Researchers Yokohama Institute for Earth Science,
8 April, 2003
2
Contents
Nonhydrostatic ICosahedral Atmospheric Model
(NICAM)

• Introduction
• Nonhydrostatic modeling
• Icosahedral grid modeling
• Runs on ES

3
Motivation

• Mission
• Development of a high resolution atmospheric
  global model on the Earth Simulator
• 10 km or less in horizontal, 100 levels in
  vertical
• Can be run with 3.5km mesh on ES
• Global cloud resolving model
• Climate study
• Strategy of development
• Quasi-uniform grid icosahedral grid
• Spectral method becomes inefficient at higher
  resolution.
• Legendre transformation
• Massive data transfer between computer nodes
• The latitude-longitude grid point method has the
  pole problem.
• Severe limitation of time interval by the CFL
  condition.
• Inhomogeneous near the poles.
• Non-hydrostatic equations system a new
  conservative scheme

4
Development procedure
Coupling with Ocean, Land, Ice, Bio models
Computational tuning on ES
2003-
2002
Dynamical Core of Global Non-hydrostatic
Model Icosahedral grid Nonhydrostatic equations
2000-01
Global Cumulus ResolvingModel Dynamical Core
Physics
Global Shallow Water Model Icosahedral or
Conformal cubic grids
Regional Cartesian Nonhydrostatic model New
dynamical scheme
Regional Climate Model with Streched grid
Study of physical processes using regional
nonhydrostatic model
Physical tuning
5
Icosahedral grid model

• 1960-1980
• Williamson(1968), Sadourny et al.(1968)
• Barotropic model
• Masuda and Ohnishi(1986)
• Shallow water model
• 1990-
• Colorad State Univ.(CSU-AGCM)
• AGCM with primitive equation model (hydrostatic)
• Ringler et al.(2000), D.Randall
• DWD(GME)
• NWP operational model since 1999 (Majewski et al.
  2002)
• Max Plank Institute(ICON)
• Under development (Bounaveture),2001-
• FRSGC(NICAM)
• Global cumulus resolving model

6
Collaboration with other institutes, project

• Collaboration with
• CCSR/NIES/Frontier AGCM
• Earth Simulator Center AFES(AGCM for the Earth
  Simulator)
• MRI/NWD NHM (Nonhydrostatic Model)
• NHM2000 (Nonhydrostatic model research group
  in Japan)
• DWD GME, Local Model
• Max Plank Institute ICON
• Colorado State University Geodesic grid model
• Kyousei Project Category 2 cloud physics,
  aerosols
• Univ. of Tokyo Study of dynamics, three
  dimensional turbulence

7
Outline of a nonhydrostatic modeling
Non-hydrostatic modeling

• Characteristics of the non-hydrostatic model
• Dry formulation and results
• Moist formulation
• Squall line experiments

8
Characteristics of the nonhydrostatic model (1)

• A subset of the three-dimensional global
  non-hydrostatic model
• A test frame of new dynamical schemes
• Development of the conservative scheme
• Physics cloud schemes (warm/ice), radiation,
  turbulence
• Study of cloud-radiation interaction and cumulus
  parameterization
• Radiative-convective equilibrium experiments
• Model hierarchy can be used as 1D-vertical,
  2D-horizontal-vertical, and 3D-regional models.

9
Characteristics of the nonhydrostatic model (2)

• Fully compressible non-hydrostatic equations
• Horizontally explicit and vertically implicit
  time integration with time splitting
• The Helmholtz equation is formulated for vertical
  velocity not for pressure
• A switch for a hydrostatic/non-hydrostatic option
  can be introduced.
• Conservation of the domain integrals.
• The finite volume method using flux form
  equations of density, momentum and total energy.
• Tracer advecion
• Third order upwind, or UTOPIA
• Consistency with Continuity
• Exact treatment of moist thermodynamics (Ooyama
  1990, 2001).
• Dependency of latent heat on temperature and
  specific heats of water substance
• Transports of water, momentum, and energy due to
  rain.
• An accurate transport scheme for rain.
• Conservative Semi-Lagrangian scheme with 3rd order

10
Characteristics of the nonhydrostatic model (3)

• Physics (mainly introduced from CCSR/NIES AGCM,
  MRI NHM)
• Warm rain (bulk method), no ice yet
• Turbulence Mellor and Yamada Level 2, 2.5
  Deardorff Smagorinsky
• Surface flux Louis(1982)
• Radiation MSTRN8 (Nakajima et al, 2000)
• Investigation of better physical processes
• Cumulus parameterization, particularly at 10-30km
  resolution
• Interaction between clouds and radiation
• Turbulence comparison with LES
• Radiation, aerosols interaction

11
Dry formulation

• Conservative flux form equations for density R,
  momentum V, and internal energy E

where and
12
Time integration scheme time splitting

• Large time step t, small time step t
• Leap-frog
• or RK2

13
The flux division method (Klemp et al.2000)
14
Small time integration(1)

• Explicit for U and V
• Implicit for R, W, E using
• 1D-Helmholtz eq. for W

a0Hydrostatic option
15
Small time integration(2)

• Integrate for R in the flux form
• Energy correction integrate for total energy in
  the flux form
• where E internal energy, K kinetic energy, and
  G potential energy

16
Density current experiment (Straka et al, 1993)
Initial cold bubble ? ?15K ?x ?z 50m ?t
0.1s
17
Moist formulation with warm rain

• Prognostic variables
• water vapor qv
• cloud water qc
• rain water ql
• total density ?
• momentum V (U, V, W) (?u, ?v, ?w)
• Sensible part of internal energy Ea
• Effects of specific heats of water substance
  are considered

18
Governing equations (Ooyama, 1990,2000)
19
Squall line exp. 2D, ?x1.25km
Cloud water and rain
Precipitation
Water Energy budgets
20
Squall line exp. 3D 100km x 125km x 21km
t150min
qc z7.3km
? z0.1km
qc z1.4km
t200min
21
Outline of an icosahedral grid modeling
Icosahedral grid modeling

• Grid generation
• Advection terms and Coriolis term
• Life cycle of extratropical cyclones experiment
• Held and Suarez experiments

22
Grid Generation Method

• Grid generation
• Each side of icosahedron whose vertices are on a
  sphere is projected onto the sphere. (glevel-0)
• By connecting the mid-points of the geodesic
  arcs, four sub-triangles are generated.
  (glevel-1)
• By iterating this process, a finer grid
  structure is obtained. (glevel-n)
• of gridpoints
• 11 interations are requried to obtain the 5km
  grid interval.

(0) grid division level 0 (1) grid
division level 1
(2) grid division level 2 (3) grid
division level 3
23
level 8 (28km)
level 7 (56km)
level 10 (7km)
level 9 (14km)
24
Advection of momentum and Coriolis terms
Only the advection term is evaluated with the
Cartesian components.
25
Grid Optimization by Spring Dynamics (2)

• Another application of spring grid
• We can construct the clustered grid by tuning the
  spring.

Example of the clustered grid (a) High
resolution hemisphere (b) Low
resolution hemisphere
? For the regional prediction or climate model
26
Model configuration
27
Life Cycle of Extratropical Cyclone Exp.(1)
Polvani and Scott(2002)
glevel 10 ?x7km
glevel 6 ?x112km
glevel 8 ?x28km
28
Life Cycle of Extratropical Cyclone Exp.(2)
glevel 6
glevel 8
glevel 10
29
Held Suarez Dynamical Core Exp.(1)

• Test configuration
• Radiation
• We use a simple radiation as Newtonian Cooling of
  temperature field
• where
• Equilibrium temperature is zonally symmetric as
• where
• Surface fricrion
• Surface friction is imposed in the lower
  atmosphere as a Rayleigh damping
• where

30
Held Suarez Dynamical Core Exp.(4)

• Zonal mean of zonal wind

• glevel-5 (b) glevel-6
  (c) glevel-7
• ?x240km 120km
  60km

GME(DWD) IFS(ECMWF)
ni64(g-level 6) T106
31
Energy spectrum
Comparison with spectral model(2)
glevel-6 vs T159 2?x2p/N240km
N 2?x
N/2 4?x
32
Computational Performance (1)

• Performance on the Earth Simulator
• Earth Simulator
• Massively parallel super-computer
• based on NEC SX-5 architecture.
• 640 computational nodes.
• 8 vector-processors in each of nodes.
• Peak performance of 1CPU 8GFLOPS
• Total peak performance 8x8x640 40TFLOPS

33
Computational Performance (2)

• Scalability of our model

• Configuration
• Horizontal resolution glevel-8
• Vertical layers 100
• Fixed
• The used computer nodes increases from
  10 to 80.
• Results
• Green ideal speed-up line
• Red actual speed-up line

? Our model has a good scalability!
34
Comparison with spectral model (1)
AFES AGCM for the Earth Simulator A very fast
spectral model in the world

• Computational time for 1step

• AFESL 2pR / N
• O(N3)
• NICAM L 2?x
• NICAM L 4?x
• O(N2)

• NICAM is more efficient than AFES
• at least for T1279 or glevel10

35
Comparison with spectral model(2)

• Maximum time step and one-day simulation time

If L 2 pR / N 4 ?x

• Time step of NICAM can be larger than AFES

At T1279 glevel-10, NICAM is faster than AFES.
36
Summary

• A new regional non-hydrostatic model using a new
  conservative scheme.
• Conservation of mass and total energy.
• A newly tuned icosahedral grid.
• Quasi uniform grid using the spring dynamics.
• A stretched grid gt a regional climate model
• A new dynamical core of the nonhydrostatic
  icosahedral grid model Validation of the
  dynamical core
• The Life cycle of extratropical cyclones
  experiment.
• the Held Suarez experiment.
• Measurement of computational performance on the
  Earth Simulator.
• A very good scalability and a good sustained
  performance ( 40 of peak performance ).
• Superior to a spectral model.
• 1 hour for 1day simulation for 7km mesh dry model
  using 80 nodes
• gt 4hour for 1 day simulation for 3.5km full
  model using 320nodes

37
References

• Icosahedral grid
• Tomita et al., (2001) Shallow Water Model on
  a Modified Icosahedral Geodesic Grid by Using
  Spring Dynamics, J. Comput. Phys., 174, 579-613
• Tomita et al., (2002) An Optimization of the
  Icosahedral Grid Modified by the Spring
  Dynamics, J. Comput. Phys., 183, 307-331
• Nonhydrostatic scheme
• Satoh (2002) Conservative scheme for the
  compressible non-hydrsostatic model with
  horizontally explicit and vertically implicit
  time integration scheme, Mon.Wea.Rev., 130,
  1227-1245
• Satoh (2003) Conservative scheme for a
  compressible non-hydrsostatic models with moist
  processes, Mon.Wea.Rev., in press.
• Global nonhydrostatic icosahedral model
• Tomita et al., (2002a) Development of a
  nonhydrostatic general circulation model using an
  icosahedral grid, Parallel CFD 2002, in press
• Goto et al., (2002) Computational performance
  of dynamical part of next generation climate
  model using an icosahedral grid on the Earth
  Simulator, Parallel CFD 2002, in press
• Tomita et al., (2002b) The Non-hydrostatic
  Icosahedral Global Model for the Earh Simulator,
  Max-Planck Institute for Meteorology technical
  Report 2002
• Tomita et al., (2002c) Global nohydrostatic
  dynamical core on the icosahedral grid Part I
  Model description and fundamental tests, in
  preparation
• Physical processes
• Nasuno et al., (2002) Resolution Dependence
  of a Tropical Squall Line, submitted to
  Mon.Wea.Rev

38
Atmospheric Model activity in Japan

• AGCM in Japan
• CCSR/NIES/Frontier AGCM sophysticalted climate
  model
• MRI AGCM (unified JMA AGCM) sophysticated NWP
  model
• AFES high resolution spectrum model
• Dennou-AGCM research oriented model
• Nonhydrostatic models in Japan
• MRI/NWD NHM
• CReSS
• Users of RAMS, MM5, ARPS
• Individual models(Yamasaki, )