High Resolution Ocean and SeaIce Data Synthesis using the Columbia Supercomputer

1
High Resolution Ocean and Sea-Ice Data Synthesis
using the Columbia Supercomputer Seminar
presented at SGIUG07 Minneapolis MN, May 23,
2007 Dimitris Menemenlis Jet Propulsion
Laboratory collaboration Chris Hill (MIT),
Chris Henze (NAS), and many more outline Motivat
ion and background ECCO2 project
description Computational aspects Scientific
visualization Some early science results
2
NASA oceanography from satellites to
supercomputers
SeaStar
TRMM
NSCAT
QuikSCAT
TOPEX
Terra
Jason
Seasat
SeaWinds
Aqua
GRACE
Aquarius
Project Columbia
ICESat
OSTM
3
Summary of 22-23 January 2007 ECCO2
meeting Overview and Motivation
ECCO, ECCO-GODAE, ECCO2 (Wunsch, MIT) The only
way to understand the complete, global,
time-evolving ocean circulation is to use all
available data and all available theory. ECCOn
seeks the best possible estimate of the
time-evolving ocean circulation, its influence on
climate, chemistry, biology, etc., understanding
of predictability, and determination of what we
do not understand. A two generation
problem! ECCO2 and NASA satellite missions (Fu,
JPL) Observations of mesoscale and sub-mesoscale
ocean variability are a key requirement for
understanding regional and global climate
processes. For this reason, wide swath altimetry
has been endorsed by the NRC Decadal Survey as a
possible new NASA mission. ECCO2 provides a
framework for utilizing high-resolution data from
existing and future NASA satellite missions.
Estimated sea level trend, spatial mean removed
(Wunsch, Ponte, and Heimbach, 2007).
Ground tracks of TOPEX/Jason tandem mission
superimposed on satellite imagery of sea surface
temperature.
4
ECCO2 High-Resolution Global-Ocean and Sea-Ice
Data Synthesis
MIT Marshall, Heimbach, Hill Wunsch JPL Fu, Kwok,
Lee Menemenlis Zlotnicki GSFC Rienecker
Suarez ARC Henze, Taft HARVARD Tziperman GFDL Adcr
oft ARGONNE Hovland, Utke
Velocity (m/s) At 15 m depth
Objective synthesis of global-ocean and sea-ice
data that covers the full ocean depth and that
permits eddies. Motivation improved estimates
and models of ocean carbon cycle, understand
recent evolution of polar oceans, monitor
time-evolving term balances within and between
different components of Earth system, etc.
5
Summary of 22-23 January 2007 ECCO2
meeting Assimilation and Modeling
Towards an ECCO2 release (Menemenlis/Zhang,
JPL) A first least squares minimization of the
global-ocean and sea-ice cube sphere (CS510)
model resulted in a 64 decrease of cost function
wrt temperature and salinity climatologies. These
state estimates are already being used in a host
of science applications Eddy permitting state
estimation (Mazloff/Heimbach, MIT) Adjoint method
state estimation in the presence of vigorous
mesoscale eddy variability has been shown to be
possible in a regional, high-resolution, Southern
Ocean model configuration. ECCO2 high
resolution modeling (Hill/Menemenlis/Henze) High-r
esolution global-ocean and sea-ice simulations
are being used to estimate error statistics, to
experiment with multiscale assimilation, to
improve parameterizations of unresolved
processes, and to drive the development of
petabit/petaflop infrastructure.
6
Using model Greens functions to fit an
eddy-permitting global-ocean and sea-ice model to
satellite and in-situ data
Drake Passage Transport
Baseline
170Sv
140Sv
2002
2005
1992
Optimized
This first optimization improves Southern Ocean
circulation and stratification (M. Schodlok)
RGPS RGPS
Baseline Optimized November
1997 April 1998 April 1998
April 1998
A Greens function approach is being used to fit
a high-resolution configuration of the MITgcm to
satellite and in-situ ocean and sea ice data.
Top panel shows temperature difference in top 700
m between WGHC climatology and a 1992-2002
baseline integration driven by NCEP.  Bottom
panel shows temperature difference from WGHC for
an integration whose initial conditions, surface
boundary conditions, and internal model
parameters have been calibrated using 30 forward
sensitivity integrations (H. Zhang).
Optimized minus Baseline time-mean wind
difference
but it degrades Arctic Ocean sea ice
distribution and freshwater budget (R. Kwok and
A. Nguyen).
7
Eddy-Permitting Southern Ocean State Estimate
Using the Adjoint Method (Mazloff and Heimbach,
MIT) Currently optimizing year 2005. First
guess initial conditions and open boundary
conditions derived from coarse-resolution ECCO
solution. First guess atmospheric state from
NCEP reanalysis. Data constraints mean and
time-variable sea surface height, sea-ice
concentration, in-situ temperature and salinity
profiles, sea surface temperature, and
hydrographic climatology.
25ºS to 78ºS 1/6º horizontal grid spacing 42
depth levels with partial cells i.e., 1/3 grid
points of global config. KPP and GM-Redi Bulk
formulae and sea-ice model
8
Investigating global ocean model solution
convergence
1/16
1/4
1/8
1/4
1/8
1/16
Hill, Menemenlis, Ciotti, and Henze.
Investigating solution convergence in a global
ocean model using a 2048-processor cluster of
distributed shared memory machines. J.
Scientific Programming, 2007.
9
Performance of key primitives used on the 1/16
resolution simulation. The exchange times are
for a sub-domain of size 96136 with OlxOly1.
10
Overall scaling and performance of the 1/16
resolution simulation on 960, 1440 and 1920
Columbia processors.
11
Scaling and performance of the 1/16-resolution
simulation on 960, 1920, and 3840 Columbia
processors (mixed Numalink / Infiniband, shared
memory / MPI implementation).
12
Developing realtime viz and storage

• Minimal impact to application (high-res already
  takes a long time to run)
• Can process every time step of multiple fields
  (want flexibility in what is recorded/animated)
• Send current simulation state for display almost
  anywhere

Chris Hill Cambridge Massachussets
Bryan Green Chris Henze Sunnyvale California
13
Approach

• Use separate processors on simulation system to
  gather data, e.g., sim. running on 1000 procs
  a custom parallel server for storage and viz
• Custom server pushes output (mapped from
  simulation) cluster for visualization/storage
• Compress frames with MPEG and stream file for
  realtime local and remote viz.
• Archive in realtime to back-end storage farm on
  dedicated server.

14
Processing pipeline
Columbia
chunnel
Rendering Cluster
Ethernet
MITgcm
MITgcm
MITgcm
MPEGs to disk and display
MITgcm
gserv
Shared Mem
Shared Mem
renderer
encoder
collector
collector
collector
coalescer
collector
Infiniband
Extraction
Visualization
Storage/MPEG Creation
15
Trying it out

• Ran a year-long MITgcm simulation
• 105,120 time steps, 5 minutes apart
• Used 1920 processors
• Captured 24 fields (2D slices)
• Two views of each Global and North Atlantic
• Saved North Atlantic data to disk
• 602 MB simulation data per time step
• Simple visualizations color mapped scalars

16
It works!
The July 31 computational run of the MIT General
Circulation Model (MITgcm) employed 48 concurrent
visualization data streams capturing features
such as ocean temperature, speed, and salinity.
The resultant five million images (requiring
nearly 13 terabytes of storage) allow the
researchers to investigate MITcgm's behavior with
unprecedented temporal resolution for example,
correlated daily and seasonal variations among
many different properties are easily
distinguished.
The ECCO concurrent visualization was made
possible by the NAS local area network (LAN)
team, which set up and fine-tuned a high-speed
LAN connection that drove 74 terabytes of data
from Columbia to the NAS hyperwall visualization
system at an impressive sustained rate of 215
megabytes per second over 100 hours. "The network
ran flawlessly throughout the entire run," said
LAN team lead Christopher Buchanan. A subset of
visualization streams was also sent across the
country to a 16-panel hyperwall display at MIT.
0.5 petabits (uncompressed) of output (0.4) and
images (0.1) in 100 hours. gt600 cell phone
minutes
17
Zonal wind speed Sea surface salinity
Sea surface height Sea surface
temperature
18
Summary of 22-23 January 2007 ECCO2 meeting Early
User Applications
Subtropical mode water (Maze, MIT) Eddy
propagation velocity (Fu, JPL) GRACE data
constraints (Zlotnicki, JPL) Errors estimates
(Forget, MIT) Eddy parameterizations (Ferreira,
MIT) Arctic freshwater budget (Condron,
WHOI) Arctic sea ice budget (Kwok, JPL) Sea ice
data/model comparison (Nguyen, JPL) Carbon cycle
modeling (Manizza, MIT) Eddy variability in
Indian Ocean (Lee, JPL) Darwin project (Hill,
MIT) Southern Ocean (Schodlok, JPL) MITgcm
assimilation efforts (Cornuelle, SIO) Adjoint
assimilation efforts (Edwards, UCSC )
19
Mode water formation
(Marshall, Maze, and Hill, MIT)
Objective is to study the dynamics of
eighteen-degree mode water formation in the North
Atlantic through analytical, numerical and
observational approaches.
Marshall (2005). CLIMODE a mode water dynamics
experiment in support of CLIVAR. Clivar
Variations, vol.3, pp. 8-14.
20
Observing and Modeling Ocean Eddies Containing
90 of the kinetic energy in the ocean, ocean
eddies (storm of currents) with scales from
10-100 km, are difficult to observe and simulate.
Using data from TOPEX/Poseidon, Jason, and ERS
radar altimeters, the energy level and
propagation velocity of eddies were estimated and
compared to a high-resolution simulation from the
MIT-JPL ECCO Model. Shown below is an example in
the Argentine Basin.
Eddy sea surface variability and propagation
velocity from altimetry
ECCO2 simulation
Coastal tides not included in the model
cm
cm
10 km/day
L-L. Fu D. Menemenlis
21
Estimating global hydrographic variability
(Forget and Wunsch, MIT) Objective is to estimate
the three-dimensional global oceanic temperature
and salinity variability, omitting the seasonal
cycle, both as a major descriptive element of the
ocean circulation and for use in the error
estimates of state estimation.
Variance of T at 200m (left) and 400m (right) in
(oC)2 as estimated from the data (upper) and as
simulated by an ECCO2 simulation (lower).
Forget and Wunsch. Estimated global hydrographic
variability. J. Phys. Oceanogr., 2007.
22
David Ferreira
Total meridional heat transport
GW
Most of the meridional eddy heat flux is achieved
in the top 200m of the ocean!
Total eddy heat transport
by diabatic eddies!
Eddy heat transport in top 200m
Not yet adequately parameterized in large-scale
coarse-resolution models
23
3400 km3/yr (0.1 Sv) freshwater added from 54
arctic rivers for 11 years into cube 47. Approx.
60 of discharge is from May-July
Lena
Mackenzie
Ob and Yenisey
2002
1992
Surface salinity anomalies (cube 47-cube 43)
(negative fresher)
River discharge penetrating into interior.
A. Condron and P. Winsor, WHOI
24
Interpretation of ICESat altimetric and
reflectivity profiles Kwok (JPL), et al.
Objective is to provide an assessment of the
ICESat altimeter for studying the Arctic Ocean
and to examine the magnitude of the large- and
small-scale expressions of geophysical processes
embedded in the elevation profiles.
Kwok, Cunningham, Zwally, Yi (2006). ICESat over
Arctic sea ice Interpretation of altimetric and
reflectivity profiles. J. Geophys. Res., vol.
111, doi10,1029/2005JC003175.
25
Synthesis of the Arctic System Carbon Cycle
McGuire (Fairbanks), Follows, Manizza (MIT), et
al.
Objective is to study Arctic region carbon cycle,
including (a) exchanges between marine and
terrestrial carbon pools and (b) possible
exchanges between these large carbon reservoirs
and the atmosphere. ECCO2 solutions will be used
to drive offline carbon/biogeochemistry models
leading to improved estimates and models of
air-sea-land-ice exchanges of CO2 in the presence
of realistic eddying flows and with active
biological and chemical processes.
26
Darwin project (Follows and Hill, MIT)
Prochlorococcus analogs
Synechococcus small eukaryotes
Diatoms
Other large eukaryotes
27
Summary ECCO2 project aims to produce
increasingly accurate syntheses of all available
global-scale ocean and sea-ice data at
resolutions that start to resolve ocean eddies
and other narrow current systems. A first public
ECCO2 ocean state estimate, obtained using model
Greens functions, is scheduled for this
summer. Adjoint state estimation in the presence
of vigorous eddying circulation has been
successfully demonstrated. A number of early
user projects have started using pre-release
ECCO2 solutions.
http//ecco2.org/