Hybrid Variational-Ensemble Data Assimilation at NCEP

1
Hybrid Variational-Ensemble Data Assimilation at
NCEP

• Daryl Kleist

NOAA/NWS/NCEP/EMC
with acknowledgements to Kayo Ide, Dave Parrish,
Jeff Whitaker, John Derber, Russ Treadon, Wan-Shu
Wu, Jacob Carley, and Mingjing Tong
Workshop on Probabilistic Approaches to Data
Assimilation for Earth Systems Banff, Alberta,
Canada February 2013
2
Outline

• Introduction
• (Brief) background on hybrid data assimilation
• Hybrid Var/Ens at NCEP
• OSSE-based hybrid experiments
• Future Work and Summary

3
Hybrid Variational-Ensemble(ignoring
preconditioning for simplicity)

• Incorporate ensemble perturbations directly into
  variational cost function through extended
  control variable
• Lorenc (2003), Buehner (2005), Wang et. al.
  (2007), etc.

bf be weighting coefficients for fixed and
ensemble covariance respectively xt (total
increment) sum of increment from fixed/static B
(xf) and ensemble B ak extended control
variable ensemble perturbations -
analogous to the weights in the LETKF
formulation L correlation matrix effectively
the localization of ensemble perturbations
4
Single Temperature Observation
3DVAR
bf-10.0
bf-10.5
5
Outline

• Introduction
• (Brief) background on hybrid data assimilation
• Hybrid Var/Ens at NCEP
• OSSE-based hybrid experiments
• Future Work and Summary

6
NOAAs NWS Model Production Suite
Oceans HYCOM WaveWatch III
Climate CFS
Hurricane GFDL HWRF
Coupled
MOM3
1.7B Obs/Day
Satellites 99.9
Dispersion ARL/HYSPLIT
Regional NAM NMM-B
Global Forecast System
Global Data Assimilation
Severe Weather
Regional Data Assimilation
WRF NMM/ARW Workstation WRF
Short-Range Ensemble Forecast
North American Ensemble Forecast System
Air Quality
WRF ARW, NMM NMM-B
GFS, Canadian Global Model
NAM/CMAQ
Rapid Update for Aviation
NOAH Land Surface Model
7
GDAS Hybrid 22 May 2012

• Package included other changes
• NPP ATMS (MW) 7 months after launch
• This is by far the fastest we have ever begun
  assimilating data from a new satellite sensor
  after launch
• GPS RO Bending Angle replaced Refractivity
• Summary of pre-implementation retrospective
  testing
• Improved Tropical winds
• Improved mid-latitude forecasts
• Fewer Dropouts
• Improved fits to observations of forecasts
• Some improvement in NA precip. in winter
• Increased bias in NA precip. decreased rain/no
  rain skill in summer (Improved by land surface
  bug fix)
• Overall significant improvement of GFS forecasts

8
TC Track Error Reduction
HYBRID TEST GFS OPERATIONAL NHC/JTWC OFFICIAL
9
Impact on Geopotential Height
1000 mb
NH
500 mb
SH
3DHYB-3DVAR
10
NAM vs NAM parallels upper air stats vs raobs
Ops NAM Solid NAMB (with Physics changes)
Dashed NAMX (with physics changes and using
global EnKF in GSI) Dash-Dot
Vector Wind RMS error
Height RMS error
Ops NAM
NAMB (improved physics)
NAMX (improved physics EnKF)
Day 1 Black Day 2 Red Day 3 Blue
Thanks to Eric Rogers and Wan-Shu Wu
11
RAP hybrid DA using global ensemble
Temp
Rel Hum
3dvar
hybrid
hybrid
3dvar
Wind
RAP GSI-hybrid vs. RAP GSI-3dvar upper-air
verification
3dvar
hybrid
6 h forecast RMS Error
28 Nov 3 Dec 2012
12
Assimilation of NOAA-P3 Tail Doppler Radar (TDR)
Data using GSI hybrid method for HWRF
TDR data for Isaac 2012082712

• HWRF Model 3 domains with 0.18-0.06-0.02 degree
  (27-9-3 km) horizontal resolutions, 43 vertical
  levels with model top at 50 hPa, with ocean
  coupling
• TC environment cold start from GDAS forecast, TC
  vortex cycled from HWRF forecast
• GSI hybrid analysis using GFS EnKF ensemble
• 80 of background error covariance from
  ensemble B.
• Horizontal localization 150 km, vertical
  localization 10 model levels for weak storm and
  20 model levels for strong storm
• Conventional data plus TDR data
• Modified gross error check, re-tuned observation
  error and rejected data dump with very small data
  coverage for TDR data
• 19 TDR missions for Hurricane Isaac, Leslie and
  Sandy

13
Outline

• Introduction
• (Brief) background on hybrid data assimilation
• Hybrid Var/Ens at NCEP
• OSSE-based hybrid experiments
• Future Work and Summary

14
Observing System SimulationExperiments (OSSE)

• Typically used to evaluate impact of future
  observing systems
• Doppler-winds from spaced-based lidar, for
  example
• Useful for evaluating present/proposed data
  assimilation techniques since truth is known
• Series of experiments are carried out to test
  hybrid variants
• Joint OSSE
• International, collaborative effort between
  ECMWF, NASA/GMAO, NOAA (NCEP/EMC, NESDIS, JCSDA),
  NOAA/ESRL, others
• ECMWF-generated nature run (c31r1)
• T511L91, 13 month free run, prescribed SST, snow,
  ice

15
Availability of SimulatedObservations 00z 24
July
SURFACE/SHIP/BUOY
SONDES
AMSUA/MSU
AIRS/HIRS
AIRCRAFT
AMVS
SSMI SFC WIND SPD
PIBAL/VADWND/PROFLR
AMSUB
GOES SOUNDER
16
3D Experimental Design

• Model
• NCEP Global Forecast System (GFS) model (T382L64
  post May 2011 version v9.0.1)
• Test Period
• 01 July 2005-31 August 2005 (3 weeks ignored for
  spin-up)
• Observations
• Calibrated synthetic observations from 2005
  observing system (courtesy Ron Errico/Nikki
  Privi)
• 3DVAR
• Control experiment with standard 3DVAR
  configuration (time mean increment compared to
  real system and found to be quite similar)
• 3DHYB
• Ensemble (T190L64)
• 80 ensemble members, EnSRF update, GSI for
  observation operators
• Additive and multiplicative inflation
• Dual-resolution, 2-way coupled
• High resolution control/deterministic component
• Ensemble is recentered every cycle about hybrid
  analysis
• Discard ensemble mean analysis
• Parameter settings

17
Time Series of Analysis andBackground Errors

• Solid (dashed) show background (analysis) errors
• 3DHYB background errors generally smaller than
  3DVAR analysis errors (significantly so for zonal
  wind)
• Strong diurnal signal for temperature errors due
  to availability of rawinsondes

500 hPa U
850 hPa T
18
3DVAR and 3DHYBAnalysis Errors
U
T
Q
3DVAR
3DHYB
3DHYB-3DVAR
19
Zonal Wind BackgroundErrors
3DVAR
3DHYB
Bf
BEN
20
3DHYB_(Retuned Spread)
U
T
New (RS) hybrid experiment almost uniformly
better than 3DVAR
Q
3DHYB_RS-3DVAR
21
4D-Ensemble-Var4DENSV
As in Buehner (2010), the H-4DVAR_AD cost
function can be modified to solve for the
ensemble control variable (without static
contribution)
Where the 4D increment is prescribed exclusively
through linear combinations of the 4D ensemble
perturbations
Here, the control variables (ensemble weights)
are assumed to be valid throughout the
assimilation window (analogous to the 4D-LETKF
without temporal localization). Note that the
need for the computationally expensive linear and
adjoint models in the minimization is
conveniently avoided.
22
Hybrid 4D-Ensemble-VarH-4DENSV
The 4DENSV cost function can be easily expanded
to include a static contribution
Where the 4D increment is prescribed exclusively
through linear combinations of the 4D ensemble
perturbations plus static contribution
Here, the static contribution is considered
time-invariant (i.e. from 3DVAR-FGAT). Weighting
parameters exist just as in the other hybrid
variants.
23
Single Observation (-3h) Examplefor 4D Variants
4DVAR
4DENSV
H-4DVAR_AD bf-10.25
H-4DENSV bf-10.25
24
Time Evolution of Increment
Solution at beginning of window same to within
round-off (because observation is taken at that
time, and same weighting parameters
used) Evolution of increment qualitatively
similar between dynamic and ensemble
specification Current linear and adjoint
models in GSI are computationally unfeasible for
use in 4DVAR other than simple single observation
testing at low resolution
t-3h
t0h
t3h
H-4DVAR_AD
H-4DENSV
25
4D OSSE Experiments

• To investigate the use of 4D ensemble
  perturbations, two new OSSE based experiments are
  carried out.
• The original (not reduced) set of inflation
  parameters are used.
• Exact configuration as was used in the 3D OSSE
  experiments, but with 4D features
• 4DENSV
• No static B contribution (bf-10.0)
• Analogous to a dual-resolution 4D-EnKF (but
  solved variationally)
• To be compared with 3DENSV
• H-4DENSV
• 4DENSV addition of time invariant static
  contribution (bf-10.25)
• This is the non-adjoint formulation
• To be compared with 4DENSV

26
Impact on Analysis Errors
U
T
Q
4DENSV-3DENSV
H-4DENSV-4DENSV
27
OSSE-based comparison of 3DHYB andH-4DENSV (with
dynamic constraints)
U
T
Something in the 4D experiments is resulting in
more moisture in the analysis, triggering more
convective precipitation
Q
4DHYB-3DHYB
28
Summary of 4D Experiments

• 4DENSV seems to be a cost effective alternative
  to 4DVAR
• Inclusion of time-invariant static B to 4DENSV
  solution is beneficial for dual-resolution
  paradigm
• Extension to 4D seems to have larger impact in
  extratropics (whereas the original introduction
  of the ensemble covariances had largest impact in
  the tropics)
• Increased convection in 4D extensions remains a
  mystery (it is not the weak constraint on
  unphysical moisture as I originally hypothesized)
• Original tuning parameters for inflation were
  utilized. Follow-on experiments with tuned
  parameters (reduced inflation) and/or adapative
  inflation should yield even more impressive
  results.

29
Scale-Dependence Motivation(Courtesy Tom Hamill)
30
Spectrum of Dual-Resolution Increment without
SD-weighting
31
Spectrum of Dual-Resolution Increment with
SD-weighting
32
Initial scale-dependent tests inand OSSE
33
Outline

• Introduction
• (Brief) background on hybrid data assimilation
• Hybrid Var/Ens at NCEP
• OSSE-based hybrid experiments
• Future Work and Summary

34
Summary

• NCEP successfully implemented hybrid
  variational-ensemble algorithm into GDAS
• NCEP aggressively pursuing application of hybrid
  to other systems
• Mesoscale (NAM), HWRF, Rapid Refresh (and HRRR
  follow on), storm scale ensemble
• Future Reanalysis
• Have already run preliminary tests for 1981-1983
  periods, attempting to capture QBO transitions (a
  difficult problem for reduced observing system
  periods)
• Extensions to the GDAS hybrid are ongoing,
  including 4DEnsVar

35
GDAS Hybrid

• EMC targeting T1148 SL GFS implementation within
  next year
• How to configure hybrid? What can be afforded
  computationally on new machine?
• Merging DA and EPS efforts
• Currently have separate EnKF (DA) and BV-ETR
  (EPS) cycles/perturbations
• Improving the ensemble for the hybrid
• Stochastic physics (Jeff Whitaker)
• TC Initialization (Yoichiro Ota)
• Hybrid developments
• Improved localization, flow-dependent (and/or
  scale-dependent) weighting
• Testing and preparing 4d-ensemble-var for
  implementation
• Helping efforts toward cloudy/precip. radiances

36
TC relocation for EnKF(work done by Yoichiro Ota)
Apply TC relocation used in deterministic
analysis to each ensemble member, but allowing TC
structure perturbations and some TC position
spread.

1. Update TC center position (latitude and
   longitude) by the EnKF
2. Use updated positions as inputs to the TC
   relocation
3. Apply this procedure before the EnKF analysis and
   GDAS analysis

The idea is to separate linear problem (TC
location space) and nonlinear problem (actual
relocation of fields).
Blue first guess positionRed Updated
positionGreen TC vital position
37
Example spaghetti diagram
Before relocation
After relocation
TC relocation of this method can reduce the
uncertainty on the TC position, maintaining the
TC structure perturbations and some of the
position uncertainty. Courtesy Yoichiro Ota
38
Comparison with GEFS TC relocation
EnKF 6 hour forecast perturbation GEFS TC
relocation
EnKF analysis with TC relocation
GEFS operational TC relocation scheme destroyed
almost all initial position uncertainty and
create very small spread around TC. Courtesy
Yoichiro Ota
39
6th WMO Symposium on DA

• NCEP will be hosting the next WMO DA Symposium
  from 7-11 October 2013 at NCWCP
• Call for papers will be out soon

40

• BACKUP SLIDES

41
Spectral Cost Function

• Assume increment cost function is in spectral
  space

• Perform variable transforms

42
New Control Variables andSpectral-dependent
Weights

• New spectral control variables then become

• GSI control variables are in physical space,
  however, so we introduce spectral transform
  operator, S

43
Extension for Dual-Resolution

• The extension to dual-resolution requires a
  further modification to apply the scale
  dependence to the ensemble-based increment, and
  not the control variable itself