Evaluation of a KiloMember Ensemble for Track Forecasting

1
Evaluation of a Kilo-Member Ensemble for Track
Forecasting

• May 5, 2004
• Jonathan Vigh
• Department of Atmospheric Science
• Colorado State University

2
Acknowledgements

• People
• Graduate Adviser Dr. Wayne Schubert
• Masters Committee
• Dr. Mark DeMaria
• Dr. William Gray
• Dr. Gerald Taylor
• Dr. Scott Fulton (MUDBAR)
• Organizations
• Schubert Research Group
• NCEP (GFS ensemble fields)
• TPC/NHC (operational guidance and best track
  data)
• Mary Haley and NCL Developers
• Funding Agencies
• Fellowship Support
• Significant Opportunities in Atmospheric Research
  and Science Program (UCAR/NSF)
• American Meteorological Society
• Grants
• NSF ATM-0087072, NSF ATM-0332197, NASA/CAMEX
  NAG5-11010, and NOAA NA17RJ1228

3
Ensemble Background

• Basic idea is to simulate sources of uncertainty
  in the forecast problem.
• Theory dictates that the mean of a well-perturbed
  ensemble should perform better than any
  comparable single deterministic forecast.
• Ensemble distribution provides information about
  the uncertainty of the forecast.
• 
  Image source NASA/GSFC

4
The MUDBAR Model

• The nondivergent modified barotropic equation
  model (MUDBAR) of Scott Fulton
• Data enter the model through the initial
  condition (specify q) and the time-dependent
  boundary conditions (specify ? on boundary, q on
  inflow)
• MUDBAR is fast
• 0.9 s for a 5-day forecast (entire ensemble runs
  in 33 min)
• and efficient
• Is as accurate as the shallow water model LBAR

5
Bogussing Procedure

• The vortex profile of DeMaria (1987) Chan and
  Williams (1987)
• This bogus vortex is blended with the GFS initial
  wind field at the operationally-estimated storm
  position with the appropriate motion vector

6
Ensemble Design

• Simple parameter-based perturbation methodology
• Magnitudes of perturbations in each class chosen
  based on sensitivity experiments for 2001 storms
• Five perturbations classes
• 11 environmental perturbations (NCEP GFS
  ensemble)
• 4 deep layer-mean winds
• 3 perturbations to the models equivalent phase
  speed
• 3 perturbations to the bogus vortex size (Vm)
• 5 perturbations of 1 m s-1 relative to the storm
  motion vector
• All perturbations are cross multiplied to get an
  ensemble of 11 x 4 x 3 x 3 x 5
  1980 members!
• Atlantic 2001-2003 seasons, 293 cases (50
  storms)
• Run in real-time for 2002-2003 seasons

7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
Conclusions

• Ensemble mean forecast does not outperform the
  control forecast (ensemble mean not that useful)
• Ensemble performance degraded by unskillful
  perturbations (perturbed GFS members, small
  bogus vortex, shallow layer mean wind)
• Vital to have perturbations be relative to the
  best guess values for each particular forecast
  case (sample the error covariance matrix)
• Weak spread-error relationship peaks at 60-h -gt
  limited ability to estimate a priori forecast
  skill
• Ensemble-based strike probabilities offer much
  more information about the forecast uncertainty
  than just the mean or spread
• All ensemble and operational guidance for
  2001-2003 seasons is at
• http//euler.atmos.colostate.edu/vigh/

25
Future Work

• Redesign the ensemble to use perturbations
  tailored to each forecast situation
• Develop a method of error recycling to calibrate
  the ensemble probabilities
• Use cluster analysis to derive most probable
  forecast scenarios
• Examine spread-error relationships of other
  ensembles (GFS, GUNA, ECMWF) can these be
  combined to get an overall estimate of forecast
  uncertainty?
• Compare strike probabilities to other ensembles
  (GFS, GUNA, ECMWF) using BSS and ROC scores

26
Questions

• Can a well-perturbed ensemble mean give a better
  forecast than any single realization?
• How many ensemble members are necessary to give
  the right answer?
• Is there a relationship between ensemble spread
  and forecast error?
• Can this relationship be used to provide
  meaningful forecasts of forecast skill?
• How accurately does the ensemble envelope of all
  track possibilities encompass the actual observed
  track?

27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)