MSC activities and plans for eps verification

1
MSC activities and plans for eps verification

• L. Wilson
• With help from Marcel Vallée, Peter Houtekamer,
  WMO and CMC

2
Outline and purpose

• Planning stages
• Principle To build verification systems which
  use what we already have and are flexible
• Motivated by needs of research but hope to cover
  operational needs too
• Outline
• CRPS Project
• Verification of weather element forecasts
• WMO standard eps verification

3
Verification of ensembles

• Goal of ensemble forecast
• It has been said many times that the goal of
  ensemble prediction of probability distributions
  is To maximize sharpness (or resolution) subject
  to calibration (reliability)
• We need measures which can evaluate this, to
  answer questions such as
• Eps system A is more reliable than system B but
  resolution is worse. Which is better?
• Is the combination of system A and system B
  better than either system alone?
• One measure that can potentially answer questions
  such as these and has other desirable properties
  is the CRPS.

4
CRPS
5
CRPS (contd)

• Advantages
• Compares overall distribution to observation
  sensitive to distance
• Converges to MAE for deterministic forecast can
  compare eps and deterministic forecast directly
• Can be partitioned into reliability and
  resolution components
• References
• Hersbach, Hans, 2000 Decomposition of the
  continuous ranked probability score for ensemble
  prediction systems. WAF 15, 559-570.
• We have the code from ECMWF for verification
  against analysis and have adapted it to
  verification vs. observations
• Candille, G. and O. Talagrand, 2005 Evaluation
  of probabilistic prediction systems for a scalar
  variable, QJRMS, in review.
• Presents an alternative version
• We have Guillem Candille

6
CRPS implementation

• Motivation
• To comparatively evaluate eps in parallel run
  with operational eps for implementation of EnKF
• Setup
• Verification of upper air data vs ensemble at
  stations, parameters and levels as in WMO
  standards for deterministic forecasts
• Data in BURP format, to be interfaced with
  Herzbach FORTRAN code
• Planned to include confidence interval estimation
  with bootstrap methods

7
Unified verification system for eps surface
variables

• Now have miscellaneous applications
• CMC verification of individual members and
  ensemble mean (Beauregard)
• Monthly ROC, rank histogram, RMSE plots (on
  website)
• CRPS applied to BMA output
• Requirements
• To respond to standard WMO verification for
  ensembles (Reliability and others)
• To conform to WMO guidelines for precipitation
  verification
• To unify verification program structures within
  MSC
• To build a flexible system for use in NAEFS
• Status
• Design stage Marcel Vallée has recently been
  assigned.

8
Design of unified system

• For verification at stations
• Use existing components as much as possible
• Based on existing UMOS verification system
• Modular, so existing verification tools can be
  interfaced
• Data format
• Separate file for each member (model), for each
  station (interpolated, upscaled or nearest
  gridpoint), for all projection times
• Raw forecast data all processing of ensembles is
  separate step.

9
Design (contd)
Original archives Fields of forecast output for
each member
Verification archive one time series file for
each station, variable, model - interpolated to
stations OR upscaled
Observation archive
MATCHING
New Variables -probability forecasts -ensemble
moments -best or worst member -output of
post-processing methods e.g. BMA
Processing ensemble forecasts
Scores which depend on eps distribution -rank
histogram -CRPS and components -RPS and RPSS
-Wilson et al -ignorance score -logarithmic
score -distribution statistics etc.
Scores for probability forecasts -Brier and
components -Reliability table -ROC graph and
score -Brier Skill Score
Scores for individual quantitative forecasts
-RMSE -MAE -linear bias -variance explained
-correlation
Station Climatology
10
Sources of verification algorithms

• CRPS and components Hans Hersbach CMC
• MAE, RMSE, bias and associated skill scores
  exist at CMC
• ROC and other scores R Project (Matt Pocernich)
• ROC, rank histogram CMC
• Reliability tables UMOS verification R Project
• Brier score and components UMOS verification

11
Reliability table example (from R Project)
12
WMO Requirements

• Probability forecasts against analysis
• Pmsl /- 1 sd, 2 sd
• Z500 /- 1 sd, 2 sd
• Wind speed 850 mb thresholds 10, 15 25 m/s
• T850 anomalies /- 2, 4, 8 degrees
• Probability forecasts against observations
• Precipitation gt1,5,10 and 25 mm / 24h for each 24
  h
• Scores for probability forecasts
• Brier Skill vs. Climatology
• Reliability table
• Potential economic value diagram
• Continuous variables
• Ensemble mean for mslp Z, T, winds for 500, 250
  mb
• Ensemble std/rmse of ensemble mean
• Scores for continuous variables
• Bias, rmse, AC, S1 skill score, rms vector wind
  error

13
WMO Requirements continued

• Comments on WMO standards
• Climatology is ERA-40 for defining anomalies
  truth is centers own analysis
• General lack of verification of surface elements
• Emphasis on verification of ensemble mean
• Where we are
• Have verification of ensemble mean against
  analysis easy to add std/rmse
• Can use CRPS project, extend to meet WMO
  requirements by adding generation of
  probabilities and reliability tables for upper
  air variables at stations
• Use unified system for precipitation verification
• Not resourced, but could do most of it from other
  initiatives

14
Conclusions

• Two main streams seem to be emerging for eps
  verification
• CRPS project for upper air evaluation against
  radiosondes
• An attempt at a unified verification for surface
  parameters
• Need for a flexible, well-defined data structure
  for NAEFS that all can use
• Plan to satisfy WMO requirements with either or
  both of these verification initiatives