Recommendations for ensemble verification

1
Recommendations for ensemble verification
2

• TIGGE recommendations
• The recommended scores and diagrams for
  probability forecasts are
• Reliability diagrams (1)
• ROC diagrams (1)
• Brier skill score (1)
• (Continuous) Ranked Probability skill score (1)
• Potential economic value (1-3)
• Ensemble spread should be assessed using
• Rank histogram
• Spread vs. skill
• The WMO standard defines "reliability tables"
  (contingency tables) as the mechanism for
  exchange of results, from which all of the
  probabilistic scores and diagrams can be
  calculated.
• In addition
• PECA (2), Wilson score (2), minimum spanning tree
  (1-3), Kolmogorov dn (2), ignorance (2), entropy
  (2), variance of spread b (2), measures of
  non-linearity (3), best/worst members (2-3)
• Importance on a scale of 1 to 3

3
Functionality issues (ensemble)

• Consider ensemble members as individual forecasts
  in design of verification system
• Ability to verify ensemble mean and median
• Function fitting option to deal with the issue of
  small or variable ensemble sizes

4
Functionality issues (general)

• Need for climatologies
• Ability to reference/score persistence
• Documentation on proper use and interpretation of
  scores
• Ability to stratify by lead time, station,
  threshold, area, season, time of day, etc.
• Provide information on observation
  representativeness uncertainty, allow user to be
  able to incorporate this information (e.g. rank
  histograms)
• Data matching to ensure consistent samples across
  multiple forecasts
• Categorical verification within bins instead of
  thresholded
• Some guidance on
• Aggregation of scores
• Non-linear scores
• Non-homogeneous regimes
• Bootstrapping to get CIs
• Ability to manipulate F-O pairs for
  post-processing / cross validation

5
New frontiers

• Methods for verifying probability forecasts of
  extremes
• Methods for verifying spatial ensemble forecasts,
  including application of object-based techniques
• Improved diagnostics for discriminating spread
  deficiencies and bias
• Metrics relevant to end users