Ensemble Verification Review of developments and plans

1
Ensemble Verification Review of developments
and plans

• By Yuejian Zhu
• June 17th 2005

2
Ensemble Verification current status

• NCEP global ensemble verification package used
  since 1995
• Comprehensive verification statistics computed
  against analysis fields
• Post most of skill scores on web-site and updated
  daily/frequently
• Inter-comparison with other NWP centers (mainly
  MSC and ECMWF)
• Saved ASCII (text) format statistics for each
  initial forecasts (or verified analysis)
• Many of them use climatology
• NCEP regional ensemble (SREF) verification
  package
• Basic measures computed routinely since 1998
• Probabilistic measures being developed
  independently from global ensemble
• Unification from global and regional
• Need to unify computation of global regional
  ensemble verification measures
• Running unified code on NCEP daily operation
  (year 2005)
• Unified framework must facilitate wide-scale
  national/international collaboration
• North American Ensemble Forecast System
  (collaboration with Met. Service Canada)
• THORPEX International Research Program
• WRF meso-scale ensemble developmental and
  operational activities
• Facilitate wider community input in further
  development/enhancements
• For example, how to establish basis for
  collaboration with NCAR, statistical community,
  etc..

3
Ensemble Verification design specifications

• Computation based on daily/6-hourly/3-hourly
  cycle
• Store the computation scores
• Compute statistics selected from list of
  available
• Point-wise measures, including
• RMS errors, PAC for individual ensemble members,
  mean and medium
• Measure of reliability (Talagrand distribution
  and outlier, spread vs. RMS error, reliability
  part of BS, RPSS, etc)
• Measure of resolution (ROC, IC, resolution part
  of BS, RPSS, potential EV, etc)
• Combined measures of reliability and resolution
  (BSS, RPSS, etc)
• Multivariate statistics (such as PECA
  (Perturbation versus Error Correlation Analysis)
  , etc. reference from Wei)
• Variable and lead time make all available that
  are used from ensemble
• Aggregate statistics as chosen in time, space,
  etc
• Select time periods (seasonal, monthly, etc)
• Select spatial domain (pre-designed or user
  specified area)
• Select lead-time (optional)
• Select variables
• Verify against observation/analysis
• Scripts running verification codes should handle
  against both O/A issues
• Use the same subroutine to compute statistics for
  either one
• Account for effect of observation/analysis
  uncertainty? (if possible)

4
Ensemble Verification design specifications

• Verify for different ensemble sizes
• Combination of multi-model ensemble
• Variable ensemble sizes, comparable
• Define forecast/verification events by either
• Observed/analyzed climatology, such as
• 10 percentile thresholds in climate distribution
• above/below normal
• User specified thresholds compute corresponding
  climate percentiles, such as
• Precipitation greater than 1 inch per 24 hours
• Temperature below frozen
• Wind shear greater than 10m/s
• Based on ensemble members (like Talagrand stats)
  compute climate percentiles
• Facilitate the use of benchmark
• Climatology, persistence, extreme or user
  specified
• Short, medium, long-rang and climate forecast
• Operational and research community
• Prioritize and find balance between
• Flexibility vs. complexity
• Operational vs. research use

5
Ensemble Verification development and plan

• Design unified ensemble verification framework
• Input data handling
• Use standard WMO formats as data input
• GRIB format for analysis, forecast and
  climatology
• BUFR format for observation
• Option to allow non-standardized user/institution
  specifying
• Computation of statistics
• Establish required software functionalities
  (scripts)
• Build up required verification statistics
  program/subroutines (source codes)
• Jointly develop and share scripts/subroutines
  with standard input/output
• Comparable scientific results from independent
  investigators
• Output daily statistics (discussion)
• Adopt WMO recommended (if any)
• VSDB format (SREF used)
• User specified
• Display of output statistic (option)
• Develop/adapt display software for interactive
  interrogation of output statistics
• FVS display system
• FSL approach to WRF verification

6
Ensemble Verification development and plan

• Develop and implement new verification framework
• Utilize existing software and infrastructure
  where possible
• Combine current global and regional verification
  software
• Use existing climatology, develop new
  climatological distribution (anomalies)
• NCEP new ensemble-related verification efforts
• Direct all of them toward new framework if
  possible
• Share NCEP works with Meteorological Service of
  Canada for
• Northern American Ensemble Forecast System
  (NAEFS) project
• Mainly exchange the subroutines
• Share NCEP works with interested collaborators
• Forecast System Lab. (FSL statistical display
  tools)
• Other institutions
• Make new software available to national/internatio
  nal community
• THORPEX international research program
• WRF ensemble verification
• Coordinate further development with wider
  community (WMO, other NWP centers)