Development of a verification methods testbed at the WRF DTC

1
Development of a verification methods testbed at
the WRF DTC

• Mike Baldwin
• Purdue University

2
Acknowledgements

• WRF Developmental Testbed Center
• visiting scientist program
• Beth Ebert
• Barbara Casati
• Ian Jolliffe
• Barb Brown
• Eric Gilleland

3
Motivation for new verification methods

• Great need within both the research and
  operational NWP communities for new verification
  methods
• High-resolution forecasts containing realistic
  detail/structure
• Ensembles/probabilistic forecasts

4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
Forecast 1 smooth
FCST 1 smooth
OBSERVED
OBSERVED
FCST 2 detailed
9
Traditional verification measures for these
forecasts
Verification Measure Smooth forecast Detailed forecast
Mean absolute error 0.157 0.159
RMS error 0.254 0.309
Bias 0.98 0.98
Threat score (gt0.45) 0.214 0.161
Equitable threat score 0.170 0.102
10
Traditional performance measures

• Often fail to provide meaningful information when
  applied to realistic forecasts
• Many of the unfavorable aspects of traditional
  measures are well-known
• Yet such measures continue to be used extensively

11
S1 score (500mb heights)
12
anomaly correlation (500mb height)
13
Threat score (QPF)
14
Sensitivity to bias and event frequency
15
Why?

• History
• Continuity
• Familiarity
• Understandable
• Comfort level
• A certain degree of credibility has been
  established after forecast performance has been
  measured over several decades

16
New methods

• Plenty of new verification methods have been
  proposed
• Features-based
• Morphing
• Scale decomposition
• Fuzzy/neighborhood
• Why havent they caught on?

17
Why havent they caught on?

• Usability has not been demonstrated
• No history
• Difficult to interpret results
• Credibility has not yet been established

18
Fuzzy verification framework
from Beth Ebert (2008)
19
Weaknesses and limitations

• Less intuitive than object-based methods
• Imperfect scores for perfect forecasts for
  methods that match neighborhood forecasts to
  single observations
• Information overload if all methods invoked at
  once
• Let appropriate decision model(s) guide the
  choice of method(s)
• Even for a single method
• there are lots of numbers to look at
• evaluation of scales and intensities with best
  performance depends on metric used (CSI, ETS, HK,
  etc.). Be sure the metric addresses the question
  of interest!

from Beth Ebert (2008)
20
Typical path to acceptance and adoption of new
verif methods

• Develop a new technique
• Test it on a small number of cases
• Publish those results and methodology
• Apply the technique to forecasts on a routine
  basis
• Build up a collection of results
• Compare the new and traditional methods
• ACCEPT when users become satisfied with the
  behavior of the new method

21
Propose a testbed for verification methods

• Provide access to a database of operational and
  experimental forecasts
• Covering a period of several years
• Compare new and traditional measures
• Collaborate with users of verification
  information
• This will help to speed up the process of
  establishing credibility and eventual use

22
Long-period database of forecasts

• NCEP Operational
• GFS
• NAM
• model grid spacing
• QPF (3h and 24h accumulations)
• truth Stage IV analyses
• CONUS region
• 00 and 1200 UTC initial times
• archive period 1999-present
• additional fields (temperature, heights) may be
  added

23
Forecast archive

• Experimental
• WRF runs produced to support SPC/NSSL HWT in
  2004, 2005, 2007, 2008
• 2004 and 2005 data already in hand
• used as part of Spatial Forecast Verification
  Intercomparison Project (ICP)
• hourly QPF Stage IV analyses
• additional fields (surface temps, reflectivity)
  to be added if feasible

24
Formats

• Forecasts will be available in several standard
  data formats (GRIB to start with)
• Archived will be maintained by DTC
• Software routines will be provided to read data,
  interpolation library
• Work with MET verification package
• traditional scores
• some new methods currently available

25
Testbed

• Fits into the WRF/DTC framework
• Provides a proving ground for new methods
• Answer operational concerns
• How much time does a method take to run?
• How much time/effort is required to analyze
  results?
• How should information be presented to users?
• Compare with traditional methods?
• How do results change before/after major model
  upgrades?

26
Collaboration with users

• Subjective component
• SPC/NSSL HWT (Spring Program) has collected
  extensive subjective/expert ratings of
  experimental WRF model forecasts
• DTC facilitates transfer from research to
  operations
• Potential use for training

27
Show me

• The testbed will allow researchers to demonstrate
  meaningful ways to apply new verification
  information
• Applied to current operational models
• accelerate the process of improving guidance
• Event-based errors for specific classes of
  phenomena
• Error scales

28
NDFD-scale surface parameters

• WRF

29
NDFD-scale surface parameters

• RTMA

30
Possible additions

• OPeNDAP/THREDDS access
• regions beyond the U.S.
• possible WGNE QPF verification data
• ensemble forecasts
• grid-to-obs capability

31
(No Transcript)
32
General verification framework

• Any verification method should be built upon the
  general framework for verification outlined by
  Murphy and Winkler (1987)
• New methods can be considered an extension or
  generalization of the original framework
• Joint distribution of forecasts and observations
  p(f,o)

33
general joint distribution

• p(f,o) where f and o are vectors containing all
  variables, matched in space and time
• o could come from data assimilation
• joint distribution difficult to analyze
• different factorizations simplify analysis
• provide information on specific aspects of
  forecast quality

34
general joint distribution

• p(Gf,Ho) where G, H are mapping/transformati
  on/operators that are applied to the variable
  values
• morphing
• filter
• convolution
• fuzzy
• some methods perform mapping of o that is a
  function of f

35
general joint distribution

• p(Gmf,Hmo) where Gm is a specific
  aspect/attribute/characteristic that results from
  the mapping operator
• measures-oriented
• compute some error measure or score that is a
  function of Gmf,Hmo
• MSE
• what is the impact of these operators on the
  joint distribution?

36
Standardize terminology

• feature a distinct or important physical
  object that can be identified within
  meteorological data
• attribute a characteristic or quality of a
  feature, an aspect that can be measured
• similarity the degree of resemblance between
  features
• distance the degree of difference between
  features
• others?

37
framework

• follow Murphy (1993) and Murphy and Winkler
  (1987) terminology
• joint distribution of forecast and observed
  features
• goodness consistency, quality, value

38
aspects of quality

• accuracy correspondence between forecast and
  observed feature attributes
• single and/or multiple?
• bias correspondence between mean forecast and
  mean observed attributes
• resolution
• reliability
• discrimination
• stratification

39
Features-based process
FCST
OBS

• Identify features

40
feature identification

• procedures for locating a feature within the
  meteorological data
• will depend on the problem/phenomena/user of
  interest
• a set of instructions that can (easily) be
  followed/programmed in order for features to be
  objectively identified in an automated fashion

41
Features-based process
FCST
OBS

• Characterize features

42
feature characterization

• a set of attributes that describe important
  aspects of each feature
• numerical values will be the most useful

43
Features-based process
FCST
OBS

• Compare features

44
feature comparison

• similarity or distance measures
• systematic method of matching or pairing observed
  and forecast features
• determination of false alarms?
• determination of missed events?

45
Features-based process
FCST
OBS

• Classify features

46
classification

• a procedure to place similar features into groups
  or classes
• reduces the dimensionality of the verification
  problem
• similar to going from a scatter plot to a
  contingency table
• not necessary/may not always be used

47
SSEC MODIS archive 10 Apr 2003
48
feature matching
49
attributes
Lake Fcst 1 Fcst 2 Obs 1 Obs 2 Obs 3
Lat 47.7 44.0 44.8 42.2 43.7
Lon 87.5 87 82.4 81.2 77.9
Area (km2) 82400 58000 59600 25700 19500
Volume (km3) 12000 4900 3540 480 1640
Max depth (m) 406 281 230 64 246
50
How to match observed and forecast objects?
dij distance between F i and O j

• missed event

O1
O3
Objects might match more than once
If dj gt dT missed event
O2
F1
for each observed object, choose closest
forecast object
for each forecast object, choose closest
observed object
If di gt dT then false alarm
false alarm
F2
51
Example of object verf
ARW 2km (CAPS)
Radar mosaic
Fcst_2
Obs_2
Fcst_1
Obs_1
Object identification procedure identifies 4
forecast objects and 5 observed objects
52
Distances between objects
ARW 2km (CAPS)
Radar mosaic

• Use dT 4 as threshold
• Match objects, find false alarms, missed events

O_34 O_37 O_50 O_77 O_79
F_25 5.84 4.16 8.94 9.03 11.53
F_27 6.35 2.54 7.18 6.32 9.25
F_52 7.43 9.11 4.15 9.19 5.45
F_81 9.39 6.35 6.36 2.77 5.24
53
ARW2
ARW4
Df .04 Dl -.07
Df .07 Dl .08
median position errors matching obs object given
a forecast object
NMM4
Df .04 Dl .22