Verifying Satellite Precipitation Estimates for Weather and Hydrological Applications

1
Verifying Satellite Precipitation Estimates for
Weather and Hydrological Applications

• Beth Ebert
• Bureau of Meteorology Research Centre
• Melbourne, Australia

1st IPWG Workshop, 23-27 September 2002, Madrid
2
val.i.date ( ) tr.v. 1. To
declare or make legally valid. 2. To mark with
an indication of official sanction. 3. To
substantiate verify. ver.i.fy ( )
tr.v. 1. To prove the truth of by the
presentation of evidence or testimony
substantiate. 2. To determine or test the truth
or accuracy of, as by comparison, investigation,
or reference "Findings are not accepted by
scientists unless they can be verified" (Norman
L. Munn)
The American Heritage Dictionary of the English
Language. William Morris, editor, Houghton
Mifflin, Boston, 1969.
3
Satellite precipitation estimates -- what do we
especially want to get right?
Climatologists - mean bias NWP data assimilation
(physical initialization) - rain location and
type Hydrologists - rain volume Forecasters and
emergency managers - rain location and maximum
intensity Everyone needs error estimates!
4
Short-term precipitation estimates

• High spatial and temporal resolution desirable
• Dynamic range required
• Motion may be important for nowcasts
• Can live with some bias in the estimates if it's
  not too great
• Verification data need not be quite as accurate
  as for climate verification
• Land-based rainfall generally of greater interest
  than ocean-based

5
Some truths about "truth" data

• No existing measurement system adequately
  captures the high spatial and temporal
  variability of rainfall.
• Errors in validation data artificially inflate
  errors in satellite precipitation estimates

6
Rain gauge observations Advantages Disa
dvantages True rain measurements May be
unrepresentative of aerial
value Verification results biased
toward regions with high
gauge density Most obs made
once daily
7
Radar data Advantages Disadvantages
Excellent spatial and Beamfilling,
attenuation, temporal resolution overshoot,
clutter, etc. Limited spatial extent
8
Rain gauge analyses Advantages Disadvantages
Grid-scale quantities Smoothes actual rainfall
Overcomes uneven values distribution
of rain gauges
9
Stream flow measurements Advantages Disadvan
tages Integrates rainfall over Depends on soil
conditions, a catchment hydrological
model Many accurate measure- Time delay between
rain ments available and outflow Hydrologists
want it Blurs spatial distribution
10
Verification strategy for satellite precipitation
estimates
Use (gauge-corrected) radar data for local
instantaneous or very short-term estimates Use
gauge or radar-gauge analysis for larger spatial
and/or temporal estimates
11
Focus on methods, not results

• What scores and methods can we use to verify
  precipitation estimates?
• What do they tell us about the quality of
  precipitation estimates?
• What are some of the advantages and disadvantages
  of these methods?
• Will focus on spatial verification

12
Does the satellite estimate look right?

• Is the rain in the correct place?
• Does it have the correct mean value?
• Does it have the correct maximum value?
• Does it have the correct size?
• Does it have the correct shape?
• Does it have the correct spatial variability?

13
Spatial verification methods

• Visual ("eyeball") verification
• Continuous statistics
• Categorical statistics
• Joint distributions
• - - - - - - - - - - - - - - - - - - - - - - - - -
  - - - - - - - - - - - - - - - - - - - - - - - - -
  - - - - - - - - - - - -
• Scale decomposition methods
• Entity-based methods

"standard"
"scientific" or "diagnostic"
14
Step 1 Visual ("eyeball") verification
Visually compare maps of satellite estimates and
observations Advantage "A picture tells a
thousand words" Disadvantages Labor intensive,
not quantitative, subjective
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
Rozumalski, 2000
15
Continuous verification statistics

• Measure the correspondence between the values of
  the estimates and observations
• Examples
• mean error (bias)
• mean absolute error
• root mean squared error
• skill score
• linear error in probability space (LEPS)
• correlation coefficient

Advantages Simple, familiar Disadvantage Not
very revealing as to what's going wrong in the
forecast
16
Mean absolute error
Mean error (bias)
Measures Average difference between forecast
and observed values
Measures Average magnitude of forecast error
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
Root mean square error
Measures Error magnitude, with large errors
having a greater impact than in the MAE
17
Time series of error statistics
24-hr rainfall from NRL Experimental
Geostationary algorithm validated against
Australian operational daily rain gauge
analysis 0.25 grid boxes, tropics only
18
Linear error in probability space (LEPS)
Measures Probability error - does not penalise
going out on a limb when it is justified.
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
19
Correlation coefficient
Measures Correspondence between estimated
spatial distribution and observed spatial
distribution, independent of mean bias
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
Danger...
20
Rozumalski, 2000 AutoEstimator validated against
Stage III 8x8 km grid boxes
21
Skill score
Measures Improvement over a reference estimate.
When MSE is the score used in the above
expression then the resulting statistic is called
the reduction of variance. The reference
estimate is usually one of the following (a)
random chance (b) climatology (c)
persistence but it could be another estimation
algorithm.
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
22
Cross-validation - useful when observations are
included in the estimates
where Yi is the estimate at point i computed
with Oi excluded from the analysis Measures
Expected accuracy at the scale of the
observations. The score is usually bias, MAE,
RMS, correlation, etc.
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
23
Categorical statistics

• Measure the correspondence between estimated and
  observed occurrence of events
• Examples
• bias score
• probability of detection
• false alarm ratio
• threat score
• equitable threat score
• odds ratio
• Hanssen and Kuipers score
• Heidke skill score

Advantages Simple, familiar Disadvantage Not
very revealing
24
Categorical statistics
25
Bias score
Measures Ratio of estimated area (frequency) to
observed area (frequency)
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
26
Probability of Detection
False Alarm Ratio
Threat score (critical success index)
Equitable threat score
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
Odds ratio
27
Hanssen and Kuipers discriminant (true skill
statistic)
Measures Ability of the estimation method to
separate the "yes" cases from the "no" cases.
Heidke skill score
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
Measures Fraction of correct yes/no detections
after eliminating those which would be correct
due purely to random chance
28
Categorical verification of daily satellite
precipitation estimates from GPCP 1DD algorithm
during summer 2000-01 over Australia
Rain threshold varies from light to heavy
29
Real-time verification example
24-hr rainfall from NRL Experimental
Geostationary algorithm
30
Real-time verification example
24-hr rainfall from NRL Experimental blended
microwave algorithm
31
Distributions oriented view
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
Advantage Much more complete picture of forecast
performance Disadvantage Lots of numbers
32
24-hr rainfall from NRL Experimental
Geostationary algorithm validated against
Australian operational daily rain gauge analysis
on 21 Jan 2002
33
ScatterplotShows Joint distribution of
estimated and observed values
34
Probability distribution functionShows
Marginal distributions of estimated and observed
values
35
Heidke skill score (K distinct categories)
Measures Skill of the estimation method in
predicting the correct category, relative to that
of random chance
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
36
Scale decomposition methods

• Measure the correspondence between the estimates
  and observations at different spatial scales
• Examples
• 2D Fourier decomposition
• wavelet decomposition
• upscaling
• Advantages Scales on which largest errors occur
  can be isolated, can filter noisy data
• Disadvantages Less intuitive, can be
  mathematically tricky

37
Discrete wavelet transforms
Concept Decompose fields into scales
representing different detail levels. Test
whether the forecast resembles the observations
at each scale.

• Measures, for each scale
• of total MSE
• linear correlation
• RMSE
• categorical verification scores
• others...

Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
38
Casati and Stephenson (2002) technique
Step 1 "Recalibrate" forecast using histogram
matching errortotal errorbias
errorrecalibrated Step 2 Threshold the
observations and recalibrated forecast to get
binary images
39
Step 3 Subtract to get error (difference) image
Step 4 Discrete wavelet decomposition of error
to scales of resolution x 2n
40
Step 5 Compute verification statistics on error
field at discrete scales. Repeat for different
rain thresholds.
41
Multiscale statistical organization
Zepeda-Arce et al. (J. Geophys. Res.,
2000) Concept Observed precipitation patterns
have multi-scale spatial and spatio-temporal
organization. Test whether the satellite estimate
reproduces this organization. Method Start with
fine scale, average to coarser scale

• Measures
• TS vs. scale
• depth vs. area
• spatial scaling parameter
• dynamic scaling exponent

Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
42
(No Transcript)
43
Upscaling verification of IR power law
rainrate16 September 2002, Melbourne
mm hr-1
IR
IR
radar
radar
44
GMSRA validated against rain gauge analyses at
different spatial scales (Ba and Gruber, 2001)
45
Entity-based methods

• Use pattern matching to associate forecast and
  observed entities ("blobs"). Verify the
  properties of the entities.
• Examples
• CRA (contiguous rain area) verification

Advantages Intuitive, quantifies "eyeball"
verification Disadvantage May fail if forecast
does not sufficiently resemble observations
Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
46
CRA (entity) verification
Ebert and McBride (J. Hydrology, Dec
2000) Concept Verify the properties of the
forecast (estimated) entities against observed
entities Method Pattern matching to determine
location error, error decomposition, event
verification

• Measures
• location error
• size error
• error in mean, max values
• pattern error

Verifies this attribute? Location Size Shape Mean
value Maximum value Spatial variability
47

• Determine the location error using pattern
  matching
• Horizontally translate the estimated blob until
  the total squared error between the estimate and
  the observations is minimized in the shaded
  region. Other possibilities maximum correlation,
  maximum overlap
• The displacement is the vector difference between
  the original and final locations of the estimate.

48
CRA error decomposition The total mean squared
error (MSE) can be written as MSEtotal
MSEdisplacement MSEvolume MSEpattern The
difference between the mean square error before
and after translation is the contribution to
total error due to displacement, MSEdisplacement
MSEtotal MSEshifted The error component due
to volume represents the bias in mean
intensity, where and are the CRA mean
estimated and observed values after the
shift. The pattern error accounts for differences
in the fine structure of the estimated and
observed fields, MSEpattern MSEshifted -
MSEvolume
49
24-hr rainfall from NRL Experimental
Geostationary algorithm validated against
Australian operational daily rain gauge analysis
50
(No Transcript)
51
(No Transcript)
52
Diagnosis of systematic errors
NRL Experimental Geostationary algorithm 289
CRAs April 2001-March 2002
53
Diagnosis of systematic errors
NRL Experimental Geostationary algorithm 289
CRAs April 2001-March 2002
54
Tropical Rain Potential (TRaP) verification?
55
Which methods verify which attributes?
56
Conclusions

• The most effective diagnostic verification method
  is still visual ("eyeball") verification.
• Categorical statistics based on yes-no
  discrimination are probably the least informative
  of all of the verification methods, although they
  remain very useful for quantitative algorithm
  intercomparison.
• The newer diagnostic verification methods (scale
  decomposition, entity-based) give a more complete
  and informative diagnosis of algorithm
  performance
• Need methods to deal with observational
  uncertainty

57
http//www.bom.gov.au/bmrc/wefor/staff/eee/verif/v
erif_web_page.shtml
__________________ UNDER CONSTRUCTION ___________
_______