Postprocessing of Ensemble MOS

1
Post-processing of Ensemble MOS

• Matthew Peroutka
• Meteorological Development Laboratory
• NWS Office of Science and Technology
• 3rd NCEP/NWS Ensemble User Workshop
• October 2006

2
Utility of Probabilities

• Probabilistic forecasts can be more useful to
  users than non-probabilistic.
• Not necessarily so.
• User must
• want to use
• be able to use
• actually use

3
Dichotomous vs. Quasi-continuous Weather Elements

• Dichotomous events are relatively simple.
• Disseminate a single probability to user.
• Examples
• Pcpn gt 0.01
• MinT lt 22 F

• Quasi-continuous case problematic.
• Concept, dissemination more difficult.
• Examples
• Temperature
• Wind Speed
• Ceiling Height

4
Temperature

• Quasi-continuousforecast and observed in
  increments of 1 F / 0.1 C.
• Wide range of values.
• Many potential users with diverse interests in
  various portions of range.
• It is not enough to set thresholds or define
  ranges. We need to forecast probability function
  (PDF, CDF, PPF).

5
Overall Goals(Characteristics of a Good
Probability Forecast)

• Unbiased (Compare E(X) to Obs)
• Reliable (Probability Integral Transform PIT
  Histogram)
• Skillful (Continuous Ranked Probability Score)
• Sharp (Credible Intervals)
• Overall shape of the distribution is informative.

6
Data Used

• Ensemble members (11) from North American
  Ensemble Forecast System (0000 UTC)
• Ensemble MOS (ENSMOS 0000 UTC)
• Operational MOS (0000 UTC)
• Verifying observations
• 1500 GFS MOS stations (CONUS and OCONUS)
• April 2004 through June 2006

7
Ensemble MOS

• MOS equations Applied to individual ensemble
  members.
• 11 separate ENSMOS text bulletins generated.
• Note that spread decreases with time projection.
• Box/whisker graphic at right from PSU.

8
Techniques

• ENSMOS Adjusted by SDe (EASe)
• Compute standard deviation of ensemble output
  (SDe) and ENSMOS output (SDm).
• Adjust spread of ENSMOS output accordingly.
• Use Kernel Density Estimation (KDE) to generate
  PDF.
• Probability Assessment using KDE, Ensembles, and
  MOS (PAKEM)
• Model relationship between forecast error and
  ensemble forecasts.
• Adjust spread of ensemble output accordingly.
• Debias adjusted values using operational MOS.
• Use Kernel Density Estimation (KDE) to generate
  PDF.

9
Kernal Density Estimation

• Extrapolates PDF (blue line) from a sample of
  data.
• N kernel functions (red dashed lines) are
  centered on data values (Gaussian functions in
  this case).

10
Effect of KDE Spread or Window Parameter on
the Sharpness of Distribution
h 3.7
h 1.1
h 0.5
h 0.1
11
Forecast Error and Ensemble Forecasts

• Variables
• MAE is mean absolute error of operational MOS.
• SDe is standard deviation of model ensemble
  temperature forecast.
• Histograms record relationship between MAE and
  SDe.
• Red bars each contain 10 of the data.
• Green bars each contain 1 of the data.
• Cumulative density plotted below on same scale.

12
Forecast Error and Ensemble Forecasts

• Data imply a linear relationship for lowest
  90-95 of the SDe values.
• MAE values seem to flatten out for highest SDe
  values.
• Note changes in relationship with longer
  projection times.
• Linear regression yields a very weak relationship.

13
Results

• Reliability/Bias (PIT Histograms)
• Sharpness (50 Credible Interval CI50)
• Skill (Continuous Ranked Probability Score CRPS)

14
Reliability/Bias

• PIT Histo-grams for EASe PAKEM techniques.
• EASe under-dispersed and cold biased.
• PAKEM seems to do better in both.

15
Sharpness

• For each forecast distribution, find T75 and T25,
  where P(TltTn) n.
• CI50 T75 - T25.
• PAKEM yields larger CI values since forecasts are
  spread more.

16
Skill

• CRPS integrates area between CDF and a perfect
  forecast.
• Perfect in this case is represented by a pulse
  or heavyside function.
• Need some standard (climatology, perhaps) for
  comparison.

17
Comments

• Obviously, a work in progress.
• Efforts to combine existing MOS techniques with
  ensemble output show promise.
• Seems possible to create probabilistic forecasts
  of temperature that are unbiased and reliable.
• Already we see trade-offs between skill,
  reliability, and sharpness.