Evaluation of a limitedarea highresolution ensemble

1
Evaluation of a limited-area high-resolution
ensemble Chiara Marsigli, Andrea Montani,
Fabrizio Nerozzi, Tiziana Paccagnella ARPA-SMR,
Bologna, Italy cmarsigli_at_smr.arpa.emr.it
2
Wednesday
Sunday
Friday
0
48
Day 5 TEPS (51 members)
Day 3 TEPS (51 members)
0
TEPS Targeted Ensemble Prediction System (120km)
3
(No Transcript)
4
Methodology
Super ensemble 2 global ensembles starting 5/3
days before the verification time
102 members (50 1)2
Hierarchical Cluster Analysis method Complete
Linkage area Southern Europe fields 4 variables
at 4 levels (3D cluster) number of clusters
fixed to 5
5 clusters

• Representative Member Selection
• one per cluster
• base on the nearest (3D fields) to the mean of
  its own cluster AND the most distant to the other
  clusters means

5 representative members (RMs)
5 LAMBO integrations nested on 5 RMs LEPS -
Limited-area (High Resolution) Ensemble
Prediction System
5
L5w or L5nw
5 RMs IC and BC
LEPS (5 members)
Hydrostatic limited-area model LAMBO (20km)
6
LEPS verification on MAP cases

• 24h cumulated total precipitation
• observed precipitation from the MAP data-base
  (around 3000 stations covering the Alpine region)
  
• 15 cases in Autumn 1999

7
L5w
L5nw
T51d3
T102
SuperEnsemble 102 TEPS members
BSS
10 mm
Most recent TEPS of 51 members
ROC area
LEPS with the 5 runs weighted with cluster
population
BSS
50 mm
LEPS with the 5 runs equally weighted
ROC area
Hypothesis testing by RESAMPLING (Hamill, 1999)
8
Cost-loss Analysis
Value of the forecast systems (expressed as a
percentage of the value of a perfect forecast
system) as a function of the cost-loss ratio. The
systems are LEPS (green lines), TEPS 102-member
(blue line) and 3-day 51-member (cyan line) and
TEPS 5-member (red lines). The considered event
is precipitation exceeding 50mm/24h.
9
How many times does the T5w best member belong to
the most populated cluster?
Z700 ana forecast geopotential height at 700 hPa
against ECMWF analysis tp ana forecast
precipitation against 24h forecast
(analysis) tp obs forecast precipitation
interpolated on station points against MAP
observations tp obs2ecbox forecast precipitation
against MAP observation averaged over ECMWF TEPS
grid boxes
Percentage of times the best member of the 5-RM
TEPS belongs to the most populated cluster, to
the two most populated clusters, to the i most
populated clusters as a function of i.
10
How many times does the L5w best member belong to
the most populated cluster?
Z700 ana forecast geopotential height at 700 hPa
against ECMWF analysis tp ana forecast
precipitation against MAP analysed precipitation
(Frei and Haller) tp obs forecast precipitation
interpolated on station points against MAP
observations tp obs2ecbox forecast precipitation
averaged over ECMWF TEPS grid boxes against
observation averaged over the same boxes
Percentage of times the best member of the LEPS
belongs to the most populated cluster, to the two
most populated clusters, to the i most populated
clusters as a function of i.
11
LEPS Cases subdivided according to the amount of
precipitation recorded
ROC
BSS
Solid line the 4 most precipitating
cases Dashed line the remaining 11 cases
12
Conclusions

• LEPS is a useful probabilistic tool for the
  probabilistic prediction of heavy precipitation
• LEPS outperforms TEPS for high precipitation
  thresholds
• cluster population seems to be related to the
  probability of occurrence of the synoptic
  scenario depicted by the RM, but some care is
  needed when high-resolution precipitation is
  analysed

13
Brier Skill Score
Brier Score

• oi 1 if the event occurs
• 0 if the event does not occur 
• fi is the probability of occurrence according to
  the forecast system (e.g. the fraction of
  ensemble members forecasting the event)
• BS can take on values in the range 0,1, a
  perfect forecast having BS 0
• The forecast system has predictive skill if BSS
  is positive, a perfect system having BSS 1.

Brier Skill Score
total frequency of the event (sample
climatology)
14
ROC area
A contingency table can be built for each
probability class (a probability class can be
defined as the of ensemble elements which
actually forecast a given event). For the k-th
probability class
The area under the ROC curve is used as a
statistic measure of forecast usefulness
15
VERIFICATION COST-LOSS ANALYSIS

• The event E causes a damage which incur a loss L.
  The user U can avoid the damage by taking a
  preventive action which cost is C.
• U wants to minimize the mean total expense over a
  great number of cases.
• U can rely on a forecast system to know in
  advance if the event is going to occur or not.
• If the forecast system is probabilistic, the user
  has to fix a probability threshold. When this
  threshold is exceeded, it take protective action.

 
Mean expense
MEk
Vk
Value
16
Brier Score decomposition