Robin Hogan

1
Quantitative verification of cloud fraction
forecasts

• Robin Hogan
• Malcolm Brooks
• Trevor Flint

2
Overview

• Now have over a year of cloud data from
  Chilbolton and Cabauw, and four models to compare
  with.
• Previous comparisons of mean and PDF of cloud
  fraction and water content evaluate the climate
  of a model but not the weather.
• In this talk I use skill scores, commonly used in
  precipitation verification, to evaluate cloud
  fraction forecasts in the four models.

3

• Chilbolton Radar
• ChilboltonLidar

Month of target classification
4
Derived cloud fraction

• Use model winds/resolution to determine averaging
  time, count pixels to get volumetric cloud
  fraction in each model grid box (different grid
  for each model)

5
Met Office ECMWF cloud fraction

• Met Office Unified Model (mesoscale version)
• ECMWF global model

6
RACMO Météo France cloud fraction

• KNMI Regional atmospheric climate model (RACMO)
• Météo France ARPEGE model

7
Contingency tables

• Comparison with Met Office model over Chilbolton,
  October 2003

Observed cloud Observed clear-sky

• Model cloud
• Model clear-sky

A Cloud hit B False alarm
C Miss D Clear-sky hit
8
Simple skill scoreHit Rate
Met Office short range forecast
Météo France old cloud scheme

• Hit Rate fraction of forecasts correct
  (AD)/(ABCD)
• Consider all Cabauw data, 1-9 km
• Increase in cloud fraction threshold causes
  apparent increase in skill.

9
Scores independent of clearsky hits

• False alarm rate fraction of forecasts of cloud
  which are wrong B/(AB)
• perfect forecast is 0

• Probability of detection fraction of clouds
  correctly forecast A/(AC)
• perfect forecast is 1

• Skill decreases as cloud fraction threshold
  increases

10
More sophisticated scores

• Equitable threat score (A-E)/(ABC-E) where E
  removes those hits that occurred by chance.

• Yules Q (?-1)/(?1) where the odds ratio
  ?AD/BC.
• Advantage little dependence on frequency of cloud

• For both scores, 1 perfect forecast, 0 random
  forecast
• From now on use Equitable threat score with
  threshold of 0.1.

11
Skill versus heightChilbolton

• Lower skill for low clouds
• Both clouds and model levels are thinner, so more
  difficult to forecast?
• As seen before, much more cloud in models than
  observations above 8 km
• Could be due to radar sensitivity

12
Skill versus heightCabauw

• Model performance
• ECMWF, RACMO, Met Office models perform similarly
• Météo France not so well, much worse before April
  2003
• Met Office model significantly better for shorter
  lead time

13
Skill versus time

• Cabauw Equitable threat score
• Cabauw mean cloud fraction
• Chilbolton Equitable threat score
• Chilbolton mean cloud fraction

Change in Météo France cloud scheme April 2003
14
Forecast degredation and scale

• Met Office forecasts available for different lead
  times
• Forecast degradation can be quantified

• Gridbox comparisons are very tough tests
• Averaging in time gives a better indication of
  skill in forecasting large scale features

15
Conclusions

• Skill scores provide a quantitative framework for
• Comparing one model against another
• Evaluating impact of a new cloud scheme
• Results so far show that
• Met Office, ECMWF and RACMO perform similarly for
  similar forecast lead times
• Short range Met Office forecasts are generally
  better
• Météo France scores are the lowest, although much
  improved since April 2003 when the cloud scheme
  was changed
• Future work
• Other scores, e.g. ROC, Brier score etc.
• Horizontal vertical averaging to test model at
  different scales
• Compare Met Office global mesoscale models, new
  cloud schemes etc.