Model Verification and Diagnostic Method

1
Model Verification and Diagnostic Method

• Yuejian Zhu
• Environmental Modeling Center
• NOAA/NWS/NCEP
• (September 12th 2006)

2
Contents

• Introduction
• Accuracy (forecast against analysis)
• Consistency (forecast against forecast)
• Precipitation verification
• Diagnostics (forecast against analysis)
• Zonal mean crossing section
• Area mean vertical plots
• Instantaneous global map plots
• Zonal mean model physics
• Surface variables
• Climatology from 20 years observation
• Conclusions
• Expectations

3
Northern Hemisphere 500hPa geopotential height
MWR Jan. 2005
Pattern Anomaly Correlation
Root Mean Square
Simple Measurement For Ensemble mean
4
NCEP ensemble mean performance for past 5-year
5
MWR Jan 2006
Based on re-analysis monthly climatology
6
Verification Accuracyforecast against analysis

• Verification
• Based on 2.5 by 2.5 degree resolution global data
• Variables include
• Height 1000hPa and 500hPa for ex-tropic (20-80)
• U and V 850hPa and 200hPa for tropic region
• RMSs for
• Analysis and forecast
• Forecast and climatology
• Mean errors for
• Analysis and forecast
• Forecast and climatology
• PACs for
• Analysis anomaly and forecast anomaly
• Climatology (NH CPC 15 years, SH CPC 8 years,
  TR NCEP/NCAR 17 years reanalysis (New
  climatology is available now)
• Based on zonal wave groups
• Wave 1-3, 4-9, 10-20, 1-20
• Output example (next slide)

7
File/global/vrfy/SCORESs.2006091000 N Hem AC
by Waves ( Z 20N-80N ) - 1-3 4-9
10-20 1-20 0 hrs after 0Z 9 10 6
500mb .9997 .9997 .9978 .9996 24 hrs
after 0Z 9 9 6 500mb .9910 .9954
.9675 .9926 48 hrs after 0Z 9 8 6
500mb .9801 .9846 .9012 .9786 72 hrs
after 0Z 9 7 6 500mb .9527 .9725
.8423 .9577 96 hrs after 0Z 9 6 6
500mb .8734 .9391 .7382 .9035 120 hrs
after 0Z 9 5 6 500mb .7282 .7083
.5722 .7084 144 hrs after 0Z 9 4 6
500mb .5204 .6188 .4591 .5663 168 hrs
after 0Z 9 3 6 500mb .3941 .5021
.1802 .4286 192 hrs after 0Z 9 2 6
500mb -.0196 .1411 .0744 .0702 216 hrs
after 0Z 9 1 6 500mb -.2254 .3384
-.1796 .1190 240 hrs after 0Z 8 31 6
500mb .1714 .3108 .2148 .2416 264 hrs
after 0Z 8 30 6 500mb .2841 .1233
-.3993 .1642 288 hrs after 0Z 8 29 6
500mb .3842 .4616 -.0117 .4111 312 hrs
after 0Z 8 28 6 500mb -.1729 .0584
.0993 -.0391 336 hrs after 0Z 8 27 6
500mb .3505 .5301 .0215 .4217 360 hrs
after 0Z 8 26 6 500mb .6005 -.0643
.1385 .2888 384 hrs after 0Z 8 25 6
500mb .4517 .2523 .2839 .3370 N Hem
Differences ( Z 20N-80N ) - (F-A)RMS Mean
(F-C)RMS Mean 0 hrs after 0Z 9 10 6
500mb 2.40 -.72 80.68 19.06 24 hrs
after 0Z 9 9 6 500mb 10.71 -4.35
79.79 15.43 48 hrs after 0Z 9 8 6
500mb 17.25 -5.44 78.71 14.34 72 hrs
after 0Z 9 7 6 500mb 23.20 -4.38
79.48 15.40 96 hrs after 0Z 9 6 6
500mb 34.55 -5.98 79.06 13.80 120 hrs
after 0Z 9 5 6 500mb 57.84 -6.58
75.40 13.20 144 hrs after 0Z 9 4 6
500mb 70.65 -5.48 78.31 14.30 168 hrs
after 0Z 9 3 6 500mb 81.17 -7.71
78.64 12.07 192 hrs after 0Z 9 2 6
500mb 107.55 -4.70 82.72 15.08 216 hrs
after 0Z 9 1 6 500mb 104.91 -11.99
82.07 7.79 240 hrs after 0Z 8 31 6
500mb 90.64 -8.82 77.07 10.96 264 hrs
after 0Z 8 30 6 500mb 99.50 -4.62
85.07 15.16 288 hrs after 0Z 8 29 6
500mb 86.16 -10.88 81.58 8.90 312 hrs
after 0Z 8 28 6 500mb 102.74 -4.69
67.51 15.09 336 hrs after 0Z 8 27 6
500mb 82.02 -15.86 66.33 3.92 360 hrs
after 0Z 8 26 6 500mb 89.51 -7.02
78.74 12.76 384 hrs after 0Z 8 25 6
500mb 83.16 -4.53 73.21 15.25
8
Plotting (GrADs)

• Computer Platform
• IBM-SP
• Time Series
• RMS errors and PAC for different lead time
• Variables include
• Height 1000hPa and 500hPa for ex-tropic (20-80)
• U and V 850hPa and 200hPa for tropic region
• Die-off plots
• RMS errors and PAC for different variables
• Variables include
• Height 1000hPa and 500hPa for ex-tropic (20-80)
• U and V 850hPa and 200hPa for tropic region
• Plotting example (next slide)

9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
What does the numbers mean to scientist?
13
(No Transcript)
14
Useful information

• http//wwwt.emc.ncep.noaa.gov/gmb/yzhu/html/vrfy_g
  rads.html
• Verification scripts, program
• Plotting scripts, program
• http//wwwt.emc.ncep.noaa.gov/gmb/yzhu/
• My web-site contains
• GFS verification plots
• GEFS deterministic and probabilistic plots
• Forecast maps.
• NAEFS products

15
Synoptic example of 500hPa height forecast
Subjective evaluation
Ini 2003021200 Valid 2003021700
NCEP analysis
NCEP forecast
NCEP.7978 DAO.8257 MSC.6068
DAO forecast
MSC forecast
16
Good Examples
17
Bad example
18
Verification consistence

• Expect to have the same/similar solution from
  different initial conditions reliability.
• Consistency is different from accuracy.
• User consideration (requirement), user likes
  consistency forecasts.
• The similar properties deterministic
  (quantitative) or ensemble (probabilistic)
  forecasts.
• Consistency forecasts should increasing/decreasing
  predictability gradually from different cycles
  (6-hour or 24-hour)
• High consistency forecast is corresponding to
  high forecast accuracy and reliability

19
Measurement of Deterministic Forecast

• Any model forecast variables
• F(t) against F(t-1) (for same validation time)
• Example tt00z, t-1t18z(yesterday) or t00z
  (yesterday)
• Pattern Anomaly Correlation (PAC)
• To compare the PAC from different model
  forecasts.
• High correlation is better
• Root Mean Square (RMS) Error
• Low RMS error is better
• Comparison
• Could be time series
• Could be period average

20
Example for deterministic forecast

• NH 500hPa geopotential height
• PAC scores
• 1d means 24 hrs forecast .vs 48 hrs
• 2d means 48 hrs forecast against 72 hrs forecast
  valid at the same time, and so on

21
DiagnosticsZonal Mean- Crossing Section

• Variables should include
• Height, temperature, zonal wind, meridional wind,
  relative humidity, vertical motion and horizontal
  divergence.
• Forecast verified against global analysis
• Analysis
• Instantaneous, 6-hr, 1-d, and 5-d average
• Forecast
• Instantaneous for different forecast lead time
• Period average for fixed lead time
• Difference
• For different variables
• Different lead times
• Examples (next slides)

22
200hPa Jet
23
ITCZ
24
DiagnosticsArea Mean- Vertical plots

• Variables should include
• Mainly temperature, and precipitation
• Forecast verified against global analysis
• Lead time
• 6 hrs, 1-, 2- 16-d forecast lead time
• areas
• Global, 30-90N, 30-90S, 20-80N, 20-80S, 20N-20S
• Over land and over ocean
• Difference
• For different variables
• Different lead times
• Examples (next slides)

25
Global .vs Mid-Latitude
26
Land .vs Ocean
27
DiagnosticsInstantaneous - global map plots

• Variables should include
• Height, temperature, winds, relative humidity,
  vertical motion and divergence/vorticity
• Forecast verified against global analysis
• Lead time
• 6 hrs, 1-, 2- 16-d forecast lead time
• areas
• Global, 30-90N, 30-90S, 20-80N, 20-80S, 20N-20S
• Over land and over ocean
• Difference
• For different variables
• Different lead times
• Examples (next slides)

28
500hPa geopotential height
29
Analysis error?
Model error?
30
DiagnosticsZonal Mean Model Physics

• Variables should include
• Precipitation, evaporation, sensible/latent
  heating, 2m and skin temperature, precipitable
  water, total cloud cover, surface stress,
  radiations and etc
• Forecast verified against global analysis
• Lead time
• 6 hrs, 1-, 2- 16-d forecast lead time
• areas
• Global, 30-90N, 30-90S, 20-80N, 20-80S, 20N-20S
• Over land and over ocean
• Difference
• For different variables
• Different lead times
• Examples (next slides)

31
(No Transcript)
32
DiagnosticsSurface Variables - Climatology

• Needs reliable resource to compare
• Reference http//wwwt.emc.ncep.noaa.gov/gmb/noor/
  indest/indest.htm
• COADS and other observations
• Zonal surface stress
• Meridional surface stress
• Sensible heat
• Latent heat
• Surface net short wave radiation
• Outgoing long wave radiation

33
Precipitation 20 years statistics Visit
http//http//wwwt.emc.ncep.noaa.gov/gmb/noor/inde
st/precip/xiearkin/xaclim.htm