Systematic Errors

1
Systematic Errors
Thomas Jung European Centre for Medium-Range
Weather Forecasts
2
Scope of the Lecture
Question Do we still have significant
systematic errors in the ECMWF forecasting
system?

• If so, what are the main problems?
• How do systematic errors grow?
• How did systematic errors evolve over the years?
• How well do we simulate specific phenomena
  (e.g., blocking, extratropical cyclones)?
• Does increasing resolution help?
• Which techniques can be used to understand
  systematic error?

3
Introduction

• Two principal sources of forecast error
• Uncertainties in the initial conditions
  (observational error)
• Model error

Two kinds of forecast error

• Random error (modelinitial error)
• Systematic error (model error)

4
Concept of Systematic Error

• Relatively straightforward to compute
• But there are pitfalls
• finite length (significance tests may help)
• Systematic error may be misleading for short
  time series (loss of predictability)
• observations might be biased
• Systematic errors can be difficult to interpret.

5
Loss of Predictability and Systematic Error
Forecast
Observed
6
Example Z500 DJF 2005/06
Spatial correlation-0.78
7
Do we know the true state?
8
Data

• Medium-range forecasts
• Primarily hindcasts from ERA-40 and ERA-Interim
  (robust statistics).
• Seasonal integrations
• 13 month long integrations started on 1st
  November of each of the years 1962-2005 (or a
  subset of this period).
• Diagnosis of standard seasons DJF, MAM, JJA, SON
  (errors had at least 1 month to grow, asymptotic
  errors).
• Most experiments at TL159 with 91 levels in the
  vertical.
• Observed lower boundary conditions (uncoupled
  integrations).
• Observational data
• ERA-40
• Other observational data sets.

9
Systematic Error Growth
How do systematic errors grow throughout the
forecast?
10
Systematic Z500 Error Growth from D1 to D10
D1
D3
D5
D10
11
Systematic Z500 Errors Medium-Range and Beyond
Asymptotic 31R2
D10 ERA-Interim
12
Evolution of Systematic Error
How did systematic errors evolve throughout the
years?
13
Evolution of D3 Systematic Z500 Errors
1981-1985
1993-1997
2006-2010
14
Evolution of Systematic Z500 Errors Model Climate
35R1
33R1
32R3
32R2
32R1
31R1
30R1
29R2
15
Systematic Z500 Errors Impact of Recent Changes
CNT-Reanalysis (33R1)
Old Convection-CNT
Old TOFD-CNT
Old Vertical Diff-CNT
Old Radiation-CNT
Old Soil Hydrology-CNT
16
Phenomena
17
Blocking Methodology
L
H
L
18
Blocking Frequency Errors (23r4)
DJF 1990-2001
19
Blocking Frequency Errors (31R2)
DJF 1990-2001
20
Blocking Frequencies DJF 1990-2005
21
Blocking Frequencies DJFM
22
Extratropical Synoptic Systems

• Two approaches
• Compute standard deviation of highpass or
  bandpass filtered time series (Eulerian
  approach).
• Track individual systems (Lagrangian
  approach).

23
Observed Tracks of Long-lived Cyclones
gt2 days
24
Sensitivity to Horizontal Resolution 29R1
25
Sensitivity to Horizontal Resolution 31R1
26
Number of Extratropical Cyclones (DJFM)
Lifetime gt 1day
27
Synoptic Activity Error Evolution
35R1
32R2
31R1
33R1
32R3
30R1
28
Resolution and Computational Effort
Increased cost compared to TL159

• 33 times for TL511
• 520 times for TL1279
• 2136 times for TL2047

29
The Madden-and Julian Oscillation
30
Schematic of the MJO
From Madden and Julian (1994)
31
Near Global Impact of the MJO (Precipitation)
Conv. Indian Ocean
Conv. Maritime Continent
Conv. Central Pacific
Conv. WH/Africa
32
The Madden and Julian Oscillation
Reanalysis
T159
T511
T1279
T2047
33
Conclusions (1)
Main systematic errors

• Concept of systematic error is very
  straightforward.
• But there are pitfalls
• Short time series (sampling issues loss
  predictability).
• True state of the atmosphere not always well
  known.
• Systematic error are (usually) a clear sign of
  model error.
• Understanding systematic error is challenging.

34
Conclusions (2)
Main systematic errors
Do we still have systematic errors in the ECWMF
model?

• Yes, we do.
• However, most systematic errors have been
  substantially reduced in recent years.
• Some key-challenges remain!

35
Conclusions (3)
Main systematic errors
Key-challenges

• Tropical circulation hydrological cycle
• Madden-Julian Oscillation
• Indian Summer Monsoon
• Quasi-Biennial Oscillation
• Euro-Atlantic blocking
• Severe weather (e.g. wind storms)
• Others (surface related fields)

36
Further Reading
Bechtold, P., M Koehler, T. Jung, M. Leutbecher,
M. Rodwell and F. Vitart, 2008 Advances in
simulating atmospheric variability with the ECMWF
model From synoptic to decadal time scales.
Quart. J. Roy. Meteor. Soc., 134,
1337-1351. Jung, T. and A.M. Tompkins, 2003
Systematic Errors in the ECMWF Forecasting
System. ECMWF Technicial Memorandum 422.
http//www.ecmwf.int/publications/library/do/refer
ences/list/14 Jung, T., A.M. Tompkins, and R.J.
Rodwell, 2004 Systematic errors in the ECMWF
forecasting system. ECMWF Newsletter, 100,
14-24. Jung, A.M. Tompkins, and R.J. Rodwell,
2005 Some Aspects of Systematic Errors in the
ECMWF Model. Atmos. Sci. Lett., 6, 133-139. Jung,
T, et al., 2010 The ECMWF model climate Recent
progress through improved physical
parametrizations. Quart. J. Roy. Meteor. Soc,
136, 1145-1160. Jung, T., 2005 Systematic errors
of the atmospheric circulation in the ECMWF
model. Quart. J. Roy. Meteor. Soc., 131,
1045-1073. Jung, T., S.K. Gulev, I. Rudeva and V.
Soloviov, 2006 Sensitivity of extratropical
cyclone characteristics to horizontal resolution
in the ECWMF model. Quart J. Roy. Meteor. Soc.,
132, 1839-1857. Hoskins, B.J. and K.I. Hodges,
2002 New perspectives on the Northern winter
storm tracks. J. Atmos. Sci., 59,
1041-1061. Palmer, T.N., G. Shutts, R. Hagedorn,
F. Doblas-Reyes, T. Jung, and M. Leutbecher,
2005 Representing model uncertainty in weather
and climate prediction. Ann. Rev. Earth. Planet
Sci., 33, 163-193. Rodwell, MJ and T Jung, 2008
Understanding the local and global impacts of
model physics changes An aerosol example. Quart.
J. Roy. Meteor. Soc., 134, 1479-1497.
37
Recent Model Changes
29R2 28 Jun 2005 Modifications to convection
30R1 1 Feb 2006 Increased resolution (L60 to L91)
31R1 12 Sep 2006 Revised cloud scheme (ice supersaturation numerics) implicit computation of convective transports introduction of orographic form drag scheme revised GWD scheme
32R1 not operational New short-wave radiation scheme introduction of McICA cloud-radiation interaction MODIS aerosol revised GWD scheme retuned ice particle size
32R2 5 Jun 2007 Minor changes to forecast model
32R3 6 Nov 2007 New formulation of convective entrainment and relaxation time scale reduced vertical diffusion in the free atmosphere modification to GWD scheme (top of the model) new soil hydrology scheme
33R1 3 Jun 2008 Slightly increased vertical diffusion increased orographic form drag retuned entrainment in convection scheme bugfix scaling of freezing term in convection scheme changes to the surface model
38
Sensitivity to Model Formulation
39
Systematic Errors AMIP Models
40
Demeter Models (DJF Precipitation)
LODYC
CERFACS
ECMWF
Meteo-France
MPI
MetOffice