IN SEARCH FOR THE BEST FORECAST 2

1
IN SEARCH FOR THE BEST FORECAST - 2
1) ESTIMATING THE FIRST MOMENT OF THE FORECAST
PDF BASED ON INFORMATION FROM HIGH RESOLUTION
CONTROL LOWER RESOLUTION ENSEMBLE
FORECASTS2) ENSEMBLE BASED ESTIMATES OF
FORECAST UNCERTAINTY AND THEIR DOWNSCALING TO
HIGHER RESOLUTION GRIDS

• Zoltan Toth, Jun Du(1), Yuejian Zhu
• Environmental Modeling Center
• NOAA/NWS/NCEP
• Ackn. Bo Cui, David Unger, Malaquias Pena
• (1) SAIC at EMC/NCEP/NWS/NOAA
• http//wwwt.emc.ncep.noaa.gov/gmb/ens/index.html

2
OUTLINE / SUMMARY

• ENSEMBLE BASED ESTIMATES OF FORECAST UNCERTAINTY
  AND THEIR DOWNSCALING TO HIGHER RESOLUTION GRIDS
• ESTIMATING THE FORECAST PDF BASED ON
• SINGLE FORECASTS
• ENSEMBLES
• STATISTICAL POSTPROCESSING
• BIAS REDUCTION WRT OPERATIONAL ANALYSIS
• DOWNSCALING
• FORECAST REPRESENTATIVE OF SMALLER SCALES
• FORECAST FORMAT
• ALL ENSEMBLE MEMBERS
• DERIVED PRODUCTS
• NDGD PROPOSAL
• REPRESENTING FORECAST UNCERTAINTY
• 10, 50, 90 PERCENTILE OF FORECAST PDF

3
INTRODUCTION

• GOAL
• Best estimate of future state
• Verify by standard probabilistic statistics
• APPROACH
• Use all available information, including
• High resolution control forecast
• Lower resolution ensemble forecasts
• Climatology of observations/analysis and
  forecasts if available
• FORMAT
• PDF for single variables
• Ensemble traces to carry temporal/spatial/cross-va
  riable covariance info

4
GENERAL FORECAST APPROACH

• BASED ON A SINGLE FORECAST
• In nonlinear regime, sub-optimal performance
• Does not provide best estimate for expected value
• Does not provide case dependent estimates of
  forecast uncertainty
• In near-linear regime, can offer case dependent
  corrections for
• Lower resolution ensemble
• BASED ON AN ENSEMBLE
• Better estimate of expected value
• Case dependent estimates of forecast uncertainty
• DATABASE
• Collection of all relevant information
• High resolution control forecast
• Lower resolution ensemble forecasts
• Climatology of observations/analysis and
  forecasts if available
• Short range
• SREF ensemble control forecasts
• Medium- extended range

5
FORECASTING IN A CHAOTIC ENVIRONMENT
PROBABILISTIC FORECASTING BASED A ON SINGLE
FORECAST One integration with an NWP model,
combined with past verification statistics
DETERMINISTIC APPROACH - PROBABILISTIC FORMAT

• Does not contain all forecast information
• Not best estimate for future evolution of system
• UNCERTAINTY CAPTURED IN TIME AVERAGE SENSE -
• NO ESTIMATE OF CASE DEPENDENT VARIATIONS IN FCST
  UNCERTAINTY

6
FORECASTING IN A CHAOTIC ENVIRONMENT -
3DETERMINISTIC APPROACH - PROBABILISTIC FORMAT

• MONTE CARLO APPROACH ENSEMBLE FORECASTING
• IDEA Sample sources of forecast error
• Generate initial ensemble perturbations
• Represent model related uncertainty
• PRACTICE Run multiple NWP model integrations
• Advantage of perfect parallelization
• Use lower spatial resolution if short on
  resources
• USAGE Construct forecast pdf based on finite
  sample
• Ready to be used in real world applications
• Verification of forecasts
• Statistical post-processing (remove bias in 1st,
  2nd, higher moments)
• CAPTURES FLOW DEPENDENT VARIATIONS
• IN FORECAST UNCERTAINTY

7
CONFIGURATION, OUTPUT CHARACTERISTICS
2005, 2006, 2007, 2008
8
PROCESS

• Collection of relevant information (model grid)
• SREF
• NAEFS
• Statistical post-processing (model grid)
• Bias correction
• Statistical
• Case dependent (using hires control)
• Weights
• Verify added value
• Manual modification of forecast guidance (model
  grid)
• Verify added value
• Downscaling (NDGD grid)
• Observation locations
• NDGD grid
• Manual modification (NDGD grid)
• Verify added value

9
RAW BASIC PRODUCT AVAILABILITY
2005, 2006, 2007, 2008
10
STATISTICAL POST-PROCESSING

• PURPOSE
• Make all forecasts look like model analysis
• Independent of lead time
• Selected set of often used variables
• All ensemble members and hi-lower resolution
  controls
• Assign weights to each member, corresponding to
  its performance
• DATA
• Medium- extended range application
• 1x1 global grid, 50 variables
• NAEFS ensemble control forecasts
• Operational analyses
• TWO METHODS
• Frequentists approach
• Compare recent statistics of forecasts and
  analyses
• In collaboration with David Unger of CPC
• Bayesian approach
• In collaboration with Roman Krzysztofowicz

11
POST-PROCESSING METHODS - 1

• FREQUENTISTS APPROACH
• First moment (expected value) June 2006
• Estimate bias in first moment
• 35 selected variables
• Compare weighted mean of recent forecasts with
  that of verifying analysis
• Kalman-filter type adaptive technique works well
  for short lead time
• Remove estimated bias from each forecast member
• Improved 1st moment Oct 07
• More efficient for longer lead times
• Second moment bias correction Oct 07
• Weights Oct 07
• Consider case dependent reduction of bias in
  lowres ensemble
• Jun Dus hybrid approach
• Compare high lower resolution (ensemble)
  control forecasts
• Difference interpreted as resolution vector
• Adjust all members by resolution vector
• Potential for short lead times, linear regime

12
(No Transcript)
13
POST-PROCESSING METHODS - 2

• BAYESIAN APPROACH
• Based on single forecast PQPF processor of
  Krzysztofowicz et al
• Modified for ensemble forecast application
• Interpret/correct latest ensemble forecast in
  context of prior information
• Apply procedure on each variable to be corrected
  separately
• Use NCEP/NCAR reanalysis climatology as prior
• Good estimate, based on 50 years, 2.5x2.5 grid
• Adjust reanalysis climate (2.5x2.5) to estimate
  operational analysis climate (1x1)
• See downscaling vector below
• Represent climate pdf with parametric
  distribution (2-3 pars)
• Compare forecast distributions with observations,
  conditioned on fcst pdf?
• Represent forecast ensemble distribution with
  parametric distribution (2-3 vars)
• Use shorter time period (90 days) with current
  model/ensemble configuration
• Use Kalman-filter type approach, update prior
  with most recent comparison
• Transform statistics to multinormal space
• Form posterior pdf, based on prior and fcst-obs
  comparison
• Estimate approximate weights for each member
• Based on Bayesian statistics combined for a set
  of selected variables
• Retain temporal/spatial/cross-variable rank
  correlations present in ensemble

14
DOWNSCALING METHODS

• PURPOSE
• Provide bias-corrected forecasts at small scales
  of interest
• Observation points
• High resolution grid
• National Digital Forecast (or Guidance) Database
  (NDFD, NDGD)
• DATA
• Input
• Bias corrected ensemble forecasts
• Output
• Observation locations
• NDGD grid
• TWO METHODS
• Climate anomalies June 2006
• Downscaling vector - Oct 2007

15
DOWNSCALING CLIMATE ANOMALIES

• Apply procedure on each ensemble member
• 19 selected bias-corrected variables
• Adjust bias-corrected forecast to look like
  reanalysis
• Use standard Kalman-filter type bias-correction
  algorithm
• Evaluate systematic difference between
• CDAS analysis (2.5x2.5 grid) and
• Operational analyses (1x1 grid)
• Remove systematic difference from bias-corrected
  forecast (2.5x2.5)
• Compare adjusted bias-corrected forecast to
  reanalysis climate pdf
• Represent climate distribution by parametric pdf
  (2-3 vars)
• Determine climate percentile corresponding to
  forecast value

16
DOWNSCALING WITH DOWNSCALING VECTOR

• Determine downscaling vector
• Systematic difference between
• Operational analysis (1x1 grid)
• Interpolate to NDGD grid using standard routine
• Real Time Meso-scale Analysis (RTMA)
• Based on RUC, not cycled analysis
• Quality measured against independent data
• 4-5 variables (2m temp, dewpoint, 10m winds,
  precip)
• 5x5km NDGD grid
• Use standard Kalman-filter type bias-estimation
  correction algorithm
• Independent of lead time
• Ship downscaling vector to users
• 5x5 km grid send only for area of interest, 1
  field per variable /day
• Ship bias-corrected forecast information
• 1x1 lat/lon grid, all lead times of interest
• Interpolate on-site to 5x5km grid, using same
  standard routine as above
• Add downscaling vector to interpolated
  bias-corrected forecast
• Performance measure

17
FORECAST FORMAT

• Different users have different requirements
• Multiple formats must be offered
• Full format
• All bias-corrected ensemble members
• In-house, NCEP service centers, interested ftp
  users
• All other formats
• Can be derived from full format (all members)
• Ensemble functionalities
• Different products can be generated and
  distributed as needed
• NDGD proposal
• Important special application
• In addition to most likely scenario
• Provide information on forecast uncertainty

18
ENSEMBLE PRODUCTS - FUNCTIONALITIES
List of centrally/locally/interactively generated
products required by NCEP Service Centers for
each functionality are provided in attached
tables (eg., MSLP, Z,T,U,V,RH, etc, at
925,850,700,500, 400, 300, 250, 100, etc hPa)
Potentially useful functionalities that need
further development - Mean/Spread/Median/Ranges
for amplitude of specific features -
Mean/Spread/Median/Ranges for phase of specific
features
Additional basic GUI functionalities - Ability
to manually select/identify members - Ability to
weight selected members Sept. 2005
19
ENSEMBLE PRODUCT REQUEST LIST NCEP SERVICE
CENTERS, OTHER PROJECTS
20
NDGD PROPOSAL

• NDFD current status
• 5x5km (or 2.5x2.5km) grid, 15-20 variables, out
  to 7 days
• Most likely value (expected value of forecast
  pdf) provided only
• Format of official NWS forecast
• NDGD new vehicle to provide guidance to be used
  in NDFD process
• Same format as NDFD
• Proposal to add forecast uncertainty information
  in NDGD
• Forecast values corresponding to 10, 50, 90
  percentile of forecast pdf
• Bias corrected forecast data, 1x1 grid June 06
• Downscaling vector for 1-2 variables, 5x5km
  grid Oct 07
• Quality to be assessed
• Can be expanded to include more NDFD variables

21
MANUAL MANIPULATION OF FORECASTS

• Manipulation needed for
• Bias-corrected forecast on model grid (national
  guidance)
• Downscaled forecast on NDGD grid (local guidance)
• Suggestion
• Initially modify only forecast value
  corresponding to 50 percentile of fcst pdf
• Use standard IFPS, NAWIPS tools
• Verify added value
• Move whole forecast distribution (10 90
  percentiles) with same adjustment
• Later attempt to modify forecast values
  corresponding to 10 90 perc.
• Verify added value

22
BACKGROUND
23
PRACTICAL APPROACH

• BIAS CORRECTION
• Adjust both hires control lores ensemble
• Ensures maximum utility of all forecasts
• Limits hires advantage to
• Flow dependent bias correction that statistical
  method cannot achieve
• IDENTIFY CROSS-OVER POINT
• Compare measures normalized by ens spread
• Measure of nonlinearity
• Control minus ens mean over ens spread
• Measure of hires advantage
• Error reduction in hires vs. lores controls over
  ens spread
• Alternatively, assess ens mean error dropping
  below hires control error
• USE OF HIRES CONTROL
• Before turnover point
• Quantitatively
• Develop methods to objectively combine info from
  hires control lores ens
• Bayesian approach?

24
INTRODUCTION

• GOAL
• Best estimate of future state
• Verify by standard probabilistic statistics
• FORMAT
• PDF for single variables
• Ensemble traces to carry temporal/spatial/cross-va
  riable covariance info
• DATABASE
• Collection of all relevant information
• High resolution control forecast
• Lower resolution ensemble forecasts
• Climatology of observations/analysis and
  forecasts if available
• Short range
• SREF ensemble control forecasts
• Medium- extended range
• North American Ensemble Forecast System (NAEFS)
• Current Canadian NCEP global ensembles (80
  members/day)
• Planned FNMOC, JMA added by FY08 (160
  members/day)

25
OUTLINE / SUMMARY

• ENSEMBLE BASED ESTIMATES OF FORECAST UNCERTAINTY
  AND THEIR DOWNSCALING TO HIGHER RESOLUTION GRIDS
• UNCERTAINTY RELATED TO THE MOST LIKELY SOLUTION
• LINK WITH TRADITIONAL FORECAST APPROACH
• STATISTICAL METHODS OF ASSESSING FORECAST
  UNCERTAINTY
• CHOICE OF THE PAST
• NOT CASE DEPENDENT NEED ANOTHER APPROACH
• MUST USE (POSSIBLY LOWER RESOLUTION) ENSEMBLE
• CAPTURE UNCERTAINTY ON RESOLVED SCALES
• COMMUNICATING FORECAST UNCERTAINTY
• MULTI-TIER APPROACH DEPENDING ON LEVEL OF USER
  SOPHISTICATION
• 10, 50, 90 PERCENTILE FORECAST VALUES FOR GENERAL
  USERS
• MORE DETAILED INFO FOR SOPHISTICATED USERS
• MULTIPLE SCENARIOS (MODES)
• FULL PDF

26
STATISTICAL ESTIMATION OF FORECAST UNCERTAINTY

• BIAS CORRECTION
• Adjust both hires control lores ensemble
• Ensures maximum utility of all forecasts
• Limits hires advantage to
• Flow dependent bias correction that statistical
  method cannot achieve
• IDENTIFY CROSS-OVER POINT
• Compare measures normalized by ens spread
• Measure of nonlinearity
• Control minus ens mean over ens spread
• Measure of hires advantage
• Error reduction in hires vs. lores controls over
  ens spread
• Alternatively, assess ens mean error dropping
  below hires control error
• USE OF HIRES CONTROL
• Before turnover point
• Quantitatively
• Develop methods to objectively combine info from
  hires control lores ens
• Bayesian approach?

27
IN SEARCH FOR THE BEST FORECAST - 1
1) ESTIMATING THE FIRST MOMENT OF THE FORECAST
PDF BASED ON INFORMATION FROM HIGH RESOLUTION
CONTROL LOWER RESOLUTION ENSEMBLE
FORECASTS2) ENSEMBLE BASED ESTIMATES OF
FORECAST UNCERTAINTY AND THEIR DOWNSCALING TO
HIGHER RESOLUTION GRIDS

• Zoltan Toth, Jun Du(1), Yuejian Zhu
• Environmental Modeling Center
• NOAA/NWS/NCEP
• Ackn. Bo Cui, David Unger, Malaquias Pena
• (1) SAIC at EMC/NCEP/NWS/NOAA
• http//wwwt.emc.ncep.noaa.gov/gmb/ens/index.html

28
OUTLINE / SUMMARY

• ESTIMATING THE FIRST MOMENT OF THE FORECAST PDF
  BASED ON INFORMATION FROM HIGH RESOLUTION CONTROL
  LOWER RESOLUTION ENSEMBLE FORECASTS
• RECONCILING INFO FROM HIRES CONTROL VS. LORES ENS
• ADVISE USER COMMUNITY
• REDESIGN NWP SUITE IF NEEDED
• ATTRIBUTES OF FORECAST SYSTEMS
• POSITIVE ATTRIBUTES OF HIRES CONTROL VS. LORES
  ENSEMBLE
• FIRST MOMENT ESTIMATION (MOST LIKELY STATE)
• WHAT LIMITES USE OF HIRES CONTROL
• LEVEL OF DIFFERENCE IN QUALITY OF HIRES VS. LORES
  CONTROLS
• EMERGENCE OF NONLINEARITIES
• PRACTICAL APPROACH
• BIAS CORRECT ALL FORECASTS
• USE OF HIRES CONTROL
• LINEAR PHASE
• Objectively combine hires control lores ens

29
INTRODUCTION

• PROBLEM
• Given high resolution single control and lower
  resolution ensemble forecasts
• What is best way to construct forecast
  probability density function (pdf)?
• Estimate first moment of distribution
• Estimate second and higher moments
• How to modify operational suite of NWP forecasts
  for best performance?
• Future choices to be made based on evaluation
  results
• ATTRIBUTES OF FORECAST SYSTEMS
• RELIABILITY
• Statistically, forecasts look like reality,
  irrespective of how skilful they are
• With large enough sample of forecast-observation
  pairs, can be improved
• RESOLUTION
• Sequence of observed events captured,
  irrespective of realism of forecasts
• Cannot be improved by statistical methods
• USER REQUIREMENTS
• Both forecast attributes important

30
HIGHRES CONTROL VS. LOWRES ENSEMBLE

• RELIABILITY
• Highres expected to be better, especially at low
  levels affected by terrain etc
• Reduced bias
• Some phenomena simulated with more fidelity (eg,
  diurnal cycle, frontal structure, etc)
• Quantify advantage of highres model before and
  after bias correction
• Bias correction benefits more the forecasts with
  larger bias gt
• Effect of bias correction expected to reduce
  advantage of highres forecast
• RESOLUTION
• At short lead time
• Highres may offer advantage?
• Never tested, needs to be evaluated
• At later lead time
• Ensemble may become advantageous
• Filtering of nonlinearly growing errors
• HOW TO RECONCILE FORECAST INFORMATION
• FROM HIRES CONTROL LOWRES ENS?

31
RECONCILING HIGHRES CONTROL VS. LOWRES ENS

• WMO/CBS Ensemble Expert Team meeting (Feb. 2006)
• Guidance sought by WMO members as to relevance of
  hires vs. lowres ensemble forecasts for
• First moment estimation
• Forecast uncertainty estimation
• Some related studies from various viewpoints in
  literature
• Hires control vs. ens mean (Zhu, Atger, Du, etc)
• Probabilistic forecasts based on hires control
  vs. lores ens
• Zhu et al, Atger, Du
• Combined use of hires control lores ensemble
• Smith et al., J. Du
• Following careful evaluation of strength
  weaknesses of both systems
• Advise user community on best practices
• Develop new tools if necessary to facilitate best
  use
• Redesign NWP forecast suite if necessary

32
ESTIMATING FIRST MOMENT

• EMERGENCE OF NONLINEARITIES
• Initially symmetric cloud of ensemble becomes
  distorted with lead time
• Measure level of nonlinearity by deviation of
  lores control and ens mean
• Normalized measure Control minus ens mean over
  ens spread
• What is critical level? 20 or what? (0linear,
  1nonlinear)
• Linearity is lost sooner for smaller scale / more
  unstable phenomena
• Once linearity is critically violated, relying on
  single control (even hires) not appropriate
• All ensemble solutions must be considered
• BEFORE NONLINEARITY BECOMES DOMINANT
• Hires control can have value beyond lores
  ensemble
• Value beyond ensemble is limited
• Resolution is typically only twice that of
  ensemble
• Bias correction of all forecasts further reduces
  value of hires control
• Quantify advantage of hires control statistically
• Error reduction (hires vs. lores controls) as
  function of lead time
• Normalize by ensemble spread?
• After cross-over time, only very special,
  qualitative use of hires model

33
PRACTICAL APPROACH

• BIAS CORRECTION
• Adjust both hires control lores ensemble
• Ensures maximum utility of all forecasts
• Limits hires advantage to
• Flow dependent bias correction that statistical
  method cannot achieve
• IDENTIFY CROSS-OVER POINT
• Compare measures normalized by ens spread
• Measure of nonlinearity
• Control minus ens mean over ens spread
• Measure of hires advantage
• Error reduction in hires vs. lores controls over
  ens spread
• Alternatively, assess ens mean error dropping
  below hires control error
• USE OF HIRES CONTROL
• Before turnover point
• Quantitatively
• Develop methods to objectively combine info from
  hires control lores ens
• Bayesian approach?

34
SCIENTIFIC BACKGROUND WEATHER FORECASTS ARE
UNCERTAIN
Buizza 2002
35
(No Transcript)
36
BACKGROUND
37
COMMENTS ON JUNS APPROACH

• Bias correction / post-processing algorithm based
  on
• Difference between hires and lores controls
• Useful as long as
• Controls evolve similarly gt
• Difference dominated by systematic differences
  between 2 model resolutions
• Once 2 controls diverge chaotically, random
  errors introduced
• How to quantify?
• Measure difference between 2 controls, normalized
  by ens spread?
• Related to limit of linearity discussed earlier?
• Stop using method beyond a point (use correction
  with tapering weights?)
• Characteristics
• Positive
• Completely flow dependent corrections
• Negative
• Correction is limited by resolution/performance
  of hires model gt
• Statistical bias correction still needed

38
EVALUATION OF FORECAST SYSTEMS

• Some statistics based on forecast system only
• Other statistics based on comparison of forecast
  and observed systems gt
• FORECAST SYSTEM ATTRIBUTES
• Abstract concepts (like length)
• Reliability and Resolution
• Both can be measured through different statistics
• Statistical properties
• Interpreted for large set of forecasts (ie,
  describe behavior of forecast system),
• not for a single forecast
• For their definition
• Assume that forecasts
• Can be of any format
• Take a finite number of different classes
• Consider empirical frequency distribution of
• Verifying observations corresponding to large
  number of forecasts of same class gt
• Observed Frequency Distribution (ofd)

39
STATISTICAL RELIABILITY STATISTICAL CONSISTENCY
OF FORECASTS WITH OBSERVATIONS

• BACKGROUND
• Consider particular forecast class Fa
• Consider distribution of observations Oa that
  follow forecasts Fa
• DEFINITION
• If forecast Fa has the exact same form as Oa, for
  all forecast classes,
• the forecast system is statistically consistent
  with observations gt
• The forecast system is perfectly reliable
• MEASURES OF RELIABILITY
• Based on different ways of comparing Fa and Oa

40
STATISTICAL RESOLUTION ABILITY TO DISTINGUISH,
AHEAD OF TIME, AMONG DIFFERENT OUTCOMES

• BACKGROUND
• Assume observed events are classified into finite
  number of classes
• DEFINITION
• If all observed classes are preceded by
  distinctly different forecasts, the forecasts
  resolve the problem gt
• The forecast system has perfect resolution
• MEASURES OF RELIABILITY
• Based on degree of separation of distributions of
  observations that follow various forecast classes
  
• Measured by difference between ofds climate
  distribution
• Measures differ by how differences between
  distributions are quantified

FORECASTS
OBSERVATIONS
EXAMPLES
41
CHARACTERISTICS OF FORECAST SYSTEM ATTRIBUTES

• Reliability resolution are general forecast
  attributes
• Valid for any forecast format (single,
  categorical, probabilistic, etc)
• Reliability
• Can be statistically imposed at one time level
• If both natural forecast systems are stationary
  in time, and
• If there is a large enough set of
  observed-forecast pairs
• Replace forecast by corresponding observed
  frequency distribution
• Not related to time sequence of forecast/observed
  systems
• Resolution reflects inherent value of forecast
  system
• Can be improved only through more knowledge about
  time sequence of events
• Statistical consistency at one time level
  (reliability) is irrelevant
• Reliability resolution are independent
  attributes
• Climate pdf forecast is perfectly reliable, yet
  has no resolution
• Reversed rain / no-rain forecast can have perfect
  resolution and no reliability
• Perfect reliability and perfect resolution
  perfect forecast system

42
FORECAST METHODS

• Empirically based
• Based on record of observations gt
• Possibly very good reliability
• Will fail in new (not yet observed) situations
  (eg., climate trend, etc)
• Resolution (forecast skill) depends on length of
  observations
• Useful for now-casting, climate applications
• Not practical for typical weather forecasting
• Theoretically based
• Based on general scientific principles
• Incomplete/approximate knowledge in NWP models gt
• Prone to statistical inconsistency
• Run-of-the-mill cases can be statistically
  calibrated to insure reliability
• For forecasting rare/extreme events, statistical
  consistency of model must be improved
• Predictability limited by
• Gaps in knowledge about system
• Errors in initial state of system

43
USER REQUIREMENTSPROBABILISTIC FORECAST
INFORMATION IS CRITICAL

44
FORECASTING IN A CHAOTIC ENVIRONMENT
PROBABILISTIC FORECASTING BASED A ON SINGLE
FORECAST One integration with an NWP model,
combined with past verification statistics
DETERMINISTIC APPROACH - PROBABILISTIC FORMAT

• Does not contain all forecast information
• Not best estimate for future evolution of system
• UNCERTAINTY CAPTURED IN TIME AVERAGE SENSE -
• NO ESTIMATE OF CASE DEPENDENT VARIATIONS IN FCST
  UNCERTAINTY

45

• FORECASTING IN A CHAOTIC ENVIRONMENT - 2
• DETERMINISTIC APPROACH - PROBABILISTIC FORMAT
• PROBABILISTIC FORECASTING -
• Based on Liuville Equations
• Continuity equation for probabilities, given
  dynamical eqs. of motion
• Initialize with probability distribution
  function (pdf) at analysis time
• Dynamical forecast of pdf based on conservation
  of probability values
• Prohibitively expensive -
• Very high dimensional problem (state space x
  probability space)
• Separate integration for each lead time
• Closure problems when simplified solution sought

46
FORECASTING IN A CHAOTIC ENVIRONMENT -
3DETERMINISTIC APPROACH - PROBABILISTIC FORMAT

• MONTE CARLO APPROACH ENSEMBLE FORECASTING
• IDEA Sample sources of forecast error
• Generate initial ensemble perturbations
• Represent model related uncertainty
• PRACTICE Run multiple NWP model integrations
• Advantage of perfect parallelization
• Use lower spatial resolution if short on
  resources
• USAGE Construct forecast pdf based on finite
  sample
• Ready to be used in real world applications
• Verification of forecasts
• Statistical post-processing (remove bias in 1st,
  2nd, higher moments)
• CAPTURES FLOW DEPENDENT VARIATIONS
• IN FORECAST UNCERTAINTY

47
NCEP GLOBAL ENSEMBLE FORECAST SYSTEM
MARCH 2004 CONFIGURATION
48
MOTIVATION FOR ENSEMBLE FORECASTING

• FORECASTS ARE NOT PERFECT - IMPLICATIONS FOR
• USERS
• Need to know how often / by how much forecasts
  fail
• Economically optimal behavior depends on
• Forecast error characteristics
• User specific application
• Cost of weather related adaptive action
• Expected loss if no action taken
• EXAMPLE Protect or not your crop against
  possible frost
• Cost 10k, Potential Loss 100k gt Will protect
  if P(frost) gt Cost/Loss0.1
• NEED FOR PROBABILISTIC FORECAST INFORMATION
• DEVELOPERS
• Need to improve performance - Reduce error in
  estimate of first moment
• Traditional NWP activities (I.e., model, data
  assimilation development)
• Need to account for uncertainty - Estimate higher
  moments
• New aspect How to do this?
• Forecast is incomplete without information on
  forecast uncertainty
• NEED TO USE PROBABILISTIC FORECAST FORMAT

49
HOW TO DEAL WITH FORECAST UNCERTAINTY?

• No matter what / how sophisticated forecast
  methods we use
• Forecast skill limited
• Skill varies from case to case
• Forecast uncertainty must be assessed by
  meteorologists

THE PROBABILISTIC APPROACH
50
SOCIO-ECONOMIC BENEFITS OFSEAMLESS
WEATHER/CLIMATE FORECAST SUITE
Commerce Energy
Ecosystem Health
Hydropower Agriculture
Boundary Condition Sensitivity
Reservoir control Recreation
Transportation Fire weather
Initial Condition Sensitivity
Flood mitigation Navigation
Protection of Life/Property
Weeks
Minutes
Days
Hours
Years
Seasons
Months
51
(No Transcript)
52
(No Transcript)
53
144 hr forecast
Poorly predictable large scale wave Eastern
Pacific Western US
Highly predictable small scale wave Eastern US
Verification
54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
FORECAST PERFORMANCE MEASURES
COMMON CHARACTERISTIC Function of both forecast
and observed values
MEASURES OF RELIABILITY DESCRIPTION Statisticall
y compares any sample of forecasts with sample of
corresponding observations GOAL To assess
similarity of samples (e.g., whether 1st and 2nd
moments match) EXAMPLES Reliability component
of Brier Score Ranked Probability
Score Analysis Rank Histogram Spread vs. Ens.
Mean error Etc.
MEASURES OF RESOLUTION DESCRIPTION Compares the
distribution of observations that follows
different classes of forecasts with the climate
distribution GOAL To assess how well the
observations are separated when grouped by
different classes of preceding fcsts EXAMPLES Res
olution component of Brier Score Ranked
Probability Score Information content Relative
Operational Characteristics Relative Economic
Value Etc.
COMBINED (RELRES) MEASURES Brier, Ranked
Probab. Scores, rmse, PAC, etc
58
EXAMPLE PROBABILISTIC FORECASTS
RELIABILITY Forecast probabilities for given
event match observed frequencies of that event
(with given prob. fcst) RESOLUTION Many
forecasts fall into classes corresponding to high
or low observed frequency of given
event (Occurrence and non-occurrence of event is
well resolved by fcst system)
59
(No Transcript)
60
PROBABILISTIC FORECAST PERFORMANCE MEASURES
TO ASSESS TWO MAIN ATTRIBUTES OF PROBABILISTIC
FORECASTS RELIABILITY AND RESOLUTION Univariate
measures Statistics accumulated point by
point in space Multivariate measures Spatial
covariance is considered
BRIER SKILL SCORE (BSS)
EXAMPLE
COMBINED MEASURE OF RELIABILITY AND RESOLUTION
61
BRIER SKILL SCORE (BSS)
COMBINED MEASURE OF RELIABILITY AND RESOLUTION

• METHOD
• Compares pdf against analysis
• Resolution (random error)
• Reliability (systematic error)
• EVALUATION
• BSS Higher better
• Resolution Higher better
• Reliability Lower better
• RESULTS
• Resolution dominates initially
• Reliability becomes important later
• ECMWF best throughout
• Good analysis/model?
• NCEP good days 1-2
• Good initial perturbations?
• No model perturb. hurts later?
• CANADIAN good days 8-10

May-June-July 2002 average Brier skill score for
the EC-EPS (grey lines with full circles), the
MSC-EPS (black lines with open circles) and the
NCEP-EPS (black lines with crosses). Bottom
resolution (dotted) and reliability(solid)
contributions to the Brier skill score. Values
refer to the 500 hPa geopotential height over the
northern hemisphere latitudinal band 20º-80ºN,
and have been computed considering 10
equally-climatologically-likely intervals (from
Buizza, Houtekamer, Toth et al, 2004)
62
BRIER SKILL SCORE
COMBINED MEASURE OF RELIABILITY AND RESOLUTION
63
RANKED PROBABILITY SCORE
COMBINED MEASURE OF RELIABILITY AND RESOLUTION
64
ANALYSIS RANK HISTOGRAM (TALAGRAND DIAGRAM)
MEASURE OF RELIABILITY
65
ENSEMBLE MEAN ERROR VS. ENSEMBLE SPREAD
MEASURE OF RELIABILITY
Statistical consistency between the ensemble and
the verifying analysis means that the verifying
analysis should be statistically
indistinguishable from the ensemble members
gt Ensemble mean error (distance between ens.
mean and analysis) should be equal to ensemble
spread (distance between ensemble mean and
ensemble members)
In case of a statistically consistent ensemble,
ens. spread ens. mean error, and they are both
a MEASURE OF RESOLUTION. In the presence of bias,
both rms error and PAC will be a combined measure
of reliability and resolution
66
INFORMATION CONTENT
MEASURE OF RESOLUTION
67
RELATIVE OPERATING CHARACTERISTICS
MEASURE OF RESOLUTION
68
ECONOMIC VALUE OF FORECASTS
MEASURE OF RESOLUTION
69
PERTURBATION VS. ERROR CORRELATION ANALYSIS (PECA)
MULTIVATIATE COMBINED MEASURE OF RELIABILITY
RESOLUTION

• METHOD Compute correlation between ens
  perturbtns and error in control fcst for
• Individual members
• Optimal combination of members
• Each ensemble
• Various areas, all lead time
• EVALUATION Large correlation indicates ens
  captures error in control forecast
• Caveat errors defined by analysis
• RESULTS
• Canadian best on large scales
• Benefit of model diversity?
• ECMWF gains most from combinations
• Benefit of orthogonalization?
• NCEP best on small scale, short term
• Benefit of breeding (best estimate initial
  error)?
• PECA increases with lead time
• Lyapunov convergence
• Nonlilnear saturation
• Higher values on small scales

70
WHAT WE NEED FOR POSTPROCESSING TO WORK?

• LARGE SET OF FCST OBS PAIRS
• Consistency defined over large sample need same
  for post-processing
• Larger the sample, more detailed corrections can
  be made
• BOTH FCST AND REAL SYSTEMS MUST BE STATIONARY IN
  TIME
• Otherwise can make things worse
• Subjective forecasts difficult to calibrate

HOW WE MEASURE STATISTICAL INCONSISTENCY?

• MEASURES OF STATIST. RELIABILITY
• Time mean error
• Analysis rank histogram (Talagrand diagram)
• Reliability component of Brier etc scores
• Reliability diagram

71
SOURCES OF STATISTICAL INCONSISTENCY

• TOO FEW FORECAST MEMBERS
• Single forecast inconsistent by definition,
  unless perfect
• MOS fcst hedged toward climatology as fcst skill
  is lost
• Small ensemble sampling error due to limited
  ensemble size
• (Houtekamer 1994?)
• MODEL ERROR (BIAS)
• Deficiencies due to various problems in NWP
  models
• Effect is exacerbated with increasing lead time
• SYSTEMATIC ERRORS (BIAS) IN ANALYSIS
• Induced by observations
• Effect dies out with increasing lead time
• Model related
• Bias manifests itself even in initial conditions
• ENSEMBLE FORMATION (INPROPER SPREAD)
• Not appropriate initial spread
• Lack of representation of model related
  uncertainty in ensemble
• I. E., use of simplified model that is not able
  to account for model related uncertainty

72
HOW TO IMPROVE STATISTICAL CONSISTENCY?

• MITIGATE SOURCES OF INCONSISTENCY
• TOO FEW MEMBERS
• Run large ensemble
• MODEL ERRORS
• Make models more realistic
• INSUFFICIENT ENSEMBLE SPREAD
• Enhance models so they can represent model
  related forecast uncertainty
• OTHERWISE gt
• STATISTICALLY ADJUST FCST TO REDUCE INCONSISTENCY
• Unpreferred way of doing it
• What we learn can feed back into development to
  mitigate problem at sources
• Can have LARGE impact on (inexperienced) users

73
(No Transcript)
74
SUMMARY

• WHY DO WE NEED PROBABILISTIC FORECASTS?
• Isnt the atmosphere deterministic? YES, but
  its also CHAOTIC
• FORECASTERS PERSPECTIVE USERS PERSPECTIVE
• Ensemble techniques Probabilistic description
• WHAT ARE THE MAIN ATTRIBUTES OF FORECAST SYSTEMS?
• RELIABILITY Stat. consistency with distribution
  of corresponding observations
• RESOLUTION Different events are preceded by
  different forecasts
• WHAT ARE THE MAIN TYPES OF FORECAST METHODS?
• EMPIRICAL Good reliability, limited resolution
  (problems in new situations)
• THEORETICAL Potentially high resolution, prone to
  inconsistency
• ENSEMBLE METHODS
• Only practical way of capturing fluctuations in
  forecast uncertainty due to
• Case dependent dynamics acting on errors in
• Initial conditions
• Forecast methods
• HOW CAN PROBABILSTIC FORECAST PERFORMANCE BE
  MEASURED?

75
Toth, Z., O. Talagrand, and Y. Zhu, 2005 The
Attributes of Forecast Systems A Framework for
the Evaluation and Calibration of Weather
Forecasts. In Predictability Seminars, 9-13
September 2002, Ed. T. Palmer, ECMWF, in press.
Toth, Z., O. Talagrand, G. Candille, and Y. Zhu,
2003 Probability and ensemble forecasts. In
Environmental Forecast Verification A
practitioner's guide in atmospheric science. Ed.
I. T. Jolliffe and D. B. Stephenson. Wiley, p.
137-164.