Title: REVIEW OF RECOMMENDATIONS FROM THE 3RD ENSEMBLE USER WORKSHOP (OCT. 2006)
1REVIEW OF RECOMMENDATIONS FROM THE 3RD ENSEMBLE
USER WORKSHOP (OCT. 2006)
- Zoltan Toth
- Environmental Modeling Center
- NOAA/NWS/NCEP
- Acknowledgements
- Louis Uccellini, Steve Lord Ensemble Team
- Paul Schultz, DJ Seo, Tom Hamill, Steve Mullen,
Julie Demargne - http//wwwt.emc.ncep.noaa.gov/gmb/ens/index.html
2MAJOR RECOMMENDATIONS - 1
- FOLLOW NRC REPORT RECOMMENDATIONS
- All major recommendations embraced
- Develop roadmap for assessing communicating
forecast uncertainty - Based on science, technology, workforce
considerations - Consistent with NOAAs mission, NWS plans,
embraced by Enterprise - Define end goal, 5-10 years horizon
- Adopt ensemble-based forecast process
- Maximize forecast skill
- Unify scientific, software, and technological
infrastructure across NOAA - Weather, water, climate applications
- Suggested in Mar 2007 to develop roadmap by Sept.
2007
3MAJOR RECOMMENDATIONS - 2
- Develop roadmap for assessing communicating
forecast uncertainty - Revise operational requirements to make them
probabilistic - Make probabilistic format the primary requirement
- For each forecast application
- Replace single value/categorical format with
probabilistic format as primary requirement - Revise/supplement corresponding performance
measures (GPRA) - Essential for orderly transition from traditional
to new forecast process - NWS is requirement-driven organization
- Without clear new requirements, process is doomed
- Phased implementation schedule in consultation
with responsible offices - Allows orderly transition to new requirements
- Suggested in Mar 2007 to prepare plan by March
2008
4MAJOR RECOMMENDATIONS - 3
- Develop roadmap for assessing communicating
forecast uncertainty (.5 yr) - Revise operational requirements to make them
probabilistic (1 year) - Design, develop, and gradually implement new
forecast process - Focus on missing scientific, technological, and
human components - Identify self-contained components with
- Clear requirements and interfaces between
components - Define basic capabilities achievable in 2-3 years
- Limited but consistent with and leading to end
goal - Full capabilities in 5-10 years
- Interface with NOAA THORPEX program
- Research/development to improve skill utility
of probabilistic forecasts - Leverage related major NOAA, national, and
international efforts - Integrate with NWS, NOAA, national activities
- NOAA-wide regional service plans
- NUOPC planning/development
- Provide long-term funding support through PPBES
- Overlap with WW High Impact Event Theme Team
5SPECIFIC RECOMMENDATIONS 1-3
- Continue development of expanded forecast process
- Focus on adaptive methods applicable for high
impact events - Collection use of observations (targeted
observations) - Data assimilation (case dependent background
error estimation) - Numerical modeling (adaptive resolution high
impact modeling) - Ensemble forecasting (case dependent variations
in membership composition) - Decision support systems (flexible user actions
depending on forecast probabilities) - Bias correction downscaling methods for
ensembles - Estimate/correct lead-time dependent bias in
ensemble forecasts (on model grid) - Generate fine resolution (NDFD grid)
uncertainty/ensemble data - Establish connection between (bias corrected)
coarse model vs. fine user grids - Use reanalysis hind-casts with operational
systems as needed - Define summary ensemble information to be used to
(ST) - Collapse vast amount of ensemble data for
inclusion in expanded NDFD/NDGD - E.g., 10, 50, and 90 percentile of forecast
distribution (in place of single value) - Manually inspect/modify summary ensemble
information
ST Short Term (2-3 yrs) LT Long Term (5-10
yrs)
6SPECIFIC RECOMMENDATIONS 4-6
- Contribute to establishment of NOAA-wide
environmental data depository - Expand NDFD/NDGD database to include forecast
uncertainty (ST) - Develop capability to hold all ensemble
trajectories (LT) - All members, variables, lead times
- Develop ensemble interrogation, modification,
product generation tools to - Derive summary information from ensemble (ST)
- Manually modify summary ensemble info (ST)
- Derive additional statistics from summary info
(product generation, ST) - Automatically modify ensemble trajectories based
on modified summary info (LT) - Derive any info from full ensemble data (product
generation, LT) - Develop telecommunication facilities to access
data - Summary info limited derived products (ST)
- All ensemble forecasts derived products (LT)
ST Short Term (2-3 yrs) LT Long Term (5-10
yrs)
7SPECIFIC RECOMMENDATIONS 7-9
- Develop unified NWS/NOAA probabilistic
verification package to (ST) - Assess statistical reliability and resolution for
- Computing official performance measures
- Evaluating value added along forecast chain
- Assessing value in newly developed vs.
operational techniques - Develop implement comprehensive training to
- Prepare all participants for their new roles in
expanded forecast process, incl. - Statistical background
- Ensemble methods
- Best forecast practices in assessing uncertainty
- Applications of probabilistic other uncertainty
information - Develop outreach program on use communication
of uncertainty - In partnership with entire Weather, Water
Climate Enterprise - Determine best ways of communicating uncertainty
- Compile sample of Decision Support Systems using
uncertainty information - Establish close partnership with public sector
users (e.g., emergency, water management) - User feedback on new activities
ST Short Term (2-3 yrs) LT Long Term (5-10
yrs)
8NEW DEVELOPMENTS SINCE WG RECOMMENDATIONS
- NOAA/NWS Forecast Uncertainty Team (NFUSE) formed
- Early 2007
- Assess and discuss status, search for solutions
- AMS ACUF Team formed
- Enterprise planning activities
- NOAA/NWS Forecast Uncertainty program formed
- Outgrowth of NFUSE
- Oversight panel established
- Budget planning for out-years
- Uncertainty-related activities wide-spread,
affect all forecasting - What is next?
- Coordinated
- Work level plans
- Related activities
92008 WORKSHOP CONTRIBUTIONS
- Agree on long term plan
- What we want in 5-10 yrs desired end
result/framework - Not how we achieve goal
- Multiple avenues can be tested within single
framework - Agree on what we can achieve in next 2-3 years
with expected resources - Limited version of long term goal interim
result/framework - Development of short term solution will
contribute to long term plan - Contribute to forming of Work Teams addressing
well defined sub-problems - Separate pieces that can be worked on
independently - Leads/principles provide overall coordination
10DISCUSS / AGREE ON LONG TERM GOAL
- Capability to answer any question related to
future weather, climate, water, including
forecast uncertainty - Considering state of related science, what do we
need for that? - Bias corrected, downscaled ensemble forecasts
Ultimate dataset - Allows derivation of single/joint probabilities
- Information on spatial/temporal/cross-variable
correlations - Trajectories (scenarios)
- Work independent of particular product/service
needs (all questions covered) - Major technical requirements
- Storage for ensemble data
- 20-100 times more data compared to current NDFD
- Distribution of / access to ensemble data
- 20-100 fold increase in bandwidth
- Technical tools to display / manipulate ensemble
data - Special AWIPS-2 capabilities
- Forecasters trained to work with ensemble data
- 2-step approach warranted
- First implement limited capability
- Will provide large portion of total benefit with
relatively small investment
11DISCUSS / AGREE ON INTERMEDIATE GOAL
- Capability to provide uncertainty/probabilistic
info on any weather element with some
approximation - Considering state of related science, what do we
need for that? - Summary info that allows the generation of
single-variable cdf/pdf - Bias corrected, downscaled
- Can be generated based on a single or ensemble
NWP - Allows derivation of single-variable
probabilities - Including information on spatial/temporal/cross-va
riable correlations - Technical requirements
- Storage for summary info
- 3-fold increase in NDFD data
- Distribution of / access to ensemble data
- 3-fold increase in bandwidth
- Technical tools to display / manipulate ensemble
data - Subset of long term toolkit in AWIPS-2
- Connect summary stats cdf/pdf
- Forecasters trained to work with ensemble data
- With right choice of summary info, forecasters
eased into new paradigm
12PHASE 1 PLAN
- Important element
- Definition of basic format for forecast
uncertainty - Advantages of using a common format
- Facilitates flow of info between three major
elements in forecast process - Numerical guidance
- Human forecasters
- External users
- Provides basis for collaboration among various
groups - Inter-comparison of different approaches
- Requirements
- Can be easily interpreted by users
- Intuitive meaning
- Can be manipulated by forecasters
- Related to current forecast modification
practices - Can be easily communicated
- Limited data volume
- Can be stored in NDFD
- Related to current forecast format
- Additional formats can be easily derived
13CHOICE OF FORMAT FOR ASSESSING AND COMMUNICATING
FORECAST UNCERTAINTY
- Background on suggested type of format
- Recommended by Predictability WG at WR/NWS
SOO/DOH Workshop (Salt Lake City, May 2003) - Discussed and recommended by Predictability
Workshop at Univ. Wisconsin (Madison, Mar 2004) - Studied in detail and adopted at UKMET Office
(2006) - Suggested format
- Beyond mid-point estimate (50, or mode, or
mean?), include - Two additional fields for each NDFD variable
- 10 percentile value of forecast distribution
- 90 percentile value of forecast distribution
- Readiness
- Numerical guidance
- Can be generated based on ensemble data, using
NAWIPS software - Forecasters
- Can use same tools to manipulate 2 new fields as
with most likely field - Users
- UKMET research indicates this is intuitive
14BACK
15EXAMPLE FROM UK METOFFICE - TEMP
Courtesy of Mark Roulston
16EXAMPLE FROM UK METOFFICE - PRECIP
Courtesy of Mark Roulston
17QUESTIONS ABOUT SUGGESTED FORMAT
- Does it make sense, reasonable format?
- No need to worry (too much)
- Just need a common format to connect three main
areas - Alternative formats can/will be generated
- What NDFD grids describe currently?
- Most likely value (mode of pdf)?
- Expected value (mean of pdf)?
- 50 exceedence value (median of pdf)?
- What mid-point parameter an expanded NDFD should
have? - What do users need?
- What is most intuitive to forecasters?
- Would median (50 percentile) be best choice?
Similar to 10, 90 percentile value - What threshold may work best for limits?
- 10 90 - not too narrow, not too wide?
- 5 95 percentile may be too extreme for
forecasters? - 20 80 may not be inclusive enough?
- What software needs to be developed to generate
other formats? - Fit parametric distribution to 3 points
- Roman Krzysztofowicz library of 40
distributions
18TOOLS NEEDED FOR INTERMEDIATE STEP
- Modify summary statistics (forecasters role)
- Fit parametric distribution to modified summary
statistics - Adjust ensemble members to be consistent with
official forecast - For users who need trajectories
- Derive probabilities from fitted cdf/pdf
- Same statistical tools must be used across
organization - Ensemble generation, display, manipulation,
product generation, etc - Same tools can be used in final stage to
manipulate ensemble distribution - Summary statistics inspected modified by
forecasters - Changes back-propagated to ensemble data set
19WORK AREAS
- All topics need to be developed in coordination
- Plenary sessions for interactions
- Topics A B are especially connected in each WG
- Same groups discussing both topics in 2 sessions
- Third WG session to focus on connections
- Working Group 1
- Topic A Operational requirements
- Topic B Corporate outreach
- Working Group 2
- Topic A Ensemble forecasting
- Topic B Statistical post-processing
- Working Group 3
- Topic A Ensemble data depository / access
- Topic B Database interrogation / forecaster tools
- Working Group 4
20WORK AREAS
- Operational requirements
- General requirements first to justify effort
- Detailed requirements later as effort unfolds
- Corporate outreach
- Link with ACUF process
- Promote quantitative use of uncertainty
information in Decision Support Systems - Ensemble forecasting
- Centerpiece of uncertainty effort
- THORPEX
- NOAA, national, international research/development
- NAEFS, NUOPC, TIGGE, GIFS
21WORK AREAS CONT
- Statistical post-processing
- Problem
- Relate coarse resolution biased forecast to user
relevant fine resolution information - Break up problem to facilitate collaboration
- Bias correction of coarse resolution forecast
grid wrt NWP analysis - Cheap
- Sample of forecasts / hind-casts needed
- Downscaling
- Relate coarse resolution NWP and fine resolution
observationally based analyses - Perfect prog approach
- No need for hind-casts
- Creation of observationally based fine resolution
analysis - Estimate of truth
- Groups can collaborate on each topic
- Addressing all 3 problems in one swoop limits
collaboration
22NAEFS RESULTS
- 8 days total gain in skill
- Downscaling more important than bias correction
- Less need for hindcasting?
- Need for local observationally based analysis
- Multicenter approach adds 1-2 days skill
NCEP biascorrected-downscaled
NCEP raw-downscaled
NCEP raw
NAEFS final
23WORK AREAS CONT 2
- Ensemble data depository / access
- Create NOAA digital forecast database (linked
with other NOAA data) - Summary statistics in phase 1
- All ensemble members in phase 2
- Provide easy access to internal / external users
- NOMADS, etc?
- Link with multi-center ensembles
- NAEFS NUOPC GIFS
- Database interrogation / forecaster tools
- Derive information from summary stats / ensembles
- Modify summary stats
- Back-propagate modified info into ensemble
- Verify added value at each step of forecast
process - Need for unified verification capability across
forecast process - Common requirement across all WGs
24WORK AREAS CONT 3
- Forecasters role
- Traditional role
- Generate forecast, or modify numerical guidance
- Changing environment
- Need to generate very large amount of forecast
data - Quality of automatic numerical guidance improves
- New role
- Direct and quality control forecast process
- Identify high impact events
- Assign adaptively configurable observational,
modeling, etc resources - Interfere with automated process only during
critical high impact events - User outreach
- Interpret probabilistic forecast
- Forecaster training
- Prepare for new role related to ensemble /
probabilistic forecasting - Modification
- Interpretation
- Outreach
25Surface Temperature MAE CONUS, Sept. 2007
12Z NDFD vs. 00Z MOS/GMOS/NAEFS
Surface Temperature MAE CONUS, Sept. 2007
00Z GMOS vs. 00Z NAEFS
RTMA Analysis
METAR obs. 1221 sites
GMOS forecast
0.5C
NAEFS products
0.5C
26Surface Temperature Area Mean Bias CONUS,
Sept. 2007 12Z NDFD vs. 00Z MOS/GMOS/NAEFS
Surface Temperature Pointwise Bias CONUS,
Sept. 2007 00Z GMOS vs. 00Z NAEFS
Pointwise Bias
Area Mean Bias
RTMA Analysis
METAR obs. 1221 sites
GMOS forecast
0.6C
0.6C
NAEFS products
27BACKGROUND
28Example for using new mask
29MDL GMOS NAEFS Downscaled Forecast Mean
Absolute Error w.r.t. RTMA Average For Sept.
2007
12-h GMOS Forecast
12-h NAEFS Forecast
For CONUS NAEFS(1.01) GMOS(1.59) 36 impr.
over GMOS
30MDL GMOS NAEFS Downscaled Forecast Mean
Absolute Error w.r.t. RTMA Average For Sept.
2007
24-h GMOS Forecast
24-h NAEFS Forecast
For CONUS NAEFS(1.45) GMOS(1.72) 15 impr.
over GMOS
31Surface Temperature MAE CONUS, Sept. 2007
12Z NDFD vs. 00Z MOS/GMOS/NAEFS
Surface Temperature MAE CONUS, Sept. 2007
00Z GMOS vs. 00Z NAEFS
RTMA Analysis
METAR obs. 1221 sites
GMOS forecast
0.5C
NAEFS products
0.5C
32Surface Temperature Area Mean Bias CONUS,
Sept. 2007 12Z NDFD vs. 00Z MOS/GMOS/NAEFS
Surface Temperature Pointwise Bias CONUS,
Sept. 2007 00Z GMOS vs. 00Z NAEFS
Pointwise Bias
Area Mean Bias
RTMA Analysis
METAR obs. 1221 sites
GMOS forecast
0.6C
0.6C
NAEFS products
33 Western Rgn Pointwise Bias
Western Rgn Area Mean Bias
Western Rgn MAE
Western Rgn MAE
34 Central Rgn Pointwise Bias
Central Rgn Area Mean Bias
Central Rgn MAE
Central Rgn MAE
35 Eastern Rgn Area Mean Bias
Eastern Rgn Pointwise Bias
Eastern Rgn MAE
Eastern Rgn MAE
36 Southern Rgn Area Mean Bias
Southern Rgn Pointwise Bias
Southern Rgn MAE
Southern Rgn MAE
37NCEP-raw
NAEFS-final
NCEP-raw-downs
NCEP-biasc-downs
BACK
Bo et al.
38BACKGROUND
39NFUSE GOALS
- Need to clarify goals (be)for(e) brainstorming
- Premise
- We accept and plan to follow major NRC
recommendations - Whats needed?
- Design, develop, and gradually implement changes
in forecast process for - assessing, communicating, and using forecast
uncertainty - Basic capabilities (by 2009-2010)
- Detailed draft plan (including low hanging fruit)
by Oct 07 - Full capabilities (by 2012-2017)
- Conceptual draft plan by Oct 07
- Corporate commitment
- Funding for both phases
- Reallocation / reprioritization for phase 1
- PPBES for phase 2
- New / revised operational requirements
- Part of planning / development
40TWO-PHASE PLAN
- Phase 1 (2-3 yrs)
- Limited time and resources - Build on existing
and emerging efforts - Not all users within/outside NWS can access
ensemble data - Limited capabilities
- Ability to express basic forecasts in
probabilistic terms - Uni-variate pdf
- Joint probab. computation assumes probabilities
for different variables are independent - Other formats derived from uni-variate pdf
- Phase 2 (5-10 yrs)
- Full capabilities
- All interested users within/outside NWS have
access to ensemble data - Joint probabilities derived from ensemble
- Exhaustive set of forecast formats/products
supported
41BACKGROUND
42LINKS WITH NFUSE PLANS
- Current system
- Single value format
- Short-term (2-3 yrs) plan 3 values format (pdf)
- Provide best (bias corrected) numerical guidance
in agreed upon format - Provide bias corrected ensemble data to external
and selected internal users - Human forecasters modify numerical guidance using
agreed upon format - Distribute and store agreed upon format
- Convert format into single variable pdf format
- External users provided with products in format
of their choice - Multiple options
- Software needed to derive user requested format
from single value pdf - Long-term (5-10 yrs) plan ensemble format
- Provide best numerical guidance in agreed upon
format - Human forecasters modify numerical guidance using
agreed upon format - Propagate information to modify bias corrected
ensemble data - Modified bias corrected ensemble data is complete
and final forecast dataset includes - Uncertainty information regarding spatial,
temporal, cross-variable co-variances - Human forecasters need access to ensemble data to
assist in
43(No Transcript)
44SYNERGIES
- Choice / acceptance of format for uncertainty
info enables inter-comparisons - Verification using common framework possible
- Added value can be traced
- Choice of THORPEX performance measures
- Related to major application areas (previous
discussion) - Use 3-value format to verify uncertainty/probabili
stic information?
45INTRODUCTION
- Plans need to be coordinated between three major
areas - Probabilistic numerical guidance for high impact
events (THORPEX) - Provides objective numerical guidance
- Human forecasters
- Add value to numerical guidance
- Interpret guidance / products
- Products to be distributed to and used by
external community - Interface with Weather enterprise
- Format of forecast uncertainty estimate can/must
link three major areas - Numerical methods
- Must be able to produce chosen format
- Forecasters
- Must be able to manipulate guidance
- Must be simple and intuitive
- Users
- Must find clear value in product given in chosen
format - Choice of exact format not critical
- As long as software available to generate
alternate formats
46PROPOSED NEW / LEGACY PRODUCTS TO BE MONITORED
- Possible new probabilistic guidance products for
high impact events - Hydrometeorology
- Extreme hydro-meteorological events, incl. dry
and wet spells (CONUS) - Exceedance of 1 5 inch thresholds
- Exceedance of 6-hr flash flood guidance (or
numerical outlook) thresholds - Quantitative extreme river flow forecasting
(OCONUS) - Tropical / winter storm prediction
- Extreme surface wind speed
- Radii of gale, storm, hurricane force wind
- Maximum surface wind (for intensity)
- Extreme precipitation (related to wet spells)
- Storm surges
- Storm surge values along US Coast
- Aviation forecasting
- Flight restriction
- Icing, visibility, fog, clear air turbulence
- Health and public safety
- Hot and cold spells
47BACKGROUND
48NFUSE PRIORITIES?
- Issues to be discussed/worked out in this order
- NRC Recommendations
- Do we agree on major recommendations?
- If not, lets discuss any issues
- If so, can we accept those as the
basis/guidelines for our work - 3rd Ensemble User Workshop Recommendations
- Three major recommendations in response to NRC
Report - High level roadmap guiding our collaboration
(Sept 2007) - New operational requirements (March 2008)
- Design, develop, and gradually implement new
forecast process - Basic capabilities (2009-2010)
- Full capabilities (2012-2017)
- Nine specific recommendations on design,
development, implementation - Work areas identified, links to be worked out
- Continue development of expanded forecast process
- Bias correction downscaling methods for
ensembles - Define summary ensemble information to be used
(ST) - Contribute to establishment of NOAA-wide
environmental data depository - Develop ensemble interrogation, modification,
product generation tools
49COMPLETING THE FORECASTASSESSING AND
COMMUNICATING FORECAST UNCERTAINTY
- Zoltan Toth
- Environmental Modeling Center
- NOAA/NWS/NCEP
- Acknowledgements
- NRC Report Louis Uccellini, Steve Lord
Ensemble Team - http//wwwt.emc.ncep.noaa.gov/gmb/ens/index.html
50OUTLINE
- REVIEW OF 3RD ENSEMLBE USER WORKSHOP
- MAJOR RECOMMENDATIONS REGARDING NRC REPORT
- WHAT, WHY, AND HOW TO CHANGE?
- SPECIFIC RECOMMENDATIONS
513rd ENSEMBLE USER WORKSHOP
- Logistics
- Oct. 30 Nov. 1 2006, Laurel, MD
- Close to 100 participants
- NWS Regions (12), Headquarters (15), NCEP (37)
- OAR (6), other government (4), private (4),
academic (8) sectors international (8) - For further info, see http//wwwt.emc.ncep.noaa.g
ov/gmb/ens/UserWkshop_Oct2006.html - Topics
- Assessing propagating uncertainty throughout
entire forecast process - From observations to users
- Working group discussions
- Ensemble configuration
- Statistical post-processing
- Data depository / Interrogation / Product
generation/dissemination / Verification - User support / Outreach / Training
- Outcome
- Enthusiastic discussions
- Convergence on number of topics
- Open questions needing further research
identified - Strong support for sustained effort/engagement,
annual meetings, etc
52MAJOR RECOMMENDATIONS - 1
- FOLLOW NRC REPORT RECOMMENDATIONS
- All major recommendations embraced
- Develop roadmap for assessing communicating
forecast uncertainty - Based on science, technology, workforce
considerations - Consistent with NOAAs mission, NWS plans,
embraced by Enterprise - Define end goal, 5-10 years horizon
- Adopt ensemble-based forecast process
- Maximize forecast skill
- Unify scientific, software, and technological
infrastructure across NOAA - Weather, water, climate applications
- Form high level planning and oversight team
March 2007 - Each NWS office to delegate one representative
- Team reports to Corporate Board
- Name programmatic and technical leads
- Develop roadmap Sept. 2007
53MAJOR RECOMMENDATIONS - 2
- Develop roadmap for assessing communicating
forecast uncertainty - Revise operational requirements to make them
probabilistic - Make probabilistic format the primary requirement
- For each forecast application
- Replace single value/categorical format with
probabilistic format as primary requirement - Revise/supplement corresponding performance
measures (GPRA) - Essential for orderly transition from traditional
to new forecast process - NWS is requirement-driven organization
- Without clear new requirements, process is doomed
- Phased implementation schedule in consultation
with responsible offices - Allows orderly transition to new requirements
- New requirements prepared by Planning oversight
team (or its designate) - Assisted by responsible NWS office
- Presented to Corporate Board for approval By
March 2008
54MAJOR RECOMMENDATIONS - 3
- Develop roadmap for assessing communicating
forecast uncertainty (.5 yr) - Revise operational requirements to make them
probabilistic (1 year) - Design, develop, and gradually implement new
forecast process - Focus on missing scientific, technological, and
human components - Identify self-contained components with
- Clear requirements and interfaces between
components - Define basic capabilities achievable in 2-3 years
- Limited but consistent with and leading to end
goal - Full capabilities in 5-10 years
- Interface with NOAA THORPEX program
- Research/development to improve skill utility
of probabilistic forecasts - Leverage related major NOAA, national, and
international efforts - Integrate with NWS, NOAA, national activities
- New NOAA CONOPS process
- NOAA-wide regional service plans
- NUOPC planning/development
- Provide long-term funding support through PPBES
55PROPOSED CHANGE
- Major paradigm shift
- Incorporate assessment and communication of
uncertainty in forecast process - Is it a major change in course of Weather Ship?
- Ie, abandon course of ever improving single
forecast scenario (expected value)? - No Expand, not abandon
- Keep improving fidelity of forecasts, PLUS
- Add new dimension
- Capture other possible scenarios ensemble
forecasting - Use a flotilla, instead of one ship, in exploring
nature - Existing activities are subset of expanded
forecast process - Single value forecast is expected value of full
probability distribution - Can keep serving forecasts in old format to users
who prefer that
56Single forecast (driven by GFS winds) example for
drifting virtual ice floe
7 September 2006
Initial position
Bob Grumbine, EMC
57Ensemble forecast for drifting ice floe for same
case
Initial position
Bob Grumbine, EMC
58Most likely forecast for drifting ice floe for
same case
Initial position
Bob Grumbine, EMC
59WHY CHANGE IS NEEDED?
- Why users (should) care about forecast
uncertainty? - They admittedly want minimal or no uncertainty in
forecasts - Distinction between no uncertainty in the
forecast, vs. not talking about it - Forecast uncertainty cannot be arbitrarily
reduced - Despite major ongoing continuing efforts, they
persist forever - Chaotic nature of atmosphere - land surface
ocean coupled system initial/model errors - Level of uncertainty is determined by nature and
level of sophistication in forecast system - Forecast uncertainty can be ignored though
- Negative consequence on informed users
- Not able to prepare for all possible outcomes
- Assumes a certain scenario and remains vulnerable
to others - Possibly serious loss in social/economic value of
forecast information - Why forecasters (should) care about forecast
uncertainty? - Imperfect forecasts are consistent w.
observations (reliable) only if in prob format - If in other format, must be brought into
probabilistic format through - Verification / bias correction
60ADVANTAGES OF PROBABILISTIC FORMAT
- More rationalized and enriched forecaster - user
interactions - Old paradigm
- Convoluted forecaster-user decision process
- User expects forecaster to make decision for them
in presence of uncertainty - Will it rain? 80 But tell me, will it
rain? - New paradigm
- Forecaster and user decision processes enhanced
and better linked - Allows forecasters to capture all knowledge about
future conditions - Provision of information related to multiple
decision levels in probabilistic format critical - Provider helps interpret probabilistic info and
modify user decision process if needed - Option to continue providing single value or
other limited info until user ready - Allows users to decide about most beneficial
course of action given all possibilities - Proper use of probability or other uncertainty
information needed - Training - User requests critical weather forecast info
depending on their sensitivity
61TRADITIONAL FORECAST PROCESS
- Focus on single forecast scenario
- Reducing uncertainty in single forecast is main
emphasis - Loss of accuracy in forecast estimate of expected
value of distribution - Mean of ensemble cloud provides better estimate
- Ignores or simplifies forecast uncertainty
- Uncertainty assessed as statistically averaged
error in single fcst (second thought) - Ensemble cloud provides better estimate of case
dependent variations in uncertainty - Use of single value / categorical forecast format
- Difficulty in formulating/communicating plausible
alternate scenarios - Ensemble member forecasts can directly feed into
Decision Support Systems - One-way flow of information from observations to
users - Not adaptable to case dependent user requirements
- Ensemble can propagate back user requirements to
adaptive - Observing, assimilation, modeling/ensemble,
post-processing and application components - Applications in planning and execution of new
CONOPS in high impact events
62PROPAGATING FORECAST UNCERTAINTY
z
Distribution
Single value
Ensemble Forecasting Central role bringing the
pieces together
63HOW CAN IT BE DONE? NEW PARADIGM
- Adopt ensemble approach across all environmental
prediction activities - Expand forecasting with new dimension of
uncertainty - Multiple scenarios (in place of single scenario)
- Provides best forecast estimate for both expected
value (as before) and uncertainty (new) - Unified scientific, technological, human approach
- Sharing resources across NWS NOAA
- Ensemble is centerpiece both symbolically and
figuratively in forecast process - Ensembles act as a glue two-way information
channel - Observing system, data assimilation, numerical
modeling - ENSEMBLES
- Statistical post-processing, product generation,
decision making - Design, develop, implement missing components
of new forecast process - Gradual, measured steps
- Basic capability - Short-term, 2-3 yrs, leading
to - Full implementation - Long-term, 5-10 yrs
64ENSEMBLES AND THE RESEARCH COMMUNITY LINKED
THROUGH THORPEX MAJOR INTERNATIONAL RESEARCH
PROGRAM GOAL Accelerate improvements of high
impact weather forecasts
INTEGRATED DATA ASSIMILATION FORECASTING
GLOBAL OPERATIONAL
TEST CENTER
GLOBAL INTERACTIVE FORECAST SYSTEM (GIFS)
Days 15-60
NWS OPERATIONS
CLIMATE FORECASTING / CTB
GLOBAL OPERATIONAL
SOCIOECON.
SYSTEM
TEST CENTER
MODEL ERRORS HIGH IMPACT MODELING
65SPECIFIC RECOMMENDATIONS 1-3
- Continue development of expanded forecast process
- Focus on adaptive methods applicable for high
impact events - Collection use of observations (targeted
observations) - Data assimilation (case dependent background
error estimation) - Numerical modeling (adaptive resolution high
impact modeling) - Ensemble forecasting (case dependent variations
in membership composition) - Decision support systems (flexible user actions
depending on forecast probabilities) - Bias correction downscaling methods for
ensembles - Estimate/correct lead-time dependent bias in
ensemble forecasts (on model grid) - Generate fine resolution (NDFD grid)
uncertainty/ensemble data - Establish connection between (bias corrected)
coarse model vs. fine user grids - Use reanalysis hind-casts with operational
systems as needed - Define summary ensemble information to be used to
(ST) - Collapse vast amount of ensemble data for
inclusion in expanded NDFD/NDGD - E.g., 10, 50, and 90 percentile of forecast
distribution (in place of single value) - Manually inspect/modify summary ensemble
information
ST Short Term (2-3 yrs) LT Long Term (5-10
yrs)
66SPECIFIC RECOMMENDATIONS 4-6
- Contribute to establishment of NOAA-wide
environmental data depository - Expand NDFD/NDGD database to include forecast
uncertainty (ST) - Develop capability to hold all ensemble
trajectories (LT) - All members, variables, lead times
- Develop ensemble interrogation, modification,
product generation tools to - Derive summary information from ensemble (ST)
- Manually modify summary ensemble info (ST)
- Derive additional statistics from summary info
(product generation, ST) - Automatically modify ensemble trajectories based
on modified summary info (LT) - Derive any info from full ensemble data (product
generation, LT) - Develop telecommunication facilities to access
data - Summary info limited derived products (ST)
- All ensemble forecasts derived products (LT)
ST Short Term (2-3 yrs) LT Long Term (5-10
yrs)
67SPECIFIC RECOMMENDATIONS 7-9
- Develop unified NWS/NOAA probabilistic
verification package to (ST) - Assess statistical reliability and resolution for
- Computing official performance measures
- Evaluating value added along forecast chain
- Assessing value in newly developed vs.
operational techniques - Develop implement comprehensive training to
- Prepare all participants for their new roles in
expanded forecast process, incl. - Statistical background
- Ensemble methods
- Best forecast practices in assessing uncertainty
- Applications of probabilistic other uncertainty
information - Develop outreach program on use communication
of uncertainty - In partnership with entire Weather, Water
Climate Enterprise - Determine best ways of communicating uncertainty
- Compile sample of Decision Support Systems using
uncertainty information - Establish close partnership with public sector
users (e.g., emergency, water management) - User feedback on new activities
ST Short Term (2-3 yrs) LT Long Term (5-10
yrs)