Outline

1
Validation of Coupled Models
Richard M. Hodur Naval Research
Laboratory Monterey, CA 93943-5502 hodur_at_nrlmry.n
avy.mil Short Course on Significance Testing,
Model Evaluation, and Alternatives 11 January
2004 Seattle, WA

• Outline
• Introduction
• Atmospheric Models
• Ocean Models
• Concluding Remarks

2
Validation of Coupled Models Acknowledgements

• Dr. James D. Doyle (NRL MRY)
• Dr. Timothy F. Hogan (NRL MRY)
• Dr. Xiaodong Hong (NRL MRY)
• Dr. John C. Kindle (NRL SSC)
• Dr. Paul May (CSC/NRL MRY)
• Dr. Jason E. Nachamkin (NRL MRY)
• Dr. Randy Pauley (FNMOC)
• Dr. Ruth H. Preller (NRL SSC)
• Dr. Julie D. Pullen (NRL MRY)
• Dr. Robert C. Rhodes (NRL SSC)
• Dr. Douglas L. Westphal (NRL MRY)

3
Validation of Coupled Models Context of Talk

• Validation is more than a skill score
• Validation implies a learning process
• Develop system
• Measure skill of system
• Seek ways to improve skill of system
• Validation is a critical component of model
  development
• Without validation there can be no improvement in
  model performance

4
How Do We Measure the Validity and Usefulness of
Atmosphere and Ocean Models? Combination of Many
Measures

• Scientific Basis
• Record of Publications, Presentations, Patents, .
  . .
• Equations, Grid Structure, Numerical Techniques,
  Representation of Physical Processes Based on
  Well-Tested, Peer-Reviewed Principles
• Reproduction of Analytic/Idealized Test Cases
• Validate Numerical Schemes (e.g., Topographic
  Flow, PGF Computation)
• Validate Physical Parameterizations (e.g.,
  Wangara, Convection)
• Measure Real-Time Predictive Performance
• No one simple metric is available
• Objective Measurements are useful (e.g., RMS,
  Bias, Anomaly Correlation, Tropical Cyclone
  Forecast Position Error, Precipitation Scores)
• May be Difficult to Measure on Mesoscale Perform
  Subjective Evaluation of Differing Episodic
  Events (e.g., Patterns, Trends, Drifters,
  Transport, Tracers)
• Measure Skill Over Long Time Periods (Months or
  more)
• Transitions require measure of skill of new
  version of system relative to a benchmark version
• Measure Utility to User(s)
• Does Output Meets User Needs?
• User-Feedback
• Robustness
• Efficiency (e.g., Wall time, Flops, . . . )

5
Validation of Coupled Models Outline

• Atmospheric Models
• Global
• Anomaly Correlation
• RMS, Bias Errors
• Tropical Cyclone Track
• Scorecard
• Mesoscale
• Idealized Flow Studies
• RMS, Bias
• Qualitative Verification (Case Studies)
• Event-Based Verification
• Ocean Models
• Global
• Features/Positions
• Sea Surface Height Validation
• Anomaly Correlation
• Sea Surface Temperature Validation
• Mesoscale
• Transport

6
Validation of Coupled Models Outline

• Atmospheric Models
• Global
• Anomaly Correlation
• RMS, Bias Errors
• Tropical Cyclone Track
• Scorecard
• Mesoscale
• Idealized Flow Studies
• RMS, Bias
• Qualitative Verification (Case Studies)
• Event-Based Verification
• Ocean Models
• Global
• Features/Positions
• Sea Surface Height Validation
• Anomaly Correlation
• Sea Surface Temperature Validation
• Mesoscale
• Transport

7
NOGAPS Annual Mean Forecast Statistics Anomaly
Correlation of 500 mb HeightsValues Greater
than 0.6 are Considered Skillful
Results indicate that forecast skill is improving
at the rate of about one day per decade
8
RMS and Bias Errors NOGAPS Navy Operational
Global Atmospheric Prediction System
Errors can be stratified by latitude bands,
hemisphere, or specified geographic area
Largest wind errors are typically found at
jet-level where the winds are the strongest
9
NOGAPS Operational 48 h TC Track Forecast
Error All Basins
NOGAPS Navy Operational Global Atmospheric
Prediction System
Time series of the annual mean 48 hour tropical
cyclone track error (nautical miles). In 2002
NOGAPS was the best performing global model for
tropical cyclone prediction (JTWC). The
improvements in skill were largely due to the
transition of improvements to the cumulus
convection scheme and the increase in resolution.
10
NOGAPS Scoring System Used for Comparing
Benchmark Run with New Version
11
Validation of Coupled Models Outline

• Atmospheric Models
• Global
• Anomaly Correlation
• RMS, Bias Errors
• Tropical Cyclone Track
• Scorecard
• Mesoscale
• Idealized Flow Studies
• RMS, Bias
• Qualitative Verification (Case Studies)
• Event-Based Verification
• Ocean Models
• Global
• Features/Positions
• Sea Surface Height Validation
• Anomaly Correlation
• Sea Surface Temperature Validation
• Mesoscale
• Transport

12
Idealized Flow Studies COAMPS Coupled
Ocean/Atmosphere Mesoscale Prediction System WRF
Weather Research and Forecast Model
Mountain Wave Test Case Linear Hydrostatic
Gravity Waves T250K, U20 m s-1, hm1 m, a10
km dx2 km, dz250m, 121L Na/U5 Nhm/U1x10-4
Linear Analytic Solution
COAMPS
WRF (EM)
13
COAMPS RMS, Bias Verification Europe 27 km Grid
14
Subjective Evaluation of COAMPS (27 km) and
NOGAPS (T159) Forecasts of Mistral 27 Hour
Forecasts of 10 meter Wind Valid at 0300 UTC 22
Aug 1998
15
Improvement of Aerosol Prediction
Capability Validation of NAAPS Using SeaWiFS and
AERONET Data
16

• The user wants a deterministic answer
• The model produces a deterministic forecast
• Unfortunately, the outcome is not deterministic!
• Verification should communicate the nature of the
  variability

17
Event-Based Verification Composite Verification
Method

• Identify events of interest in the forecasts
• Rainfall greater than 25 mm
• Wind greater than 10 m/s
• Event contains between 50 and 500 grid points
• Define a kernel and collect coordinated samples
• Square box located at center of event
• 31x31 grid points (837x837 km for 27 km grid)
• Compare forecast PDF to observed PDF
• Repeat process for observed events

18
Event-Based Verification Collecting the Samples
19
Event-Based Verification Mistral Speed Statistics

• 66-hour wind speed forecasts for 2000-01 over the
  Mediterranean Sea
• Speed greater than 12 m/s, dir 270-70 deg.,
  covering 50-500 grid points
• Verified against SSM/I satellite observations

20
Event-Based Verification CONUS Warm Season
Precipitation

• 24-hour precipitation forecasts for
  April-September 2003 over full CONUS
• Rain events greater than 25 mm covering 50-500
  grid points
• Verified against River Forecast Center
  precipitation analysis

21
Validation of Coupled Models Outline

• Atmospheric Models
• Global
• Anomaly Correlation
• RMS, Bias Errors
• Tropical Cyclone Track
• Scorecard
• Mesoscale
• Idealized Flow Studies
• RMS, Bias
• Qualitative Verification (Case Studies)
• Event-Based Verification
• Ocean Models
• Global
• Features/Positions
• Sea Surface Height Validation
• Anomaly Correlation
• Sea Surface Temperature Validation
• Mesoscale
• Transport

22
Validation of Gulf Stream Position in the Navy
Layered Ocean Model (NLOM) Validation of the Gulf
Stream Position
23
Validation of Eddy Kinetic Energy (EKE) in
1/8-degree global NCOM Mean EKE at 700 m depth
during 1998-2000
NCOM Navy Coastal Ocean Model
In comparison to the free-running case, EKE at
700 m in the assimilative case is generally
higher and in closer agreement to historical
observations, showing the two regions of
relatively high EKE south of Nova Scotia and
Newfoundland.
24
Validation of the Navy Layered Ocean Model
(NLOM) Anomaly Correlation 42 30-day forecasts
from Dec 20 2000 to Oct 24 2001 Blue Line
Persistence, Red Line NLOM
25
NOGAPS/POP Air-Ocean Coupling Air-Ocean with Data
Assimilation/Forecast Cycle
Analysis-only produces significant errors in
coastal boundary currents
Reduced errors demonstrate importance of model to
data assimilation
OMVOI Ocean Multivariate Optimum Interpolation
Analysis POP Parallel Ocean Program Prediction
Model
26
Validation of Coupled Models Outline

• Atmospheric Models
• Global
• Anomaly Correlation
• RMS, Bias Errors
• Tropical Cyclone Track
• Scorecard
• Mesoscale
• Idealized Flow Studies
• RMS, Bias
• Qualitative Verification (Case Studies)
• Event-Based Verification
• Ocean Models
• Global
• Features/Positions
• Sea Surface Height Validation
• Anomaly Correlation
• Sea Surface Temperature Validation
• Mesoscale
• Transport

27
Validation of the Intra-Americas Sea
Nowcast/Forecast System (IASNFS) Run Daily to 72
h (http//www7320.nrlssc.navy.mil/IASNFS_WWW/)

• MODAS Modular Ocean Data Assimilation System
• 2D Optimum Interpolation Analysis
• Synthetic T/S profiles generated, used as
  observations
• All observations assimilated during 12-hour
  pre-forecast period
• Domain/Bathymetry

28
Validation of the Intra-Americas Sea
Nowcast/Forecast System (IASNFS) NCOM Predicted
Transport (2001 Mean) vs. Observations
Key IASNFS/Observation
29
Validation of the IASNFS Predictions of Sea Level
Height Comparison to Tide Gauges
30
Validation of the IASNFS Predictions of Sea Level
Height Comparison to Persistence
31
Validation of the IASNFS Temperature and Salinity
(T/S) Profiles Comparisons to (non-assimilated)
CTD data
32
Air-Ocean Coupling Coastal Issues Validation of
Wind Stress for 9 km Nest in EPAC
Black Line Stress calculated from
observations Blue Line Stress from operational
COAMPS interpolated to lat/lon grid Red
Line Stress from COAMPS reanalysis on native grid
Figure courtesy of John Kindle, NRL SSC
Results indicate that unfiltered, native grid
fields are required for proper forcing and
validation along coasts
33

• Ocean-Atmosphere Nested Modeling of the Adriatic
  Sea during Winter and Spring 2001
• Meteorology and Oceanography in the Adriatic

• Atmosphere
• Bora Strong, localized northeasterly winds
  around Istrian peninsula
• Scirocco Strong, warm southeast winds
• Ocean
• Cyclonic cells in the central and southern
  regions
• River runoff and strong winds create large
  variability in the northern Adriatic

34
Collaboration with Adriatic Circulation
Experiment (ACE)
1. Generate 27 km atmospheric forcing fields over
the Med 2. Generate 6 km, 2-year spin-up of the
Med using forcing from 1, then 12-hour data
assimilation for October 1999 3. Generate 4 km
atmospheric forcing fields over the Adriatic
Sea 4. Generate 2 km Adriatic forecasts using
initial conditions and inflow from 2, and
atmospheric forcing from 3

• Objectives
• Simulate Adriatic atmospheric and oceanic
  circulation at high resolution
• Document and understand response of the shallow
  northern Adriatic waters to forcing by the Bora
  and Po river run-off
• Quantify the effects of coupling (e.g., one-way,
  two-way, frequency, resolution) on atmosphere and
  ocean forecasts
• Aid in planning and interpreting Adriatic
  Circulation Experiment (ACE) observations

81 km
COAMPSTM
36 km
COAMPSTM
27 km
12 km
4 km
1
4
3
2
Initial conditions and lateral boundary forcing
6 km NCOM
2 km NCOM
35
Ocean-Atmosphere Nested Modeling of the Adriatic
Sea during Winter and Spring 2001 COAMPS Wind
Stress (Mean and RMS vector amplitude) 28 January
- 4 June 2001
36
Ocean-Atmosphere Nested Modeling of the Adriatic
Sea during Winter and Spring 2001 COAMPS/NCOM
Model Circulation EOFs NCOM 2 km Grid Spacing
37
Comparison of observed 10 m winds to observations
(top) and 25 m ocean current to observations
(bottom) Comparison using 36 km (blue) and 4 km
(red) atmospheric forcing
Results suggest that the consideration of the
effects on an ocean model should be a metric in
the validation of atmospheric models and that
high-resolution forcing fields improve ocean
forecasts
38
Importance of Temporal Resolution of Ocean
Forcing Comparison of NCOM runs using 1 h, 6 h,
and 12 h COAMPS forcing
39
Real-Time COAMPS Support for AOSN II
AOSN II Adaptive Ocean Sampling Network II
Quadruple-nest grid built for AOSN area

• Twice Daily Forecasts to 72 h with Data
  Assimilation
• NOGAPS Lateral Boundary Conditions
• SGI Origin 3900 at FNMOC DoD HPC DC Facility
• Real-Time Winds and Fluxes Used to Force Multiple
  Ocean Models

40
Concluding Remarks Atmospheric Model Validation

• Many Tools Available
• RMS, Bias
• Anomaly Correlation
• Idealized Tests
• Threat Scores
• Event-Based Validation
• Qualitative/Quantitative Case Studies
• Long-Term Studies are Mandatory
• Avoid Simplistic Answers with Single Case Study
• Minimum Requirements for Evaluation of Systems
• 2-week Periods for Summer and Winter
• Test Over Several Different Geographical Areas
• Simple Questions/Complex Answers to Validation
• Grid Structures
• Formulation of Dynamics
• Physical Parameterizations/Interactions
• Data Assimilation Issues (i.e., QC, Analysis
  Techniques, Initialization, First-Guess)
• Sensitivity in specific grid-point validation
• Represent-ativeness of what is being validated
  (i.e., Resolution)

41
Concluding Remarks Ocean/Coupled Model Validation

• Many Tools Available
• RMS, Bias
• Anomaly Correlation
• Qualitative/Quantitative Case Studies
• Idealized Test Cases
• Validation/Performance Affected by Atmospheric
  Model
• Resolution (Spatial and Temporal)
• Grid Native vs. Interpolated/Filtered
• Long-Term Studies are Mandatory
• Unique Validation Parameters
• Transport
• Sea Surface Height
• Tides
• Simple Questions/Complex Answers to Validation as
  in the Atmospheric Models

42
Concluding Remarks Challenges

• Demonstrating improved skill is becoming more
  difficult to do
• Models have improved tremendously
• Modeling systems are much more complex
• Requires thorough understanding of model(s), no
  black box mentality
• More validation metrics are needed, especially
  for mesoscale modeling
• Higher resolution does not always translate to
  improved skill scores
• Phase/Pattern shifting validation?
• Expect dramatic increase in remotely-sensed data
  - How to apply to validation of models?
• Coupled modeling complicates the validation
  process
• Air/Ocean interactions/feedbacks
• What if atmosphere forecasts are better (worse)
  and ocean forecasts are worse (better)?
• Additional resources needed
• Commitment of more resources to validation (Also
  commit more resources to preparing efficient code)

43
Concluding Remarks Lessons Learned from Model
Validation/Development

• Important, Do Right
• Listen to the Customer
• Data Assimilation
• Configuration Management
• Lower Boundary Condition
• Physical Parameterizations
• Validation/Verification
• Efficiency
• Be Creative, Build Flexibility in System
• Important, Dont Do Wrong
• Numerics
• Grid Configuration/Flexibility/Relocatability
• Upper, Lateral Boundary Conditions
• Horizontal Diffusion
• Database Issues
• Portability
• Resolution (terrain, coastlines, etc.)
• Plug-compatible code
• Use Standard Sane FORTRAN, UNIX

44
Validation of Coupled Models
Richard M. Hodur Naval Research
Laboratory Monterey, CA 93943-5502 hodur_at_nrlmry.n
avy.mil Short Course on Significance Testing,
Model Evaluation, and Alternatives 11 January
2004 Seattle, WA

• Outline
• Introduction
• Atmospheric Models
• Ocean Models
• Concluding Remarks