Metrics for Model Skill Assessment

1
Metrics for Model Skill Assessment

• Model Error time series (model-data misfit)
• ME(i) model - data
• Total Root-Mean-Square Error RMS_Total
• RMS_Total2 RMS_Bias2 RMS_Variability2
• RMS_Bias difference between means
• RMS_Variability (centered pattern RMS)
• mean difference of deviations from mean
• RMS_Variability Correlation, Amplitude --gt
  Taylor diagram
• Amplitude of deviations
• Correlation of deviations

2
Taylor Plot Graphical relationship between time
series based on four statistics 1) Overall
Mean Bias 2) Seasonal Variance Standard
Deviation 3) Timing/Phase Correlation
Coefficient 4) Root Mean Square Error Centered
RMS Distance (RMS_V)

3
Taylor Diagram Example
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
new Run 801
old Run 751
8
(No Transcript)
9
Target Diagram as Skill Assessment Tool
RMS_T2 RMS_B2 RMS_V2
SAB SST climatology
model-data misfit variability in data
model-data misfit error in data
10
SST
11
Chlorophyll
12
Chlorophyll
13
Summary
Taylor Target diagrams are two complimentary
ways of assessing model skill - Taylor
Correlation of variability Amplitude
of variability (Bias) - Target
Total RMS Relative bias and
variability components
14
Summary (cont.)

• SST
• Correlation 0.9 always
• satellite, in situ, 2004, climatology
• Amplitude of variability good
• especially for satellite 2004 comparisons
• underestimate in FL, GA
• overestimate everywhere north of SC
• Bias low
• underestimate in SAB in climatology, better
  using 2004
• RMS_bias RMS_variability
• MLD
• Correlation always positive
• Higher in MAB (.8) than SAB (.5)
• Higher in outer SAB (gt.6) than inner SAB (lt.4)
• Amplitude of variability overestimate
  variability
• Except for MAB Outer shelf
• Bias generally low

15
Summary (cont.)

• Surface chlorophyll - much greater challenge!
• Correlation -0.6 to 0.9 (same for Clim and
  2004)
• lower off NC, SC, NY
• Higher off FL, DE, NJ
• Amplitude of variability so-so (worse for 2004
  in SAB)
• underestimate in SAB
• overestimate in MAB
• Bias large negative bias everywhere
• underestimate in GA, SC (benthic production?)
• underestimate on inner MAB shelf
• RMS_bias gtgt RMS_variability
• Little correlation between where MLD/SST is
  modeled
• well (poorly) and where chlorophyll is modeled
  well (poorly)

16
Future Work

• Use these Taylor/Target diagrams to compare runs
• With/without tides
• With/without DOM
• Plot other quantities
• kPAR, productivity, oxygen, salinity
• Examine other regions
• Gulf of Maine
• Gulf Stream/Sargasso
• Use these for the OCRT meeting?
• Use these for the Oceanography article?