Marine Verification at WFO Key West

1

• Marine Verification at WFO Key West
• Where to go and how to get there
• Andy Devanas, SOO and
• Matt Parke, Verification Focal Point

2
There are three kinds of lies lies, damned
lies, and statistics. Benjamin Disraeli (1804
- 1881)
3
USF Buoy Not Operational
X
X
X
X
X
X
X No Surface Data Available NO WAVE DATA AT ALL
Marine Area approx. 22000 square miles
Massachusetts 10555 square miles
4
(No Transcript)
5
(No Transcript)
6
Key West Verification System Needs
Need to integrate verification into the forecast
process Need to provide meaningful, frequent and
timely verification data to forecasters ( e.g.
Percent Correct vs. ESS) Need to identify
forecast and guidance biases Enhanced
discrimination in critical categories
7
How To Get There
Use MVerify for rapid feedback to
forecasters -Local AWIPS application -Provides
most flexibility in stratification of desired
parameters Minimum forecast period
adjustable Can segregate by forecaster Invest
igate various statistical measures/distributions
which best track forecast performance Set
reasonable goals of forecast performance. Monito
r forecast performance. -Accountability Use
verification statistics in post-mortem
evaluations as feedback to forecasters.
8
APPLICATION OF THE NATIONAL MARINE VERIFICATION
PROGRAM (NMVP) TO LOCAL MARINE VERIFICATION
DRAWBACKS NMVP is not timely. Must wait until
after end of month to access statistics. NVMP
does not allow defining of periods. Minimum
period is one month. Cant define inter-month
periods. NMVP does not provide access to
individual forecast-observation pairs. -Cannot
do post-mortem analysis. -Do not know which
matched pairs are producing statistics. -No
Quality control of forecasts and
observations. Some measures (ESS) are heavily
weighted towards rare climatological
events. -ESS is volatile. -Critical missing
matched pairs can have large impact on
ESS. BENEFITS Provides relatively easy and
quick statistics spanning gt 3 year dataset Can
stratify data with regards to forecast period,
buoy (cman), and season
9
Equitable Skill Score reward/penalty
Characteristics (from NMVP)-

• A relatively small reward for correctly
  forecasting common events
• A large reward for correctly forecasting rare
  events
• Less penalty is assigned to an incorrect forecast
  of a rare event than a similar size error of a
  common event. "Near hits" of rare event receive a
  modest reward.
• Uses a graduated reward/penalty system, whereby a
  large forecast error is penalized more than a
  small forecast error for a given category of
  events.

Whats wrong with this picture??
10
ESS Does Not Account For Value of Forecast to Our
Customers
MOST CRAFT REMAIN IN PORT
S C E C
S C A
11
Before Matched Sets at NMVP Site
12
This Forecast Was Never Made Where Did This
Number Come From?
EYW Forecasts January 2004
This Will Severely Skew ESS Score. As Well As
Other Statistical Measures.
13
Percent Correct Definition From NOAA
Verification Glossary PC (AD)/(ABCD).
14
Percent Correct Definition From Marine
Verification Program Glossary
Percent Correct Definition Percentage of the
correct forecasts (hits) from the total number of
events. Formula                   
In Other Words A/(ABCD)
15
gaming
16
Some EYW verification results

• The majority, almost 89, of our observations
  (marginal distribution p(x)) were 17 kts.
• Our forecast system had a hard time
  discriminating between light winds (lt 8kts) and
  winds of 8 12 kts.
• Our forecast system is slightly better
  discriminating in the 8 12 kt range, but still
  tends to overforecast more often than
  underforecast.
• At 13 17 kts, our system is the least biased.
• We underforecast (bias gt 1) moderate to high wind
  events.
• Forecasters tend to game light wind forecasts
  which do not have much value to customers.

17
Gridded Verification at EYW.
Otherwise known as
What are you Kidding me??
18
From Murphy 1993
Since Forecasters judgments necessarily contain
an element of uncertainty, their forecasts must
reflect this uncertainty accurately in order to
satisfy the basic maxim of forecasting. In
general, then, forecasts must be expressed in
probabilistic terms.
19
Melbournes ADAS Closer to grid
verification Supported by relatively dense
surface data network over land Buoy in Marine
Zones
20
LAPS analysis highly influenced by single data
point. Winds in this region most likely NE 10-15
knots.
21
Again note influence of single data point over
huge region
22
We do not have ANY guidance to forecast wave
height inside the reef at this resolution and
detail, and little skill therefore we do not
attempt to forecast this region in our grids and
zero out this region but do include in our
CWF. Wavewatch III is poor at best outside of the
reef and does not account for gulf stream
interactions.
AND NOT ONE WAVE HEIGHT DATA POINT!! So How Do
You Verify Wave Height?? Answer You dont.
23
Marine Forecast 730 PM EST SUN MAR 14 2004 SMALL
CRAFT ADVISORY TONIGHT EAST WINDS AROUND 20
KNOTS. SEAS OUTSIDE THE REEF 5 TO 7 FEET...EXCEPT
HIGHER IN THE GULF STREAM. SEAS INSIDE THE REEF 2
TO 3 FEET. NEARSHORE WATERS CHOPPY TO ROUGH.
SCATTERED SHOWERS. MONDAY EAST WINDS 15 T0 20
KNOTS...DECREASING TO 10 TO 15 KNOTS BY NOON.
SEAS OUTSIDE THE REEF 4 TO 7 FEET...EXCEPT HIGHER
IN THE GULF STREAM. SEAS INSIDE THE REEF 2 TO 3
FEET. NEARSHORE WATERS CHOPPY. SCATTERED SHOWERS.
MONDAY NIGHT SOUTHEAST WINDS NEAR 15 KNOTS.
SEAS OUTSIDE THE REEF 3 TO 5 FEET. SEAS INSIDE
THE REEF 2 TO 3 FEET. NEARSHORE WATERS A MODERATE
CHOP. ISOLATED SHOWERS.
24
GRID VERIFICATION THOUGHTS

• You must have relatively dense surface data
  network to support gridded verification field.
• Perception that numerical models are the better
  performer are incorrect and not supported.
• Need high resolution shallow water wave guidance.

Given the current state of model guidance and
surface data available for the EYW marine area
- It is entirely possible that our forecasters
will have to DEGRADE our forecasts to match
verification fields