OSSEs and testbeds

1
OSSEs and testbeds Why were going there
2
New technology must be validated across the globe
Wave Height
3
New technology must be validated across the globe
Wave Height
4
Operational Validation and Verification

• Administrative Modeling Oversight Panel (AMOP)
• Ensure development advances from TRL 5
  (validation in a realistic research environment)
  to TRL 8 (validation in an operational
  evironment)
• Ensure computational and personnel resources will
  be available
• Chaired by Commander Naval Meteorology and
  Oceanography Command (CNMOC)
• Members from ONR, NRL, NAVO, FNMOC, N84, SPAWAR

5
Operational Validation and Verification

• Administrative Modeling Oversight Panel (AMOP)
• A clearly defined process is necessary but not
  sufficient
• Rigor develops over time
• Initial delivery is easy
• Replacing components is difficult

6
Operational Validation and Verification

• Administrative Modeling Oversight Panel (AMOP)
• Example of a small transition
• Predicting position of tidally generated internal
  waves in the South China Sea
• Uses mostly operationally accepted model, data,
  assimilation

7
Example of Planning Necessary
Non-linear internal waves forecasting Tasks
Implement an internal waves alarm capability at
NAVO
ONR NLIWI
Density Measurement by Shipboard CTD
NRL LZS ONFS IW Prediction
Satellite SAR Image
Dongshan Ko, 6.1 NLIWI
8
Example of Planning Necessary
Non-linear internal waves forecasting Tasks
Implement an internal waves alarm capability at
NAVO
ONR NLIWI
MODIS
Chao, S.Y., D.S. Ko, R.C. Lien, and P.T. Shaw,
Assessing the west ridge of Luzon Strait as an
internal wave mediator, J. of Oceanog., 63,
897-991, 2007.
Dongshan Ko, 6.1 NLIWI
9
Example of Planning Necessary
Non-linear internal waves forecasting Tasks
Implement an internal waves alarm capability at
NAVO

• Validate Nested System in South China Sea to
  Examine Skill in Prediction of Internal Waves
  Position and Amplitude
• Set up Nested system for SCS and perform
  validation tests
• Input data files have been assembled and checked
  
• Validation test period will be April-May 2005
• Upgrade internal waves prediction capability and
  document differences from standard Nested system

10
Example of Planning Necessary
Non-linear internal waves forecasting Tasks
Implement an internal waves alarm capability at
NAVO

• Implement Internal Waves Alarm Capability
• Develop operational metrics for significant
  internal waves
• Implement metrics to produce alarm capability
• Test user interaction functionality with
  NAVOCEANO personnel
• Test and verify alarm capability at NAVOCEANO

11
Example of Planning Necessary
Non-linear internal waves forecasting Tasks
Implement an internal waves alarm capability at
NAVO
FY11
FY10
FY09
FY08
FY07
Milestone
S------D--------R----C
ONR NLIWI Project 1. Demonstrate internal waves
prediction using research model
2. Validate and upgrade RELO internal waves
prediction capability for SCS
S-----D-------R-----C
3. Establish Internal Waves Validation Panel
R
Note S -Start, D -Demonstration, R -Report, MS
-Milestone, T -Transition, C -Complete.
12
Example of Planning Necessary
Non-linear internal waves forecasting Tasks
Implement an internal waves alarm capability at
NAVO
S--TD2
5. Transition internal waves alarm capability
applying upgraded RELO
Note S -Start, D -Demonstration, R -Report, MS
-Milestone, T -Transition, C -Complete.
13
Example of Planning Necessary
Non-linear internal waves forecasting Tasks
Implement an internal waves alarm capability at
NAVO
Deliverable Capabilities

• Capability to predict non-linear internal waves
  in SCS using RELO
• Alert operators when internal waves exceed
  predefined levels
• Improved internal waves prediction capability
  incorporated into RELO model at NAVOCEANO

14
Example of Planning Necessary
Non-linear internal waves forecasting Tasks
Implement an internal waves alarm capability at
NAVO

• Projected Transition Dates for MS1 MS2
• Transition Deliverables Status
• Prior to Milestone 1
• Source Code Source Code 65 complete
• Transition Plan Complete on 31 December 2008
• Validation Test Report Complete on 31 March 2009
• Preliminary DOD-STD Documentation Complete on 31
  March 2009
• Prior to Milestone 2
• Final DOD-STD Documentation Complete 30 May 2009
• OPTEST Report 30 August 2009

15
Operational Validation and Verification

• Administrative Modeling Oversight Panel (AMOP)
• Validation Test Panel decides if system
  performance is satisfactory
• Technically sound capability
• Validated for specific purposes
• Test report is satisfactory
• Composed of developers, operation center,
  external independent experts
• Must pass the smell test and the Washington Post
  test
• Necessary but not sufficient
• Does everyone always make it work correctly?

16
Global Navy Coastal Ocean Model Timeline for
development and transition of operational system
6-23-2008 GOMS V2.6 declared operational
includes NCODA assimilation of in situ data into
a first guess of MLD-modified synthetics, code
version 4.0, coupled CICE model. VTR in progress.
Hurricane Katrina 8-29-2005 Older
operational models never restarted.
July 2002 Formed Validation Test Panel. Begin
model improvements, transition, experiments for
VTR.
Aug 17 2000 Began running 0.25 global NCOM in
real time for NAVOCEANO to use in Barents Sea for
Kursk response. NCOM code v1.5
2-19-2006 AMOP declares global ocean modeling
system (GOMS) V2.0 operational
12-11-2003. Last meeting of GNCOM validation test
panel. Model, all supporting documentation, and
Navy VTR submitted to NAVO. Recommend begin
OPTEST.
Paul Martin et al. published NRL report comparing
several coastal ocean models. This led to
developing NCOM.
3-6-2006 GOMS V2.5 declared operational (NCOM
upgrades 1/32 NLOM)
Barron, et al. 2004. (GNCOM SSH)
2004-2005 Model running in real time on NAVO
operational accounts but not declared
operational. NRL evaluating new capabilities
(versions 2i-2l).
NRL river climatology added
November 2001 Began running 0.125 global NCOM
in real time on NRL account. NCOM code v1.9.
GNCOM v glb8_1g.
Barron, et al. 2007. (Evaluation of modified
assimilation)
October 2003 Global NCOM version glb8_2f began
running in real time in nowcast/forecast mode
under operational account at NAVO. Used code
version 2.3.
Barron, et al. 2006. (Model formulation) Kara et
al., 2006. (Model validation)
June 1999 John Harding letter to Eric Hartwig
proposing development of a global model based on
NCOM. Based on Bob Rhodes idea 2 years earlier.
Feb 2002 Used data from glb8_1g for support of
Strong Resolve 2002.
Barron, 6.4 LargeScale / ODA
17
Operational Validation and Verification

• Administrative Modeling Oversight Panel (AMOP)
• How rigorous demonstration / validation is
  sufficient
• What statistical confidence level is required
• Is sufficient data available to state with
  required confidence that system performance meets
  requirements

18
Ocean Data Assimilation (end FY08)
MLD is treated as a proxy for sonic layer depth.
Errors in MLD and SLD are proxies for surface
duct transmission loss errors (Helber et al., JGR
Oceans, in press).
If the SLD is deep enough, acoustic energy is
trapped in a surface duct.
If the SLD is too shallow, acoustic energy
spreads spherically, giving short transmission
ranges.
Barron, Helber, 6.2 ISOP
19
Ocean Data Assimilation (end FY08)
hydrographic proxies indicate expected acoustic
fidelity
Mixed Layer Depth (MLD) Kara et al. (2000)
density threshold equivalent to ?T0.8C
Sonic Layer Depth (SLD) Helber et al.
(submitted) near surface sound speed maximum
appropriate for frequency range
Below-Layer Gradient (BLG)
More details in paper in press Helber, Barron,
Carnes and Zingarelli Evaluating the sonic layer
depth relative to the mixed layer depth, J.
Geophys. Res.
Barron, Helber, 6.2 ISOP
20
Large Scale Prediction
Temperature vs. depth error analysis
Locations of 8400 unassimilated profiles used
in the error analysis spanning June-July 2007
Metzger, 6.4 LargeScale
21
Ocean Data Assimilation NCOM assimilating
MLD-modified synthetic vs. synthetic
For MLD, where is MLD-modified GNCOM better than
a standard synthetic (MODAS)?

• 11 m GNCOM median improvement
• GNCOM better in 80 of the regions. (shown in
  red and yellow)

Statistics based on 43,474 T,S profiles from 2006.
Barron, Helber, 6.4 ODA
22
Ocean Data Assimilation Global
Median statistics relative to 43,474 T,S profiles
over the global ocean.
Statistics based on 43,474 T,S profiles from
2006. Profiles are compared with nearest
climatology, synthetic, and GNCOM nowcasts. None
of the products here assimilate profiles.
Barron, Helber, 6.4 ODA
23
Operational Validation and Verification

• Administrative Modeling Oversight Panel (AMOP)
• It only gets harder

24
Large Scale Prediction
SSH and independent IR frontal analysis 10 March
2008
Kuroshio
Frontal analysis lt 4 days old white, analysis
4 days old black
Gulf Stream
www7320.nrlssc.navy.mil/GLBhycom1-12/skill.html
Hurlburt, Metzger, 6.2 NOPP GODAE
25
Large Scale Prediction
V3.0 Phase 1 validation tasks

• HYCOM/NCODA (V3.0) is performing equal to or
  better than
• GNCOM-02f (V2.5) with regard to
• Large scale circulation features
• Determine correct placement of large scale
  features
• Eddy kinetic energy / sea surface height
  variability
• determine if the system has a realistic level and
  distribution of energy at depths
• Sea surface temperature
• evaluate whether the models are producing
  acceptable nowcasts and forecasts of sea surface
  temperature
• Coastal sea level
• assess the models ability to represent observed
  sea surface heights

26
Temperature vs. depth error analysis
V3.0 best experiment using CH (black) vs. V3.0
using MODAS synthetics vs. V2.6 NCOM-03a (blue)
Mean error (C)
RMSE (C)
Based on 8400 unassimilated profiles over global
ocean for the period Jun-Jul 2007 Black curve
uses NCODA modifications 1-5 Red curve uses NCODA
modifications 2, 3 6 (only at altimeter pts of
large SSH change) Blue curve uses MODAS
synthetics at ALL model gridpoints
Metzger, 6.4 LargeScale
27
Large Scale Prediction
MLD/SLD/DSC/BLG error analysis
MODAS synthetics HYCOM versus NCOM
Metzger, 6.4 LargeScale
28
Operational Validation and Verification

• Administrative Modeling Oversight Panel (AMOP)
• It only gets harder
• For nested models, there is not sufficient data
  to reach statistically sound conclusions
• Data necessary to constrain dynamics is not
  available
• Level of performance is not certain

29
RELO NCOM - Regional forecasting
Globally relocatable, 1- to 3-km resolution,
regional 3D temperature, salinity and currents
Surface temperature (?C) and currents (m/s) on a
6km grid.

• Globally relocatable
• 72h forecasts
• 1-6 km horizontal resolution
• Forecast component
• Navy Coastal Ocean Model (NCOM)
• Assimilation component
• NRL Coupled Ocean Data Assimilation (NCODA)
• Altimetry, satellite SST, in situ surface and
  profile observations
• Forcing
• Surface fields from NOGAPS, COAMPS
• Lateral boundary from Global NCOM
• Tides using Oregon State databases
• River fluxes using the NRL global river database

Latitude (?N)
Longitude (?E)
Rowley, 6.4 SmallScale
30
LBSFI CTD Fusion Integration

• Focusable systems for tactical Navy applications
• Testbed construction
• Sensor QC of data to correct vehicle motion
  effects, bio-fouling
• OcnQC of threaded data
• Super-observations from high horizontal
  resolution threaded data
• Construction of representativeness errors (how
  much sub-gridscale variability is in the data)

ASAP-MBARI-2006
Spence, 6.4 LBSFI
31
Expected Error Levels
No winds No Assim NOGAPS winds NOGAPS,
Assim NOGAPS, Assim 2 A1/A2
Hogan, 6.1 SEED
32
Expected Error Levels
Temperature 0m
Temperature 200m
Ensemble RMS T 0m
Ensemble RMS T 200m
Rowley, Coelho, Fabre 6.2/6.4 Adaptive Sampling
33
Acoustic evaluation of ensemble represenation
Does ensemble SSP provide acoustic propagation
encompassing observed?
Fabre 6.2/6.4 Adaptive Sampling
34
Acoustic evaluation of ensemble represenation
125Hz
Fabre 6.2/6.4 Adaptive Sampling
35
Targeting Observations for Forecast Optimization
Rowley, Coelho, Fabre 6.2/6.4 Adaptive Sampling
36
Observation System Control
Asset Allocation
Nature Ocean
Sampling Strategy
Glider Navigation
OSSE
Observations
Model Ocean
Assimilation
Performance Assessment
Mission Requirements
System Evaluation

• Does the model ocean meet mission requirements
  given the present asset allocation and sampling
  strategy? If not,
• increase asset allocation and try again
• modify sampling strategy and try again
• reduce mission area, duration, or performance
  requirements

Barron, 6.2 GOST
37
Operational Validation and Verification

• Administrative Modeling Oversight Panel (AMOP)
• Plans to implement new capabilities into
  operations (2-5 year plans)
• 3DVar
• 4DVar
• Glider sensor QC
• Glider specific OcnQC
• Glider data thinning / super-obs
• Representativeness errors
• BCs
• To reach statistical significance, test over more
  events and areas TEST BEDS

38
System Assessment Test Beds
Building historical data sets, test beds, case
scenarios

• AOSN_II_2003/ - Autonomous Ocean Sampling Network
  II,2003, Monterey Bay, California - Input data
  for 01 July - 30 September 2003
• TASWEX_2004/ , 2004, East China Sea - Input data
  for 01 September - 15 November 2004
• SCS_2005/ - South China Sea, ONR Nonlinear
  internal wave initiative, 2005
• ASAP_2006/ - Adaptive Sampling and Prediction,
  2006, Monterey Bay, California - Input data for
  01 July - 30 September 2006
• RIMPAC_2006/ - Rim of the Pacific, 2006 Hawaii
  region Input data for 01 July 30 July 2006
• ShallowWater_2006/ - Shallow Water, 2006, New
  Jersey Shelf, USA - Input data for 01 May - 31
  August 2006
• RADAR_2007/ - Ocean-Acoustics Trial, 2007,
  Setubal Canyon, Portugal - Input data for 01 June
  - 31 July 2007
• BP-LASSIE_2007/ - REA, Acoustics and Air-Sea
  interaction, 2007, Italy - Input data for 01
  March - 30 June 2007
• ValiantShield_2007/ - Valiant Shield, 2007,
  Micronesia (Guam area) - Input data for 01 July -
  31 August 2007
• Okinawa Trough_2007 / - Navy survey Sep-Nov 2007

Spence, Coelho, 6.4 LBSFI
39
Example LBSFI Data Structure (AOSN-II)?
/u/LBSFI/Glidr_Exp_Data/
AOSN_II_2003/
BC's
host/
gncom_cutouts/
spence.AOSN/
Atm Forcing
met/
NOGAPS/
COAMPSg/
altim/
nogaps0.50/
nogaps1.00/
glider/
2003/
Obs Data
ocnqc/
goes/
AUV/
lac/
MBARI/
mcsst/
glider/
profile/
ship/
ship/
godae/
ssmi/
Spence, Coelho, 6.4 LBSFI
40
LBSFI CTD Fusion Integration

• Focusable systems for tactical Navy applications
• Testbed construction
• Sensor QC of data to correct vehicle motion
  effects, bio-fouling
• OcnQC of threaded data
• Super-observations from high horizontal
  resolution threaded data
• Construction of representativeness errors (how
  much sub-gridscale variability is in the data)

ASAP-MBARI-2006
Spence, 6.4 LBSFI