MIW METOC Metrics: Ops Analysis

1
MIW METOC Metrics Ops Analysis
Tom Murphree and David Meyer Naval Postgraduate
School (NPS) murphree_at_nps.edu Bruce Ford and
Manuel Avila Clear Science, Inc.
(CSI) bruce_at_clearscienceinc.com Paul Vodola,
Matt McNamara, Luke Piepkorn, and Ed Weitzner
Systems Planning and Analysis (SPA) pvodola_at_spa.c
om
Presented at MIW METOC Metrics Symposium III
CNMOC, Stennis Space Center, MS 1 February 2008
2
MIW METOC Metrics Ops Analysis

• Outline
• Previous Ops Analysis Modeling
• Weather Impact Assessment Tool (WIAT)
• Linking METOC and Customer Performance
• Role of Ops Analysis in the MIW Metrics Process

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
3
MIW METOC Metrics Ops Analysis

• WIAT Overview
• WIAT is
• A pc-based, stochastic simulation tool used for
  assessing the impacts of METOC forecasts and
  METOC phenomena on strike operations
• WIAT can
• Simulate modern naval strike operations with a
  focus on fixed strike (STK) and kill box
  interdiction (KI) and close air-support (CAS)
• Take as input both actual weather conditions and
  forecasts of conditions with varying lead times
  and accuracies
• Emulate actual mission planning processes days
  and hours ahead of mission execution
• Model customer decisions and outcomes based on
  weather predictions and other factors
• Allows users to
• Simulate impacts of METOC forecasts and phenomena
  on spectrum of operational situations
• Assess mission effectiveness employing a variety
  of METOC support situations
• Determine the payoff/ROI for increased
  forecasting accuracy

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
4
MIW METOC Metrics Ops Analysis
WIAT Overview
From NPS thesis research of LCDR Jake Hinz, USN,
2004
For official use only
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
5
MIW METOC Metrics Ops Analysis

• WIAT Time-Error Forecast Analysis
• Metrics collected through the online real-time
  collection system can be used as input to the ops
  analysis models to show the link between METOC
  performance and customer performance
• Purpose of the following parametric analysis is
  to show that improved timing of forecasted
  features leads to improved operational
  effectiveness in Strike warfare
• Perfect forecasting case is compared with timing
  errors of one and three days

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
6
MIW METOC Metrics Ops Analysis
METOC PERFORMANCE METRICS
WIAT Time-Error Forecast Analysis

• Large amount of weather near the RYG thresholds
  decreases FAC POD
• Higher accuracy rate in forecasting Green Wx than
  Yellow or Red
• Three-day errors not significantly worse than the
  one-day error

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
7
MIW METOC Metrics Ops Analysis
OPERATIONALMETRICS
WIAT Time-Error Forecast Analysis

• Effectiveness per sortie is degraded due to
  inaccurate forecasting
• Results in 11 decrease in targets killed per
  mission

• Sorties with degraded BDA increased from 42 to 68
• Increase from 16 of all missions flown to 27

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
8
MIW METOC Metrics Ops Analysis
WIAT Time-Error Forecast Analysis
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
9
MIW METOC Metrics Ops Analysis
WIAT Time-Error Forecast Analysis
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
10
MIW METOC Metrics Ops Analysis

• Role in MIW Metrics
• Develop and apply a set of operational analysis
  and modeling tools for
• Modeling MIW operations, especially METOC inputs
  to planning process and impacts on operational
  outcomes
• Evaluating operational implications of METOC
  metrics based on real world data
• Developing quantitative operational thresholds
  for the performance of METOC products
• Identifying gaps and excesses in METOC support,
  and recommend mitigation strategies (DOTMLP-F)
• Assessing alternative CONOPS for METOC support
• Informing budgetary decisions at directorate and
  resource sponsor levels
• Support initial operational use of tool set to
  provide assessment of METOC impact on customer
  performance

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
11
MIW METOC Metrics Ops Analysis

• Application
• Tool allows for assessment of the impacts of
  METOC support products and services across a wide
  range of MIW scenarios
• Determine sensitivities to increased Accuracy
• Effect of enhanced Timeliness

Scenario
Offensive Mining MCM Sweeping MCM Hunting
A
B
C
D
Multiple combinations of METOC products and
services support MIW
METOC Product
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
12
MIW METOC Metrics Ops Analysis

• Application
• Use collected data to determine performance
  relative to the determined sensitivities
• Answers the question HOW WELL ARE WE DOING?

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
13
MIW METOC Metrics Ops Analysis

• Application
• For each Scenario METOC Product pair, determine
  current performance in relation to the defined
  benchmarks
• Identify gaps excesses
• Provide strategy recommendations (DOTMLP-F)

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
14
MIW METOC Metrics Ops Analysis

• Application Overview

Periodic re-evaluation provides a feedback loop
between collected data performance benchmarks
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
15
MIW METOC Metrics Ops Analysis
QUESTIONS?
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
16
MIW METOC Metrics Ops Analysis

• Outline
• Role of Ops Analysis in the MIW Metrics Process
• Previous Ops Analysis Modeling
• Weather Impact Assessment Tool (WIAT)
• Linking METOC and Customer Performance

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
17
MIW METOC Metrics Ops Analysis

• Notional Tool
• Notional MIW Weather Impact Assessment Tool
• Simulates modern Mine Warfare operations
• Inputs both actual environmental conditions and
  predictions with varying lead times and
  accuracies
• Emulates actual mission planning processes days
  and hours ahead of mission execution
• Models customer decisions and operational
  outcomes based on environmental predictions and
  other factors
• Value
• Simulates impacts of METOC predictions and
  phenomena on spectrum of MIW situations
• Facilitates determination of phenomena which have
  significant impact on MIW operations
• Evaluates MIW effectiveness employing a variety
  of METOC support situations

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
18
MIW METOC Metrics Ops Analysis

• Outline
• Role of Ops Analysis in the MIW Metrics Process
• Previous Ops Analysis Modeling
• Weather Impact Assessment Tool (WIAT)
• Linking METOC and Customer Performance

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
19
MIW METOC Metrics Ops Analysis
WIAT Role in METOC Metrics
Currently, WIAT has been used to experiment with
various levels of METOC support to STW
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
20
MIW METOC Metrics Ops Analysis
WIAT METOC Features

• Forecasted and actual conditions over time
• 0.5o spatial resolution
• Temporal resolution varying, down to 2 hrs
• METOC features modeled
• Cloud coverage (three layers)
• Precipitation
• Visibility
• Wind speed
• Includes inputs for other forecasted and observed
  conditions
• Weather features impact all components of the
  simulations
• Threat friendly forces
• Weapon probability of kill
• Sensor probability of detection
• Allows for the input of various forecast
  accuracies to capture operational effects of
  forecast uncertainty

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
21
MIW METOC Metrics Ops Analysis
METOC IMPACTMETRICS
WIAT Time-Error Forecast Analysis

• Mission change accuracy decreases by 40 60
  percent

• Over 30 fewer missions are saved when forecast
  timing errors are introduced

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
22
MIW METOC Metrics Ops Analysis

• Model Development Approach
• Develop realistic vignettes and scenarios for
  MIW, including mission-level metrics of success
• Identify the environmental factors that most
  significantly influence MIW platform performance
  and operational effectiveness
• Describe and model the flow of METOC information
  from reachback, through in-theater support, to
  the MIW commander
• Describe and model the MIW mission planning
  process, to include how and when METOC
  information is incorporated
• Develop operational model(s) for MIW
• Assess METOC support performance and impact on
  customer performance

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
23
MIW METOC Metrics Ops Analysis
WIAT Model Schematic
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
24
MIW METOC Metrics Ops Analysis
WIAT Mission Planning Process
Flight Plan Generation and Modification Timeline
Flight Ops
Execution Forecast
Sortie Generation
Planning Forecast

• Primary mission modifications

• Secondary modificationsand sortie cancellations

• Initial flight plan generation

Multiple Flight Plan Updates Over Time due to
Forecasting ISR
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
25
MIW METOC Metrics Ops Analysis
WIAT Output Metrics

• WIAT contains an automated output processing tool
• WIAT provides an automated link between METOC
  support and operational performance

WIAT Captures METOC Metrics
and Operational Impact Metrics
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
26
MIW METOC Metrics Ops Analysis

• Application
• Based on these determinations, METOC support
  benchmarks can be developed to establish goals
  and evaluate real world performance

MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com
27
MIW METOC Metrics Ops Analysis
Linking METOC to Customer Performance
WIAT Time-Error Forecast Analysis

• Forecast Accuracy with Three-day errors not
  significantly worse than the One-day error
• Decrease in Forecast Accuracy leads to an
  increase in sorties that could not perform BDA
  due to Wx

0
1
3
Time Shift (Days)
MIW METOC Metrics Ops Analysis, Feb 08,
pvodola_at_spa.com