METOC Metrics

1
METOC Metrics Operational Modeling
Serial XXXXX

• Prepared for NSW Metrics Meetings
• 22-23 March 2007

2
NSW Metrics
or other products
3
NSW Operational ModelingNotional Tool

• Notional NSW Weather Impact Assessment Tool
• Simulates modern Naval Special Warfare operations
• Inputs both actual environmental conditions and
  forecasts of conditions with varying lead times
  and accuracies
• Emulates actual mission planning processes days
  and hours ahead of mission execution
• Models customer decisions and outcomes based on
  weather predictions and other factors
• Value
• Simulates impacts of METOC forecasts and
  phenomena on spectrum of NSW situations
• Facilitates determination of phenomena which have
  significant impact on NSW operations
• Evaluates NSW effectiveness employing a variety
  of METOC support situations

4
NSW Operational ModelingApproach

1. Develop realistic vignettes and scenarios for
   NSW, including mission-level metrics of success
2. Identify the environmental factors that most
   significantly influence NSW platform performance
   and operational effectiveness
3. Describe and model the NSW mission planning
   process, to include how and when METOC
   information is incorporated
4. Develop engagement model(s) for NSW
5. Perform sensitivity analysis on METOC product
   accuracy and timeliness

5
NSW Operational Modeling Application
Useful as a Laboratory for Experimentation

• Allows for assessment of the effects of METOC
  products and services across a wide range of NSW
  scenarios
• Effect of Increased Accuracy
• Effect of Enhanced Timeliness

Scenario
SBT Operations Direct Action Mission Special Recon Mission
A
B
C
D
Multiple combinations of METOC products and
services support NSW
METOC Product
6
NSW Operational Modeling Application
Can be used to develop METOC support benchmarks
to establish goals and evaluate real world
performance
Resulting METOC Metrics
Scenario
SBT Operations Direct Action Mission Special Recon Mission
A
B
C
D
No NSW payoff from additional accuracy /
timeliness
/- 1
NSW Performance is Improved
/- 3
METOC Product
UNSAT - No Impact above Historical
/- 5
Never
H-12
H-24
H-2
Optimal combinations of accuracy and timeliness
lie on the blue line
7
NSW MetricsOverview

• MS Role in 2 of 3 major steps of metrics
  development program
• What metrics to collect
• How to collect them
• What standards/benchmarks to measure against
• Performance benchmarks
• NSW lends itself well to the establishment of
  go/no-go criteria, similar to STW
• Working defn. of SAT Any increased accuracy
  and/or timeliness would not improve the
  scenario-level metrics
• Working defn. of UNSAT The accuracy and
  timeliness provided provides no meaningful
  benefit above the use of historical databases /
  existing uncertainty
• NSW mission planning process / modification
  timeline development

8
SPA Team Contact Info

• Paul Vodola
• pvodola_at_spa.com
• paul.vodola_contractor_at_spa.dtra.smil.mil
• 703-399-7255
• Luke Piepkorn
• lpiepkorn_at_spa.com
• 703-399-7239
• Tom Pentony
• tpentony_at_spa.com
• 703-399-7268

2001 North Beauregard Street Alexandria, Virginia
22311-1739 Fax [phone] www.spa.com
9
Environmental Impact on NSW
Multiple Combinations of Sensors, Weapons, and
Environmental Features
SAMPLE FOR ILLUSTRATION ONLY
Wx Phenomena-NSW System matrix allows for
identification of most significant METOC phenomena
10
Costs of Inaccurate Information
Mission Outcome Uncertainty (e.g. detections,
kills)

• Opportunity Costs
• Actual is better than predicted
• Deploy more assets than necessary
• Resources better used for other locations/missions

Accurate Predictions (Optimal of Searchers are
Provided)

• Effectiveness Costs
• Actual is poorer than predicted
• Deploy too few assets
• At risk for losses

Actual
Predicted
11
Forecasting Mission Outcomes
Symmetry exists between forecasting and
operational outcomes
12
Forecasting Mission Metrics
Accuracy of Recommended Mission Changes D / B
Accuracy of Forecasting METOC Events D / B
13
Operational Modeling METOC Metrics
Improving METOC support products allows for
improvement in both system and operational
performance
14
Impacts of Environmental Uncertainty on
Operational Metrics
Mission Planning Uncertainty
Forecast Uncertainty
Mission Performance
Forecast Performance
Predicted (better outcome)
Actual (worse outcome)
Actual (worse environment)
Predicted (better environment)
Accurate Predictions
Mission Outcome Uncertainty

• Opportunity Costs
• Actual is better than predicted
• Over-prepared for environmental risks
• Spend too many resources, too much time
• Resources better used for other locations/missions

• Effectiveness Costs
• Actual is worse than predicted
• Under-prepared for environmental risks
• Deploy too few resources, too little time
• Increased risk of mission failure