Evolution of NPS and SPA METOC Metrics Work

1
(No Transcript)
2
(No Transcript)
3
Evolution of NPS and SPA METOC Metrics Work
Study Approach Selected Major Results
Cantu 2001 Analysis of NAVSAFECEN Class A aircraft mishaps Losses per year from weather-related mishaps 11 deaths 74M in damages
Martin 2002 Analysis of NAVSAFECEN Class A-B-C afloat mishaps Losses per year from METOC-related mishaps 2 deaths 900 lost/light duty days.
Hinz 2004 Analysis of forecasts, verifying observations, and mission planning and performance data from OIF CAOC Forecast POD 55. Negative ops impacts from mis-timing of late Mar 03 dust storm.
Anderson 2004 Analysis of model output, forecasts, and verifying obs for major OIF dust storm, late Mar 03 Mesoscale model errors led to errors in dust storm forecasts.
Jarry 2005 Analysis of performance and operational impacts of forecasts for Air Mobility Command (AMC) No go forecast accuracy 66. Weather mitigated missions saved by mitigation 67.
Butler 2005 Development of online near real time system for forecast performance and ops impacts metrics for NPMOD Fallon / NSAWC System in operational use since summer 2005. Missions saved or potentially saved by forecasts 36
Darnell 2006 Adaptation of Butler (2005) system for Air Combat Command (ACC) No go forecast accuracy 60. Missions saved or potentially saved by forecasts 37.
Callahan 2006 Adaptation of Butler (2005) system for carriers and amphibs Siprnet system created. Red POD 40 (preliminary).
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
4
Evolution of NPS and SPA METOC Metrics Work
Study Approach Selected Major Results
SPA 2001-2006 15 METOC operational impact studies for STK, ASW, MCM, EXW, NSW, and overall campaign Based on OPNAV planning scenarios Designed to investigate operational impacts of - uncertainty regarding environmental conditions - tactical utility of timely and accurate METOC information Uncertainty regarding METOC conditions has great impact on high-level mission success metrics. Mission planners require accurate information to predict performance and generate CONOPs. POM assessment community requires accurate information for CONOPs generation, force structure and allocation decisions, and requirements analysis.
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
5
(No Transcript)
6

• Definitions
• Metrics Objective, quantitative, data based
  measures of products and services. Examples
• Metrics of product quality
• Metrics of effects of products on customers
• METOC Metrics Metrics of METOC organizations
  products and impacts. Two main types
• Performance metrics metrics of capacity,
  readiness, quality, efficiency / return on
  Investment
• Impacts metrics impacts on warfare customer
  operations (planning, execution, post-assessment)
• Methods for Generating METOC Metrics
• ? Collect / analyze real world data on METOC and
  customer ops
• ? Model METOC and customer ops

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
7

• Goals
• 1. Develop and transition to operational use
  systems for
• a. collecting data from METOC units and
  their customers
• b. quantifying METOC performance and
  impacts on customer
• operations
• c. modeling and predicting impacts of METOC
  support on
• war fighting operations
• 2. Identify methods for improving quality and
  efficiency of METOC support.
• 3. Recommend
• a. focus directions for METOC resources
• b. methods to incorporate METOC into OPNAV
  assessments

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
8
Effects of Environmental Uncertainty on Mission
Planning and Execution
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
Accurate Predictions

• Opportunity Costs
• Actual is better than predicted
• Over-prepared for environmental risks
• Spend too many resources, too much time
• Resources better used for other locations/missions

• Effectiveness Costs
• Actual is worse than predicted
• Under-prepared for environmental risks
• Deploy too few resources, too little time
• Increased risk of mission failure

Need to know relationships between forecasts,
planning, and outcomes in order to quantify, and
eventually reduce, outcome uncertainty and
associated costs.
9

• Develop automated system to collect and
  analyze real world
• data.
• Develop and apply warfare mission model.
• Use real world data to modify and verify
  model.
• Use model to identify additional data to
  collect and analyses to
• conduct.

10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
Forecast Performance and Operational Impacts
Metrics OIF
From NPS thesis research of LCDR Jake Hinz, USN,
2004
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
14
(No Transcript)
15
Forecast Performance Metrics AMC
Overall performance very good. But high overall
FAC comes mainly from relatively common Go
events. Persistent Go forecasts would have given
almost the same overall FAC. FAC and skill for
relatively uncommon No Go events much lower. In
No Go cases of critical importance to operators,
forecast performance much lower than overall.
From NPS thesis research of Capt Jeff Jarry,
USAF, 2005
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
16
Operational Impacts Metrics AMC
Saved Mission a. No Go forecast for original
plan is accurate and b. Go forecast for
accepted mitigation plan is accurate and c.
Mission successful using mitigation plan
and d. Mission would have failed using
original plan.
From NPS thesis research of Capt Jeff Jarry,
USAF, 2005
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
Operational Impacts Metrics Responses to
Forecasts of Negative Impacts ACC PACAF

• Percent of missions for which
• Planning forecasts indicated negative weather
  impacts 36
• Planning forecasts led to mission plan changes
  21
• Execution forecasts indicated negative weather
  impacts 39
• Execution forecasts led to mission plan changes
  21
• Missions that experienced negative weather
  impacts 36

Missions that avoided, or could have avoided,
negative weather impacts based on forecasts
issued ? 36 for all our real world Air Force
and Navy data sets.
From NPS thesis research of Maj Karen Darnell,
USAF, 2006
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
31

• Online Real World Metrics System Users
• In operational use by Naval Pacific Meteorology
  and Oceanography Detachment at NAS Fallon in
  support of Naval Strike and Air Warfare Center
  (NSAWC).
• System in operational testing by Enterprise,
  Eisenhower, Stennis, and Iwo Jima.
• Web site and database components of system being
  transitioned from NPS to FNMOC. Beta version
  expected on FNMOC development server winter 2007.
• System adapted and tested for use by USAF Air
  Combat Command (ACC) and Pacific Air Forces
  (PACAF).
• Adaptations of system for other warfare areas
• System being adapted for use in NSW
  (collaboration with NOSWC)
• Discussions underway to adapt system for
  automation of ocean model product metrics
  (collaboration with NRL-SSC)
• Planning underway to adapt system for use in ASW
  (collaboration with ASW directorate)

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
32
See brief by CDR Steve Woll for more on uses of
metrics by METOC leadership.
33
See brief by CDR Steve Woll for more on uses of
metrics by METOC leadership. See brief by CDR
Mike Angove for more on uses of metrics in
assessment / budget processes.
34
Integration of Real World and Model Metrics
See briefs by Luke Piepkorn Matt McNamara for
more on our metrics modeling efforts.
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
35

• Future Extensions - Approach to Other Warfare
  Areas
• Develop metrics dashboard for all directorates
• STW work directly applicable to other
  mission-centric warfare areas
• NSW, ISR, EXW / AMW
• Outcomes measure performance and impacts (e.g.,
  missions saved), improve mitigation
  recommendations, improve CONOPS
• Applicable with modification to campaign style
  warfare areas
• ASW, MIW
• Model metrics with fewer real-world exercises
  and operations to analyze, real-time realistic
  modeling provides opportunity to explore options,
  what-ifs
• Well-positioned if/as JFMCC moves to Maritime
  Tasking Order concept
• In all warfare areas, need to
• Go from anecdotal, often post-mission assessments
  to objective, automated, real-time, assessments
  that can be incorporated into planning process at
  strategic, operational, and tactical levels
• Inject realistic METOC factors and forecasting
  into Naval modeling, simulation, and assessments

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
36
Future Extensions - ASW
Metrics as tools for evaluating different METOC
support strategies
From ASW Coordination/CONOPs Conf 14 Mar 05 CAPT
Jeff Best, CNMOC Director for ASW CDR Van
Gurley, CNMOC Deputy Director for ASW
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
37
Future Extensions - ASW
Metrics as tools for evaluating different METOC
support strategies
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
38
Future Extensions - ASW
Metrics as tools for evaluating different METOC
support strategies
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
39
Future Extensions - ASW

• Preliminary Proposal
• METOC metrics program for ASW should be designed
  to
• Develop ASW real world data collection, analysis,
  and reporting system
• Develop ability to model ASW operations and METOC
  support of them
• Use real world and model systems to test and
  experiment (e.g., with variations in objectives,
  weighting of objectives, means to meeting
  objectives, success measures, etc.)
• Report results rapidly, automatically, and in
  optimal formats to both METOC personnel and ASW
  customers
• Support facts-based decision making at multiple
  levels by both METOC personnel and ASW customers

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
Backup Slides
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
44
Dollars and Metrics

• Metrics are Quantitative Measures of Performance
• May relate performance to other areas
• Visibility forecast accuracy and CVN green deck
  for air ops
• Performance to Customer Dollars
• Iterative Process
• Do not need to be dollar related initially!

From NPS thesis research by LCDR Jake Hinz, 2004
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
From NPS thesis research of Capt Jeff Jarry,
USAF, 2005
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
WIAT Simulation of Adaptive Strike
MissionPlanning Process
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
57
WIAT METOC Features

• Includes forecasted and actual conditions over
  time from OIF weather briefing maps
• Spatial resolution 25 x 25 nm
• Temporal resolution varying, down to 2 hrs
• Represents forecasted and observed cloud density,
  cloud layers
• Includes inputs for other forecasted and observed
  conditions
• Simulates interactions of OA division and mission
  planners
• Allows forecasts of various accuracies to capture
  operational effects of forecast uncertainty

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
58
WIAT TACAIR Features

• Force Composition
• Size Carrier Air Wing
• Sorties modeled as TACAIR missions
• 2 AC per mission
• Type F-18
• Objectives
• Provide CAS / KI to ground forces in transit and
  during engagements
• Engage fixed strike targets targets of
  opportunity
• Behavior
• Fly missions according to the adaptive ATO
• Sample Metrics
• Mission modifications due to weather
• 24 hrs prior 1 hr prior
• Sortie missions aborted / sorties successfully
  flown
• Weapons correctly selected / dropped
• TACAIR mission success (targets destroyed / KI
  stations maintained / CAS successes)
• Opportunity costs (targets that were incorrectly
  avoided due to erroneous forecasts)

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
Future Extensions - Approach to Other Warfare
Areas

• Determine METOC inputs for warfare area
• Current support products
• Mission planning timelines
• Collect product and observational data
• Historic database information
• Current operational observations
• Develop applications which quantify
• Value of METOC support
• Impacts of improved environmental analysis and
  prediction
• Capitalize on this information to improve METOC
  support
• New or improved support products
• New methods of feature analysis

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
66
Future Extensions - ASW

• ASW Metrics Operations Analysis Task Areas
• Task 1 Develop realistic vignettes and
  scenarios for ASW.
• Consistent with both MCOs and exercises /
  operations
• High level metrics motivated by OPNAV
• Task 2 Identify the environmental factors that
  influence ASW platform and system performance.
• Identify impacts and prioritize
• Leverage existing vetted assessments for OPNAV
  and METOC communities
• Task 3 Describe and model the ASW mission
  planning process.
• Identify decision points and structure the ASW
  decision process in terms of where the decision
  is being made, the nature of the decision and
  options available,
• Identify the likely options selected as a result
  of alternative environmental intelligence, and
  the uncertainty associated with the METOC
  products provided.

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
67
Future Extensions - ASW

• ASW Metrics Operations Analysis Task Areas
• Task 4 Develop an engagement model for ASW.
• Realistic and credible representation of forces,
  tactics, and outcomes
• Output measures of effectiveness that are
  consistent with OPNAV N81/N84 high-level
  measures of ASW success and what-it-takes-to
  win criteria
• Representation of the environmental context of
  the mission with particular attention to the
  critical environmental features (collaboration
  w/NRL-SSC)
• Representation of the mission planning process
  and the ability to input and implement the
  metrics collected from operating forces by NPS
  for their portion of the collaboration

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
68
Future Extensions - ASW

• ASW Metrics Operations Analysis Task Areas
• Task 5 Perform sensitivity analysis on METOC
  product accuracy and timeliness.
• Coordinate with NPS and NRL-SSC to develop a set
  of cases and sensitivities to assess the tactical
  utility of METOC support services and to identify
  areas where accuracy and timeliness should be
  improved
• Perform sensitivities to scenario variables such
  as threat, U.S. forces, and rules of engagement
  as well as sensitivities to the accuracy of
  forecasts, timeliness of METOC support, season,
  and other environmental uncertainties. Modify
  models and databases as required as a result of
  these tests and sensitivities.
• Task 6 Provide reports, briefs, and analysis
  results
• Deliver modeling tools and datasets together with
  documentation and training as directed.

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
69
Future Extensions - NSW
OOD Effects on NSW Number of Missions Impacted
by Ocean
FID Missions Minimally Impacted
ISR Missions Strongly Impacted
Number of Missions
MIssion Type
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
70
Future Extensions ASW / MIW
OOD Effects on ASW / MIW Number of Contact
Hours Added by OOD
OOD shows higher than expected MDRs. Screen/sensor
spacing increased. Significant improvement in
Contact Hours.
OOD shows lower than expected MDRs. Screen/sensor
spacing reduced. Significant improvement in
Contact Hours.
MDRs
Contact Hours Added
OOD Historical Traditional Minimal
improvement in Contact Hours.
Day
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
71

• Future Extensions - CNMOC Use of Metrics
• METOC Metrics Dashboard During Realignment

ASW MIW NSW ISR FLT AVN MAR NAV PTA
Capacity
Readiness
Quality
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
72

• Future Extensions - CNMOC Use of Metrics
• METOC Metrics Dashboard After Realignment
  Completed

ASW MIW NSW ISR FLT AVN MAR NAV PTA
Capacity
Readiness
Quality
Impact
Efficiency
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
73
Future Extensions - ASW
ASW Example
Mission Planning Steps

• Determine length and position of barrier desired
  between threat ports and op areas
• Determine the barrier performance objective
  (e.g., 0.90 P(detect) against quiet transitor)
• Compute forces required based on environment,
  sensor performance, and threat/searcher speed

• METOC Issues
• What is the impact of accurate knowledge and
  modeling of bottom
• properties on mission planning?
• Is the age of the information important?
• Are more bottom surveys necessary?

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
74
(No Transcript)
75
Future Extensions - ASW
Exercise Support Results
How did the actual search performance change as
a result of the improved environmental
information and subsequent CONOPs change?
The total actual performance improvement of 9
hours comes from two sources

• Improvement due to a more favorable environment
  (4 hours)
• Improvement due to understanding the environment
  and changing the CONOPs (5 hours)

• Search performance does improve nearly as much
  as expected
• Median time to detect both threats improves from
  23 hours to 14 hours
• Probability of detecting both threats by the end
  of the exercise improves from 75 to 85

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
76

• Future Extensions - NSW
• Proposal Submitted to SPAWAR for FY07-FY09
  Project
• Project Performers
• NPS Tom Murphree, CDR Rebecca Stone
• SPA Paul Vodola, Luke Piepkorn
• NRL-SSC Rick Allard, Gregg Jacobs
• CSI Bruce Ford
• Project Objectives
• Develop METOC metrics data collection methods,
  databases, data analysis, modeling, and metrics
  computation systems for METOC support of Naval
  special warfare (NSW) operations.
• Apply systems to
• collect and analyze data
• simulate impacts of METOC products on warfighting
  operations,
• determine metrics that quantify the performance
  and operational impacts of METOC forecasts
• Transition to METOC community a metrics toolset
  composed of data collection, database, data
  analysis, and modeling systems for NSW.
• Develop recommendations for METOC leadership.

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
77

• CNMOC Roles
• Advocacy at CNMOC
• Systematic application of integrated real world
  and model based metrics methods to all
  directorates
• N6 / EED staffing
• Coordination with TD
• Assistance in establishing
• METOC metrics working group
• METOC metrics advisory board
• Assistance in establishing METOC metrics as an
  NPS thesis research priority
• METOC metrics topics (meteorological and
  oceanographic) on future thesis topics lists
• Guidance to NPS students (e.g., email)
• Support for further funding

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu