Approximating Sensor Footprints

1
Fall 2005 Simulation Interoperability
Workshop 05F-SIW-039 An Application and
Evaluation of a Methodology for Testing
Operational Level Training Simulations MAJ Karl
D. Neal Virginia Modeling, Analysis, and
Simulation Center Old Dominion University Wesley
N. Colley, Ph.D. and Mikel D. Petty, Ph.D.
Center for Modeling, Simulation, and
Analysis University of Alabama in Huntsville
2
Presentation Outline

• Introduction
• Operational-level training simulation
• The VVA problem
• Methodology
• The Missions/Means Paradigm
• Evaluation
• Application
• Terminal Fury 05
• Controlled Testing
• Conclusions

3
Operational-level Training Simulations

• Operational Level of War
• Operational-level training simulations
• Simulations used in constructive components of
  training exercises for operational level
  commanders and their staffs in conducting warfare
  at the operational level.

The operational level links the tactical
employment of forces to strategic objectives. The
focus at this level is on operational art the
use of military forces to achieve strategic goals
through the design, organization, integration,
and conduct of strategies, campaigns, major
operations, and battles (DoD Joint Pub 3.0,
2003).
4
Verification, Validation, and Accreditation

• Verification
• Determination whether simulation meets design
  goals
• Validation
• Determination whether simulation reproduces
  reality at desired fidelity
• Accreditation
• Official certification that a simulation may be
  used for a specific purpose

5
VVA Problem 1What does does it work mean?

• Stimulates correct training goals?
• Produces valid entity interactions?
• Is sufficiently usable for the training staff?
• Even the concept of correct is tricky.
• How good is good enough?
• Is aggregate behavior okay even if micro-behavior
  isnt?
• What level of error mitigation by training staff
  is acceptable?

6
Possible Testing Boundaries

• Between trainees and entire system
• Between operator-controllers and federation

Observers
Observers
Trainers

Trainers

Interface A

Means 2
Interface D




Trainees
Role Players
Simulation

Trainees
Role Players
Simulation

Interface B
Interface F
Operator


Operator


Interface
Interface


Controllers
Controllers






CCWS
-
like
CCWS
-
like




operator
operator
Response Cell
Response Cell
Interface I


interfaces allow
interfaces al


simulation
simulation




command/
msg
command/
msg
Interface E


Technical

Technical

I/O not
I/O not


Interface C
Support

necessarily
Support

necessarily


operational
operational


format
format
Interface J
Interface H


C4ISR

C4ISR

Computer
Computer
Interface G
Systems

Systems

Simulation

Simulation



Real C4ISR

Real C4ISR
1

2


system
system


connected to
connected to


simulation
simulation
Simulation
7
The Controller/Federation Boundary

• Advantage Well-defined border
• Machines on one side people on other
• Best place to evaluate simulation alone, rather
  than blend of simulation staff
• Only logical place to evaluate simulation
  usability

8
VVA Problem 2

• Vast range of possible scenarios to evaluate
• Environment
• Urban, Mountain, Jungle, Desert
• Geography, Topography, Climate, Day/Night
• Force (Air, Ground, Sea, Special)
• Equipment (sub, fighter, tank)
• Action (support, attack)
• Posture (R.O.E., etc.)

9
Presentation Outline

• Introduction
• Operational-level training simulation
• The VVA problem
• Methodology
• The Missions/Means Paradigm
• Evaluation
• Application
• Terminal Fury 05
• Controlled Testing
• Conclusions

10
The Use Case Strategy

• Develop a suite of strategically chosen cases
• Each case tests the simulation in a range of
  tasks
• Problem
• Number of cases difficult to hold down
• Example Air support with F-16 is a different
  case from air support with F-18, even with all
  other factors identical
• Need strategy to organize use cases.

11
Structuring the Myriad

• Group related cases hierarchically
• Small changes dont require completely new cases
• Source of Use Cases
• Universal Joint Training List (UJTL)
• Provides a hierarchically organized list of
  operational level joint military tasks (missions)
• These are the tasks the training audience is
  intended to learn from using the simulation.

Missions Decomposition
12
UJTL-Based Missions Decomposition
Task 2.3
Task 2.2
Task 2.3
Subtask 2.3.2
13
Can the Simulation Support the Task?

• Simulations are not organized by task or case
• Instead, simulations have capabilities or means
• What can the simulation play?
• Forces air, naval, ground
• Equipment tanks, subs, ships, planes, guns
• Action movement, engagement, support
• Hierarchical list of capabilities is the

Means Decomposition
14
Means Decomposition

• Much more difficult than missions
• Subject Matter Experts (SME) required
• Consider necessary means for each task
• Need intimate knowledge of military operations
  related to the task
• Survey requirements documents of simulation
• Ideally, an expert on the simulation, also
• Recognize and develop hierarchy
• Global Hawk ? Air Intelligence ? Air Forces

This is the most difficult and costly part of the
method
15
Missions-Means Schematic
16
Missions-Means Example
17
Evidence Items

• Each low-level subtask, and submean has
• Input and/or Output
• Related missions and means interact through
  these inputs and outputs
• A task output from the trainees or staff becomes
  a mean input to the simulation
• A simulation output becomes an input to the staff
  or trainees
• These matrix elements are Evidence Items
• Low-level interactions to be evaluated

18
Evidence Item Purpose

• Provide specific cases for evaluation
• Broad Goals of Evaluation
• Does Simulation provide input to stimulate
  trainees performance of UJTL tasks?
• Does simulation provide capabilities to execute
  decisions made during performance of UJTL tasks?
• Method generates a very large set of evidence
  items
• Specific to operational level training, not
  specific to simulation

19
Presentation Outline

• Introduction
• Operational-level training simulation
• The VVA problem
• Methodology
• The Missions/Means Paradigm
• Evaluation
• Application
• Terminal Fury 05
• Controlled Testing
• Conclusions

20
Evaluating Evidence Items

• To evaluate evidence items, we created a set of
  Test Indicators in four categories

• Capabilities/Execution
• Existence Degree to which E.I. is supported by
  the simulation (as is not role-played)
• Correctness Degree to which simulation is
  accurate and realistic
• Completeness Fraction of critical elements
  simulated.
• Continuity Degree to which simulation produces
  logically and temporally consistent results.

• Interface Usability
• Satisfaction How pleasant is the interface to
  use?
• Currency Timeliness of simulation in providing
  needed information to operators.
• Modality Appropriateness of push/pull type
  used by simulation.

21
Test Indicators, cont.

• Operator Errors
• Severe of severe errors that
• Prevent a training goal
• Cause a major pause in the simulation
• Moderate of errors that affect trainees,
  multiple operators, or multiple staff members but
  are not severe.
• Negligible of errors whose effects are
  confined to a single operator and have little
  effect on training or the simulation as a whole.

• Operator Workload
• Currency Timeliness of simulation in providing
  needed information to operators.
• Modality Appropriateness of push/pull type
  used by simulation.

22
Test Indicator Evaluation

• Evaluations made on worksheets for each evidence
  item
• Most Ratings 05

• 0. Item not supported by simulation
• Minimal support/very difficult for operators
• Incomplete support/cumbersome usability
• Acceptable support/usability
• Good support/usability
• Complete support/ Easy/Excellent

23
Presentation Outline

• Introduction
• Operational-level training simulation
• The VVA problem
• Methodology
• The Missions/Means Paradigm
• Evaluation
• Application
• Terminal Fury 05
• Controlled Testing
• Conclusions

24
Terminal Fury 05

• Operated by USJFCOM
• HQd at Pacific Command (PACOM)
• Main hub of exercise (70 personnel)
• BLU Naval forces 6 JTLS operators
• RED forces 7 JTLS operators
• 24-hour testing for 10 days
• Hurlburt Field
• BLU air 8 JTLS operators
• 8-hour testing for 3 days

25
Joint Theater Level Simulation (JTLS)

• Main Driver of TF 05.

26
Generating Evidence Items with Method
Effective

• Broad outline of TF 05 training goals allowed
  focus of effort on tasks in a subset of UJTL and
  corresponding means
• Evidence items incumbent to exercise were
  identified and indexed
• Evaluation sheets were prepared automatically
  within the database
• Missions/Means Decomposition effective for
  producing relevant evidence items for evaluation

27
Applying Method at Live Exercise
Ineffective

• Early in the exercise, a handful of evidence
  items were evaluated
• Operators generally not busy yet
• After exercise ramped up, evidence items could
  not be evaluated
• Operators were too busy to rate items like
  satisfaction for the many evidence items that
  corresponded to their ongoing minute-to-minute
  work
• More to come on TF 05

28
Applying Method in Controlled Testing
Effective

• Sets of evidence items were carefully selected
  to correspond to single vignette/sub-scenario
• Single operator on a JTLS terminal
• Two 4-hour sessions
• Testers guided operator through vignette and
  evaluated each evidence item along the way
• Very successful the method is very good in
  controlled scenarios with no time constraints

29
Lessons Learned

• Even in controlled testing, prepared vignettes
  were necessary for effective testing
• Starting to sound again like use cases
• Future work likely to involve blending the
  Missions-Means paradigm with the use-case
  paradigm
• MM will form an index of available means and
  pertinent missions
• Use cases will be generated and mapped onto that
  index

30
TF 05 Observations

• Ancillary observations of note
• Often, federates behaved nominally w.r.t.
  requirements, but generated subtly inaccurate
  behaviors
• Seemingly small issues caused major headaches
• Frequent hand-jams required to maintain
  training stimuli
• Many operators favored other simulations and
  became very frustrated with the JTLS interface
  and behaviors
• Often the issue was operators desiring more
  tactical control than is provided in JTLS.
• Such observations would not likely occur under
  controlled testing Live exercise observation
  still important!

31
Presentation Outline

• Introduction
• Operational-level training simulation
• The VVA problem
• Methodology
• The Missions/Means Paradigm
• Evaluation
• Application
• Terminal Fury 05
• Controlled Testing
• Conclusions

32
Conclusions

• VVA is complicated by the enormous number of
  possible scenarios, forces, equipment and
  interactions
• The Missions/Means paradigm is a two-dimensional
  hierarchical model that brings structure and
  organization to the problem
• Method effective for generating evidence items
  for testing simulation
• Our M/M based evaluation proved ineffective in a
  live exercise due to intense activity, but did
  provide ancillary insights
• Evaluation during controlled testing was very
  effective when vignettes were prepared carefully
• A method with use cases indexed to the
  Missions/Means system is recommended

33
End of presentation