... Design and Analysis Issues for Field Settings Ran

1
Draft (not releasable for general
distribution, for discussion only)
The Logic of Warfighting Experimentation
Presented to Information Age Metrics Working
Group (IAMWG) 16 July 2004
Rick Kass Chief, Analysis Division Joint
Experimentation Directorate, J9 US Joint Forces
Command
2
Take-Aways
Experimentation is uniquely suited to Capability
Development Develop Capabilities to cause
increased effectiveness and design experiments
to assess causality Logic of Experimentation is
not difficult 2, 3, 4, 5, 21 Can apply
principles of science and achieve robust
defensible results in Experiments Able to
empirically justify the value of new capability
recommendations Can maximize information from
individual Experiments and accumulate rigor in
Experiment Campaign using multiple experiment
venues and continuous simulation in
model-exercise-model paradigm
3
Scientific Method And Experimentation
4
Taxonomy of Sources of Knowledge
Knowledge
5
Impact of Experimentation on Science and
Technology

• Experimentation only 400 years old
• 400 BC to 1600 AD
• Science Observations and Theory
  (Aristotelian)
• 1600 Galileo to Present
• Science Observations and experiments to test
  Theory
• Example
• Aristotle reasoned that heavy objects fall
  faster than lighter objects
• Galileo dropped different weights and observe
  the results
• Results contradicted Aristotles 2000 yr-old
  belief

Galileo Galilei
New Method Experiment Do something under
specified conditions and observe what
happens. let the data decide
In 400 years experimentation has
revolutionized knowledge and technology. Can it
have the same impact on Military Transformation?
6
Scientific Method and the Joint Concept
Development and Experimentation Process
Scientific Method
Publish Paper in Scientific Journal
Evaluation Phase
8. Ascertain Impact on Problem
If Inconclusive
7. Evaluate the Hypothesis
Experiment Phase
6. Analyze Data
5. Conduct Experiment
4 Design Empirical Test of Hypothesis
3. Formulate Hypotheses
Clarification Phase Clarify Problem Possible
Solutions
2. Review of Literature
1. Identification of a Problem
7
Why Experiment?
Transformation is about-- changing something
to increase Effectiveness effectiveness/efficien
cy
8
What is an Experiment?
9
Simplest Experiment (If A, Then B)
B
A
10
Useful Definition of Experiment
35 different definitions at WWW. One-Look
Dictionary Search Common Themes A
test done in order to learn something or to
discover whether something works or is true
(Cambridge Advanced Learning Dictionary). An
operation carried out under controlled conditions
in order to discover an unknown effect or law, to
test or establish a hypothesis, or to illustrate
a known law (Merriam-Webster Dictionary)
Experiment To explore the
effects of manipulating a variable.
Shadish, Cook, Campbell,. Experimental and
Quasi-Experimental Designs for Generalized Causal
Inference p. 507)
Warfighting Experiment To examine the effects
of varying proposed warfighting capabilities or
conditions. Joint Warfighting Experiment To
examine the effects of varying proposed joint
warfighting capabilities or conditions.
11
What Will an Experiment Do for You?
A proposed solution B operational problem to
be overcome C another possible solution

• Does A affect B?
• Is A important for solving B?
• How much A is necessary to solve B?
• How much of B is alleviated by A?
• What is the best way to do A to solve B?
• Is C also necessary for A to work?
• Is A more important than C to solve B?

Not either-or, need both experiments and
experience
Analysis of historical data and the use of
military experts is critical to understanding the
real problem and proposing potential solutions.

• But Experts History
• Sometimes produces contradictory implications
• May not include future environment
  characteristics
• Can not quantify potential effects of new
  solutions on historic problem
• Can not resolve cause and effect
  retrospectively

12
Experiment Rigor References
William R. Shadish, Thomas D. Cook and Donald T.
Campbell,. Experimental and Quasi-Experimental
Designs for Generalized Causal Inference
(Houghton Mifflin Co 2002) Thomas D. Cook and
Donald T. Campbell. Quasi-Experimentation
Design and Analysis Issues for Field Settings
Rand McNally, 1979) Donald T. Campbell and
Julian Stanley. Experimental and
Quasi-Experimental Designs for Research (Rand
McNally, 1963)
Experiment rigor requirements based on 40 years
of writing about non-laboratory experiment
requirements. Adapted ideas and terminology for
joint warfighting experiments Apply traditional
scientific principles to Joint Experimentation
in innovative ways
13
The Logic of Warfighting Experiments
2 3 4 5 21
14
Experiment Hypotheses educated guesses of what
might happen

• Useful
• Help to clarify what experiment is about
• Identify logical thread of the experiment
• Guide experiment design and data collection
• Nothing magic
• If ________________ then ________________.

proposed solution(s) independent
variable potential cause
problem to be overcome dependent
variable possible effect
Sea Basing Collaboration Global Cell Robust ISR
Rapid deployment Adaptive planning Inter-theater
coordination Deny sanctuaries
15
Experiment Hypothesis avoidance?

• False Concerns about use of Hypotheses
• Too generalone over the worldnot helpful.
• Not justified, derived from theory no
  war-fighting theory of war fighting.
• Too constrictivedetrimental to discovery.
• Not appropriate for messy military field
  experimentsonly useful in laboratory
  experiments.
• Dont have enough information to formulate
  hypothesis
• Demand too rigorous data and analysis to
  reject/accept

You know enough to construct hypothesis!!
All you needis some idea of problems you are
trying to overcome or missions or tasks
attempting to conduct and the tools or
capabilities you are proposing to attempt to
solve the problem or execute the task.
16
Different Levels of Hypotheses
Capability Level (overarching)
If Robust ISR is employed then the threat
will have no sanctuaries...
Experimental Level (measurable-MOE/MOP)
If the Advanced XX System is employed then
threat will be continuously tracked.
MOE/MOP
Ho T ? YY Ha T ? YY
Statistical Level
17
Logic of hypothesis resolution

• Logic of hypothesis resolution
• Did A occur?
• Did B occur?
• Was B due to A ?

Internal Validity of an experiment
18
Four Requirements for Good (valid) Experiment
Evidence for Validity
Threat to Validity
Requirement
19
Five COMPONENTS of any EXPERIMENT
TREATMENT (A)
20
21 Threats to a Good Warfighting Experiment
4 Requirements 5 Components
Ability to Use Capability
Ability to Detect Results
Ability to Relate Results to Operations
Ability to Isolate Reason for Results
Single Group Multiple Groups
18. Nonrepresentative capability Is the
experimental surrogate functionally
representative?
11. Capability changes over time Are there
system (hardware or software) or process changes
during the test?
1
5. Capability variability Is systems (hardware
and software) and use in like trials the same?
1.Capability not workable Does the hardware and
software work?
NA
Treatment
2
15. Player differences Are there differences
between groups unrelated to the treatment?
19. Nonrepresentative players Is the player
unit similar to the intended operational unit?
6. Player variability Do individual
operators/units in like trials have similar
characteristics?
12. Player changes over time Will the player
unit change over time?
2. Player non-use Do the players have the
training and TTP to use the capability?
Players
3
20. Nonrepresentative measures Do the
performance measures reflect the desired
operational outcome?
7. Data collection variability Is there a large
error variability in the data collection process?

13. Data collection changes over time Are there
changes in instrumentation or manual data
collection during the experiment?
16. Data collection differences Are there
potential data collection differences between
treatment groups?
3. No potential effect in output Is the output
sensitive to capability use?
Effect
4
4. Capability not exercise Does the scenario and
Master Scenario Event List (MSEL) call for
capability use?
8. Trial conditions variability Are there
uncontrolled changes in trial conditions for like
trials?
21. Nonrepresentative scenario Are the Blue,
Green, and Red conditions realistic?
14. Trial condition changes over time Are there
changes in the trial conditions (such as weather,
light, start conditions, and threat) during the
experiment?
17. Trial condition differences Are the trial
conditions similar for each treatment group?
Trial
9. Violation of statistical assumptions Are the
correct analysis techniques used and error rate
avoided? 10. Low statistical power Is the
analysis efficient sample sufficient?
5

• The purpose of an experiment is to verify that
  A causes B.
• A valid experiment allows the conclusion A
  causes B to be based on
• evidence and sound reasoning
• - by reducing or eliminating the 21 known
  threats to validity.

Analysis
NA
21
Implication of Experiment Logic 2, 3, 4, 5, 21
How does the LOGIC help our understanding of
Warfighting Experimentation?

• Provides logical framework for
• Organizes good practices for planning and
  executing individual experiments
• 2. Provides rationale for developing
  experimentation campaigns

22
Four Requirements To Designing Warfighting
Experiments
Internal Validity 1. Capability Used 2. Detection
of Change in Effect 3. Isolation of Reason for
Change External Validity 4. Relating Results to
Military Operations
23
Ensuring the experimental Capability is employed?

• Most consistent lesson learned reported after
  warfighting experiments completed
• The system did not work as well as promised.
• The players did not know how to use it properly.
• The scenario outcome would not have shown a
  difference if it was used.
• The scenario play did not give the players the
  opportunity to use

Ensuring that the experimental capabilities are
used and can make a difference is the first
logical step in designing a valid experiment.
24
Threats to the Ability to Use the Capability
THREAT
PREVENTION
Treatment

• Capability not workable
• Does the HS/SW work?

• Ensure functionality of experimental capability
  is
• present.
• Ensure player organized, equipped, and trained
  for capability use.
• Provide sufficient doctrine and SOPs for
  capability use
• Provide sufficient pre-experiment "practice
  time."
• Pilot-test impact on experiment outcome
• Verify model input-output logic
• Pilot-test scenario and MSEL
• White cell specific scenario injects and
  monitor for use

Unit

• 2. Player non-use
• Do the players have the training and TTP to use
  the capability?

Effect

• 3. No potential effect in output
• Is the output sensitive to capability use?

Trial

• 4. Capability not exercise
• Does the scenario and MESL call for capability
  use?

25
Four Requirements To Designing Warfighting
Experiments
Internal Validity 1. Capability Used 2. Detection
of Change in Effect 3. Isolation of Reason for
Change External Validity 4. Relating Results to
Military Operations
26
Detecting Change in the Effect

• Given that A was employed
• Next Question Did B (effect) change when A was
  applied ?

Ability to detect change in B Statistically
Valid Experiment Detect Change Detect
COVARIATION B changes when A applied
27
Ability to Detect Change-- statistical validity
Threats
Treatment
Unit

Effect
Trial

• 9. Low statistical power
• Small sample
• Low alpha risk (5)
• Inefficient statistical test

Analysis

• 10. Violate assumptions of statistical test
• Statistical techniques have sensitive
  assumptions
• Error rate problem (fishing)
• Large number of statistical tests
• Large alpha risk (eg 20)

28
Four Requirements To Designing Warfighting
Experiments
Internal Validity 1. Capability Used 2. Detection
of Change in Effect 3. Isolation of Reason for
Change External Validity 4. Relating Results to
Military Operations
29
Isolating the Reason for Change

• Given that A was employed
• Given that B changed as A was applied
• Next Question What really produced the change
  in B?

Design Validity -- A alone caused change in B

• Threat -- Something other than A caused change
  in B
• confounded results
• -- Threat depends on type of experimental design

30
Isolating the Reason for Change
SINGLE GROUP DESIGN
31
Isolating the Reason for Change
SINGLE-GROUP DESIGN ORDER EFFECTS
PREVENTION
THREAT
11. System changes over time System or
process improves or degrades over time
Treatment

12. Player unit changes over time
Performance improves during later trials
due to experience rather than treatment
presentation
Unit




13. Data collection changes over time
Data collector or instrumentation
improve or degrade over time ---
artificially changing results
Effect
14. Trial condition changes over time
Weather, OPFOR, and simulations improve or
degrade over time
Trial

• General prevention/check
• Counterbalance presentation sequence
• Check for increase/decrease over time

32
Isolating the Reason for Change
MULTIPLE GROUP DESIGNS

• Different player units receive
• different treatments

33
Isolating the Reason for Change
MULTIPLE-GROUP DESIGN UNINTENDED DIFFERENCES
THREAT
PREVENTION
Unit
Effect
Trial
34
Four Requirements To Designing Warfighting
Experiments
Internal Validity 1. Capability Used 2. Detection
of Change in Effect 3. Isolation of Reason for
Change External Validity 4. Relating Results to
Military Operations
35
Ability to Relate Results to Actual Operations
DEFINITION

• Given that A was employed
• Given that B changed as A was applied
• and A alone probably caused change in B
• Next Question Are these findings related to
  actual operations?

36
Ability to Relate Results to Actual Operations
Experiment Operational Realism Validation
similar to MS validation
Validation of MS ...determining the degree to
which a model is an accurate representation of
the real world (DOD VVA Recommended Practice
Guide, 1996) Techniques Face Validation-
experts provide subjective assessments Predict
ive Validation-comparisons to actual system
performance e.g. model-test-model
37
Threats to Results Experiment to Actual Operations
THREAT
PREVENTION

• 18. Nonrepresentative capability
• Not functionally representative
• 19. Nonrepresentative unit
• Level of training --undertrained
• or overtrained (golden crew)
• Nonrepresentative players
• 20. Nonrepresentative measure
• Use of approximate measures
• Time versus in time
• Inadequate data source for measure
• Single data collector
• Qualitative measures only
• 21. Nonrepresentative scenario
• Blue operations inappropriate
• Threat unrealistic
• Unrealistic setting

• Ensure functionality of experimental surrogate
  capability is present.
• Use actual end users.
• Provide sufficient pre-experiment "practice
  time."
• Use "typically trained" units
• Use simulation to address complex measure based
  on component measure input (model-test-model).
• Use multiple data collectors.
• Show correlation to related quantitative
  measures
• Provide combat developer accreditation
• Provide adaptive independent accredited threat
• Provide appropriate political and military
  background
• Adaptive free play threat enhances scenario
  setting and uncertainty

Treatment
Unit
Effect


Trial
38
Implication of Experiment Logic 2, 3, 4, 5, 21
How does the LOGIC help our understanding of
Warfighting Experimentation?

• Provides logical framework for
• Organizes good practices for planning and
  executing individual experiments
• 2. Provides rationale for developing
  experimentation campaigns

39
Top-down Logic Organizes Experiment Good
Practices
Transformation Managing change requires cause
and effect Science and Experimentation
Empirically assessing cause and
effect Experiment Definition To explore the
effects of manipulating a variable (cause)
2. Components to a Hypothesis If cause, then
effect
3. Resolving Hypothesis resolve cause and
effect
5. Experiment Components to execute and analyze
experiment
4. Experiment Validity Requirements to resolve
hypothesis and generalize
21. Threats to Experiment Validity to execute
and analyze experiment
40
LOGIC Framework Provides Training Goals to
Decrease Validity Threats

• Note
• decreases
• threat
• increases
• threat

Isolate Reason
Detect Change
Relate Results
Use Capability
Multiple Group
Training Goals
Single Group

• player unit changes over time

Ensure all users are at highest skill level

• non representative unit

• player non-use of capability

Player Operators Experimental Unit
Ensure users in different groups have equivalent
skill level

• player unit differences

Ensure all users in same group are at similar
skill level

• player unit variability

• non representative unit

Ensure comparable ratings among data collectors
for similar event

• non representative measure

• data collection differences

• data collection variability

Ensure consistent ratings of same event at
different times

• non representative measure

Data collectors Subject Matter Experts

• data collection differences

• data collection changes

• data collection variability

• no potential effect in output

Ensure dissimilar events receive dissimilar
ratings

• non representative measure

41
Implication of Experiment Logic 2, 3, 4, 5, 21
How does the LOGIC help our understanding of
Warfighting Experimentation?

• Provides logical framework for increasing
  experiment rigor in
• Individual Experiments Organizes good
  practices for increasing rigor in individual
  experiments
• Experiment Campaigns Provides good ideas for
  increasing experiment rigor through
  experimentation campaigns
• Use multiple experiment methods
• Different experiment methods have different
  strengths and weaknesses
• Use different experiment methods at different
  stages of concept and prototype development
• Experiment requirements are different at
  different stages
• Use multiple methods in single experiment
• Combining constructive experiments and wargame
  experiments into a M-W-M paradigm increases rigor
  of both
• Use constructive simulation experiments to
  analytically integrate good ideas across
  dissimilar events
• Provides a common crucible for comparing apples
  and oranges

42
Implication of Experiment Logic 2, 3, 4, 5, 21
PROBLEM No single experiment can meet all 4
requirements.
Accumulate Experiment Rigor through an Experiment
Campaign (integrating multiple experiment
methods during development)
43
Understanding 4 Experiment Requirement
provides insights into Experiment Design TRADEOFFS
All Experiments are tradeoffs -can not
eliminate all threats to validity The 100 valid
Experiment does not exist
44
Rigorous Experiment Campaign RequiresMultiple
Methods to Meet the Four Requirements
Requirements for a Good Experiment
Employ Capability Detect Change in Effect
Isolate Reason for Effect Relate
Results to Operations
45
Four Experiment Requirements
Emphasizing Exp Requirements During Concept and
Prototype Development
Isolate Reason for Change
Relate Results to Operations
Use Capability
Detect Change
Capability Implementation in Joint Force

• Prototype Validation Experiments
• Demonstrate applicability to Combatant
  Commanders mission
• Examine predicted effectiveness in joint
  operational force
• Embed experiments within JTF exercises or
  training events

• Prototype Refinement Experiments
• Investigate incorporation of latest HW SW
  improvements
• Examine interoperability with existing fielded
  systems,
• and develop detailed tactics, techniques, and
  procedures (TTP)

• Concept Assessment Experiments
• Examine robustness across different scenarios and
  threat conditions
• Compare to other alternatives or baselines to
  quantify gains in effectiveness

• Concept Refinement Experiments
• Investigate optimal integration of piece-parts
  into most effective comprehensive solution
• Examine tradeoffs and synergistic effects between
  alternative combinations

• Concept Discovery Events
• Describe future operational problem and propose
  solutions in coherent framework
• Operational lessons learned, military history,
  industry and academia workshops, conferences,
  wargames

46
Methods for Different Concept and Prototype Phases
Prototype
Concept
Methods
Discovery Assessment
Refinement
Refinement
Validation
External Input Opn Lessons Learned RCCs Services I
ndustry Internal Events Workshops Conferences Sem
inars Seminar Wargames Analysis Analytic
models Experiments Constuctive Wargame HITL Field
Exercises JNTC JCS
X X X X X X X X x x x
x x x x x x x x x X X X x
x x x x x x x x x X x X X
x x x x x x x x x x X X x
X X x x x x x x x X X X X
47
Experiment Campaign employing M-E-Mcan Optimize
Design of Future Wargames(and increase
applicability of Wargame results)
Experiment Model-Exercise-Model (M-E-M)
Exercise
Event-Simulation (constructive or HITL)
Model
Model

• Real staff/operators
• Reactive threat
• One trial
• No Alternative Comparison
• No Baseline

Pre-Event Constructive Simulation
Post-Event Constructive Simulation
48
Integrating Results (Good Ideas) Across Eventsin
an Experiment Campaign
Sources of Good Ideas
(Mil Expr, History) Current
Event X Seminars Industry Operations
Joint Experimental Focus Areas Achieving Info
Superiority Joint ISR Joint Maneuver and
Strike Forcible Entry Operations Force
Protection? Deployment, Employment
Sustainability
x
x
x
x
x
x
x
x
x
x
x
x
x

• Simulation hot Test Bed
• Incremental Baseline
• Quick turn-around results
• Common Scenarios/Model
• Quantify Degree of Impact for Cost Benefit
  Analysis

49

• Integrated Experiment Campaign
• maximizes usefulness of experiments
• Continuous community integration
• Optimizes experiment results
• maximizes credibility of experiment
• Optimizes design of experiments
• Multiple experiment venues

Co-sponsored Seminar Wargames
Test Evaluation
Current Operations
Multinational Events
Industry
Continuous Discovery (continually refining
problem and proposing new solutions)
Joint Community Continuous Integration
Integration Workshops
Integration Workshops
Continuous Experimentation (Joint and Service)
Constructive Simulation Experiments (quantify
good ideas)
Analytical Wargames Experiment
HITL Experiments
Field Experiments
50
Implication of Experiment Logic 2, 3, 4, 5, 21
How does the LOGIC help our understanding of
Warfighting Experimentation?
Accumulate Experiment Rigor through an Experiment
Campaign

• Provides logical framework for increasing
  experiment rigor in
• Individual Experiments Organizes good
  practices for increasing rigor in individual
  experiments
• Experiment Campaigns Provides good ideas for
  increasing experiment rigor through
  experimentation campaigns
• Use multiple experiment methods
• Different experiment methods have different
  strengths and weaknesses
• Use different experiment methods at different
  stages of concept and prototype development
• Experiment requirements are different at
  different stages
• Use multiple methods in single experiment
• Combining constructive experiments and wargame
  experiments into a M-W-M paradigm increases rigor
  of both
• Use constructive simulation experiments to
  analytically integrate good ideas across
  dissimilar events
• Provides a common crucible for comparing apples
  and oranges

51
Experiment Logic 2, 3, 4, 5, 21 to support
Capability Development
Executing 5 Components
52
Take-Aways
Experimentation is uniquely suited to Capability
Development Develop Capabilities to cause
increased effectiveness and design experiments
to assess causality Logic of Experimentation is
not difficult 2, 3, 4, 5, 21 Can apply
principles of science and achieve robust
defensible results in Experiments Able to
empirically justify the value of new capability
recommendations Can maximize information from
individual Experiments and accumulate rigor in
Experiment Campaign using multiple experiment
venues and continuous simulation in
model-exercise-model paradigm
53
Discussion
54
Sorting Through Terminology
A New Sensor B detections
Goal Stimulating Event
Event
Purpose of Event
Operation to assist entity in acquiring ability
to do A. Operation to show/explain how A
works. Operation to confirm presence or quality
of A. Operation to confirm/quantify a proposed
relationship between B and something else (A).

• Practice on A to get B.
• Show how A works to produce B.
• Determine if A works (produces B).
• How effective is A?
• Can entity do A?
• Determine how B is produced.
• Is A related to B?
• How much does A affect B?

Training Demonstration Test Experiment