Interpreting Precursor Events Phase I

1
Interpreting Precursor EventsPhase I

• Robin Dillon-Merrill (PI)
• Catherine H. Tinsley (Co-PI)
• The McDonough School of Business
• Georgetown University
• NASA-CPMR 1st Annual Fellows Conference
• January 2005

2
Precursors to Catastrophes

• In most cases of catastrophes, some forewarning
  is available (Turner, 76)
• When catastrophes occur that were preceded by
  near miss events, the question becomes, why
  were conditions allowed to continue?
• Foam separation in the Space Shuttle
• Tire bursts deflations on the Concorde
• Mud slides in California
• Were near miss events ignored?

3
Precursors Influence

• Decision Makers attend to near miss
• Near miss information is incorporated into
  decision calculus
• Near misses will systematically bias decision
  making
• Research Questions for Phase I
• How do near misses bias decisions under risk?
  (H1)
• How does cognitive load affect the decision
  process? (H7)
• Can we do anything to ameliorate the near miss
  bias? (H5)

4
How does Near Misses InformationBias Decisions?

• Hot Hand Analogy
• Representative Heuristic (Kahneman Tversky,
  1972)
• H-T-H-T-H-T vs. H-H-H-T-T-T
• Representative thinking implies structure where
  none exists (Dawes, 1988)
• Outcome independence
• Outcomes as necessary consequence of antecedent
  conditions or as draws from a distribution of
  possible outcomes (March, Sproull, Tamaz, 1991)

5
How does Near Misses InformationBias Decisions?

• See a non-random pattern when several non-fatal
  outcomes (near misses) occur in a row
• Fail to recognize independence of events (that
  outcomes are draws from a distribution)
• Discount or ignore or revise probability
  information
• Infer an underlying structure (hot hand, sturdy
  rover)
• Reason the past is predictive of the future
• Feel overly optimistic about chances of survival
• Accept more risky decisions based on heuristic
  decision making

6
How does Cognitive LoadInfluence Decisions?

• Occupies working memory
• Disrupts non-automatic cognitive processes
  (Gilbert, Pelham Krull, 1988)
• Encourages automatic, heuristic cognitive
  processes (Ignoring probability information and
  expected value models)
• Creates interaction effect between cognitive load
  and near miss information on decision making

7
Phase I HypothesesExperiment 1 (Sept. 2004)

• H1 People with near miss information are more
  likely to choose a risky alternative than people
  without near miss information
• H2 Interaction between near miss information
  and cognitive load, such that those without
  near miss information or cognitive load will make
  less risky decisions than the other three groups.

8
Phase I Design
9
Method

• Simulation of a Mars Rover mission
• Limited battery life (8 days)
• 5 travel days to destination
• Rewarded 5 extra dollars for each battery day
  extra
• Weather forecast for each day
• Mild weather or 95 chance of severe storm
• Severe dust storms can cause catastrophic failure
• 40 catastrophic failure if drive through severe
  storm
• 100 safe if stop deploy wheel guards
• Operational decisions (stop/ go) for day 6-13
• Decide to drive or stop deploy wheel guards
• Manipulation check, risk propensity, and
  engagement

10
Manipulations

• Near Miss
• Of 5 days before you started operating the rover,
  had 3 days of severe storms and rover had driven
  successfully through these
• Of 5 days before you started operating the rover,
  all mild weather
• Cognitive load
• Memorize 7 facts about Mars (no time to write
  down), need to remember for later at press
  conference
• No facts to memorize

11
Analysis ResultsExperiment 1 H1
12
Analysis ResultsExperiment 1 H2
13
Other Results

• No differences across any of the condition in
  task engagement
• No differences across any of the conditions in
  risk propensity
• Risk propensity had no influence on decisions to
  drive or not
• Task engagement significantly influences
  decisions to drive (ß .43, p.02, Cox Snell
  R2.12)

14
Experiment 2 Modifications(Nov. 2004)

• Ask participant the basis of their decision
• RQ Did they use probability information?
• Make probability information more salient
• RQ Can we ameliorate the near miss bias?
• Count whether participants searched for more
  information (8 buttons including wheel failure
  data)
• RQ Will participants with near miss information
  engage in less search for additional information
  (even when information is costless)?

15
Analysis ResultsExperiment 2

• What was the basis for your decision?
• Calculated probabilities (N55)
• There was a 60 chance of success, Ill be ok
• Some other reason (N59)
• I survived before
• Its too early to risk things
• I just wanted to take a chance
• Were confused or gave a reason we couldnt
  understand (N8)
• No differences in reasons across conditions

16
Analysis and ResultsExperiment 2 Those who
USED probability information
17
Analysis and ResultsExperiment 2 Those who did
NOT use probability information
18
Experiment 2Search for Information

• 8 information buttons
• Overall range 0-40, µ12.37, sd8.5, median11,
  mode8
• No near miss information µ13.49, sd8.9
• Near miss information µ10.85, sd7.9
• Wheel failure data button
• Overall range 0-6, µ2.04, sd1.5, median2,
  mode1
• No near miss information µ2.36, sd1.5
• Near miss information µ1.72, sd1.3

19
ConclusionsPhase I

• Near miss bias exists encouraging riskier
  decisions
• This bias appears to be a heuristic decision
  process
• Discourages search for additional (costless)
  information
• Does not crowd out probability information
• Does not appear to be used to update probability
  information (or those that used probability
  information and had near miss information should
  be making riskier decisions)
• Cognitive load appears to narrow decision focus
  and also encourage heuristic decision making
• Salient probability information, if used by
  decision maker, will eliminate the near miss bias

20
Limitations Moving Forward (Phase II)

• Mediating variables why does near miss
  information bias decision making?
• Direct test of how probability information is
  used
• Information search and information quality
• Moderating variables what exacerbates/ameliorates
  this bias?
• Near miss to self versus other
• Timing of near miss
• Frame of near miss
• Counterfactuals
• Decision tools
• External validity
• Group decision making

21
Benefits to NASA

• Improving P/PM Decision Making
• Awareness of Near Miss Bias
• Suggestions for Managing Near Miss Bias
• Developing an Effective Lessons Learned System
• Effectiveness of LL systems are dependent on
  completeness of data
• Events transformed into data when they stimulate
  counterfactual thought
• Lessons are learned with self-focused, downward,
  counterfactual thought
• A complete data set requires re-framing and
  incorporating positive near miss information
• Identifying NASA-specific P/PM templates (cases)
• Develop case based training tools based on
  experimental materials

22
Engagement and Outreach Plan

• Goals
• Help NASA managers learn from all the decisions
  that they make, not just those that result in a
  failure
• Complement other risk management efforts at NASA
• Offer various educational delivery methods to
  maximize exposure of the results to the broadest
  community (seminar and web-based)
• Reduce risk across NASA programs by improving
  decision making skills
• Content
• Training people to be aware of the near miss bias
  
• Teaching decision aids to ameliorate the bias

23
NASA/USRA CENTER FOR PROGRAM/PROJECT MANAGEMENT
RESEARCH
A Virtual Center for Research in Program/Project
Management for Aeronautics and Space Sponsored by
NASA/Academy of Program and Project
Leadership Operated by The Universities Space
Research Association July 23, 2004