Interpretation: The Columbia Shuttle Problem

1
Interpretation The Columbia Shuttle Problem

• Roger Dunbar
• Raghu Garud

2
The Interpreted Event

• NASA obtained an unclear photo showing that
  something had fallen from the external fuel tank
  and hit the shuttle. Falling foam causes minor
  damage to the thermal protection tiles on most
  launches and is repaired after return to earth
  and so Mission Control did nothing. Engineers
  were concerned about the size and speed of the
  debris and whether it damaged the shuttle. They
  requested close-up photos of the shuttle in
  orbit.

3
The Investigators Conclusion

• Management decisions reflect missed
  opportunities, blocked or ineffective
  communications channels, flawed analysis, and
  ineffective leadership. Perhaps most striking
  management including Shuttle Program, Mission
  Management Team, Mission Evaluation Room, and
  Flight Director and Mission Control displayed
  no interest in understanding a problem and its
  implications. (The CAIB Report, 2003170)

4
Agenda

• An organizational history of NASA
• What happened on the Columbia Flight
• What solutions are there?
• Search Options
• Mindful Management Processes
• Distributed Cognition Issues

5
The 1960s The Golden Age

• Kennedys NASA mission a man on the moon by the
  end of the decade
• Impossible without changing how engineers worked
  and organized
• Introduced project planning and all-up testing
• 3 astronauts killed in 1967 men on the moon in
  1969

6
After 1960s Is NASA needed?

• Kennedys inspiring mission NASA can do
• Congress has to be convinced NASA must do
• Promises amazing technological feats
• Promises high reliability, low costs
• Increasingly, a business ideology emerged with
  deadlines, production cycles, cost and efficiency
  concerns, speed, etc

7
The Challenger Disaster

• Flight Readiness Reviews
• Requires everything to be checked
• Requires responsible engineers to report all
  objective evidence subjective hunches not
  allowed
• FRR process did identify anomalies to stop flight
• Managers felt pressure to approve flight

8
Ron Dittemore, Space Shuttle Project Manager,
summarizes

• I think were in a mixture of RD and operations.
  We like to say that were operating the fleet of
  Shuttles. In a sense we are, because we have a
  process that turns the crank and were able to
  design missions, load payloads into a cargo bay,
  conduct missions in an operating sense with crew
  members who are trained, flight controllers who
  monitor people on the ground, processing arenas
  who process we can call that operations because
  it is repeatable, fairly structured and its
  function is well known.

9
Ron Dittemore continued

• The RD side of this is that were flying
  vehicles were blazing a new trail because
  were flying vehicles that are getting more
  experienced. Theyre getting a number of flights
  on them, and theyre being reused. Hardware is
  being subjected over and over again to the
  similar environments. So you have to understand
  whether or not there are effects from reusing
  these vehicles -- back to materials, back to
  structure, back to subsystems. (The CAIB Report,
  200320)

10
Foam Shedding

• The original shuttle design required that no foam
  shedding would occur
• In fact, foam shedding always occurred and was
  documented and resolved after each shuttle flight
• Foam shedding was re-categorized as an
  in-family event and an acceptable risk in 1992

11
The Event

• 82 seconds into the launch, a 640 cu inches block
  of insulation foam fell from the external fuel
  tank and shattered the shuttles protective tiles
  on its left wing.
• Cameras produced unclear images that were
  sharpened but were still unclear
• Did something significant happen?

12
Yes says the Intercenter Photo Working Group

• Classifies the event as out-of-family
• Orders on-orbit photos from Defense Department
• Requests data on foam-loss events as these
  occurred on previous flights to reanalyze and see
  if any patterns in data were overlooked

13
Not so fast says Mission Control

• Is very familiar with foam shedding events
• Based on 1992 decision,sees the new event as
  in-family, a turnaround issue.
• Worried about keeping shuttle flights on time and
  avoiding delays
• Node 2 of the International Space Station is due
  to be complete by February 19, 2004.
• Appoints Debris Assessment Team to explore

14
Lets find out - Debris Assessment Team

• Requests on-orbit photos from Dept. of Defense
  through engineering friends at Johnson Space
  Center
• Uses Crater model to assess damage to the
  shuttle wing (scale inappropriate)
• Uses another model to calculate angle debris
  would have to hit to penetrate
• Reassesses analysis to determine whether can be
  absolutely shuttle is damaged

15
Mission Control

• Cancels requests for on-orbit photos, as the
  requests do not come from Mission Controls
  Flight Dynamics Officer, the only person Mission
  Control authorized to make such requests
• Hears Debris Assessment Report
• As DAT Report did not prove events were not
  in-family, maintained this categorization and
  continued flight as usual

16
Search

• Organization theory
• Theory should describe how conflicts can be
  converted into coordinated efforts
• Given different views, MC appoints DAT
• Appropriateness criteria influence search
• DAT wants to find out about shuttle
• MC wants to know if its categorization is right
• Presentation rules confirm categorization

17
Mindfulness Processes

• Mindfulness is focused on a subject, e.g.,
  shuttle condition, categorizations
• DAT is mindful of the shuttles condition
• DAT reinvents the shuttle experience and avoids
  pre-conceived concepts or categories. Flexible.
• MC is mindful of operations
• MC preplans everything and then checks it. Flight
  is expected to follow pre-planned sequence. MC
  categories identify events. Categories are
  modified during reviews, not during flights. MC
  not flexible.

18
Distributed Cognition Issues

• Organizations divide into units with specialist
  skills, knowledge (Hutchins)
• Action nets within each unit defines a way of
  looking at things
• An isolated action net has no problems
• If action nets interact, same event will be
  interpreted differently

19
Distributed Cognition in NASA

• One action net focuses on operations and keeping
  flights on schedule
• Sees shuttle technology as a tool
• Wants stable, reliable technology
• One action net focuses on understanding the wear
  and tear on the shuttle fleet
• Sees shuttle technology as evolving
• Wants to understand the evolution

20
Categories

• Any category occupies space between situated
  reality and action (Bowker and Star), and there
  are many possibilities.
• Categorizing can be
• relative to technology evolution
• relative to operating schedule

21
Conclusion

• With distributed cognition, search will have
  different purposes, dialog among those who are
  mindful will generate heavy conflict, and
  categorizing will reflect different action nets.
• Situational data is inevitably indeterminate
• A policy decision independent of data can
  override interpretations and determine action

22
INTRODUCTION

• The Columbia and Challenger flights were NASA
  tragedies
• What can NASA learn from these extreme events?
• Two Investigative Commissions have sought to find
  out - The Rogers Commission (1986) and the CAIB
  Report (2003)
• Are there organizational process implications?

23
Epistemology of Extreme Events

• Revelatory organizational audit
• Unearth patterns of behavior and possible
  limitations or contradictions inherent in
  real-time processes
• Findings may establish a basis from which to
  develop future designs

24
The decision-making paradigm

• Generate action alternatives
• Predict consequences of alternatives
• (Evaluate alternatives and pick best the
  decision-making problem)
• Eliminate all alternatives with negative
  consequences

25
Organizational action

• Action requires reduced behavioral variety,
  achieved by rules, procedures, roles and, more
  generally, ideologies
• Action depends on enhanced confidence due to
• Consistent Expectations
• People expect their coordinated effort to
  generate action
• Motivation
• People believe their action has value
• Commitment
• People have some basis for shared control over
  one another and so they can rely on one another

26
Action rationality decision irrationality

• Limit the alternatives e.g., just one decision
  to do or not do something
• Look for positive consequences, only, in support
  of the desired alternative
• Start with expected consequences and invent
  objectives, i.e., justifications that fit the
  consequences
• Gets action may not be right action

27
Ideologies facilitate action

• Objective ideologies are beliefs shared by all
• Ideologies that are conclusive (include a few
  normative statements), complex (include
  contingency statements) and consistent solve the
  choice problem and so provide a good basis for
  action
• As ideology attributes causes internally, want
  commitment and motivation in order to achieve
  action

28
To facilitate action, ideologies can replace
decision rationality

• Should enable easy action unchallenged by
  decision uncertainty or analysis
• Easy action likely as objective ideologies are
  conclusive, complex and consistent because then
  thinking is separated from acting
• Based on choices already made and action
  conditions (ideology) already created, repeat
  actions

29
Action and Change

• Change calls for a new choice
• New choices require not new information but a
  lack of confidence in existing information
• New actions depend on new expectations,
  motivations and commitments that must evoke
  uncertainty
• Ideologies can solve the uncertainty problem and
  direct choices that block or support new actions.
  Conclusive, consistent and complex ideologies
  eliminate more behavioral variety and so better
  support actions

30
Information Uncertainty

• Rationalistic vs. Impressionistic mode
• Incomplete vs. incorrect information
• Lack of confidence in a cognitive mapping reduces
  motivation to act
• Lack of confidence in preferences limits ability
  to judge
• Estimation uncertainty reduces ability to assess
  the affects of possible actions
• The impressionistic mode builds on attributes
  that are known unusual attributes are not
  recognized. Other uncertainties emerge less

31
Risk uncertainty x stakes

• Individual stakes depend on the degree to which
  people believe themselves to be personally
  responsible for the effects of action on the
  organization
• Risk threatens motivation more than uncertainty
  alone
• People reduce uncertainty by learning about
  action effects, breaking actions into smaller
  steps, and claiming that negative effects werent
  known but are a part of normal risk taking.
• People reduce stakes and personal responsibility
  by pointing out uncertainties and drawbacks to
  actions, and by not participating in a decision
  or expressing opposition to it
• Uncertainty-reducers are usually highly motivated
  and speculate in action success. Stake-reducers
  speculate in action failure.
• The rational decision making mode generates
  uncertainty and so a need for responsibility
  evasion., It also provides a method to achieve
  reduced responsibility. The impressionistic mode
  generates less uncertainty and so people feel
  less need to evade responsibility

32
Processes leading to action

• Arguments can unexpectedly favor an alternative
  not to be accepted. Bring in new criteria to
  override relevance of positive facts and
  eliminate alternative
• Participation enhances commitment
• Compromises enhance commitment
• Withholding commitment enhances side payments
• As only one choice is provided by legitimate
  authority, people expect they must accept it

33
Ideologies and processes

• With an inconclusive ideology, the firm
  considered the environment critical to the fate
  of any proposal. It used a rationalistic decision
  mode and it was difficult to get new products
  accepted in this firm
• With a conclusive ideology, the firm held that
  its own efforts were critical to the fate of a
  proposal. The firm used an impressionistic mode
  and its products were highly innovative

34
Ideological inconsistency

• Ideological inconsistencies within an
  organization lead to social deadlocks and
  obstructed action
• Ideological inconsistencies can center on how
  (routines/proper procedures), what (objective vs.
  ontology) and why (using environmental,
  organizational or individual attributions) things
  are or should be many possibilities
• As the recognized ideological inconsistencies
  grow in number, social deadlock, frustration and
  confusion along with a strongly felt need for
  change are likely coupled with no way out

35
NASA Processes in Detail

• Based on MER, specialists find cause of the
  problem, design a solution and test it
• Set safety standards based on this solution
• This standard is used to assess future risk
• If the standard is not infringed the problem is
  solved and it gets no more attention
• NASA informs all levels who can always raise
  questions based on sound engineering, but not
  based on simple doubts, feelings, etc.

36
NASA Process Example

• Flight Readiness Review
• A formal, open, bottom-up process that is
  designed to identify and bring all known risks to
  higher managements attention
• Everyone has one opportunity to voice their
  concerns
• No gut feelings or observations
• Instead, dispute resolution by numbers
• the process is very thorough (Vaughan, 1996)

37
NASA Process Assessment

• All-up testing saves costs and time but
  introduces risk this intended outcome is what
  makes disasters possible
• The Investigative boards always report surprise
  at this, e.g., with no engineering analysis,
  Shuttle managers used past success as a
  justification for future flights (CAIB,
  2003126)

38
NASA Meta-Processes

• Deviations are normalized
• Over time Exceptions Waivers
• Elastic waistband for specifications
• Real-time categorization of anomalies
• In-family or not (investigate)
• Acceptable risk or not (stop launch)
• Real-time investigative responses then depend on
  urgency categorization
• E.g., Appoint a DAT vs. a Tiger team

39
The NASA Process Question

• Is engineering intuition without complete data
  a signal, in fact, of a real, emerging problem
  (exploration), or an unfounded fear that will
  delay NASA progress (exploitation)?
• Investigative commissions favor centralized
  control and standard routines
• Centralization routines take power away from
  intuiting decision-makers

40
Conclusion

• Operating in dual modes generates data
  indeterminacy in real-time
• So far, NASAs routines fail to deal with data
  indeterminacy by eliminating the intuitions of
  its best knowledge sources
• If unexpected events emerge in real time, expert
  intuitive understanding is best able to interpret
  data and resolve indeterminacy

41
Dual Operations
42
Foam Strike with STS-107
43
Out-of-family Event

• Despite normalization, foam strike event was
  considered to be out-of-family.
• Question was whether or not the foam strike posed
  safety-of-flight issue
• Senior mission control said No
• Junior engineering groups said Yes

44
What Explains these Missed Opportunities in
Real-time?
45
What Explains these Missed Opportunities in
Real-time?
46
Examples of Indeterminacy

• Blurry pictures
• Camera lens is out of focus
• Activation of DAT instead of Tiger team
• Use of a analytic model that is not meant to be
  used for this type of event
• Focus on flight schedules instead of safety

47
Indeterminacy of Data for Real-time Decision
Making
48
DAT instead of Tiger team
49
Resolution through inaction

• Ticking clock 7 day window
• Burden of proof on DAT to show need for on-orbit
  photograph
• 3 requests for photographs did not lead to
  organizational response
• 8 other missed opportunities

50
Focus on flight schedule
Most of the Shuttle Programs concerns about
Columbias foam strike were not about the threat
it might pose to the vehicle in orbit, but about
the threat it might pose to the schedule.
51
Resolution through inaction

• Ticking clock 7 day window
• Burden of proof on DAT to show need for on-orbit
  photograph
• 3 requests for photographs did not lead to
  organizational response
• 8 other missed opportunities

52
Influence of Path Dependencies on Real-time
Decisions
53
Impact of Real-time Pressures on Dual
Organizational Modes
54
Implications for learning

• Can we learn when we are operating in dual mode?
• What can we learn?
• Distinction between know-how and know-why
• In real time with emerging events, you dont
  immediately know how or why but you know you
  dont know and can intuit that emerging events
  could be serious

55
Distributed Arrangements for Navigating a Ship