Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers

1
Usage of Decision Analysis Methods Outside of a
Classroom Environment by Aerospace Researchers
National Aeronautics and Space Administration

• Sharon Monica Jones
  
  NASA Langley
• Rafael E. Landaeta, C. Ariel Pinto and Resit Unal
  
  Old Dominion University
• James T. Luxhøj
  
  Rutgers University
• Hampton Roads Area INCOSE Conference on Decision
  Analysis and Its Applications to Systems
  Engineering, Newport News, VA (November 17-18,
  2009)

www.nasa.gov
2
Outline

• Background/Definitions
• Data Collection Process
• Results
• Concluding Remarks

3
Problem Definition

• Aerospace technology managers need
• To predict future technologies
• To assess progress toward RD goals
• Aerospace technology portfolio decisions are
  difficult because
• Very little time to acquire background data
• Uncertainty factors (e.g., politics, global
  economy, environment, funding)
• Prescriptive decision analysis methods
• Have been used for financial portfolio assessment
• Value for policy related decisions has been
  questioned

4
Ralph Keeneys Suggestions for Making Better
Decision Makers

• Develop concepts, tools and procedures to help
  decision makers
• My experience is that many people, including
  well-educated people, have a very difficult time
  in structuring their decisions. They can get
  mixed up about the difference between fundamental
  concepts such as alternatives and objectives.
• Use real decisions, not just laboratory problems
  in decision research
• We have learned a great deal from all the
  laboratory settings where decision experiments
  have been conducted. There have also been some
  research studies of real decision problems. I
  feel there is much more to be gained by having
  more of this type of research.
• Teach people what they can and will learn and use
• hundreds and thousands of people have had at
  least a course that included a substantial part
  on decision analysis and very few have probably
  ever conducted a formal decision analysis. Once
  we find out what people can and will learn and
  use, that should constitute the basis for much of
  our teaching of decision analysis.

Source Ralph L. Keeney, Making Better Decision
Makers, Decision Analysis, 14 (2004)
5
Decision Analysis Usage in Aerospace Portfolio
Development

• Aerospace managers have investigated the use of
  decision analysis methods for portfolio
  investment decisions
• Commercial Aviation Safety Team (CAST)
• NASA Aviation Safety Program
• Future Aviation Safety Team (FAST)
• These technology assessments involved
• Resource commitments (e.g., employee time, travel
  money, software acquisition, training)
• Assumption that decision analysis methods would
  improve aerospace technology assessment process

6
Technology Assessment

• A process for measuring the impact of
  established or new technologies

Hans Mohr, Technology Assessment in Theory and
Practice, Society for Philosophy and Technology,
44 (Summer 1999)
7
Aerospace Technology Assessment

• Three different processes for examining impact of
  a set of technologies
• Technology assessment
• Technology forecasting
• Technology foresight

Aerospace Technology Assessment
8
Conceptual Model
Adapted from Robert T. Clemen, Making Hard
Decisions, 2nd Edition (1995)
9
Purpose of Study

• What is known
• Decision analysis methods in financial portfolio
  assessment
• Decision experiments in laboratory settings
• Technology assessment in medical RD

• What is unknown
• Decision analysis methods for policy related
  decisions
• Decision analysis methods in real decision
  problems
• Technology assessment in aerospace

10
Study Overview

• Population was aerospace researchers with
  experience in one or more of the following
• Aerospace program/project management
• Aerospace technology assessment
• Aerospace technology selection
• Aerospace RD portfolio development
• Methods that were investigated in the study
• Decision trees
• Influence diagrams
• Criteria aggregation methods
• Explicit tradeoff approaches
• Participants were questioned about their usage of
  these methods for aerospace technology assessment

11
Decision Trees
decision node
chance node
consequence node
12
Influence Diagrams
chance node
consequence node
decision node
13
Criteria Aggregation Methods

• Methods in which two sets of aggregated indices
  are developed and used to evaluate the
  alternatives in the decision problem
• Methods in the category include
• Analytical Hierarchy Process (AHP)
• Weighted Sum Model (WSM)

Example of Simple Weighted Sum Model
E. Triantaphyllou, Multi-Criteria Decision
Making Methods A Comparative Study(1995)
14
Explicit Tradeoff Approaches

• Decision analysis methods that are based on value
  functions
• Methods in this category include
• Multi-Attribute Utility Theory (MAUT)
• Simplified Multi-Attribute Rating Approach
  (SMART)

15
Excluded Decision Analysis Methods

• Avoided decision analysis methods that are not
  popular in U.S.
• Real world applications are complex with large
  amounts of uncertainty
• Specific decision analysis methods that were
  excluded from study
• Outranking methods (e.g., ELECTRE, TOPSIS)
• Optimization methods
• Analytic network process (ANP)

16
Data Collection and Analysis Process
Refine List of Candidate Survey Participants
Analyze Results
Conduct Survey

• Legend
• Survey Development
• Data Collection
• Data Analysis

17
Web-Based Instrument Development

• Several web-based services examined
• Questions developed based on several sources
• Short surveys at professional meetings
• Validated research in decision analysis
  literature
• Identities of survey participants remained
  anonymous

18
Pilot Survey

• Conducted with subset of population (10 persons)
• Think aloud cognitive interviewing techniques
  used
• Solicitation of all thoughts and comments
• Manual recording of information during completion
  of online survey
• De-identification of subjects in final
  documentation

19
Survey Instrument Modification

• Questions were modified, added or eliminated from
  the survey based on input from
• Pilot survey comments
• Data analysis of pilot survey data
• Additional comments from other reviews (e.g., ODU
  IRB)
• Number of survey questions reduced from 70 to 65

20
Data Collection Overview

• Approval to conduct survey was obtained from NASA
  Langley and ODU Institutional Review Boards
  (IRBs)
• E-mail invitation was sent to 260 persons
• 154 total visits to survey website
• 16 partial responses
• 99 completes surveys
• Out of the 99 completed surveys
• 76 male, 24 female
• Highest degree level was 60 Masters, 21
  Bachelors, 18 Doctorate, 1 Associates
• 72 employed as government civil servants

21
Job Functions
22
Aerospace Experience
23
Decision Trees
24
Influence Diagrams
25
Criteria Aggregation Methods
26
Explicit Tradeoff Approaches
27
Usage of Decision Analysis Methods Outside of a
Classroom Environment
28
Categories of Non-ATA Usage of DA Outside of a
Classroom Environment
29
Additional Questions
30
Additional Questions (contd)
31
Additional Questions (contd)
32
Additional Questions (contd)
33
Additional Questions (contd)
34
Likelihood of Future Usage of DA Methods
35
Concluding Remarks

• This is a subset of the total data
• There are many additional questions in the study
• More formal analysis of the data was conducted
  using structural equation modeling techniques to
  test a set of hypotheses
• Survey participants believed that the successful
  use of decision analysis methods depends on
• Selection criteria in the decision model
• Experience of the person that implements the
  method
• Reliability of the input data
• Training/education does not guarantee future use
  of a decision analysis method

36
Questions?
37
SignOffPage