Methodology Matters: Doing Research in the Behavioral and Social Sciences

1
Methodology MattersDoing Research in the
Behavioral and Social Sciences

• ICS 205
• Ha Nguyen
• Chad Ata

2
The way we do research matters

• Because the meaning of the discovered evidence is
  based on the way it is obtained
• 3 domains are always involved when doing research
• Substantive ? content of interest
• Conceptual ? ideas that give meaning to the
  content
• Methodological ? techniques for conducting
  research
• 3 important keywords
• Relations which are between elements within a
  context or embedding system

3
The Substantive Conceptual Domains

• Substantive Domain
• Elements Phenomena
• Relations Patterns of phenomena
• Studying the actors behaving towards objects in
  context
• Conceptual Domain
• Elements Properties of the actors behaviors...
• Relations Interconnections between elements

4
The Methodological Domain

• Elements methods Modes of Treatment
• Includes techniques for
• Measuring
• Manipulating
• Giving instruction to participants
• Imposing constraints on the environment
• Selecting materials for use
• Giving feedback about prior performances
• Using experimental confederates
• Controlling
• Distributing impact

5
Methodological Domain

• Relations Comparison Techniques
• Involve 3 features of the system under study
• Dependent variables or features
• These are measured
• Independent variables or features
• These are manipulated
• All of the rest of the systems features
• These are relevant but not the focus of the study
• Comparison performed to assess covariation or
  association between dependents and independents

6
The Domains
7
Research Methods Are Like Tools

• Each method offers opportunities not available by
  other means
• Each method also has inherent limitations
• Therefore you should use multiple methods in
  order to counterbalance the flaws in each
  individual method
• Make sure to chose methods such that one strength
  can compensate for anothers weakness

8
The Research Methods
Abstract
Precision Maximum
II
I
III
Unobtrusive
Obtrusive
IV
Realism Maximum
Generalizability Maximum
Concrete
9
Quadrant I Field Strategies
Abstract
Precision Maximum
II
III
Unobtrusive
Obtrusive
IV
Realism Maximum
Generalizability Maximum
Concrete
10
Quadrant I Field Strategies

• Field Study
• Natural unobtrusive
• Field Experiment
• Manipulating one variable in an otherwise natural
  system ? decrease in realism
• A bit obtrusive

11
Quadrant II Experimental Strategies
Abstract
Precision Maximum
I
III
Unobtrusive
Obtrusive
IV
Realism Maximum
Generalizability Maximum
Concrete
12
Quadrant II Experimental Strategies

• Laboratory Experiment
• Concocted system
• Increased precision
• Decreased realism generalizability
• Experimental Simulation
• Similar to lab experiment, but
• strives to increase realism

13
Quadrant III Respondent Strategies
Abstract
Precision Maximum
II
I
Unobtrusive
Obtrusive
IV
Realism Maximum
Generalizability Maximum
Concrete
14
Quadrant III Respondent Strategies

• Sample Survey
• Sampling of the population ? high
  generalizability
• Low precision
• Judgment Study
• Here the focus is on the stimulus material,
  rather than respondents attributes ? High
  precision
• actors of convenience ? Low generalizability
  over the population
• Both often done in neutral settings ? low realism

15
Quadrant IV Theoretical Strategies
Abstract
Precision Maximum
II
I
III
Unobtrusive
Obtrusive
Realism Maximum
Generalizability Maximum
Concrete
16
Quadrant IV Theoretical Strategies

• Formal Theory
• Formulating general relations between variables
  of interest ? high generalizability
• No concrete system involved ? low in realism
• No observation ? low precision
• Computer Simulation
• Like experimental simulation, except human
  participants are not included
• Modeling the real world ? high realism
• Outcomes are predictable ? low precision
• Modeling a particular system ? low
  generalizability

17
Questions to keep in mind

• Does the presented material coincide with the
  strengths weaknesses of the research strategies
  that were used?
• Is the research evidence well supported by a
  single or by multiple research strategies?
• Are the strategies weaknesses and strengths
  offset such that they counterbalance one another?

18
Study Design

• For every empirical study, observations have to
  be gathered, combined, and compared
• Ex Given a pair of properties, X and Y, can we
  tell if X caused Y?
• X is independent variable, Y is dependent
  variable
• What techniques are available for comparison?
• Baserates
• Correlation
• Difference

19
Comparison Technique 1

• Baserates
• How often (at what rate, or what proportion of
  time) does Y occur?
• If I do not know how often Y occurs in the
  general case, then I cannot decide whether the
  rate of Y in the study is significant
  (particularly high or low).

20
Comparison Technique 2

• Difference Question
• Is Y present under conditions where X is present?
  Is Y absent when X is absent?

21
Comparison Technique 3

• Correlational Question
• Do the values of property X covary with the
  values of property Y?
• If X is high, is it likely that Y will also be
  high?
• If X is low, is it likely that Y will also be
  low?
• High positive correlation, high negative
  correlation, little or no correlation
• This technique tells you whether X and Y go
  together but it does not help in determining
  causality.

22
Randomization

• Randomization using a random assignment
  procedure to allocate cases to conditions
• Each case must be equally likely to end up in any
  given condition
• This procedure does not guarantee to rule out all
  other factors that could affect the results.

23
Validity of Findings

• 4 types of validity
• Internal validity
• How close can you come to asserting that the
  presence of X caused the altered level of Y?
• Statistical conclusion validity
• Is the given result due to chance?
• Construct validity
• How well defined are the theoretical ideas in the
  study?
• External validity
• How well will the results hold up for replication
  and how generalizable are the results?

24
Measure Techniques

• Categorized by two questions
• Who makes the record of the behavior?
• Is the participant aware that his/her behavior is
  being recorded for research purposes?

25
6 Major Types of Measures
26
Strengths and Weaknesses of the Measures

• Self-reports
• Strength - Low costs, does not require a lot of
  time to create and use
• Weakness - Reactive
• Trace measures
• Strength unobtrusive
• Weakness not very versatile, time consuming to
  gather

27
Strengths and Weaknesses of the Measures

• Observations
• Strength relatively versatile
• Weakness can only be used on overt behavior,
  vulnerable to observer errors, can be reactive
  for visible observers, possible ethical issues
  for hidden observers
• Archival records
• Strength good for researching the past, studies
  for extensive periods of time
• Weakness public records can be reactive,
  versatility is limited

28
Techniques for Manipulating Variables

• Selection selecting cases with desired values
  and assigning them to appropriate conditions
• Direct Intervention manipulating the specific
  variable directly

29
Techniques for Manipulating Variables

• Inductions
• 3 types
• Misleading instructions
• False feedback
• Experimental confederates
• Ethical issues

30
Concluding Comments

• Results depend on methods. All methods have
  limitations. Therefore, any set of results is
  limited.
• It is not possible to maximize all desirable
  features of different methods in any one study.
• Each set of results must be interpreted in
  relation to other evidence relating to the same
  questions.