Methodology Matters: Doing Research in the Social and Behavioral Sciences Joseph E. McGrath

1
Methodology Matters Doing Research in the Social
and Behavioral SciencesJoseph E. McGrath

• Gary Suh
• Vesna Memisevic

2
Outline

1. Introduction
2. Some Basic Features of the Research Process
3. Substantive Domain
4. Conceptual Domain
5. Methodological Domain
6. Research Methods as Opportunities and Limitations
7. Research Strategies Choosing a Setting For a
   Study
8. Quadrant I The Field Strategies
9. Quadrant II The Experimental Strategies
10. Quadrant III The Respondent Strategies
11. Quadrant IV The Theoretical Strategies
12. Some Strategic Issues

3
Outline

1. Study design
2. Comparison techniques
3. Baserates
4. The Correlation Question
5. The Difference Question
6. Randomization and True experiments
7. Sampling, allocation and statistical inference
8. Validity of findings
9. Potential classes of measures in social
   psychology
10. Strengths and Weaknesses of Types of Measures
11. Techniques for Manipulating Variables
12. Selection, direct intervention, induction
13. Conclusion

4
Introduction

• Doing research the systematic use of some set
  of theoretical and empirical tools to try to
  increase our understanding of some set of
  phenomena or events
• Some of the tools with which
• researchers in the social and
• behavioral sciences go about
• doing research (strategy,
• tactics, and operations)

5
Some Basic Features of the Research Process

• Always involves bringing together three sets of
  things
• some content that is of interest,
• some ideas that give meaning to that content, and
• some techniques or procedures by means of which
  those ideas and content can be studied.

6
Some Basic Features of the Research Process

• These three sets of things more formally, as
  three distinct, though interrelated domains
• The Substantive domain, from which we draw
  contents that seem worthy of our study and
  attention
• The Conceptual domain, from which we draw ideas
  that seem likely to give meaning to our results
  and
• The Methodological domain, from which we draw
  techniques that seem useful in conducting that
  research.

7
Some Basic Features of the Research Process
8
Substantive Domain

• Phenomena and Patterns of phenomena are the
  object of our study
• The phenomena of interest involve the states and
  actions of some human systems and the conditions
  and processes that give rise to and follow from
  those states and actions.

9
Substantive Domain

• Example an individual casting a vote in a
  county election

10
Conceptual Domain

• Properties of the states and actions of those
  human systems that are the focus of study
• Relations refer to any of a variety of possible
  ways in which two or more elements can be
  connected
• Examples causal connections, logical relations,
  chronological relations

11
Conceptual Domain

• Examples attitude, cohesiveness, power, social
  pressure, status

12
Methodological Domain

• Basic sets of elements or tools by which social
  and behavioral scientists systematically gather
  empirical information
• Measuring
• Manipulating
• Controlling
• Distributing Impact

13
Methodological Domain

• Measuring
• For assessing the state or magnitude of some
  property of some actors-behaving-in-context so
  that the researcher can determine what value or
  level that feature has for each case to be
  studied
• Examples questionnaire, rating scale,
  personality test, instruments for observing and
  recording communications, techniques for
  assessing the quality of some products resulting
  from individual or group task performance

14
Methodological Domain

• Manipulating
• Techniques for manipulating some property of an
  actor-behavior-context
• Experimental manipulation making a feature have
  one particular predetermined value or level for
  certain cases to be studied and another
  specific preordained value or level for certain
  other cases, so that the effect of differences
  in that property can be assessed by comparing
  those two sets of cases

15
Methodological Domain

• Manipulating
• Social psychologists have tried to manipulate
  features of the systems they study by a number of
  techniques, such as
• (a) giving instruction to participants
• (b) imposing constraints on features of the
  environment
• (c) selecting materials for use
• (d) giving feedback about prior performances
• (e) using experimental confederates
• (More is said about techniques for manipulating
  variables near the end of this chapter)

16
Methodological Domain

• Controlling
• A set of techniques for controlling the impact of
  features that are important but that you are not
  going to measure or manipulate in a particular
  study
• These include
• Techniques for experimental control you make
  certain features take the same predetermined
  value for all cases in the study
• Techniques for statistical control you try to
  nullify the effects of variations in a given
  property within a study by removing those
  variations by statistical means
• Techniques for distributing the impact so that
  such impact can be taken into account in
  interpretation of results

17
Methodological Domain

• Distributing Impact
• Techniques for distributing the impact of a
  number of features of the system and its
  context-without directly manipulating or
  controlling any one of them-so that such impact
  can be taken into account in interpretation of
  results
• Randomization the most prominent means refers
  to procedures for the allocation of cases among
  various conditions within the study

18
Research Methods as Opportunities and Limitations

1. Methods enable but also limit evidence.
2. All methods are valuable, but all have weaknesses
   or limitations.
3. You can offset the different weaknesses of
   various methods by using multiple methods.
4. You can choose such multiple methods so that they
   have patterned diversity that is so that
   strengths of some methods offset weaknesses of
   others.

19
Research Methods as Opportunities and Limitations

• The fundamental principle, in behavioral and
  social science is that credible empirical
  knowledge requires consistency or convergence of
  evidence across studies based on different
  methods.

20
Research Strategies Choosing a Setting For a
Study

• When you gather a batch of research evidence, you
  are always trying to maximize three desireable
  features or criteria
• A. Generalizability of the evidence over the
  populations of Actors.
• B. Precision of the measurement of the behaviors
  that are being studied (and precision of control
  over extraneous factors that are not being
  studied).
• C. Realism of the situation or Context within
  which the evidence is gathered, in relation to
  the contexts to which you want your evidence to
  apply.

21
Research Strategies Choosing a Setting For a
Study

• Although you always want to maximize all three of
  these criteria, A, B and C simultaneously, you
  cannot do so.
• This is the fundamental dilemma of the research
  process.

22
Research Strategies Choosing a Setting For a
Study
23
Quadrant I The Field Strategies
24
Quadrant I The Field Strategies

• The two research strategies in quadrant I are the
  Field Study and the Field Experiment.
• Field study the researcher sets out to make
  direct observations of natural, ongoing
  systems, while intruding on and disturbing those
  systems as little as possible.
• Field experiment also works within an ongoing
  natural system as unobtrusively as possible,
  except for intruding on that system by
  manipulating one major feature of that system.

25
Quadrant I The Field Strategies

• The essence of both of the strategies in quadrant
  I, the field study and the field experiment, is
  that the behavior system under study is
  natural, in the sense that it would occur
  whether or not the researcher were there and
  whether or not it were being observed as part of
  a study.

26
Quadrant II The Experimental Strategies
27
Quadrant II The Experimental Strategies

• Laboratory experiment the investigator
  deliberately concocts a situation or behavior
  setting or context, defines the rules for its
  operation, and then induces some individuals or
  groups to enter the concocted system and engage
  in the behaviors called for by its rules and
  circumstances.
• Experimental simulation the researcher attempts
  to achieve much of the precision and control of
  the laboratory experiment but to gain some of the
  realism (or apparent realism) of field studies.

28
Quadrant II The Experimental Strategies

• The two strategies in Quadrant II, in contrast to
  those of Quadrant I, involve concocted rather
  than natural settings.
• The laboratory experiment and the experimental
  simulation are strategies that involve
  actor-behavior-context systems that would not
  exist at all were it not for the researchers
  interest in doing the study.

29
Quadrant III The Respondent Strategies
30
Quadrant III The Respondent Strategies

• Sample survey the investigator tries to obtain
  evidence that will permit him or her to estimate
  the distribution of some variables, and/or some
  relationships among them, within a specified
  population
• Examples public opinion surveys on voting
  intentions, political preferences, buying
  intentions

31
Quadrant III The Respondent Strategies

• The strategies of Quadrant III concentrate on the
  systematic gathering of responses of the
  participants to questions or stimuli formulated
  by the experimenter, in contrast to the
  observation of behaviors of the participants
  within an ongoing behavior system
• Studies are usually done under neutral
  conditions of room temperature, lighting, chair
  comfort to nullify any effects of the behavior
  setting or context on the judgments that are the
  topic of study.

32
Quadrant IV The Theoretical Strategies
33
Quadrant IV The Theoretical Strategies

• Formal theory the researcher focuses on
  formulating general relations among a number of
  variables of interest
• Computer simulation a complete and closed
  system that models the operation of the concrete
  system without any behavior by any system
  participants

34
Quadrant IV The Theoretical Strategies

• The inclusion of these two strategies reminds us
  of the importance of the theoretical side of the
  research process.
• One of the more powerful general strategies for
  research is the simultaneous use of one of the
  theoretical strategies (say, the formulation of a
  general theory) and one of the empirical
  strategies (for example, a laboratory experiment).

35
Some Strategic Issues

1. Does the material, as presented, properly reckon
   with the strengths and weaknesses of the research
   strategies it encompasses?
2. To what extent is the research evidence on each
   problem or topic based on use of only a single
   research strategy, and therefore limited by the
   weaknesses of that strategy and to what extent
   is that body of evidence based on use of
   multiple, complementary strategies, with
   agreement or convergence among the findings
   attained via the different strategies?

35
Fall 2007
Informatics 231 Human-Computer Interaction
36
Study design

1. Gathering observations
2. Aggregation and partitioning
3. Comparison on the data set

37
Study design

• Comparison depends on
• Whats included in the study (what phenomena,
  what properties, what model of treatment for
  variables)
• What system works (which substantive system we
  study, paradigms, strategies)
• What conceptual relations have been posted (which
  properties are linked)
• What comparison techniques are available

38
Comparison Techniques

• Three base forms
• Baserates (how often?)
• Correlations (are properties related do they
  occur together?)
• Differences (comparison or difference)

39
Base rates

• How often Y occurs in the general case, as a
  basis for deciding whether the rate of Y in some
  particular case is notably high or low
• Problems - difference in interpretation - various
  political, economic and social issues

40
The Correlation Question

• Is there covariation in the values of two
  properties or features of system?
• Correlations High Low Positive Negative
  Zero
• Linear or nonlinear relation between two or more
  variables
• Can asses conceptual relations that imply
  covariation between two or more variables, but
  cannot asses any conceptual relations that are
  causal in their implications

41
The Difference Question

• Whether Y is present under condition where X is
  present and whether Y is absent when X is absent?
• Samples are separated in two groups, one with
  members which have performance X, and another
  with members who dont have performance X.
• Some set of tasks is given to both of groups,
  and average task performance is compared for
  both groups.
• Groups must be comparable on factors that might
  affect task performances!

42
Randomization and True Experiments

• Randomization using random assignment procedure
  to allocate cases to conditions (in order to
  remove artifacts and observations that happen by
  chance)
• In order for the study to be called a true
  experiment, study must include some
  randomization
• Random allocation procedure doesnt guarantee an
  equal distribution of any of the potential
  factors among the conditions being compared

43
Randomization and True Experiments

• Possible problems
• Reduce the scope of study, as some variables are
  hold constant, and therefore experimental
  variables will occur only at a few levels
• Reduce the realism of context of your study,
  designed the tasks serves us - not the
  participants' - purposes.

44
Sampling, Allocation and Statistical Inference

• The way we choose cases which will be included in
  our study (from larger population of potential
  cases) effect credibility of the evidence
  resulting from the study
• How to choose right nature of the random sample
  population?

45
Sampling, Allocation and Statistical Inference

• We need to use random sample in that way that we
  can apply results of the study to the population
  of which cases constitute a random sample.

46
Sampling, Allocation and Statistical Inference

• Sampling case procedure which cases from larger
  population will be included in study.
• Allocation case procedure which condition every
  given cases (already selected as a part of the
  study) will be assigned to
• Random refers to procedure, not outcome!

47
Sampling, Allocation and Statistical Inference

• Size of the samples?
• The larger the number the more distributed those
  cases will approach the idealized random
  distribution
• Uneven distribution doesnt occur often if only
  chance is operating.

48
Validity of Findings

• Four different types of validity (Cook
  Campbell)
• Statistical conclusion validity
• Internal validity
• Construct validity
• External validity

49
Validity of Findings

• Statistical conclusion validity
• Difference arisen just by chance?
• Relationship between cause and effect
• Internal validity
• How close can you come to
• asserting that the present of X
• caused the altered level of Y values?
• Difference in Y associated with a
• difference in X does not necessarily
• imply a causal role for X
• Are there other factors which may
• have been covary with X and they, rather
• than X, might have produced the change in Y

50
Validity of Findings

• Construct validity
• How well defined are the theoretical ideas in our
  study?
• How clearly understood are the conceptual
  relations
• being explored?
• How well are mapping of concepts and relations
• External validity
• How confident you can be that your
• findings will hold upon replication or
• how general are your our findings.
• How confident you can be that
• you can make predictions ?

51
Potential Classes of Measures In Social Psychology

• Different techniques to measure presence or
  values of the specific features of the human
  system
• Participants awareness
• Natural or unnatural behavior
• Reactivity of measures
• Who makes which record of behavior
• Participants
• Knowingly make (self reports)
• Unwittingly make (trace measures)
• Investigators
• Observations (visible or hidden observer)
• Third party (for non research purposes)
• Archival records (private or public behavior)

52
Strengths and Weaknesses of Types of Measures
Self reports Observations by visible observer Observations by hidden observer Archival records of public behavior Archival records of private behavior Trace measures
Reactivity High potential (Some participants are aware that their response will be recorded, that may influence how they response they might try to make good impression, to give socially desirable answers or to try to help investigation) High Low High Low None
Versatility Versatile to potential contents and to the population to which they would apply to Can only be used on overt behavior (not feelings, expectations, etc) Med-Low Low Low Low (many concepts not available)
Costs Low High High Medium Medium High (sometimes)
Dross Rates (discarded info.) Low High (per observer hour bases) High(per observer hour bases) High High Very high (links to concept sometimes very week)
Time Low High High Medium Medium High
Errors Vulnerable to observer error Ethical concerns, Vulnerable to observer error No possibility to cross validate No possibility to cross validate
53
Techniques for Manipulating Variables

• Techniques for manipulating variables are not
  very well specified
• Three classes of techniques
• Selection
• Direct intervention
• Inductions
• Manipulating variables are use to create an
  experimental manipulation of features of a
  situation

54
Selection

• Advantage Selection is the most convenient means
  to make sure that all cases of a given conditions
  are alike on a certain variable and that all
  cases of another comparison condition differ o
  that variable
• Disadvantage conclusion drawn is more or less
  unreliable unless all other potential factors are
  removed (uncertainty in nature of variable
  manipulated (additional meaning))

55
Direct intervention

• Advantages
• Surest way of achieving a definite and specific
  variable manipulation
• Permits random allocation (distributing impact of
  other factors and variables, that we are not
  studying)
• Nor likely to be costly or time consuming
• Low dross rate
• Disadvantages
• Applicable only for relatively overt
• and tangible variables
• May suffer from reactivity effects
• and experimental demands

56
Induction

• Three major forms
• Use of misleading instruction to the participants
  
• Use of false feed back
• Use of experimental confederates
• Advantages
• Fairy versatile in context of variables
• Normally low in cost and time
• Disadvantages
• Involves deception (raises ethical concerns)
• Risk of detection
• Risk of experimental demands

57
Conclusion

• Results depend on methods. All methods have
  limitations. Hence, any set of results is
  limited.
• It is not possible to maximize all desirable
  features of method in any one study tradeoffs
  and dilemmas are involved.
• Each study (each set of results) must be
  interpreted in relation to other evidence bearing
  on the same questions.

58
Questions?