Purposes of Research

1
Purposes of Research

• Purposes of Research
• Exploration
• Description
• Explanation
• Our focus
• Evaluation
• Explanation makes evaluation much better and
  easier, but evaluation can also be done without
  explanation
• Examples from our teenage drug use study

2
Experiments and Evaluation Research

• Experiments are the most direct way to address
  hypothesis testing, particularly for evaluations
• We often cannot do experiments
• Experiments also have weaknesses
• Understanding how one would do an experiment to
  address a research question is critical to
  designing good non-experimental research

3
Experiments

• Experiments consist of
• (1) Taking Action
• (2) Observing Consequences
• Experiments are useful for testing hypotheses
• Hypothesis Example Dads spending more time with
  teenagers reduces teenage drug use
• Sub-hypothesis Example Time reduces drug usage
  through improving the father-teenager relationship

4
Advantage of Experiments

• Why will an experiment answer our question about
  Dads time and teenage drug use better than the
  survey (observational data) did?
• What if teenage drug usage causes poor
  relationships with fathers?
• What if drug usage and poor relationships are
  both caused by low socioeconomic status? What if
  both are caused by genetically-based low
  self-esteem?

5
Advantage of Experiments

• Experiments allow us to disentangle correlation
  and causation
• Experiments allow us to directly observe
  causation

6
Classical Experiments

• Cleanest and easiest in the natural sciences
• Independent and Dependent Variables
• Pre-testing and Post-testing
• Experimental and Control Groups

7
Independent and Dependent Variables

• An experiment looks at the effect of an
  independent variable on a dependent variable.
• The independent variable is the cause.
• The dependent variable is the effect.
• Example
• Dependent variable teenage drug usage
• Independent variable time spent by Dads with
  teenagers

8
Independent and Dependent Variables Issues

• The meaning of independent variable and dependent
  variables is clear in experiments, by design.
  Things are less clear with non-experimental data
  analysis.
• Dependent and Independent variables must be
  operationalized. The actual measures we have may
  be poor proxies for what we want to know.
• Example
• Self-reported drug use vs. actual drug use
• What constitutes drug use? Ever trying?

9
Post-test vs. Pre-test
Pre-test
Post-test
The Treatment Change Independent
Variable (Experimental Stimulus)
Measure Dependent Variable
Measure Dependent Variable
How has dependent variable changed?
10
Problems with simple pre-post comparison

• The simple pre-test and post-test comparison
  interpretation assumes that the change in
  dependent variable is due to the change in
  independent variable, the treatment.
• What else could cause a change in dependent
  variable?
• Response to attention of observers Hawthorne
  Effect
• Changes which would have happened anyway, due to
  other causes

11
Treatment and Control Groups

• Treatment Group Those receiving change in
  independent variable (experimental stimulus),
  the treatment.
• Control Group Those who do not receive change in
  independent variable, the treatment.
• Attribute difference (between treatment and
  control) in differences (pre- and post-) to the
  treatment.

12
Comparability of Treatment and Control Groups

• Counterfactual what would have happened to the
  treatment group if they had not received the
  treatment.
• The control group is supposed to tell us about
  the counterfactual.
• The control group should be comparable to the
  treatment group in every way which matters
• starting dependent variable
• relevant observable variables
• relevant unobservable variables

13
Treatment-Control Design
Treatment Group
Control Group
Measure Dependent Variable
Measure Dependent Variable
Compare same?
Administer Treatment
Compare different?
Remeasure Dependent Variable
Remeasure Dependent Variable
14
Difference in Differences
15
Example Share of Teenagers Using Drugs
16
Why does the control change?

• Placebo and Hawthorne Effects Effect of being
  studied, having attention paid, thinking that you
  are being helped, etc.
• General Trends Affecting Everyone e.g.,
  improving economy, anti-drug campaigns on TV

17
Blind Experiments

• To make sure that a treatment effect is real and
  not due to thinking that you are going to be
  helped, experiments, particularly drug trials are
  often done blindly, so that people do not know if
  they are in the treatment or control group.
• This is now being done for some surgical trials,
  amid great controversy.

18
Double-Blind Experiments

• Sometimes researchers will treat or evaluate the
  treatment and control groups differently,
  frequently unconsciously. To avoid this problem,
  studies, particularly drug trials are done
  double-blind, i.e. without the researcher
  knowing who is treatment and who is control.

19
Methods of Ensuring Treatment-Control
Comparability

• Randomization (the gold standard)
• Sampling from a given population subjects are
  randomly allocated to treatment and control
  groups
• No systematic differences between treatment and
  control
• What if people refuse the treatment?
• Lack of comparability due to chance variation gt
  Sample-size issues
• Matching
• Make the treatment and control groups as alike as
  possible in observable characteristics

20
Randomization over Matching?

• Matching on particular variables is difficult
  because you do not know which ones are important
• Statistics assume randomization
• Can combine through stratification-- to be
  covered after sampling

21
Generalizability

• How are experiment subjects chosen?
• In medical trials, often those with no other hope
  or those who physicians think are most likely to
  benefit
• For statistical reasons, the groups are made as
  homogeneous as possible-- although recent
  controversy has prompted inclusion of women and
  minorities
• But we want results which are applicable
  (generalizable) to a much more general
  population.

22
Problems with Experiments

• Generalizability
• Ethical Considerations
• Expense
• Samples too small
• General Equilibrium Effects Its different if
  the whole system changes
• Utterly impossible to do

23
Selection Bias and the Importance of Theory
Dependent Variable Teen Drug Use
Independent Variable Dad Time with Teen
More Time Treatment
No Drug Use
Drug Use
Less Time Control
Why do some Dads spend more time with their teens
than other Dads?
Could the same factors influence drug usage?
24
Selection Bias and the Importance of Theory
Dependent Variable Teen Drug Use
Independent Variable Dad Time with Teen
More Time Treatment
No Drug Use
___________ ___________
Drug Use
Less Time Control
Why do some Dads spend more time with their teens
than other Dads?
Could the same factors influence drug usage?
25
Selection Bias and the Importance of Theory

• No ability to do randomized experiments
• ?
• Theory is very important to decide why
  treatment and controls differ
• Are the same factors relevant in determining
  outcomes?

26
Exogeneity and Endogeneity

• Exogenous Determined Outside the System
• Endogenous Determined Inside the System
• Depends on the Dependent Variable
• Is time with Dad exogenous to Drug Usage?
• If and only if drug usage does not cause change
  in time of Dad with Teen
• If and only if factors that influence drug usage
  do not also influence time of Dad with Teen