Non-Experimental designs: Developmental designs

1
Non-Experimental designs Developmental designs
Small-N designs

• Psych 231 Research Methods in Psychology

2
Announcements

• Journal Summary 2 due date moved back a week
  (instead of this week, they are due next week)

3
Exam 2
4
Error in survey research

• Sampling error - Are there differences in your
  sample compared to the population as a whole?
• Response rate
• What proportion of the sample actually responded
  to the survey?
• Hidden costs here - what can you do to increase
  response rates
• Non-response error (bias)
• Is there something special about the data that
  youre missing? From the people who didnt
  respond
• Measurement error (validity and reliability)
• Are your questions really measuring what you want
  them to?

5
Quasi-experiments

• What are they?
• Almost true experiments, but with an inherent
  confounding variable
• Design includes a quasi-independent variable
• Examples
• An event occurs that the experimenter doesnt
  manipulate
• Interested in subject variables
• Time is used as a variable

6
Quasi-experiments

• What are they?
• Almost true experiments, but with an inherent
  confounding variable

• Advantages
• Allows applied research when experiments not
  possible
• Threats to internal validity can (sometimes) be
  assessed

7
Quasi-experiments

• What are they?
• Almost true experiments, but with an inherent
  confounding variable

• Disadvantages
• Threats to internal validity may exist (which
  can not be addressed)
• Be careful when making causal claims
• Statistical analysis can be difficult
• Most statistical analyses assume randomness

8
Quasi-experiments

• What are they?
• Almost true experiments, but with an inherent
  confounding variable

• Common types
• Nonequivalent control group designs
• Program evaluation
• Interrupted time series designs

9
Quasi-experiments

• Nonequivalent control group designs
• With pretest and posttest (most common)

• But remember that the results may be
  compromised because of the nonequivalent control
  group

10
Quasi-experiments

• Program evaluation
• Research on programs that is implemented to
  achieve some positive effect on a group of
  individuals.
• e.g., does abstinence from sex program work in
  schools
• Steps in program evaluation
• Needs assessment - is there a problem?
• Program theory assessment - does program address
  the needs?
• Process evaluation - does it reach the target
  population? Is it being run correctly?
• Outcome evaluation - are the intended outcomes
  being realized?
• Efficiency assessment- was it worth it? The the
  benefits worth the costs?

11
Quasi-experiments

• Time series designs
• Basic method Observe a single group multiple
  times prior to and after a treatment

treatment
Obs
Obs
Obs
Obs
Obs
Obs

• The pretest observations allow the researcher
  to look for pre-existing trends
• The posttest observations allow the researcher
  to look for changes in the trends
• Is it a temporary change, does it last, etc.?

12
Quasi-experiments

• Time series designs

• A variation of basic time series design
• Addition of a nonequivalent no-treatment control
  group time series

13
Developmental designs

• Used to study changes in behavior that occur as a
  function of age changes
• Age typically serves as a quasi-independent
  variable

• Three major types
• Cross-sectional
• Longitudinal
• Cohort-sequential

14
Developmental designs

• Cross-sectional design
• Groups are pre-defined on the basis of a
  pre-existing variable
• Study groups of individuals of different ages at
  the same time
• Use age to assign participants to group
• Age is subject variable treated as a
  between-subjects variable

15
Developmental designs

• Cross-sectional design

• Advantages
• Can gather data about different groups (i.e.,
  ages) at the same time
• Participants are not required to commit for an
  extended period of time

16
Developmental designs

• Cross-sectional design

• Disavantages
• Individuals are not followed over time
• Cohort (or generation) effect individuals of
  different ages may be inherently different due to
  factors in the environment
• Example are 5 year old different from 13 year
  olds just because of age, or can factors present
  in their environment contribute to the
  differences?
• Cannot infer causality due to lack of control

17
Developmental designs

• Longitudinal design

• Follow the same individual or group over time
• Age is treated as a within-subjects variable
• Rather than comparing groups, the same
  individuals are compared to themselves at
  different times
• Repeated measurements over extended period of
  time
• Changes in dependent variable reflect changes due
  to aging process
• Changes in performance are compared on an
  individual basis and overall

18
Developmental designs

• Longitudinal design

• Advantages
• Can see developmental changes clearly
• Avoid some cohort effects (participants are all
  from same generation, so changes are more likely
  to be due to aging)
• Can measure differences within individuals

19
Developmental designs

• Longitudinal design

• Disadvantages
• Can be very time-consuming
• Can have cross-generational effects
• Conclusions based on members of one generation
  may not apply to other generations
• Numerous threats to internal validity
• Attrition/mortality
• History
• Practice effects
• Improved performance over multiple tests may be
  due to practice taking the test
• Cannot determine causality

20
Developmental designs

• Cohort-sequential design

• Measure groups of participants as they age
• Example measure a group of 5 year olds, then the
  same group 5 years later, as well as another
  group of 5 year olds
• Age is both between and within subjects variable
• Combines elements of cross-sectional and
  longitudinal designs
• Addresses some of the concerns raised by other
  designs
• For example, allows to evaluate the contribution
  of generation effects

21
Developmental designs

• Cohort-sequential design

• Advantages
• Can measure generation effect
• Less time-consuming than longitudinal
• Disadvantages
• Still time-consuming
• Still cannot make causal claims

22
Small N designs

• What are they?
• Historically, these were the typical kind of
  design used until 1920s when there was a shift
  to using larger sample sizes
• Even today, in some sub-areas, using small N
  designs is common place
• (e.g., psychophysics, clinical settings,
  expertise, etc.)

23
Small N designs

• One or a few participants
• Data are not analyzed statistically rather rely
  on visual interpretation of the data
• Observations begin in the absence of treatment
  (BASELINE)
• Then treatment is implemented and changes in
  frequency, magnitude, or intensity of behavior
  are recorded

24
Small N designs

• Baseline experiments the basic idea is to show
• when the IV occurs, you get the effect
• when the IV doesnt occur, you dont get the
  effect (reversibility)
• Before introducing treatment (IV), baseline needs
  to be stable
• Measure level and trend

25
Small N designs

• Level how frequent (how intense) is behavior?
• Are all the data points high or low?
• Trend does behavior seem to increase (or
  decrease)
• Are data points flat or on a slope?

26
ABA design

• ABA design (baseline, treatment, baseline)

• The reversibility is necessary, otherwise
• something else may have caused the effect
• other than the IV (e.g., history, maturation,
  etc.)

27
Small N designs

• Advantages
• Focus on individual performance, not fooled by
  group averaging effects
• Focus is on big effects (small effects typically
  cant be seen without using large groups)
• Avoid some ethical problems e.g., with
  non-treatments
• Allows to look at unusual (and rare) types of
  subjects (e.g., case studies of amnesics, experts
  vs. novices)
• Often used to supplement large N studies, with
  more observations on fewer subjects

28
Small N designs

• Disadvantages
• Effects may be small relative to variability of
  situation so NEED more observation
• Some effects are by definition between subjects
• Treatment leads to a lasting change, so you dont
  get reversals
• Difficult to determine how generalizable the
  effects are

29
Small N designs

• Some researchers have argued that Small N designs
  are the best way to go.
• The goal of psychology is to describe behavior of
  an individual
• Looking at data collapsed over groups looks in
  the wrong place
• Need to look at the data at the level of the
  individual

30
Next time

• Statistics (Chapter 14)