Multiple-Group Threats to Internal Validity

1
Multiple-Group Threats to Internal Validity
2
The Central Issue

• When you move from single to multiple group
  research the big concern is whether the groups
  are comparable.
• Usually this has to do with how you assign units
  (for example, persons) to the groups (or select
  them into groups).
• We call this issue selection or selection bias.

3
The Multiple Group Case
Alternative explanations
Administer program
Measure outcomes
Measure baseline
Do not administer program
Measure outcomes
Measure baseline
Alternative explanations
4
Example

• Compensatory education in math for 1st graders
• Pre-post program-comparison group design
• Measures (O) are standardized achievement tests
  (at start of grade 1 and start of grade 2 forms
  A B)

5
Selection Threats

• Any factor other than the program that leads to
  posttest differences between groups.
• For example, because of group differences, kids
  in one group watch Sesame Street more frequently
  and pick up more math concepts.

6
Selection-History Threat

• Any other event that occurs between pretest and
  posttest that the groups experience differently.
• For example, kids in one group pick up more math
  concepts because they watch more Sesame Street.

7
Selection-Maturation Threat

• Differential rates of normal growth between
  pretest and posttest for the groups.
• They are learning at different rates, even
  without program.

8
Selection-Testing Threat

• Differential effect on the posttest of taking the
  pretest.
• The test may have primed the kids differently
  in each group or they may have learned
  differentially from the test, not the program.

9
Selection-Instrumentation Threat

• Any differential change in the test used for each
  group from pretest and posttest
• For example, change due to different forms of
  test being given differentially to each group,
  not due to program

10
Selection-Mortality Threat

• Differential nonrandom dropout between pretest
  and posttest.
• For example, kids drop out of the study at
  different rates for each group.

11
Selection-Regression Threat

• Different rates of regression to the mean because
  groups differ in extremity.
• For example, program kids are disproportionately
  lower math scorers and consequently have greater
  regression to the mean.