Validity and Threats to Validity in Research

1
Validity and Threats to Validity in Research
2
Extraneous Variables and Confounding

• Extraneous variables
• Example
• An investigation of the effect of parental
  involvement on students GPA.
• What is the relationship between parental
  involvement and intellect (or socioeconomic
  status)?
• What is the relationship between intellect (or
  socioeconomic status) and GPA?

3
Confounding variables

• When an extraneous variable systematically varies
  with the independent variable it becomes a
  confounding variable.
• Extraneous variables can influence a confound in
  subtle ways.
• Example Colors and selection of objects.

4
Confounding and Validity

• Confounding variables threaten the validity of
  quantitative research results.
• A goal of research design is to guard against
  these threats to validity.
• Four types of validity
• Statistical conclusion validity.
• Internal validity.
• Construct validity.
• External validity.

5
Statistical Conclusion Validity

• Concerned with the inference that there is a
  relationship between an independent variables and
  a dependent variable (and the strength of that
  relationship).
• Questions the appropriateness of the statistical
  analysis used.
• Typically involves computing effect size.

6
Internal Validity

• Most research is focused on determining whether a
  causal relation exists between independent
  variables and dependent variables.
• Two types of causal relationships
• Causal description describe the consequences of
  manipulating an independent variable.
• Causal explanation explain the dynamics of a
  causal relationship.

7
Criteria for Inferring Causation

• Evidence (usually statistical) of a relationship
  between two variables.
• Correlation.
• Comparisons.
• Temporal ordering (knowing which variable comes
  first).
• Elimination of competing causes (confounding
  extraneous variables).
• Consider the relationship between attendance and
  grades.

8
Threats to Internal Validity History

• Events, other than the independent variable(s),
  that occur between the pretest and posttest and
  influence the postest.
• Example A study to investigate the effect of
  peer tutoring on spelling performance.
• Teacher provides instruction on students and
  monitor the tutoring process.
• Monitoring may influence students motivation to
  do well (for both tutor and tutee).

9
Threats to Internal Validity Maturation

• Developmental (physical, experiential, cognitive)
  changes that occur between the pretest and
  posttest that affect performance on the posttest.
• Example The effect of SAT training on SAT
  performance.
• There exists documented evidence that SAT scores
  improve by, on average, 50 points from one year
  to the next.

10
Threats to Internal ValidityTesting and
Instrumentation

• Testing A change in scores on the posttest as a
  result of having taken the pretest.
• Tests can serve as advanced organizers.
• Tests can sensitize individuals to particular
  aspects of the treatment.
• Instrumentation Any change that occurs in the
  way the dependent variable is measure.
• Test norms can change over time.
• Raters can get better over time.

11
Threats to Internal Validity Regression

• An artifact of selection according to some
  criterion on the pretest and then using the same
  (or similar) measure for the posttest.
• A function of the unreliability of measuring
  instruments.
• Example remedial instruction for children
  scoring low on the ABC tests.

12
Threats to Internal ValidityDifferential
Selection

• Selection of intact groups that may have
  different characteristics to receive different
  treatments.
• Intact groups can differ in many ways, e.g.
• Age Intelligence Reading ability
• Anxiety Level Learning style Religious belief
• Curiosity Maturity Self esteem/concept
• Coordination Motivation Socioeconomic status
• Hearing ability Personality Testwiseness
• Home environment Eyesight Time spent on homework

13
Threats to Internal Validity mortality

• Differential attrition.
• Differential loss of participants from the
  different comparison groups.
• NOT random attrition.
• Can occur coincidentally, or
• As a direct consequence of the treatment
  intervention.

14
External Validity

• Concerned with the extent to which a studys
  results can be extended (generalized) to other
  populations, settings, outcomes, and treatment
  variations.
• Three types of external validity
• Population validity.
• Ecological validity.
• Outcome validity.
• Treatment variation validity.

15
External Validity Population validity and
ecological validity

• Population validity
• To which populations can results obtained from a
  sample be generalized?
• To the Target population?
• To populations more broadly defined?
• Ecology validity
• Can results be generalized to other settings?
• To what extent is subject reactivity a factor?

16
External Validity Outcome validity and
Treatment variation validity

• Outcome validity.
• Can the results of a study be generalized to
  different but related dependent variables.
• E.g., ITED vs CTBS and dependent vars.
• Treatment variation validity.
• Can the results be generalized across variations
  in the independent variables?
• E.g., Traditional instruction in one school vs
  traditional instruction in another school?

17
Construct Validity

• Concerned with the extent to which higher-order
  constructs are represented in a particular study.
• Can be affected by operational definitions.
• Identifying specific operations is often
  convenient (and usually necessary), but tends to
  narrow the construct.
• Bandwidth-fidelity problem.
• Jingle/jangle fallacies.

18
Threats to Internal ValidityAdditive and
Interactive Effects

• Several of the threats to validity can combine
  with differential selection to produce additive
  or multiplicative bias.
• Selection-history.
• Selection-maturation.
• Selection-mortality