Reliability and Validity of Dependent Measures

1
Reliability and Validity of Dependent Measures
2
Validity of Dependent Variables

• Does it measure the concept?
• Construct Validity Does DV really capture what
  you want to measure (good operational
  definition?)
• Or does it include mood, culture or gender bias,
  confusing wording, observational bias, etc.

3
Indicators of Construct Validity

• Face Validity Does it appear to be a good
  measure (do experts think so?)
• Predictive Validity Predict later behavior-
  GREgrad school success?
• Concurrent Validity Are those known to diverge
  different in scores (Self Monitoring)

4
Indicators of Construct Validity

• Convergent Validity do other kinds of ratings
  agree? Similar responses to similar scales
  
• Divergent validity is it different from other
  constructs? (measures intell, not SES or gender
  bias) shy isnt lonliness
• Reactivity- knowing you are being studied changes
  behavior

5
Reliability of DV

• Are results repeatable?
• All measurement contains true score plus error of
  measurement
• Not an issue of replication- same subjectssame
  scores

6
Types of Reliability

• Inter-rater reliability- calculate r for
  observers or Cohens Kappa
• Internal consistency- split half reliability
  Cronbachs Alpha calculates ave of all possible
  corr.
• Temporal consistency- test-retest reliability
  with SAME people
• Restaurant example

7

• Can a variable be reliable and not valid?
• Valid and not reliable?
• How do you know you have a good DV?
• Mental Measurements Yearbook

8
Validity of Experimental Designs

9
Survey Design
10
Internal validity

• Does the design test the hypothesis we want it to
  test? Did IV manipulation cause change in DV?
  Can we infer causality?
• What if internal validity is low?

11
External validity

• Does your study represent a broad population?
• Caution with Discussion Section if weak
• Random Sampling
• Stratified Sampling
• Block Randomization

12
Ecological validity

• Does study reflect the real world-
• do people really behave this way?
• Can you study anything without changing it?

13
Threats to Internal Validity

• In pre-post design
• Test participants
• Administer IV
• Post test for effect of IV
• Compare pre vs. post results to look for effect
  of IV

14
History

• World events may cause change in attitudes or
  behavior over time.
• Tests of patriotism pre/post 9/11
• Views of President pre/post Katrina
• Attitudes of adolescents pre/post Cobain suicide

15
Maturation

• Individuals change over time as they mature.
• Issue for studies of children, but also huge
  growth in freshman year- change of attidues and
  behavior.

16
Testing

• The study you use may cause differences in
  behavior.
• Similar to REACTIVITY, but for entire study not
  just DV. Parenting study for example

17
Instrumentation

• Use of instrument may get better or worse with
  time
• Observation studies
• Testing skill/ interviewing

18
Regression toward the mean

• Extreme scores do not tend to be repeatable-
  those who score very high or very low on a test
  will be closer to the average if tested again.
• A big issue for any study where pretest is used
  to select subjects for post test.

19
Mortality

• Those who drop out of your study may differ from
  those who choose to continue.

20
Placebo effect

• If given any treatment, behavior will change,
  even if treatment was not meaningful. (fake
  drugs get some results)

21
How can we improve internal validity?

• History
• Maturation
• Testing
• Instrumentation
• Regression toward the mean
• Mortality
• Placebo effect

22
Improved Design

• In pre-post design
• Test participants
• Administer IV
• Post test for effect of IV
• Compare pre vs. post results to look for effect
  of IV

• Two Group design
• Pretest (do you need to do this?)
• RANDOMIZED assignment to levels of IV
• Compare post test results of IV and Control groups

23
Extraneous Variables

• Any variable that you have not measured or
  controlled (RA) that may impact the results of
  your study

24
Demand Characteristics

• Participants behave in ways demanded by the
  situation or experimental set-up. Behavior does
  not reflect actual beliefs or attitudes.
• Issue of Ecological Validity

25
Subject Bias

• Bias brought on by subjects beliefs
• (Overhead of mood and menstrual cycle)

26
Social desirability

• Subjects want to do the right thing and try to
  guess what the experimenter wants, and do not
  behave naturally.
• How to reduce Subject biases?

27
Experimenter Bias

• Experimenters behavior and expectations can sway
  results of test.
• How to reduce these biases?

28
Floor Ceiling Effects

• If measures are too easy or too difficult you
  will not see differences between groups.
• Pilot test with similar subjects!

29
Order effects

• When using within subjects designs, order of
  presentation can affect results in several ways.
  Practice effects Subjects get better at task
  with successive trials Fatigue effects
  Subjects get tired and do worse or lose
  interest Carryover effects subjects experience
  in one condition impacts results of another
  condition- subject bias or anchoring and
  adjustment issues.

30
How to reduce order effects

• Counterbalancing
• Does not get rid of effects, it just makes them
  equal for all groups. Can do complete
  counterbalancing if small number of conditions.
• Latin Square counterbalancing
• A, B, skip, C, skip, D, etc. then fill back
• A, B, N, C, N-1, D, N-2, E etc.

31
A Latin Square for 6 conditions
Order1 A B F C E D
2 B C A D F E
3 C D B E A F
4 D E C F B A
5 E F D A C B
6 F A E B D C
32
Pretest Vs. Pilot test

• When do you use a pilot test?
• When do you use a pre test?

33
Can a DV be reliable but not valid?
34
Experimental Validity

• What to do if low Internal Validity?
• What are impacts of low External Validity?
• What if Ecological Validity is low?