Validity Reliability Control

1
ValidityReliabilityControl
2
Establishing Cause and Effect

• Temporal Precedence
• Cause has to happen before the effect
• Covariation of Cause and Effect
• Show there is a relationship between X and Y
• No Plausible Alternative Explanations
• Any other cause that can bring about the effect

3
Establishing Cause and Effect
4
Validity and Reliability

• Measure-Related Concerns
• Construct Validity
• Are you measuring what you think youre
  measuring?
• Reliability
• Is your measure consistent in its evaluation of
  the same individuals?

5
Types of ReliabilityInter-rater reliability

• Assesses the degree to which different
  raters/observers give consistent estimates of the
  same phenomenon
• agreement
• Correlation between observers scores

6
Types of ReliabilityTest-retest

• Correlation between two observations (i.e., set
  of scores) on the same test administered to the
  same (or similar) sample on two different
  occasions
• Assumes no change in construct being measured
• Time between observations is crucial

7
Types of ReliabilityParallel-forms

• Correlation between two observations (i.e., set
  of scores) on parallel forms of a test
  administered to the same sample
• Requires generation of many items that measure
  the same construct
• Two forms of test can be used independently
• Assumes randomly divided halves are equivalent

8
Types of ReliabilityInternal consistency

• Single test administered to a sample on one
  occasion
• Assesses the consistency of the results for
  different items for the same construct within the
  measure
• Average inter-item correlation
• Average item-total correlation
• Split-half reliability
• Cronbachs Alpha

9
Relationship between reliability and validity
10
Validity and Reliability

• Method-Related Concerns
• Internal Validity
• Is the independent variable the only possible
  explanation of the results shown?
• External Validity
• How far can the study generalize?

11
Validity

• Internal Validity
• Is the independent variable the only possible
  explanation of the results shown?
• Protecting internal validity allows us to
  eliminate potential alternative explanations for
  the outcome of an experiment
• Extraneous variables
• Influence the DV without your intent or consent!

12
Experimental Control

• Extraneous Variables
• Nuisance variables
• Make the effects of the IV more difficult to
  determine by affecting variation within groups
• Participant characteristics
• Unintended influences of experimental situation

13
Experimental Control

• Extraneous Variables
• Confounding variables
• Vary systematically with the IV
• Influence the difference between groups
• The differences observed between groups could be
  attributed to the IV or the confound

14
Experimental Control

• Extraneous Variables
• Experimenters
• Physiological differences
• Rosenthal effects

15
Experimental Control

• Extraneous Variables
• Participants
• Demand characteristics
• Good participant effect
• Response biases

16
Internal Validity

• Exercise control to protect internal validity
  before the experiment is conducted
• Random Assignment
• Elimination
• Constancy
• Counterbalancing
• Single- and Double-blind Experiments
• Altering Response Sets

17
Basic Control Techniques

• Randomization
• Individuals have an equal chance of being
  assigned to any group in the experiment
• Allows us to assume groups are roughly equivalent
  before administering the IV

18
Basic Control Techniques

• Elimination
• Extraneous variables are completely removed from
  the experiment

19
Basic Control Techniques

• Constancy
• An extraneous variable is reduced to a single
  value experienced by all participants
• Standard control technique
• Hold environment, temperature, lighting constant

20
Basic Control Techniques

• Balancing and Counterbalancing
• Balancing
• Achieves group equality by distributing
  extraneous variables to all groups
• Counterbalancing
• Controls order effects by presenting treatments
  in a different sequence
• Order or sequence effects
• Carryover effects
• Differential carryover

21
Basic Control Techniques

• Within-Subject Counterbalancing
• Within-Group Counterbalancing
• Each treatment must be presented to each
  participant an equal number of times
• Each treatment must occur an equal number of
  times at each testing session
• Each treatment must precede and follow each of
  the other treatments an equal number of times

22
The Cola Wars!

• Participant 1
• Participant 2
• Participant 3
• Participant 4
• Participant 5
• Participant 6

• P C
• P C
• P C
• C P
• C P
• C P

23
The Cola Wars!

• Participant 1
• Participant 2
• Participant 3
• Participant 4
• Participant 5
• Participant 6

• P C DP
• P DP C
• C P DP
• C DP P
• DP P C
• DP C P

24
Basic Control Techniques

• Single- and Double-Blind Experiments
• The experimenter (single-blind) or both the
  experimenter and participants (double-blind) are
  unaware of the participants condition

25
Basic Control Techniques

• Altering response sets
• Rewrite survey items so that some negative
  responses indicate agreement and some positive
  responses indicate disagreement
• Check for socially desirable responses in your
  survey and/or potentially undesirable cues

26
Internal Validity

• Exercise control to protect internal validity
  before the experiment is conducted
• Assess the internal validity of the experiment
  afterward

27
Threats to Internal Validity

• History
• Significant events occur between measurements

28
Threats to Internal Validity

• Maturation
• Participants change over time in the experiment

29
Threats to Internal Validity

• Testing (Pretest Sensitization)
• Repeated tests cause changes in DV practice
  effects

30
Threats to Internal Validity

• Instrumentation
• Changes in measurement due to changes in the
  measurement device (human or machine)

31
Threats to Internal Validity

• Statistical regression
• Extreme scores regress toward the mean
• Difficult to maintain extreme scores over
  repeated measures

32
Threats to Internal Validity

• Selection
• Selecting participants into groups that were not
  equivalent before the experiment

33
Threats to Internal Validity

• Attrition or Mortality
• Experimental dropouts

34
Threats to Internal Validity

• Diffusion of treatment
• Participants in one group communicate information
  from the IV to the other group

35
Threats to Internal Validity

• Interactions with Selection
• Maturation
• History
• Instrumentation

36
How Important is Internal Validity?

• It is the most important property of any
  experiment
• An experiment without internal validity cannot
  have external validity
• With no internal validity, you can have no
  confidence in your results

37
External Validity

• How far can you generalize your results beyond
  your experiment?
• 3 types
• Population
• Environmental
• Temporal

38
Threats to External Validity

• Testing/Treatment Interaction
• Pretest may make reaction to the IV different
  from those not tested
• Selection/Treatment Interaction
• Effect is found only for a specific group of
  participants

39
Threats to External Validity

• Reactive Arrangements
• A la Hawthorne effects
• Demand characteristics
• Participant Characteristics
• Specific animal species

40
(No Transcript)
41
Threats to External Validity

• Reactive Arrangements
• A la Hawthorne effects
• Demand characteristics
• Participant Characteristics
• Specific animal species
• Experimental participant populations
• College students
• White male Americans

42
Is External Validity Necessary?

• May want to find out if something can happen
• We may be predicting from the real world to the
  lab
• If we can show something happens in the labs
  unnatural setting, we may have more confidence in
  the phenomenon
• We may study something without a real-world
  analogy

43
Validity

• Measure-Related Concerns
• Construct Validity
• Reliability
• Method-Related Concerns
• Internal Validity
• External Validity