Matters of Design

1
Matters of Design
2
Internal Validity

• Internal validity the extent to which we can
  accurately state that the IV produced the
  observed effect

3
External Validity

• External validity the extent to which the
  results of an experiment can be applied to and
  across different persons, settings, and times

Is it more important to be able to generalize the
results to other populations, settings, etc.?
4
What do we mean by control?

• Exerting a constant influence.

5
Control Techniques

• Randomization
• Random selection
• Random assignment
• Experimental control
• Elimination
• Make IV
• Matched groups
• Control group

6
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

7
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

8
Experimenter Expectancy

• Experimenter bias (expectancy effect, Pygmalion
  effect, self-fulfilling prophecy effect)
• Solution
• Training practice
• Run all conditions simultaneously
• Automated procedures (videotape, computer
  scoring)
• Blind the investigator

9
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

10
Subject Expectancy

• Hawthorne effect (placebo effect, guinea pig
  effect, novelty effect, gee whiz effect)
• Rating Effect (Halo effect, overrater error,
  underrater error, central tendency error)
• Avis Effect
• John Henry effect

11
Subject Expectancy Solutions

• Deception
• Placebo
• Blind the subjects (double-blind)
• Select subjects without prior associations
• Inform the subjects of their roles

12
Expectancy Interactions

• Does the investigator respond to male subjects
  better than female subjects?
• Do the subjects respond better to one
  investigator better than another?
• Solution
• randomization

13
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

14
History

• Outside influences that occur between the pretest
  and posttest in addition to the treatment
• Solution
• Randomization
• Control group
• Length of study

15
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

16
Maturation

• Morphological changes that occur during the study
  that change performance or behavior
• Solution
• Randomization
• Control group
• Length of study

17
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

18
Instrumentation

• Bias stemming from the process of measurement in
  the research setting
• Types
• Measurement device influences
• Testing influences

19
Measurement device influences

• Measurement error of devices
• Instrument sensitivity

20
Measurement device influences

• Reliability repeatability, consistency of
  instrument
• Validity - accuracy
• Objectivity consistency of rater, measurer

21
Reliability
85
100
115
Score on test
22
Types of Reliability
23
Reliability

• Assessment
• Pearson r correlation
• Reliability of scores, not tests
• Homogeneity of scores
• Report for current study conditions

24
Validity

• Construct Validity adequacy of the operational
  definition of variables

25
Indicators of Construct Validity of a Measure

• Face Validity content of measure appears to
  reflect the construct being measured
• Criterion-Oriented Validity scores on the
  measure are related to a criterion
• predictive scores on measure predict behavior
  on a criterion
• concurrent people in groups known to differ on
  the construct score differently on measure
• convergent scores on the measure are related to
  other measures of the same construct
• discriminant scores on the measure are not
  related to other measures that are theoretically
  different

26
Instrument Sensitivity

• Ceiling floor effects
• Must consider sample characteristics (age, skill
  level, etc.)

27
Measurement device influences

• Solution
• Calibration
• Use of multiple observers
• Training of observers
• Establishment of reliability, validity, and
  objectivity of instruments

28

• A researcher developed a test of the trait
  optimism and then compared students who were
  judged to be happy versus not happy to see if
  they had different levels of optimism as expected
  if the Happiness Theory were credible.

29

• A researcher administered the new Test of
  Recreation Programming Potential to graduating
  recreation majors and then correlated the test
  scores with recreation directors of programming
  effectiveness after one year on the job to see
  whether the test scores were related to job
  performance.

30

• A researcher tested a sample of students using
  the Occupational Interest Test. The test was
  administered again six months later to see if
  interests were fleeting.

31
Instrumentation

• Bias stemming from the process of measurement in
  the research setting
• Types
• Measurement device influences
• Testing influences

32
Testing Influences

• Learning effect
• Order effect
• Pretest sensitization

33
Testing Influences Solution

• Familiarize with testing
• Randomize order (counterbalancing)
• Eliminate pretest
• Use subjects unaware of measurement
• Use Solomon 4-group Design
• Randomization
• Control group

34
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

35
Regression

• Tendency of repeated measurements to shift away
  from extreme values toward more normal ones
• Ceiling/floor effect
• Solution
• random selection, random assignment, control group

36
Blood Lipid Study
Pre Post
Mean 200 180.5 SD 66 49
37
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

38
Mortality

• The differential loss of subjects from various
  experimental groups
• Solution
• Large sample size
• Inform subjects of importance
• Incentives
• Length of study

39
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

40
Selection

• Errors resulting from differences in groups at
  beginning of study
• volunteer subjects
• intact groups
• Solution
• Random selection
• Random assignment
• Control group

41
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

42
Intra-subject Interaction

• Subjects influenced by other subjects
• Solution
• Ask subjects not to discuss study

43
Threats to Internal Validity

• Experimenter expectancy
• Subject expectancy
• History
• Maturation
• Instrumentation

• Regression
• Mortality
• Selection
• Intra-subject interaction
• Factor interaction

44
Factor Interaction

• The cumulative effect resulting from the
  combination of two or more threats.
• selection maturation
• selection history
• selection instrumentation
• history subject mortality
• selection subject mortality