Research Design in Clinical Psychology

1
Research Design in Clinical Psychology

• Lecture 2
• Reliability, Validity, and Artifact/Bias
• (Chapters 2-4 in Kazdin)

2
Internal Validity

• To what extent can the intervention, and not
  other factors, account for study results
• History
• Maturation
• Testing (Practice)
• Instrumentation
• Statistical Regression
• Selection Bias
• Attrition
• these can occur across all groups or only to
  select groups
• Diffusion/Imitation of treatment
• Special treatment/reaction of controls

3
External Validity

• To what extent can the study results be
  generalized to other samples with different
  characteristics than the study sample

4
External Validity Examples 1

• Sample characteristics
• Differences b/n study sample and other samples
• Include age, gender, culture, education
• Stimulus characteristics and settings
• Extension across characteristics of the study
• Includes setting, experimenter,
  materials/apparatus
• Ecological validity?
• Reactivity of experimental arrangements
• Awareness of being in a study may affect behavior
• Are responses relevant to those who are not in a
  study
• Multiple-treatment interference
• Subjects receives multiple interventions
• Relevant to those who did not receive other
  interventions?

5
External Validity Examples 1

• Novelty
• Is effect due to newness
• Reactivity of assessment
• Similar to experimental arrangements, but focuses
  on awareness of what the measures are tapping
• Test sensitization
• Does pre-testing or the test itself (in the case
  of post-test sensitization) alter subject
  experience and responses
• Timing of Measurement
• When assessments are given could alter results

6
Parsimony

• Least complex explanations first
• Frequently, a threat to internal validity is most
  parsimonious
• When considering limitations in both internal and
  external validity, however, parsimony suggests
  that findings are the best statement of a
  relationship, unless there are clear reasons to
  think otherwise.

7
Why internal validity precedes external validity

• Cant have ExtVal, without IntVal
• It would be like asking, gee can we generalize
  these results we have no confidence in to a wide
  variety of individuals?
• One can still have important findings that
  elucidate basic principles without much ExtVal.
• A lack of generalization of a finding across
  samples may be very important and can span other
  research

8
Construct Validity

• What is the causal agent and conceptual basis
  underlying an effect (what is the intervention
  and why did it lead to change?)
• In clinical research methods, CV is different
  from in test construction where CV is the extent
  that a measure captures a construct of interest.

9
Threats to construct validity

• Contact time
• Placebo effects
• Nonblind (single and double) designs
• Single operations and narrow stimulus sampling
• Is the effect due to the selected IV
• Does other aspects of the intervention have an
  effect beyond the aspect identified by the
  experimenter
• Expectancies
• Cues and Demand characteristics
• Others exist based on conceptual relevance

10
Statistical Conclusion ValidityAKA Data
Evaluation Validity

• Refers to the facets of the quantitative
  evaluation that influence the conclusions reached
  about experimental conditions and effects (to
  what extent are real effects demonstrated and
  interpreted)
• Does one understand the stats used
• Has one done stats correctly

11
Rejecting the Null Hypothesis I

• Alpha probability of rejecting the null when
  you shouldnt (Type 1 error)
• Saying groups are different when in reality are
  same
• Beta probability of accepting the null when you
  shouldnt (Type 2 error)
• Saying groups are same when in reality are
  different
• Power probability of correctly accepting or
  rejecting the null

12
Rejecting the Null Hypothesis II

• Standard deviation variability around mean
• sqrt (each observation mean)2 / (N -1)
• SS/df
• Effect size Beyond significance, it is the
  magnitude of difference b/n groups
• Expressed in terms of SD units
• ES (M1 M2) / SD

13
Threats to statistical conclusion validity

• Low power
• Inability to detect real differences
• Variability in the procedures
• Subjects in same group get different treatment
• Subject heterogeneity
• Subjects in same group differ on potentially
  confounding variables
• Unreliability of measures
• Multiple comparisons/error rates
• Experiment-wise error

14
Experimental Precision

• Controlling vs holding constant
• Tradeoffs

15
Sources of Artifacts and Bias
16
Rationale, scripts, procedures

• Imprecision in carrying out procedures
• Experimenter may not have clear a protocol
• Loose protocol effect
• Experimenter may ignore protocol
• Strategies for overcoming include
• developing a clear protocol
• actually testing the protocol out ahead of time
• Get feedback from participants
• Train experimenters together
• Document all deviations

17
Experimenter Expectancy

• Experimenters knowledge of study hypotheses may
  bias study implementation
• Strategies for overcoming include
• Keep experimenters naïve to purpose
• Double blind procedures for group assignment
• Success of these strategies can be empirically
  reviewed

18
Sample selection

• Convenience samples and Volunteers
• Do these sample differ in ways that affect the
  results or their generalizability
• May select certain types of convenience or
  volunteer samples to answer particular questions
• Attrition
• Difference in of dropouts across groups could
  have huge effect on results
• Reminders and Commitment tactics (backloading
  )
• Select those unlikely to drop-out?
• Plan ahead from past research

19
Other sources

• Experimenter characteristics
• Situational and context cues
• Demand characteristics clues on how to respond
• Subject roles (pg 95 in book)
• Data recording and analysis