Controversies in Assessment

1
Controversies in Assessment

• Clinical vs. statistical prediction
• Validity of projective techniques
• Attention confined solely to Rorschach,
  especially Exners Comprehensive System for
  scoring interpreting Rorschach

2
CLINICAL vs. STATISTICAL PREDICTION
3
Stages of Prediction

• Two serial stages of prediction/judgment must be
  sharply distinguished (Meehl, 1954)
• Data gathering ( encoding)
• Data combining
• This literature is about data combining
• Example predicting college GPA
• Get SAT, high school GPA, recommend-ation
  letters, extracurricular activities
• combine potentially conflicting information
  somehow, to yield single prediction

4
Data Combination Two Methods

• Clinical (intuitive)
• Formal, algorithmic, mechanical
• Sometimes statistical formula
• Sometimes actuarial table
• Sometimes computer program, etc.
• No necessary connection between kinds of data,
  and ways of combining the data

5
One Must Choose Between Methods

• When you have to make a decision, the formula
  will sometimes give one prediction, and clinical
  judgment a different one
• You cant follow both
• Averaging the two predictions is just another
  kind of algorithmic prediction
• Selectively overruling formula is a clinical
  methodbecause clinician has the last word

6
How to Choose a Method?

• Use a different method for each caseBUT we dont
  know how to do this rationally, so this has to be
  rejected
• ?Always use method with greatest long-run
  accuracy
• So Which method gives greatest long-run
  accuracy?

7
Studies Reviews of Studies

• Sarbin (1927) said a clerk with formula
  outperforms counselor did study of GPA that
  showed this
• Meehl (1954) book reviewed 23 studies, almost
  uniformly favoring actuarial prediction
  tabulated studies again in 65
• Holt 1958 conceptual response offered ? new data,
  cited ? studies on point

8
Post-Meehl Literature Reviews

• Other reviews Sawyer (1966, also inter-ested in
  type of data being combined), Sines (1971),
  Wiggins (1981), Marchese (1992)
• All concluded clinician does same as, or worse
  than, actuary, in (essentially) all studies
• If conclusion is true, cheapness of actuarial
  approach generally argues for using it
• But is the conclusion true? And if its typically
  true, is it always true? (Are there pockets of
  clinical expertise?)

9
Grove et al. Study

• Many more studies (137 vs. 45, most before) that
  predicted human health or behavior
• Meta-analysis methodology used
• Results
• 129/137 studies, actuarial prediction as good as
  or better than clinical prediction (9 hit rate)
• 63 studies show actuarial superior (gt9)
• 8 studies show clinician superior, none of which
  replicated each other (whats predicted, whos
  predicting, data used to predict, etc.)

10
Why Do Clinicians Persist?

• Ignorance of literature
• Immediate reinforcement (pay, prestige)
• Lack of readily-to-hand mechanical prediction
  method for particular problem
• Belief they can personally beat the actuary
• Philosophical muddle-headedness about single-case
  probabilities

11
VALIDITY OF RORSCHACH
12
History of Rorschach in U.S.

• Favored among clinicians since 1940s in U.S.
• Deprecated by researchers from 1940s through
  1970s at least
• Seven competing systems, differing even in
  administration methods Beck, Klopfer, Hertz,
  Piotrowski, Rapaport, Schafer

13
Rorschach Criticisms 1940s On

• Unreliability of scores (especially ratio scores,
  e.g., CFFC Color-Form determinants)
• Subjective scoring interpretation
• Scores dependent on number of responsesthis
  cant be fixed with present instructions/
  stimulus set
• Nonexistent norms
• Most scores unvalidated/very poorly validated

14
John Exner

• Many competing systems synthesized and replaced
  by Exners Comprehensive System (CS) in the 1970s
  based on his work starting in the 1950s
• He brought out three-volume book in successive
  editions (now 3rd), dealing with adult (and
  later, adolescent) Rorschach interpretation

15
Exners Studies and Reputation

• Hundreds of studies cited in his 3-vol. book
• Many studies on CS in journals (mostly in one
  pro-Rorschach journal, Journal of Personality
  Assessment)
• Conventional wisdom by 1980s Exner saved
  Rorschach from scientific graveyard by putting
  its scoring and interpretation on sound basis
• CS became only Rorschach system taught in most
  accredited clinical programs

16
Exners Contributions

• Makes administration, recording uniform
• Makes scoring of some categories more reliable
• Addresses some earlier Rorschach score
  criticisms, except
• confounding of many scores with R (number of
  responses), but
• this cant be fixed within existing Rorschach
  format (i.e., only 10 cards or so)

17
Exner Contributions, contd

• Offers some normative data, but
• sample not too large (about 700)
• not a census-like sample, as with WAIS,
  Stanford-Binet, MMPI-2, etc.
• unclear how sample was recruited
• only selected summary statistics available on
  norm group members you cant check score
  distributions, etc.

18
Cracks in the Edifice

• Wood, Nezworski, Stejskal noticed great majority
  of studies Exner cites in his books
• Are written by him, and
• Were never published in any peer-reviewed venue
• Each investigator identifies an area (e.g.,
  narcis-sism) to review
• Each independently writes to Exner for copies of
  the unpublished studies
• No studies are sent

19
Cracks Widen, Let in Daylight

• Requests repeated finally, assistant writes that
  she cant send Exners studies because the
  majority of these are not written in a
  publishable form. Instead, they usually include
  a brief statement concerning the methodology of
  the study handwritten on sheet of paper, or
  computer printout of analyses

20
More Bad News

• Since then, Exner has also refused to let those
  outside circle of supporters see his raw
  normative data
• Recently, it was disclosed that 221 (40) of his
  normative subjects were mistakenly duplicated in
  data set this wasnt detected for decades
• Others community/normative samples (e.g.,
  Schaffer Erdbergs) dont resemble Exners for
  some key scoresExners norms con-siderably
  overpathologize patients

21
Early Rorschach Meta-Analyses

• Parker, Hunsley Hanson (1976)
• Method Find studies in two main journals for
  several years on WAIS, MMPI, Rorschachalmost all
  Rorschach studies from one specialty journal
• Compare reliability, validity coefficients
  between instruments
• Note Doesnt cover Exner system

22
Early Meta-Analyses, contd

• Parker, et al. report conclusions based on
  selected studies (selection said not to have been
  based on study outcomes)
• Rorschach as reliable as WAIS (!)a conclusion
  essentially no one believes
• Rorschach as valid as MMPI (based on 5-study
  subset)
• Wierzbicki chapter (mistakenly, in my opinion)
  lauds study

23
Early Meta-Analyses, contd

• I wrote Parker for explanation of selection
  method. He referred me to Atkinson, who couldnt
  explain it precisely. I couldnt understand
  selection I couldnt replicate it
• Garb, et al. (1998) correct Parker et al.s
  statistical errors, look at all studies,
  conclude
• Avg. validity MMPI .39, Rorschach .21

24
Recent Meta-analyses

• Hiller, et al. (1999) report average validities
  for new sample of MMPI Rorschach studies.
  These are not studies of the CS
• .30 MMPI, .29 Rorschach
• Garb, et al. correct Hiller, et al. statistics
• .37 MMPI, .26 Rorschach

25
Consensus Conclusions

• Neither Rorschach nor MMPI-2 very highly valid,
  on average
• Validities vary, with a couple of Rorschach
  scores (e.g., non-CS Rorschach Prognostic Rating
  Scale) pretty valid (.7), many other score have
  near-zero validity
• Most CS scores need more validation via well-
  designed studies in peer-reviewed journals
• Norms could stand considerable improvement