Learning to make decisions with incomplete feedback

1
Learning to make decisions with incomplete
feedback
Brunswik Society Meeting Houston November 16, 2006

• Tom Stewart
• Jim Holzworth
• Jeryl Mumpower

2
Conditional probabilities of outcome feedback
Full
Conditional
p selection rate for subjects given conditional
feedback
Partial
Random feedback on same proportion of trials as
corresponding conditional feedback condition
3
Cue display for screening task
4
Weights for screening task
5
Value conditions
6
Decision scoring reminder
7
Score feedback after each block of 25 cases
8
Cue display for hiring task
9
Weights for hiring task
10
Independent variables

• Feedback (full, conditional, partial)
• Uncertainty (Re2 .5, .7, or .9)
• Base rate (.1, .2, .5, and .8)
• Payoffs -- Points lost for errors

11
Task construction Taylor-Russell vs. Detection
Theory

• Detection theory
• Create noise distribution
• Add d to every observation to produce signal
  distribution
• Set p(Signal) base rate
• Variance (Noise) Variance (Signal)
• Taylor-Russell
• Create Ye with desired R2
• Choose cutoff to yield desired base rate
• Variance (Noise) ? Variance (Signal) unless base
  rate .5

12
Base rates determined by thresholds on criterion
Base rate
.1
.2
.5
.8
r2 .7
13
Noise and signal distributions from
Taylor-Russell for different base rates
ra .83
ra .83, base rate .80
ra .83, base rate .10
Probability density
14
Distribution of signal and noise distributions
derived from Taylor-Russell, by base rate (ra
.83)
15
Procedure

• Step 1. Instructions
• Describe correct weights
• Describe payoffs
• Step 2. Make judgments and decisions
• 500 trials (20 blocks of 25 cases each)
• Make judgment, then yes/no decision
• Outcome feedback correct decision on each trial
  (full), trials when decision is Yes
  (conditional), or random trials (partial)
• Scores and accuracy feedback after each block
• Performance bonus up to 10
• Step 3. Subjective reporting
• Subjective weights, accuracy, threshold, percent
  correct, subjective base rate

16
Performance depends on

• Accuracy of judgment (sensitivity)
• Predictability of environment
• Irreducible uncertainty
• Fidelity of information system
• Judgment process
• Weights, function form, organizing principle
• Reliability of information acquisition and
  processing
• Threshold (bias)
• Threshold location
• Reliability of threshold use
• Threshold movement
• Threshold violations

17
Results Decision threshold

• Selection rate proportion of Yes decisions
• Selection rate for last 250 cases indicates what
  subjects learned.
• Consider the following condition
• Base rate .5
• Values equal penalty for false positives and
  false negatives
• Uncertainty .7 (Re2)

18
Results Decision threshold

• Base rate .5
• Values equal penalty for false positives and
  false negatives
• Uncertainty .7 (Re2)

Conditional probabilities of feedback
19
Results for last 250 cases hiring task (UConn),
base rate .5, equal values
SDT representations of tables
A
Tables B, C, and D have essentially the same
graph.
B, C, D
Which condition produced which table?

• Full
• Conditional
• Partial
• Ideal

20
Results for last 250 cases hiring task (UConn),
base rate .5, equal values
SDT representations of tables
A
Tables B, C, and D have essentially the same
graph.
B, C, D
Which condition produced which table?
B. Full A. Conditional C. Partial D. Ideal
21
Mean selections rate (proportion of positive
decisions) for last 250 casesEqual
penaltiesRe2 .7
UConn, hiring task
UAlbany, screening task
UConn, screening task
22
Accuracy/Sensitivity screening task (UAlbany)
data, Az, equal values
23
End