Engineering Psychology PSY 378S

1
Engineering PsychologyPSY 378S

• University of Toronto
• Spring 2004
• L3 Signal Detection Theory

2
Outline

• Signal detection theory
• Example App
• Background
• Detection experiment
• Theory distributions of neural activity
• Sensitivity and bias
• Factors affecting bias
• ROC Curve
• Return to Example

3
Example Baggage Inspection

• Suppose have two baggage inspection systems

Acme
SuperX
4
Example Baggage Inspection

• With Acme, inspectors produce higher hit rate (95
  out of 100 targets vs. 85 for SuperX)

Acme
SuperX
5
Signal Detection Theory

• SDT developed by engineerpsychologist
• Green Swets at U. of Michigan
• Psychologist trying to measure absolute
  thresholds--what was the minimum amount of sound
  necessary to be detected--wanted to measure
  sensitivity
• How sensitive is the human organism in detecting
  sound
• Had prob.--how to parcel out effects of
  bias--some observers more inclined to say they
  heard a tone than others

6
Signal Detection Theory

• Take-home message
• This is the main feature of SDT--separating out
  our measure of sensitivity from bias
• Turns out to have application in many domains
  medical diagnosis, airport inspection, eyewitness
  testimony, industrial inspection, vigilance

7
Signal Detection Theory

• Imagine the following experiment
• On each trial a low level sound is either present
  or absent (50/50)
• You must judge its presence/absence
• 2 x 2 matrix of different responses results

8
Signal Detection Theory
9
Signal Detection Theory

• If the tone loud enough or if participant has
  very sensitive hearing then all responses Hits or
  CRs
• But this is a threshold experiment-- Therefore
  diff. to detect tone
• Sometimes FAs, Misses will occur

10
Neural Activity

• Imagine we can directly measure the neural
  activity corresponding to presence or absence of
  tone
• Variation in activity may be a direct result of
  presence of tone
• May also result from noise in the environment, or
  from neural noise
• Evidence variable X

11
Neural Activity
Evidence variable X vs. time

• Intensity of signal (S.L.meter) vs. time

12
Distributions

• Can construct signal distribution from On
  region and noise frequency distribution from
  Off region

Signal On Frequency Count
X
Signal Off Frequency Count
X
13
Distributions

• Draw separate signal and noise distributions
• Signal distribution sometimes called
  SignalNoise Distribution
• Draw neutral criterion onto distributions, label
  sections

14
Sensitivity

• Distance between two means is d'
• d' measure of sensitivity of observer, or
  strength of signal relative to background noise
  (how easy the task is)
• Bringing distributions together lowers d',
  increases both kinds of errors, misses and false
  alarms

15
Sensitivity
d
16
Bias

• Formal measure of bias is ?
• Height of signal distribution over height of
  noise distribution at criterion point (Xc)
• Thus ? P(XS) / P(XN)

17
Bias

• Factors that affect criterion
• 1) individual differences
• 2) probabilities
• 3) payoffs

18
Individual Differences

• Some people more inclined to say they hear tone
  than others
• low criterion (threshold) -- liberal
• high criterion -- conservative

19
Probabilities

• Criterion can be adjusted by adjusting how likely
  the signal is
• ?(opt) P(N)/P(S)
• Lots of stimulus present trials, you will lower
  criterion
• Few stimulus present trials, you will raise
  criterion

20
Payoffs

• ?(opt) (V(CR)C(FA)) / (V(H)C(M))
• (Assume signal and noise are equally likely)
• If I pay you 100 for detecting a signal, but
  penalize you only a little for a FA (say 1) then
  you will produce a low criterion--call almost
  anything a signal
• If I pay you 1 for detecting a signal, but
  penalize you 100 for a FA then you will produce
  a high criterion--be absolutely sure before you
  call something a signal

21
Putting It Together

• ?(opt) P(N)/P(S) X (V(CR)C(FA)) / (V(H)C(M))
• Once you know the probabilities or the payoffs,
  you can predict an optimal beta value
• This means you can compare peoples performance
  against this optimal value

probabilities
payoffs
22
Sluggish Beta

• People dont optimize as they should especially
  at extremes, and especially with probabilities
• Called sluggish beta

23
Sluggish Beta
Conservative
Liberal
Conservative
Liberal
24
Sluggish Beta

• Why?
• People dont like to respond with lots of Y or
  N in a row
• (for probabilities)
• People have to keep track of how often signal
  occurred may fail to attend, forget,
• People dont understand probabilities--they tend
  to overestimate rare events
• People often show bias in proportion judgments
  (internal psychophysics may reflect biases in
  perceptual judgment) (Hollands Dyre, 2000)

25
Confidence Ratings

• Time-consuming to replicate an experiment several
  times, varying the criterion using prob/payoffs
• Can use confidence ratings instead
• Built in bias measure

26
Confidence Ratings

• With three levels can obtain two beta settings
• Can use as many as six levels and computer can
  generate the ROC curve

27
ROC Curve

• If you run an experiment and obtain the 2 x 2
  table, you actually only need 2 of the numbers
• P(Hit) and P(FA)
• Other two are redundant (perfectly predictable
  from P(Hit) and P(FA))

28
ROC Curve (contd)

• Can plot the two values in a Hits by FAs data
  space
• If vary criterion obtain a number of points
• Connect the dots and obtain an ROC curve
• ROCReceiver Operating Characteristic

29
Probabilities
Receiver Operating Characteristic (ROC)
30
ROC Curve (contd)

• The ROC curve is an isosensivity curve--that is,
  it connects points of equal sensitivity with
  differing bias
• Tangent to curve at particular point indicates
  value of bias at that point
• Height of curve above positive diagonal along
  negative diagonal represents d'

31
Normal Probability Paper

• Can plot the ROC curve on normal probability
  paper in z units
• Has nice feature that ROC curve becomes straight
  line
• Difference between z(H) and z(FA) is d'
  --sensitivity

32
Problem

• However, if the slope of the ROC curve plotted in
  z units is different from the positive diagonal
  (45 degree slope), how do you know what d' is?
• Cant talk about a single d' , because it varies
  with beta
• One approach is to use da, which is d' at zero
  bias
• But you have probably violated the normality
  assumption 

33
Non-Parametric Approaches

• In that case there are a number of parameter
  free approaches to SDT
• e.g., (P)A-- area under triangle in Fig 2.7
• Can use the parameter free measures with just one
  data point

34
Other Measures of Bias

• C better than beta, because less sensitive to
  changes in d than beta
• C 0.5 (z(FA) z(H))
• Conservative biases produce ve C values (liberal
  biases ve)
• See et al., 1995 Psych Bulletin
• See et al., 1997 HF

35
In Practice

• You can compute any of these measures from pairs
  of hit and false alarm values

36
Take-Home Message

• This is the main feature of SDT--separating out
  our measure of sensitivity from bias
• We have independent measures of sensitivity and
  bias

37
Example Baggage Inspection

• Our two baggage inspection systems

Acme
SuperX
38
Example Baggage Inspection

• Inspectors produce higher hit rate with Acme(95
  out of 100 targets vs. 85 for SuperX)

Acme
SuperX
39
Not So Fast

• but also higher false alarm rate (5 non-targets
  out of 100 for Acme vs. 2 for SuperX)

??
Acme
SuperX
40
Example Baggage Inspection

• Sensitivity about the same for SuperX and Acme
  (d 3.2)

??
Acme
SuperX
41
Example Baggage Inspection

• Risk of liberal bias increasedlong airport
  lineups with Acme

??
42
Summary

• Signal detection theory
• Background
• Detection experiment
• Theory distributions of neural activity
• Sensitivity and bias
• Factors affecting bias
• ROC Curve