Rewardrelated Neural Circuitry

1
Reward-related Neural Circuitry

• Julie Fiez, Ph.D.
• Departments of Psychology Neuroscience

2
Acknowledgements
Karin Cox Mauricio Delgado Corrine Durisko Mary
Conway Kate Fissell Chris May Alison Moed Susan
Ravizza Elizabeth Tricomi Steve Wilson
Bruce McCandliss James McClelland Athanassio
Protopapas Michael Sayette Andy Stegner
3
Dopamine Plays a Crucial Role in Reward-Related
Processing
Dopamine neurons respond to unexpected rewards.
Schultz et al. (1997). Science, 2751593-1599
Animals will work for delivery of drugs that
stimulate dopaminergic signalling.
4
Dopamine neurons project into distinct
fronto-striatal-thalamic loops
PFC
Orbitofrontal
THALAMUS
Dorsal Striatum (Caudate/Putamen)
Ventral Striatum (Nucleus Accumbens)
SNpc
VTA
5
Is Dopamine a Pleasure Signal?Liking vs.
Wanting
Cannon Bseirki (2004). Physiol Behav,
81741-7428.
6
Does Dopamine Support the Development of
Associations That Yield Increased Reward?
Even simple behaviors have multiple opportunities
for habit formation
light -gt lever press -gt food
delivery stim -gt response -gt outcome
Stimulus-outcome consequences (feedback) may
alter the value of neutral stimulus Response-outc
ome consequences may alter motor (and
cognitive) activity Stimulus-response-outcome c
onsequences may alter the relationship between a
stimulus a response Stimulus-response after
learning, behavior may be no longer governed by
outcomes
7
The Dopamine Signal May be Ideal to Support Such
Reinforcement Learning
Egelman et al. (1998). J Cogn Neurosci,
10623-30.
Schultz Montague(1997). Science, 2751593-1599
8
Do ventral dorsal striatum support different
aspects of reinforcement learning?
(e.g., Elliott et al., 2004 ODoherty et al.,
2004 Robbins et al., 1992)

• Training
• initial Pavlovian training
• CS light paired with drug delivery
• CS- clicks presented non-contingently
• 2nd order conditioning
• each lever press leads to light (CS)
  delivery
• 10 lever presses earns drug delivery
• drug delivered after a fixed (20 min)
  interval

Ito et al. (2002). J Neurosci, 226247-6253
9
Emerging Issues for fMRI
What striatal response properties are observed
in humans? Are there dissociations between
ventral vs. dorsal activity that converge with
the animal literature? What insight might such
dissociations provide into the nature of human
reward-related processing?
10
Do striatal regions respond to the unpredictable
delivery of reinforcers?
Yes, especially at or near the nucleus accumbens
Schultz et al. (1997). Science, 2751593-1599
Berns et al. (2001). J Neurosci, 211793-2798
11
Do striatal regions respond to delivery of
unexpected monetary outcomes?
No significant differences between reward,
punishment, and neutral trials were observed.
12
How might we reconcile these findings?
The study by Berns colleagues involved the
delivery of a primary reinforcer.
The oddball study made use of an abstract,
unconditioned cue (red or green arrow) to
indicate gain or loss of a secondary reinforcer
(delivered later).
Schultz et al. (1997). Science, 2751593-1599
Will delivery of an unexpected, conditioned cue
activate the ventral striatum?
13
Unexpected delivery of conditioned cues
Male heavy smokers (at least 20
cigarettes/day) Participants abstained from
smoking for 8 hours Compliance assessed by
expired CO Three neutral and one conditioned
cue exposure
Notepad
Golf ball
Notepad
Golf ball
Run 1
Run 1

Runs separated by approximately 23 minutes

Runs separated by approximately 23 minutes
Tape (neutral)
Cigarette
Tape (neutral)
Cigarette
Run 2
Run 2
14
Interim Summary
Consistent with prior neurophysiological
findings, the ventral striatum responds to the
unexpected delivery of primary reinforcers and
conditioned cues. These findings support claims
that the ventral striatum plays an integral role
in reward-related signaling under normal
conditions, and that it may contribute to
pathological states such as addiction.
15
What about the dorsal striatum?
Reward-responsive dopamine neurons also project
to the dorsal striatum.
The dorsal striatum has typically been observed
to respond weakly in paradigms that drive the
ventral striatum. However, robust reward-related
differences have been found in the dorsal
striatum using other paradigms.
16
The Card Guessing Task
Indicated monetary gain
7
?
Indicated monetary loss
Trial Events
Card
Outcome
Card
REWARD TRIAL

Scanning Sequence
Scan 1
Scan 2
Scan 3
Scan 4
Scan 5
Scan 1
Choice Period
Post-Outcome Period

TEMPORAL SEQUENCE
3
15
0
6
9
12
Seconds
17
Robust dorsal striatal activity is found during
the card guessing task
18
Which aspects of the task account for activation?
Oddball task
Guessing task
7
?
7
?
Unlike the ventral striatum, delivery of
reinforcer or conditioned cue is not sufficient
to activate dorsal striatum. Activation
during guessing task shows such delivery is not
necessary.
Is it the mere need for an instrumental
response?
Blue circle single keypress
Or must there be a real or perceived
contingency between the the response the
outcome?
Yellow circle choose a keypress
19
The dorsal striatum is sensitive to perceived
response-outcome contingency.
20
Involvement in response-outcome signaling may
apply to complex situations.
Caudate Activity
LateTrials
Early Trials
21
Do the contributions of the dorsal striatum
extend to cold cognition?
22
The Development of Speech Categories May Be
Self-Organizing
When one neuron A participates in firing another
neuron B, the strength of the effect of A on the
firing of B is increased. - paraphrased from
Hebb, 1949 Or, put more simply Neurons that
wire together, fire together.
23
Once perceptual categories have been formed, can
they be reshaped?
Difficulties caused by a self-reinforcing
tendency to hear two speech sounds as the same,
thus

• Exaggerating the differences between sounds could
  overcome barrier.
• Learning should not require explicit feedback.

24
An Empirical Test of the Theory
Adaptive Training Condition
100
90
80
Pretest
70
60
Posttest
50
40
30
20
10
0
0.0
0.5
1.0
l
r
l
r
Anchor
Anchor
Anchor
Anchor
25
Is the model complete?
Difficulties caused by a self-reinforcing
tendency to hear two speech sounds as the same,
thus

• Exaggerating the differences between sounds could
  overcome barrier.
• Learning should not require explicit feedback.
• But what if feedback is given?

26
With feedback, both the adaptive and fixed
techniques are effective.
(McCandliss et al., 2002)
27
Could the differences in learning reflect the
engagement of the dorsal striatum?

• Hypothesis
• In a motivated learner, performance feedback may
  be rewarding (correct response) or non-rewarding
  (incorrect response).
• Outcomes may engage striatal reinforcement
  learning mechanisms.
• Perceptual representations and associated
  responses that lead to rewarding outcomes are
  strengthened.
• Test by having Japanese subjects perform the /r/
  vs. /l/ task with and without feedback.
• Compare activation in perceptual identification
  task to activation in the guessing task.

28
A comparison across tasks.
Guessing Task
Categorizaton Task
29
Increased Caudate Activation During Feedback
Training
The striatum is more active in the feedback as
compared to the no-feedback condition.
30
Performance Feedback Acts Like Gambling
Reward/Punishment
The activation is similar in location and pattern
to that observed with the guessing task.
31
Temporal cortex may be affected by top-down
outcome signals.
32
Can we see pre vs. post training differences?
No explicit task Subjects listen passively to
stimuli An oddball response is presented every
16-24 ms
Use fMRI to determine which areas of the brain
respond to the oddball stimulus. If the sounds
are perceived as the same, there should be no
response to the oddballs.
33
Examine the Neural Response to Native vs.
Non-native Phoneme Contrast

• Subjects native Japanese speakers (n9)

34
Before training, auditory regions responded most
to the native oddballs.
35
After training, the largest responses were to the
non-native oddballs.
36
Implications for Perceptual Organization

• The organization of perceptual categories may be
  mediated by both Hebbian-based and
  reinforcement-based learning mechanism.
• During development, both mechanisms may come into
  play.

Rewarding outcome
Adaptive input
Proportion of canonical syllables
Baseline
Social response
Extinction
Test periods (10 min)
Kuhl, Nature Neuroscience Reviews, 5831-843.
Goldstein et al., PNAS, 100830-835.
37
Feedback may invoke learning that cuts across
both implicit explicit memory tasks.
38
Implications for Normal Development
The striatum appears to be part of a
reinforcement learning system. This system may
use rewarding outcomes (broadly construed) to
shape
- perceptual representations of environmental
stimuli - affective (motivational) responses
evoked by stimuli associated
contexts - overt (motor) covert (?)
responses elicited by stimuli - episodic
memory associations or retrieval processes
39
Dysfunction/abnormal input into this system may
result in developmental disorders.
- susceptibility to drug abuse and drug
addiction
- OCD
- stress during early developmental periods
40
Conclusions
Ventral striatum is responsive to the mere
presentation of primary reinforcers and
conditioned cues thus, the ventral striatum may
play an important role in representing the
incentive value of stimuli. Dorsal striatum is
sensitive to whether there is a perceived
contingency between a response and an
outcome thus, dorsal striatum may contribute to
selecting and shaping behavior by associating
actions with their outcomes. The dorsal
striatum and prefrontal cortex may work together
to provide substantial cognitive control over
representations of incentive value induced by
stimulus events.
41
(No Transcript)
42
The dorsal striatal response is multi-faceted.
The outcome period shows a sensitivity to outcome
value
The choice period shows a sensitivity to
motivational state
Large Reward Trial
e
7
7
7
7
?
?
?
?
Positive Feedback 4.00
Positive Feedback 4.00
Positive Feedback 4.00
Positive Feedback 4.00
Low reward trial
?
?
?
?
3
3
3
3
Positive Feedback 0.00
Positive Feedback 0.00
Positive Feedback 0.00
Positive Feedback 0.00
Cue
Choice
-
Period
Outcome
Feedback
Cue
Choice
-
Period
Outcome
Feedback
Cue
Choice
-
Period
Outcome
Feedback
Cue
Choice
-
Period
Outcome
Feedback
High or Low
High or Low
43
Caudate neurons show selective activation for
trials in which the monkeys movement will be
rewarded
(Schultz, Tremblay, and Hollerman, 2000)
44
Modulation of cue-induced craving
All participants refrained from smoking for 8
hours 10 participants expected to smoke midway
through scanning session 10 participants did not
expect to smoke
Notepad
Golf ball
Notepad
Golf ball
Run 1
Run 1

Runs separated by approximately 23 minutes

Runs separated by approximately 23 minutes
Tape (neutral)
Cigarette
Tape (neutral)
Cigarette
Run 2
Run 2
Expectancy modulates the cue-induced response
affects measures self-reported craving affects
facial expressions evoked in response to a
conditioned cue affects performance on tasks
requiring executive control
45
The dorsal striatum may act in concert with
prefrontal regions.
Leon Shadlen (1999). Neuron, 24415-425.
46
Expectancy modulates prefrontal activity
Dorsolateral PFC
Ventrolateral PFC
47
The dorsal striatum is sensitive to perceived
response-outcome contingency.
48
Behavioral Results
After the imaging study, subjects completed
extended training.
With presentation of fixed (non-adpative
stimuil), robust learning occurred only with
feedback.