Attention

1

CNS186 - Face Gaze 030408 Faces, Gaze, and
higher-order
2

• 1. Biological functions of faces
• - Expression / perception / memory.
• - Categorical perception / identification /
  emotional cognition.
• Social interaction, non-verbal and verbal
  communication,
• attractiveness.
• - Biological significance of facial
  attractiveness?
• a) Darwinian
• b) Incidental effects of sensory bias from
  adaptation to object
• perception.

3
c) Adaptationist view (Thornhill Gangestad,
'01) Facial features for attractiveness
judgments as special-purpose adaptations to
discriminate the mate value (i.e. fitness,
health). Three lines of evidence 1)
Symmetry. Early supporting evidence, though
contaminated. (Rhodes, et al. '98, Mearly,
et al. '99 ) 2) Averageness (related to
Leopold's prototype theory Rhodes, et
al. '99) 3) Secondary sex traits as hormone
markers.
4

• 2. Face perception as a specific "module"
• Face has traditionally been regarded as a
  specific biological module,
• just as biological motion and speech perception.
  Though not as
• popular recently, some latest brain imaging
  evidence are not
• necessarily inconsistent with this view.
• - Evidence for specialty of face processing
• a) Holistic (neuropsychology, Lincoln's face,
  visual search, upside-
• down effect, etc.)
• b) Unusual memory (recognition) (Bahrick, et al.
  '75)
• c) Prosopagnosia (face selective cognitive
  deficit Tipett, et al.'00)
• d) Development (infant studies Slater Quinn,
  '01, Mondloch,
• et al., '99, Lundy, '00)
• e) Specific brain locus (Electrophysiology
  Rolls, Perret, etc. fMRI
• PET Kanwisher, et al., 99), fusiform
  gyrus.

5
(No Transcript)
6
Vegitables, or?
The Illusion Live 2000
7
(No Transcript)
8
Thacher Illusion
Weird face ?
9
Same, or different person?
10
(No Transcript)
11

• 3. Face as a non-specific complex stimulus?
• fMRI counter-evidence against the
  face-specificity. The "face area"
• (the fusiform gyrus) may not be as specific as
  believed, it may be
• specialized for fine object identification
  (Gauthier, et al, '99, and
• many more).
• Psychophysics prototyping and averaging,
  "anti-Adam effect"
• (Leopold, '01). Faces behave just as other
  complex yet familiar
• visual stimuli.
• -Subliminal influence of facial expressions. A
  solid chunk of
• evidence, both from patients and normals.
• Explicit processing evoked significantly
  more activity in temporal lobe cortex than
• implicit processing, whereas implicit
  processing evoked significantly greater activity
  in
• amygdala region. (Critchely, et al., 00
  Whalen, et al., 98)
• Also see Hacjin Kims unpublished data later.

12
- A third view distributed encoding/decoding.
(Huxby, et al., 2001)
13

• Locations of face-selective neurons
• in the macaque (summary of 8
• studies)

(b) Locations of neurons selective for facial
expressions (open circles), identitiy (close
circles), both (half- filled), or interaction
(squares).
(Haxby, et al. 00)
14
Locations of face-responsive regions in the
fusiform gyrus (PET and fMRI studies)
Haxby et al. (PET), Clark et al. (fMRI) face
vs. non-sense pictures. Sergent et al. (PET)
Facial identity task. Kanwisher et al.,
MacCarthy et al. (fMRI) face vs. non-face.
(Haxby, et al. 00)
15
Regions activated by faceless animals and faces
overlap. The former weaker, yet broader.
(Haxby, et al. 00)
16
Regions showing enhanced responses to birds or
cars in experts. Right occipital fusiform
face- responsive regions (slices 2-4). Right
left parahippocampal place areas (slices 4,5).
Specialized for faces, or specialized for fine
pattern discrimination?
(Haxby, et al. 00)
17
There are face- and house-responsive regions
identified in inferior occipital ventral
temporal areas. Inversion effect on the
responses to houses in the house-responsive
regions (b-left) shows the same pattern as the
inversion effect on the responses to faces in
the face-responsive regions. The only effect
specific to the face inversion was an increased
response to inverted faces in the
house- responsive regions (a-right).
(Haxby, et al. 00)
18
The core system (circuit) for visual analysis of
faces.
(Haxby, et al. 00)
19

• The intraparietal sulcus activated by averted
  gaze.
• The amygdala activated by fear in the face.
• The auditory superior temporal regions by hearing
  speech.

(Haxby, et al. 00)
20
Distributed system for face perception
(Haxby, et al. 00)
21
Negative Face Aftereffect (Leopold et al., 01)
22
Adaptation makes the neutral face appear to be an
anti-face (negative aftereffect).
23
Adapting to anti-face makes identification more
sensitive. Inversion makes identification
harder, but it still improves with adaptation
(left). Translation invariance (non-retinotopy)
of face adaptation effect (right).
24
Face Adaptation / Aftereffects

• Identification (Leopolod, et al., 01)
• Emotional expression (Halberstadt, et al., 01)
• Attractiveness (Rhodes, et al., 02, 03)

Face perception is just like other perception,
such as luminance, color, orientation, shape,
motion, depth, etc.
25
- Another look at the debate over face as a
unique module or not -
Familiarity-Novelty Paradox
? Both familiarity and novelty are critical
factors for preference decision making in
infants and adults. ? Sometimes we prefer
familiar stimuli, but other times we prefer
novel ones. ? What determines which would be
dominant? Are there separate brain mechanisms
for F and N which modulate preference decision
making? ? As a first step to resolve this
puzzle, we examined whether repeated
exposure could have different effects for
different categories of visual object.
(Shimojo,
Park, LeBon, Schleim Shimojo, VSS 07)
26
2 AFC
ltOne trialgt
0
1
Free visual inspection until decision
Reporting relative preference in a 7-point scale
using a mouse
Ready (1s)
ltNext trialgt
0
2
Free visual inspection until decision
Ready (1s)
Reporting relative preference in a 7-point scale
using a mouse
27
Behavioral study Experimental design
? Three object categories 1) Faces - Total 16
subcategories 2 age (young/old) X 2 gender
(male/female) X 4 race (African, Asian,
European, Indian) - Subjects were tested with 8
subcategories own race one other race
Young
Old
(Indian)
(African)
Male
(European)
(Asian)
Female
28
2) Geometric figures - 4 subcategories 2
complexity (simple/complex) x 2 symmetry
(symmetric/asymmetric) - subjects were
tested twice for each subcategory with different
sets of images
symmetric
asymmetric
simple
complex
29
3) Natural scenes - 8 subcategories mountain,
desert, ocean, river/lake, animal, flower,
food, sky
Mountain
Flower
Desert
Animal
30
I. Behavioral study experimental design
? 3 blocks in one session - Each object category
was tested separately in 3 blocks with testing
order counterbalanced between subjects ? 8
subcategories in each block - Each object
category was tested 8 times with images from
different subcategories ? 26 trials in each
subcategory - Each subcategory has 27
images - Attractiveness of each image was
pre-rated by independent raters (for face N12
for geometric shape and natural scene
N5) - Among 27 images, one with median
attractiveness was selected as old stimulus
- 26 pairs of old vs. new images were
prepared - Position of old stimulus was
randomized (L or R) - For each pair,
participants judged relative preference in a
7-point scale (-3 to 3)
31
Familiarity
Novelty
32
Summary results - Faces old vs. new, Nat.
Scenes old vs. new

• - Faces-familiarity lt--gt Nat.scenes-novelty
  (Geom.shapes-neutral)
• Upside-down nearly identical results --gt
  lower-level (local and
• not orientation-specific) coding?

33

• 4. Gaze and emotional judgment
• Are there somatic precursors preceding and
  preparing for
• conscious, cognitive emotional judgment, such as
  preference?
• - The answer is YES. Eg. Orienting behavior (gaze
  shift)
• (Shimojo, et al., '03)
• 5. Object perception and recognition
• How do humans and animals perceive object 3-D
  structure and
• identify it as a familiar object (from nearly
  2-D retinal images)?
• - Prototype and familiy resemblance (Rosh)
• - Classical theory (Marr, '82)
• 2 1/2 D-sketch matched with 3-D models of
  object in LTM.
• Significance of canonical axis.

34
6. Current issues in object perception (1) The
issue of constancy How could object
recognition be accomplished over a wide range
of view-dependent changes? (2) View-variant(i.e.
image-based) vs. view-invariant processes.
--gt Geon theory (Biederman) --gtAspect graph
(Koenderink) (3) Spatio-temporal liminations in
detection of visual events. --gt RB
(Repetition Blindness) (Kanwisher) --gt CB
(Change Blindness) (Rensink)
35
Back to face, again

• Emotion
• Behavioral/physiological state
• Experience (social psychology)
• Expression (facial, bodily)
• Perception / cognition
• (3), (4) ---gt Social, non-verbal
  communication

• Perception
• Hierarchy of neural visual pathways
• Feature vs. holistic
• Specialized vs. generic
• Generation vs. perception (analysis-by-synthesis,
  
• mirror neuron)?
• (5) Explicit vs. implicit

36

• Emotion is contagious.
• Mimicking facial expressions (yawning)
• (Yoshikawa, et al.)
• (2) Implicit priming by seeing faces (Shimojo, et
  al., 02)

37
Seeing Is Liking Gaze Cascade Effect towards
Preference Shinsuke Shimojo1,2 , Claudiu
Simion1, Eiko Shimojo3, Christian Scheier1 1-
California Institute of Technology 2 - NTT Comm.
Sci. Laboratories, Atsugi, Kanagawa, Japan 3 -
Bunkyo Gakuin University, Saitama,
Japan Supported by Genesis Research Institute,
Inc.
38
Introduction

• Emotions operate along the dimension of approach
  or aversion.
• Overt orienting may be intrinsically related to
  mental representation
• of the emotional state, i.e. feeling.
• How they are causally related?
• Classical psychology does one cry because one
  feels sad or
• one feels sad because one cries? (cf.
  James-Lange Theory )
• Psychophysics do we see something more because
  we like
• it, or we like it more because we see it?

39
Broader Background

• 1. Mere exposure effect (Zajonc, 1980).
• 2. Two-stage theory of emotion (Schacter, 1964).
• 3. Preferential looking in infants (Fantz, 1959).
• 4. Perceptual memory and eye movements (Althoff,
  1999).
• 5. Role playing paradigm in social psychology.

40
Task Compare freely and decide which face is
more attractive? Question Can we find any eye
movement pattern prior to the decision which
predicts it?
41
Experimental paradigm
Stimulus free inspection, no time limit
Fixation point (1 s)
Key press upon decision (dep. on task)
Faces in a pair were matched for attractiveness
(pre-experiment rating), gender, race and
age. Task Which face is more attractive?
42
Facegen / EyeLink 2 Example of Raw Data
Easy calibration, high spatial-temporal
resolution.
43
Faces Liking, Matched
0.9
Average for N5 subjects
R20.91 max - 0.83
Interpolated curve - 4 parameter sigmoidal
Chance level (50)
0.8
0.7
Likelihood that the chosen face is inspected
0.6
0.5
Decision
Decision
0.4
0.3
1.80
1.60
1.40
1.20
1.00
0.80
0.40
0.20
0.00
Time before decision (s)
44
Face Pairs Liking/Roundness/Disliking
Effect is not solely due to selection
1.0
Like
0.9
Max - 0.83
Not explained by selection bias
Round
0.8
Dislike
0.7
R20.91 max - 0.62
Likelihood that the chosen face is inspected
0.6
R20.80 max - 0.57
0.5
Decision
0.4
0.3
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
Time before decision (s)
45
The cascade effect - possible interpretations

• 1) The effect could be artifactual
• Gaze bias Attractiveness bias Selection
  bias
• (not necessary to create preference)
• 2) Our hypothesis (VSS 01) Gaze bias - bodily
  basis for subjective decisions

Exposure, perceptual facilitation
Attractiveness template
Gaze bias
cascade effect
Preferential looking
46
Do we see the effect with stimuli other than
faces?

• Fourier Descriptor-generated shapes (abstract,
  symmetric, complex) (Sakai Miyashita, 1991).
• Unfamiliar stimuli (no direct memory)- decision
  should be more difficult.
• Same experimental paradigm and task, but
  comparison is between abstract shapes.

47
Fourier descriptor pairs, attractiveness task
Fourier Descriptors Liking
R20.98 max - 0.95
1.0
0.9
Fourier Descriptors
0.8
Larger cascade effect
0.7
Likelihood that the chosen face is inspected
0.6
Faces
0.5
Decision
Decision
0.4
0.3
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
Time before decision (s)
48
Facilitation in the memory trace
Preferential
looking,
Gaze bias,
cognitive
exposure
assessment
Eye movements
(orient to the stimulus, inspect more)
49
Gaze Manipulation Experiment

• Pre-rating of attractiveness --gt (2) pairing. (3)
  Gaze
• Manipulation (2, 6 or 12 loops) . (4) Press a
  button for preference.

(N 15, or 13)
2 loop
Short (300 ms)
6 loops
12 loops
Control experiments suggest The preference bias
can not be attributed to mere exposure.
Long (900 ms)
50
Gaze Manipulation
Longer gaze leads to preference in general.
Like Horiz. x 2
Like Horiz. x 6
Like Horiz. x 12
Like Vert. x 6
Tasks, Conditions
N 15 15 13
15
choosing the gazed longer
51.2 59.0 59.2 60.2
P value (t-test)
0.31 lt0.001 lt0.005 lt0.0001
51
?????????(????????)
?????????????????? (delay 2ms)?
52
Peep-hole exp. Results
Preference
Consistent only with the Gaze intrinsic
prediction
Roundness
53
Perception, gaze, and emotion are intermingled
from the outset!
cf. Very quick feedback from the amygdala
to the visual cortex (teens of ms in EEG).
54
A Related fMRI Study (Kampe, et al., (2001)
Nature, 413, 589.)
No single brain region whose activity is
correlated with attractiveness in general
(although the ventral striatum showed a
positive, when gaze in contact, and a negative,
when gaze away, correlation with attractiveness).
More attractive when gaze in contact (?).
55
(No Transcript)
56
(1)Gaze cascade in all cases. (2)Pref. for away /
forward. (3) Mixed-away is special.
57
Scenario of Co-evolution
58
Implications

• Development.
• (2) Clinical Psychology.
• (3) Neurology.
• (4) Robotics.
• (5) Media (CM, internet, A-V, etc.).

59
(No Transcript)
60
Time -frequency diagram based on Morlet wavelet
analysis for coherence (EEG)
Beta, Gamma (-2s)
(Bhattacharya, Shams, Shimojo, 02)
61
Mid-Frontal Region (Fz)
Like (abstract)
Like (face)
Passive
Clear separation!
ERP Response (microV)
No separation
Dislike
Passive
Round
Time (sec)
Button Press
62
fMRI study of facial preference
Preference

Preference
Variable interval (3 sec)
Preference
Preference
Preference
Brief Fixed duration (100 msec)
Preference
Presentation order random but balanced in
terms of number of repetitions
DECISION!
63
Decision
C2
C3
C1
U3
U2
U1
T3 (C3 - U3)
T2 (C2 - U2)
T1 (C1 - U1)
C Chosen stimuli U Unchosen stimuli
Predictions
Module 1 (Attractiveness encoding area) ? Early
discrimination (Implicit?)
Module 2
Signal Difference (C - U)
Module 1
Module 2 (Preference decision making area) ?
Late discrimination (Explicit?)
T1
T2
T3
64
Chosen vs. Unchosen Faces in EARLY cycle (CE
UE)) Two-cycle trials only
NAC
p lt 0.01
p lt 0.005
p lt 0.001
L
R
Nucleus Accumbens ??? (x 15, y 3, z -12)
No other areas such as PFC or mOFC! ???????????
65
2cycle trials alone
Early cycle (CE-UE)
Late cycle (CL-UL)
NAC
mOFC
Nuculeus Accumbens
Medial orbito frontal cortex
L
R
Late cycle (CL-UL)
OP/INS
D
Coefficient
NAC mOFC OP/INS
CE?UE CL?UL Response
Operculum/Insular
66
Group 1 preference decisions on NEW faces
--gt mOFC and NAC Group 2
preference decisions on FAMILIAR faces
--gt mOFC, but No NAC
Does NAC make IMPLICIT or SUBCONSCIOUS
preference decisions on new faces even during
ROUNDNESS decision task?
67
???????????????????????????????????
Implicit
Explicit
68
Implicit Preference Decision from Group 2 (Early
cycle during Roundness decision)
Explicit Preference Decision from Group 1 (Early
cycle during Preference decision)
y 3
L
R
???? (NAC) ??? (?????) ?????????
?????????????
69
Implicit Preference Decision from Group 1 (Early
cycle during Roundness decision)
mOFC
R
L
mOFC, but not NAC! --gt First impression only.
NAC??????????????????????
70
NAC
mOFC
71
OP/ INS
NAC
NAC communicates with OP/INS directly when
decision is clear
72

References Biederman, I. (1987)
Recoginition-by-components A theory of human
image understanding. Psychological Review,
94, 115-147. Bruce, V. and Young, A. (2000) In
the eye of the beholder The science of face
perception. Part 1. British journal of
psychology, 91, 141-142. Haxby JV, Gobbini MI,
Furey ML, Ishai A, Schouten JL, Pietrini P.
(2001) Distributed and overlapping
representations of faces and objects in ventral
temporal cortex. Science, 293(5539)
2425-30. Kanwisher, N. (2000) Domain specificity
in face perception. Nature neuroscience, 3, 759-
763. Mondloch, C. J. , Lewis, T. L., Budreau, D.
R., Maurer, D., Dannemiller, J. L., Stephens,
B. R., Kleiner-Gathercoal, K. A. (1999) Face
perception during early infancy. Psychological
science, 10, 419-422. Palmer, S. E. (1999)
Vision Science Photon to Phenomenology,
Cambridge, MA, MIT Press, Chap. 9, Perceiving
function and category, 408-461. Shimojo, S.,
Simion, C., Shimojo, E., and Scheier, C. Seeing
and Liking Gaze Cascade towards Preference.
Nature Neuroscience, 6, 1317-1322,
2003. Thornhill, R. Steven W. Gangestad, S. W.
(2001) Facial Attractiveness. Trends in
Cognitive Science, 3, 452-460. Tippett, L. J.,
Miller, L. A., Farah, M. J. (2000) Prosopamnesia
A selective impairment in face learning.
Cognitive neuropsychology, 17, 241-255. Whalen,
P.J., Rauch, S. L., Etcoff, N. L., et al. (1998)
Masked presentations of emotional facial
expressions modulate amygdala activity without
explicit knowledge. Journal of Neuroscience,
18, 411-418.