How We Cognize Space

1
How We Cognize Space

• Zenon Pylyshyn
• Rutgers Center fir Cognitive Science
• Rutgers University,
• New Brunswick, NJ

2
How we cognize spaceIs There an Image Space in
the Head?

• The most common approach to the question of how
  we represent a spatial layout is that we
  represent it in the form of a mental image. The
  format of mental images is supposed to be
  particularly suited for representing spatial
  information.
• The minds great illusion That you see a world
  inside your head when you imagine
• the Picture Theory of mental imagery

3
The four-part plan of this lecture

1. First I will talk a little about the imagery
   debate and introduce recent neuroscience
   evidence.
2. I will then focus on what I consider the core of
   the debate How mental images represent space.
3. Then I will talk about the special case of images
   that are projected onto the perceived world.
4. To show the generality of projected images that I
   will need to spend a few minutes to introduce the
   idea of visual indexes or FINSTs, as a type of a
   deictic or demonstrative reference.
5. Finally I will combine the idea of indexes with
   the evidence on mental imagery and suggest how
   generalized indexes may allow spatial properties
   of the currently-perceived world to translate
   into apparent spatial properties of images.

4
I. The imagery debate A capsule overview

• The main question is whether thoughts experienced
  as mental images or as seeing with the minds
  eye are different from other thoughts, and if so
  how.
• The dominant view is the picture theory of
  mental images, which assumes that images stored
  in a spatial medium and are examined by the
  visual system the same way that the original
  scene would be.

5
Does imagery use the visual system and if so,
what does that tell us about the nature of images?

• There is some evidence that the visual system is
  active during imagery
• This has led to the view that the visual system
  must be examining some not-yet-interpreted image,
  just as it was thought to do in visual
  perception.
• But the last step is unwarranted because even if
  the visual system was involved, it would only
  mean that both vision and imagery use some of the
  same processes and the same kind of
  representations, but neither need be pictorial.

6
Failure of the picture-theory in vision
In vision the picture theory was meant to explain
why our perception is panoramic and stable while
the visual inputs are highly local, partial and
constantly changing

• But the picture theory of vision has been
  thoroughly discredited There is no rich
  panoramic display in vision (e.g., see change
  blindness, superposition studies, )

7
The picture theory of visionis a non-starter,
even for cats
(Cartoon by Kliban)
8
A more plausible theory of vision(even for cats)
9
II. The newest round of the imagery debate

• In recent years the picture theory has been
  revived, due largely to two neuroscience
  findings
• The visual cortex (V1) is activated during
  imagery
• The visual cortex is retinotopically organized
  (i.e., it appears to map the retina in a
  topographically continuous or homeomorphic
  manner).
• From this, people have concluded that mental
  imagery uses a literal spatial display, located
  in V1.

10
The goal of neuroscience research on mental
imagery is to find a display of the imagined
pattern in visual cortex
We already know that there is a topographical
projection of retinal activity in visual
cortex The tool of choice has been the use of
brain scans (esp fMRI, PET)
Tootell, R. B., Silverman, M. S., Switkes, E.,
de Valois, R. L. (1982). Deoxyglucose analysis of
retinotopic organization in primate striate
cortex. Science, 218(4575), 902-904.
11
What do recent neuroscience results tell us about
mental imagery?
None of the brain-scan (fMRI, PET) results
supports the picture theory of mental images for
reasons that I will discuss next

1. Even if there is a 2D mapping of retinal activity
   in visual cortex (V1), this should not be
   identified with the mental image.
2. Patterns in V1 do not function the same way as
   mental images for several reasons.
3. Even if dynamic 3D patterns were found in V1 it
   would not explain most mental imagery research
   findings.

12
The topographical structure of the visual cortex
could not support mental images

1. Even of there is a 2D mapping of retinal activity
   in V1, this cannot be identified with the mental
   image which is panoramic, 3-dimensional, dynamic
   and has many other properties that could not be
   mapped onto V1, so we would need a different
   theory for them.

13
Why activity in visual cortex could not
correspond to a mental image

• Patterns in V1 are different from mental images
• Patterns in V1 are foveal and retinocentric while
  mental images are panoramic and allocentric
• There is no spontaneous 3D interpretation of
  patterns in mental images ltparallelogram examplegt
• There is no amodal completion of patterns in
  mental images ltKanizsa examplegt
• Order of access of information in a mental image
  is not free ltname letters of a familiar word
  backwordsgt
• Emmerts law does not hold for images ltunlike
  afterimagesgt
• There is no visual (re)interpretation of images
  ltSlezak examplegt

14
Why activity in visual cortex could not
correspond to a mental image

• Patterns in V1 are different from mental images
• Patterns in V1 are foveal and retinocentric while
  mental images are panoramic allocentric
• There is no spontaneous 3D interpretation of
  patterns in mental images ltparallelogram examplegt
• There is no amodal completion of patterns in
  mental images ltKanizsa examplegt
• Order of access of information in a mental image
  is not free ltletter reading examplegt
• Emmerts law does not hold for images
• There is no visual (re)interpretation of images
  ltSlezak examplegt

15
Imagine two parallelograms (as below) one above
the another
16
Close your eyes and then imagine these two
parallelograms
Connect the corresponding top and bottom vertices
What do you see? Keep looking to see if anything
changes
17
Did you see this? Did it flip?
18
Why activity in visual cortex could not
correspond to a mental image

• Patterns in V1 are different from mental images
• Patterns in V1 are foveal and retinocentric while
  mental images are panoramic allocentric
• There is no spontaneous 3D interpretation of
  patterns in mental images ltparallelogram examplegt
• There is no amodal completion of patterns in
  mental images ltKanizsa examplegt
• Order of access of information in a mental image
  is not free ltletter reading examplegt
• Emmerts law does not hold for images
• There is no visual (re)interpretation of images
  ltSlezak examplegt

19
Amodal completion in imagery?
20
Amodal completion in imagery?
21
Why activity in visual cortex could not
correspond to a mental image

• Patterns in V1 are different from mental images
• Patterns in V1 are foveal and retinocentric while
  mental images are panoramic allocentric
• There is no spontaneous 3D interpretation of
  patterns in mental images ltparallelogram examplegt
• There is no amodal completion of patterns in
  mental images ltKanizsa examplegt
• Order of access of information in a mental image
  is not free ltletter reading examplegt
• Emmerts law does not hold for images
• There is no visual (re)interpretation of images
  ltSlezak examplegt

22
Slezak figures
Pick one (or two) of these animals and memorize
what they look like. Now rotate it in your mind
by 90 degrees clockwise and see what it looks
like.
23
Rotated Slezak figures

• No subject was able to recognize the mentally
  rotated figure
• Subjects remembered the figures well enough so if
  they drew it they could recognize the rotated
  figure

24
Even if patterns in visual cortex were isomorphic
to those in the mental image, it still would not
explain most results of mental imagery research!

1. The reason that patterns of activation in striate
   cortex would not explain most of the results of
   mental imagery research is that the results are
   largely cognitively penetrable and therefore
   require the appeal to knowledge, goals,
   utilities, etc and inferences over them. In
   other words they require a cognitive explanation.

25
Task Demands and the tacit knowledge explanation

• The task of imagining X is the task of
  pretending that you are seeing X and simulating
  as much of that event as seems relevant to the
  task using your tacit knowledge about how the
  event might unfold. The task also requires
  certain other skills (e.g., estimating
  time-to-collision, generating time intervals,
  etc) but it does not require that you use a
  spatial display.
• Examples

26
There are many examples showing that the result
that was attributed to the mental image format is
actually due to tacit knowledge

• Color mixing example to illustrate the difference
  between the two sources of observations ltslidegt
• Imagine dropping weights from different heights
• Mental Image size (It has been shown that it
  takes longer to report small details from a small
  image than from a large on. What does this mean?
  What would you think if the result showed the
  opposite?)
• Mental scanning ltexample slidegt

27
Color mixing example
28
Studies of mental scanningA window on the mind?
(Pylyshyn Bannon. See Pylyshyn, 1981)
29
Are all imagery findings a result of simulation
using tacit knowledge?

• Simulation is the main explanation of results in
  the vast imagery literature. A few results
  require other explanations (see my recent BBS
  article). A special group of experiments that
  do not appear to be due to tacit knowledge will
  be discussed next. These are experiments that
  appear to involve projecting an image onto the
  visible world. Because this idea of projecting
  an image is central to my main thesis I will
  discuss some of the studies next.

30
Mental representation of spaceThe core of the
imagery debate

• It seems to be almost impossible to deny that
  thinking using mental images exploits spatial
  properties of images in some important sense. In
  what sense? (Do images preserve metrical
  spatial properties?)
• It is always possible to encode spatial relations
  in any form of representation that has a numeral
  system, so why assume that the representation of
  space is itself spatial?
• Phenomenology we see things as laid out in
  space!
• Psychophysical evidence from projected images
  (illusions, S-R compatibility)
• Clinical evidence (visual/imaginal neglect)

31
Use of real visible space in projected mental
imagery

• Projected images serve to directing attention
  and to associate thoughts with selected visible
  objects. Examples
• Robust version of mental scanning (scanning with
  eyes open)
• Visual illusions involving projected images
  ltBernbaum Chung, 1981gt
• Projected memory images act like displays
  ltPodgorny Shepherdgt
• S-R Compatibility with images (Tlauka McKenna,
  1998)
• Visuomotor (prism) adaptation from mental images
  ltFinke, 1980gt

32
Visual illusions with projected images
Bernbaum Chung. (1981)
Alternative explanations include response bias
and attentional allocation (which may be
responsible for the visual illusion as well).
33
Shepard Podgorny experiment
Both when the displays are seen and when the F is
imagined, RT to say whether the dot was on the F
was fastest when the dot was at the vertex of the
F, then when on an arm of the F, then when far
away from the F and slowest when one square off
the F.
34
S-R Compatibility Effect with display
35
S-R Compatibility Effect with Images
36
Might all spatial images work like projected
images?

• There are three key ideas behind the proposal
  that spatial mental images are the projection of
  the spatial layout of imagined objects onto a
  perceived scene
• Recognition that the spatial properties exhibited
  in experiments with projected images depend only
  on the location of a few items and not on other
  visual properties
• The idea of a limited-capacity amodal indexing
  mechanism or deictic reference FINSTs and
  Anchors.
• The idea of a primitive amodal spatial sense that
  allows us to perceive and recall the location of
  objects in an allocentric frame of reference,
  independent of the objects perceptual properties
  or of sense modality, and automatically updated
  by our movements

37
We dont need a spatial display in our head if we
have the right kind of deictic contact with real
(perceived) space

• None of the experiments that are alleged to show
  the existence of a spatial display (in visual
  cortex) need to appeal to anything more than a
  small number of imagined locations. (e.g.,
  Shepherd Podgorny, Finke, Tlauka,)
• If we can index a small number of (occupied)
  locations in real space (using FINSTs) we can use
  them to allocate attention or to program motor
  commands.
• If these indexed objects are also bound to
  objects of thought this will result in our
  thoughts (i.e. images) having persisting spatial
  relations.

38
Aside on Visual Index (FINST) Theory

• FINSTs are direct, unmediated, nonconceptual
  connections between objects in the world and
  mental symbols
• FINSTs serve as visual demonstratives (like
  this or that).
• Such direct references are essential for solving
  the correspondence problem in vision especially
  in the case of visual representations built up
  incrementally over different glances or
  noticings.
• Some instances where we need Indexes
• Visual stability, recognizing n-place relations,
  subitizing, and multiple-object tracking

39
Several objects must be picked out at once in
relational judgments

• When we judge that certain objects are collinear,
  we must have picked out the relevant individual
  objects first.

40
Several objects must be picked out at once in
relational judgments

• The same is true for other relational judgments
  like inside or on-the-same-contour etc. We must
  pick out the relevant individual objects first.

41
A concrete demonstration of what visual indexes
can do

• Multiple Object Tracking studies (MOT)
• Basic finding People can track up to 5
  individual objects that do not have a unique
  description
• We have shown that it is unlikely that the
  tracking is done by updating locations but rather
  that individuating and keeping track of certain
  kinds of individuals is a primitive visual
  operation
• Tracking is primitive and likely both
  preconceptual and preattemtive
• The mechanism for tracking is the same as the
  mechanism that is used for picking out elements
  when images are projected onto a scene.

42
How do we do it? What properties of individual
objects do we use?
43
How do we do it? What properties of individual
objects do we use?
44
But you can also imagine in the dark or with your
eyes closed!

• Does imagery work differently in the dark or with
  eyes closed?
• Must indexes be visual?

45
The Sense of Space

• This phrase is meant to denote an extremely
  well-developed human capacity to recall and
  orient to locations in space a space that is
  independent of modality and is anchored to real
  allocentric space.
• There is a major difference between a sense of
  space and a visual image. The sense of space is
  not a subjective experience but a skill that is
  largely unconscious. There has long been a
  suspicion that what has been studied under the
  name mental imagery is really spatial ability
  (e.g., unconscious images?).
• The sense of space does not need an internal
  spatial medium it can derive spatial properties
  by binding mental particulars to real perceived
  space.
• Perceptual Indexes (I.e., FINSTs and Anchors) are
  mechanisms that allow representations to inherit
  some of the spatial properties of the perceived
  world.

46
Some illustrations of the sense of spaceMany
phenomena that have been cited in support of the
picture theory of mental imagery only implicate a
spatial sense, not the visual perception of a
mental display

• Sense of space is not specific to (or parasitic
  on) vision
• Blind people exhibit all the observed phenomena
  of mental imagery
• Responses to images exhibit S-R compatibility and
  the Simon effect i.e., reactions made towards a
  stimulus are faster than ones made away from it.
• The space that is relevant to the Simon effect is
  amodal (you get cross-modal Simon effects)
• Hemispatial Neglect is a deficit in orienting
  attention to real locations thats why it may
  be mirrored in imagery
• Mental Images can induce visuomotor adaptation
• But only location, not visual pattern, plays a
  role (R. Finke)
• Observations such as the mental scanning effect,
  when they are not due to task demands, can be
  explained in terms of scanning through perceived
  space

47
Conclusion

• Many of the mental imagery findings in the
  literature are the result of subjects using their
  tacit knowledge to simulate what it would be like
  to see the situation described.
• The neuroscience evidence does not show that
  there is a 2D display in visual cortex on which
  we draw images when we imagine. The activity
  in visual cortex is of the wrong kind to
  underwrite mental imagery.
• More interesting are the studies in which people
  project images onto perceived scenes because
  these studies do show the involvement of spatial
  properties. But these experiments never need to
  assume that a picture-like pattern is projected.
  All they need to assume is that a few objects in
  the visual scene are indexed and associated with
  objects of thought. The rest of the spatial
  properties come from perception.
• Although the clear cases are when images are
  projected onto a visual scene, the same is likely
  true of other modalities that contribute to our
  sense of space.

48
Representing space

• The spatial character of mental images (and other
  spatial representations) comes from binding
  objects of thought to real objects in 3D space.
  The space in mental imagery comes from real
  concurrently-perceived spatial relations, which
  give us our exquisite sense of space.

49
References
Bernbaum, K., Chung, C. S. (1981). Müller-Lyer
illusion induced by imagination. Journal of
Mental Imagery, 5(1), 125-128. Kanizsa, G.,
Gerbino, W. (1982). Amodal completion Seeing or
thinking? In B. Beck (Ed.), Organization and
Representation in Perception (pp. 167-190).
Hillsdale, NJ Erlbaum. Pylyshyn, Z. W. (1973).
What the Mind's Eye Tells the Mind's Brain A
Critique of Mental Imagery. Psychological
Bulletin, 80, 1-24. Pylyshyn, Z. W. (1981). The
imagery debate Analogue media versus tacit
knowledge. Psychological Review, 88,
16-45. Pylyshyn, Z. W. (1989). The role of
location indexes in spatial perception A sketch
of the FINST spatial-index model. Cognition, 32,
65-97. Pylyshyn, Z. W. (2000). Situating vision
in the world. Trends in Cognitive Sciences, 4(5),
197-207. Pylyshyn, Z. W. (2001). Visual indexes,
preconceptual objects, and situated vision.
Cognition, 80(1/2), 127-158. Pylyshyn, Z. W.
(2002). Mental Imagery In search of a theory.
Behavioral and Brain Sciences, 25(2),
157-237. Pylyshyn, Z. W. (2003). Return of the
Mental Image Are there really pictures in the
brain? Trends in Cognitive Sciences, 7(3),
113-118. Pylyshyn, Z. W. (in press). Seeing and
visualizing It's not what you think. Cambridge,
MA MIT Press/Bradford Books. Pylyshyn, Z. W.,
Storm, R. W. (1988). Tracking multiple
independent targets evidence for a parallel
tracking mechanism. Spatial Vision, 3(3),
1-19. Shepard, R. N., Metzler, J. (1971).
Mental rotation of three dimensional objects.
Science, 171, 701-703. Shepard, R. N.,
Podgorny, P. (1978). Cognitive processes that
resemble perceptual processes. In W. K. Estes
(Ed.), Handbook of learning and cognitive
processes (Vol. 5). Hillsdale, NJ
Erlbaum. Slezak, P. (1995). The philosophical'
case against visual imagery. In P. Slezak, T.
Caelli R. Clark (Eds.), Perspective on
Cognitive Science Theories, Experiments and
Foundations (pp. 237-271). Stamford, CT
Ablex. Tlauka, M., McKenna, F. P. (1998).
Mental imagery yields stimulus-response
compatibility. Acta Psychologica, 67-79.