The Illusion of Mental Pictures

1
The Illusion of Mental Pictures

• Zenon Pylyshyn
• Rutgers University,
• Center for Cognitive Science
• http//ruccs.rutgers.edu/faculty/pylyshyn.html

2
The illusion of mental pictures

• There is no question that people (except maybe
  2 of the population) experience mental images
  when they recall, plan, anticipate and otherwise
  enjoy life in the absence of the things and
  people that they imagine
• Not only are we able to picture some object or
  scene in our minds eye but it seems that we
  must do so in order to solve certain kinds of
  problems
• Books are full of examples of how images helped
  people to discover and create theories and works
  of art creations that would not have happened
  without the capacity to use mental imagery. I
  will not rehearse all the examples, but they
  include Einstein, Kikule,

3
The illusion about the causal role of mental
pictures in thought

• What happens when we create and inspect mental
  images? This is a deep problem because it
  touches on the mind-body duality and other
  potentially unsolvable problems. But it is
  important that as scientists we consider what is
  entailed by talk of creating, recalling,
  examining and transforming mental images.
• I have argued, and still believe, that there is a
  powerful illusion behind not only our folk
  understanding of mental imagery, but also behind
  our attempts to build scientific theories of it.
  
• The illusion is this When we engage in what we
  call visualizing there is, somewhere (presumably
  in our head), a thing that we view more or less
  the way we view the world a thing that we might
  as well call a mental picture since a picture is,
  after all, something that looks like the thing
  that is pictured.

4
Some common mistakes in thinking about mental
imagery

• The intentional fallacy Confusing properties of
  the imagined world with properties of the
  imagination or the mechanism or medium of imagery
• ? Examples size, distance, and especially
  temporal duration
• Task demands. Insufficient attention is paid to
  how subjects interpret instructions in imagery
  experiments.
• Imagine x ? Pretend that you are seeing x
  happening
• This is a task demand and is not a case
  of subjects being acquiescent and trying to give
  the result they think the experimenter wants.
  This is a rational and appropriate understanding
  of the task to imagine something. The
  consequence is that subjects will make as many
  properties as they know of and as they are able
  to control come out as it would have in reality.

5
Two mistakes in imagery research

• The intentional fallacy confounding properties
  of the representation with properties of the
  represented
• Task demands neglecting the fact that subjects
  are being asked to pretend that they are seeing
  something

6
1. A common mistake in thinking about mental
imagery is called the intentional fallacy
Confusing properties of the imagined world with
properties of the imagination or the mechanism or
medium of the image

• An image of X with property P can mean
• (An image of X) with property P or
• An image of (X with property P)

7
2. Demands of the task to imagine x

• Most of the behavioral research into dynamic
  properties of mental imagery is explained by
  noticing how one studies properties of imagined
  processes. When we ask subjects to imagine that
  X where X is some process (like looking at a
  small mouse or watching a spot move across a map
  from one place to another), what we are inviting
  the subject to do is pretend that they are seeing
  X happening. In that case, how X unfolds is
  dictated by what observers believe would happen
  if they were seeing it. This belief is often
  tacit and unconscious.
• This is not a case of subjects being
  disingenuous, or of acquiescing to experimenter
  demand. Rather its the rational response to
  the task as presented.
• After a few examples I will turn to the real
  question Do people represent space when they
  imagine spatial layouts, and if so what does that
  entail, and how do they do it?

8
Imagine various events unfolding before your
minds eye
Examples to probe your intuition and your tacit
knowledge

• Imagine turning a heavy wheel. Now a light
  wheel. Which is faster?
• Imagine a baseball being hit. What shape
  trajectory does it trace? It is coming towards
  you Where would you run to catch it? You have
  considerable tacit knowledge of what to do in
  this case.
• Imagine a coin dropping and whirling on its edge
  as it eventually settles. Describe how it
  behaves. (Eulers Disk problem solved in 2000)
• Imagine a heavy ball (a shot-put) and a light
  ball (a tennis ball) being dropped at the same
  time from a building (e.g., the leaning tower of
  Pisa). Indicate when they hit the ground.
  Repeat at different heights.
• Imagine a clear glass containing a colored
  liquid. Tilt it 45º to the left
  (counter-clockwise). What is the orientation of
  the liquid?

9
What color do you see when two color filters
overlap?
10
Where would the water go if you poured it over a
full beaker of sugar?
Is there conservation of volume in your image?
If not, why not?
11
Aside What can we conclude from the contents of
conscious experience?
12
Representation of Space in mental images
And now for something more serious

• This is the question I am most interested in and
  it is a major topic in imagery research

13
Spatial character of mental images

• Among the more impressive findings of research on
  mental imagery are ones that suggest that images
  have spatial properties (e.g., mental rotation,
  mental scanning, mental size effects,
  psychophysical measures of the minds eye).
• Intuitively we feel that we can reason by
  imagining things laid out in space and then by
  examining the display we can often read off the
  solution. Yet there have been few attempts to
  say exactly what being laid out in space means,
  either formally or physically.
• One of the most explicit statements concerning
  the spatial properties of images has been a
  statement by Steve Kosslyn about what he calls
  the depictive nature of mental images.

14
Images as displayed in functional spaceA
statement of the picture theory (Kosslyn, 1994)

• A depictive representation is a type of
  picture, which specifies the locations and values
  of configurations of points in a space.
• The space in which the points appear need not
  be physical, but can be like an array in a
  computer, which specifies spatial relations
  purely functionally. That is, the physical
  locations in the computer of each point in an
  array are not themselves arranged in an array it
  is only by virtue of how this information is
  read and processed that it comes to function as
  if it were arranged into an array (with some
  points being close, some far, some falling along
  a diagonal, etc). (p5)
• I will argue that it is important why the
  information is read in one way rather than in
  another since that determines whether the account
  is explanatory or descriptive or merely circular.

15
The illusion of mental (picture) space

• There have been two options for accounting for
  the spatial properties of images
• Assume a physical display in the brain, or
• Assume a mechanism that simulates spatial
  properties but is not itself a literal space.
  This is referred to as functional space.
• Neither of these options is consistent with
  empirical evidence The cortical space assumption
  is not consistent with neural or behavioral
  evidence and the functional space assumption is
  either metaphorical or circular.
• Because a functional space has no inherent
  constraints, and exhibits whatever properties we
  stipulate it to have, it is not explanatory.
• Later I will suggest that spatial properties are
  not in the head but in the relation of thought to
  concurrently-perceived space.

16
What does being spatial entail?
Images and space some possible constraints

• Are images spatial? Do they have spatial
  properties such as size, distance, and relations
  such as above, next-to, in-between? Do the
  axioms of Euclidean geometry and measure theory
  hold of patterns displayed in them? e.g.,
• ab bc ? ac and ab ba
• If ?abc 90, then ab2 bc2 ac2
• If such axioms are true of images, what would
  that entail about how they must be instantiated
  in the brain?
• Could they be analogue? What constraints does
  that impose?
• Is the space 2-D or 3D?
• Is there a coherent notion of a functional
  space, as something with the formal properties
  of space yet without being instantiated in real
  physical brain-space?

17
The spatial-metrical character of images

• The claim that images have spatial properties
  comes from our phenomenology, and also from a
  number of experiments suggesting that images must
  actually have metrical properties, particularly
  spatial ones (not just represent metrical
  properties, but have them).
• The most commonly cited experiments are ones that
  seem to involve continuous spatial properties
• Image size
• Mental rotation of images
• Mental scanning across an image

18
Do images have size?

• There are many studies showing that when subjects
  imagine something small it takes them longer to
  detect small features (e.g. the mouses whiskers)
  than when they imagine them as large. What do
  these tell us about what size is?
• There are two possibilities The size is either
  the size of the image or it is the size of the
  thing imagined. The first needs either a
  physical size or some theoretical idea about what
  constitutes image size that has yet to be
  provided, and the second can yield the observed
  result simply because the subject knows what
  would happen in the a viewing, namely if
  something is seen to be small the details will
  not be as clear or you will need to come closer
  (or zoom in on the object) to see the details,
  so an observer will surely ensure that this
  happens. (What if it didnt?)
• Suppose, instead, you were asked how long it took
  to report details in a large blurred
  low-resolution image versus a small high
  definition image? Why is such a control not
  done?

19
Do mental images have (as opposed to represent)
size?

• Imagine a mouse across the room so its image
  occupies a small part of your total image
  display.
• Now imagine it close to you so it fills your
  image display
• Of these two conditions, in which do you see
  small details most clearly? In which does it take
  longer to see the mouses whiskers?
• What does this result tell you about the size
  of a mental image?
• Imagine a horse. How close can you imagine
  coming to the horse before it starts to overflow
  your image? Repeat with a toaster, a table, a
  persons face, etc.
• Does this provide a measure of the visual angle
  of the minds eye and a measure of the mental
  size of the horse?
• Do these concepts have any meaning without the
  literal space view of images?
• Clearly these results are the ones you would
  expect if the subject is telling you what it
  would be like to see a real horse, mouse, etc

20
One of the least controversial examples of image
transformation Mental rotation
Time to judge whether (a)-(b) or (b)-(c) are the
same except for orientation increases linearly
with the angle between them (Shepard Metzler,
1971)
21
Imagine this shape rotating slowly
Is this how it looked to you?
When you make it rotate in your mind, does it
seem to retain its rigid 3D shape without
re-computing it?
22
The missing obligatory constraint

• What is assumed about the format or architecture
  of the mental representation in the examples of
  mental rotation?
• According to philosopher Jesse Prinz (2002) p
  118,If visual-image rotation uses a spatial
  medium of the kind Kosslyn envisions, then images
  must traverse intermediate positions when they
  rotate from one position to another. The
  propositional i.e., symbolic system can be
  designed to represent intermediate positions
  during rotation, but that is not obligatory.
• This is a very important observation to which I
  will return. But the statement is incomplete.
  It needs to answer the question What makes it
  obligatory that the object must pass through
  intermediate positions when rotating in
  functional space, and what constitutes an
  intermediate position? These terms apply to
  the represented world, not to the representation!

23
The important distinction between architecture
and represented content?

• It is only obligatory that a certain pattern must
  occur if the pattern is caused by fixed
  properties of the architecture as opposed to
  being due to properties of what is represented
  (i.e., what the observer tacitly knows about the
  behavior of what is represented)
• If it is obligatory only because the theorist
  says it is, then score that as a free empirical
  parameter
• The important consequence is that if we allow one
  theory to stipulate what is obligatory without
  there being a principle that mandates it, then
  any other theory can stipulate the same thing.
  So the theories are unconstrained and explain
  nothing.
• This failure of image theories is quite general
  all picture theories suffer from the same lack of
  principled constraints

24
How are these obligatory constraints realized?

• Image properties, such as size and rigidity are
  assumed to be inherent in the architecture (of
  the display)
• That raises the question of what kind of
  architecture could possibly enforce rigidity of
  shape?
• Notice that there is nothing about a spatial
  display, let alone a functional space, that makes
  it obligatory that shape be rigidly maintained as
  orientation is changed.
• Also such rigidity could not be part of the
  architecture of an imagery module because we can
  easily imagine situations in which rigidity does
  not hold (e.g. imagine a rotating snake!).
• There is also evidence that mental rotation is
  incremental, not holistic, and the speed of
  rotation depends on the conceptual complexity of
  the shape and the comparison task.

25
Mental rotation the real story

• In mental rotation the phenomenology
  motivates the theory of rotation but what the
  data show is that,
• Mental rotation is only found when the
  comparisons are enantiomorphs (3D mirror-images)
• No rotation occurs if the figures have landmarks
  that can be used to identify the relations among
  their parts or if differences are specifiable in
  allocentric coordinates
• Records of eye movements show that mental
  rotation is done incrementally It is not a
  holistic rotation as often reported. If fact
  even the phenomenology is not of a smooth
  continuous rotation (your experience just now)
• The rate of rotation depends on the conceptual
  complexity of the figure and also of the
  recognition task so that, at least, is not a
  result of the architecture (Pylyshyn, 1979).
  There are even demonstrations that it depends on
  how the subject interprets the figure (Kosslyn,
  xx).

26
Rate of mental rotation depends on task
complexity
Good subset
Bad subset
Not a subset
27
Example 2 Mental Scanning

• Some hundreds of experiments have now been done
  demonstrating that it takes longer to scan
  attention between places that are further apart
  in the imagined scene. In fact the time-distance
  relation is linear.
• These have been reviewed and described in
• Denis, M., Kosslyn, S. M. (1999). Scanning
  visual mental images A window on the mind.
  Cahiers de Psychologie Cognitive / Current
  Psychology of Cognition, 18(4), 409-465.
• Rarely cited are experiments by Liam Bannon and
  me (described in Pylyshyn, 1981) which I will
  summarize for you.

A window on the mind !!
28
Studies of mental scanningDoes it show that
images have metrical space?
(Pylyshyn Bannon. Described in Pylyshyn, 1981)

• Conclusion The image scanning effect is
  Cognitively Penetrable
• i.e., it depends on goals and beliefs, or on
  Tacit Knowledge.

29
What is assumed in the mental picture
explanations of mental scanning?




? The central problem with imagistic
explanations?

• In actual vision, it takes longer to scan a
  greater distance because real distance, real
  motion, and real time is involved, therefore this
  equation holds due to natural law
• Time distance speed
• But what ensures that a corresponding relation
  holds in an image? The obvious answer is
  Because the image is laid out in real space!
• But what if that option is closed for empirical
  reasons? Well you might appeal to a Functional
  Space which imagists liken to a matrix data
  structure in which some pairs of cells are closer
  and others further away, and to move from one to
  another it is natural that you pass through
  intermediate cells
• Question What makes these sorts of properties
  natural in a matrix data structure?

30
What warrants the obligatory constraint?

• To use Prinzs term, it is not obligatory that
  the well-known relation between distance, speed
  and time hold in functional space or in a matrix.
  There is no natural law or principle that
  requires it. You could imagine an object moving
  instantly or according to any motion relation you
  like, and the functional space would then comply
  with that since it has no constraints of its own.

31
Where does the obligatory constraint come from?

• There are at least two reasons why the
  following equation holds in the mental image
  scanning task, even though, unlike in the real
  vision case, it does not follow from a natural
  law.
• Time Representation of distance
  Representation of speed
• Because subjects have tacit knowledge that this
  is what would happen if they viewed a real
  display, and they understand the task to be one
  of reproducing properties of this viewing, or
• Because the matrix is taken to be a simulation of
  real space. In that case the reason that the
  equation holds is that it is supposed to be
  simulating real space and the equation holds in
  real space.
• In that case it is not something about the form
  of the representation that provides the
  principled constraint, its the fact that it is
  supposed to be simulating real space which is
  where the obligation comes from. But the same
  thing can be done for any form of representation.

32
Real and functional space

• What is assumed by picture accounts of mental
  imagery experiments, including those involving
  image scanning, image size and image rotation, is
  that images have the properties of a real spatial
  display as viewed by the minds eye this is what
  provides a principled explanation.
• But as we will see, this explanation carries a
  number of assumptions, including that images are
  2D patterns laid out in real space (presumably on
  visual cortex). Because the evidence does not
  support this assumption, imagists appeal to a
  functional space.
• What is a functional space and how does it
  explain the scanning or image size findings?

33
What is functional space?

• Because functional space is cited by almost every
  imagery theorist, it deserves some attention.
• The main point about a functional space is that
  it has whatever properties we want to assign it
  i.e., it can be made to fit any data. We
  stipulate that it takes longer to scan greater
  distances since the law relating distance, time
  and speed does not apply to a functional space.
• For that reason a functional space does not
  differ from any other proposal about how space is
  represented the properties we assign to
  functional space can be assigned to any other
  theory. So the concept of functional space does
  no explanatory work.
• The assumption that functional space must have
  certain obligatory (as Prinz put it) spatial
  properties, relies on one of several mistakes
  which result in these properties seeming more
  natural in a functional space.

34
Why is it natural to assume that functional
space is like real space?

• There are several possible reasons why a
  functional space, such as a matrix data
  structure, appears to have natural spatial
  properties (e.g., distances, size, empty places)
• Because when we think of incarnations of
  functional space, such as a matrix, we think of
  how we picture them on paper.
• In fact a matrix does not intrinsically have
  distance, empty places, direction or any other
  such property, except in the mind of the person
  who draws it or uses it!
• Moving from one cell to another does not require
  passing through intermediate cells unless we
  stipulate that it does. The same goes for the
  very concept of intermediate cell.

35
Why is it natural to assume that functional
space is like real space?

• Because when we think of a functional space, such
  as a matrix, we think of it as being a way of
  simulating real space inside the model making
  it more convenient to build the model which
  otherwise would require special hardware
• This is why we think of some cells as being
  between others and some being farther away,
  etc. This makes properties like distances seem
  natural because we interpret the matrix as
  simulating real space.
• In that case we are not appealing to a functional
  space in explaining the scanning effect, the size
  effect, etc. The explanatory force of the
  explanation comes from the real space that we are
  assuming.
• This is just another way of assuming a real space
  (in the brain) where representations of objects
  are located in neural space
• All the reasons why the assumption of real brain
  space cannot be sustained in explanations of
  mental imagery phenomena apply to this version of
  functional space.

36
Functional space and explanatory power

• There is a notion of explanatory power that needs
  to be kept in mind. It is best illustrated in
  terms of models that contain empirical
  parameters, as in fitting a polynomial curve to
  data.
• The general fact about fitting a model to data is
  that the fewer parameters that need to be
  estimated from the data to be fitted, the more
  powerful the explanation. Thus the lower the
  order of the polynomial fit the better the
  explanation.
• In terms of the current example of explaining
  results of experiments involving mental imagery,
  appealing to a functional space leaves open an
  indeterminate number of empirical parameters, so
  it provides a very weak (or vacuous) explanation.
• A literal (brain) space, on the other hand, is
  highly constrained since it must conform to
  Euclidean axioms and Newtonian physics
  otherwise it would not be the space of natural
  science. But that kind of space implies that
  images are displayed on a surface in the brain.

37
What next?

• So we turn now to the only place where we might
  be able to find properties that explain the
  experimental imagery results the brain
  because it is the only place where there is
  literal physical space that could function to
  underwrite such operations as scanning or
  rotation. No wonder the more recent work on
  imagery has been carried out in collaboration
  with neuroscience.

38
I. Is there any reason to be optimistic about
finding mechanisms of imagery in visual cortex
The good news

• There is neuroanatomical evidence for a
  retinotopic layout in the earliest visual area of
  the brain (V1)
• Neural imaging data shows that V1 is more active
  during mental imagery than during other forms of
  thought
• Transcranial magnetic stimulation (TMS) of visual
  areas interferes more with imagery than other
  forms of thought
• Clinical cases of visual agnosia show that some
  impairments of vision have associated impairments
  of imagery (Bisiach, Farah)
• Recent psychophysical observations of imagery
  show parallels with corresponding observations of
  vision, and these can be related to the receptive
  cells in V1 (e.g., oblique effect)

39
Neuroscience evidence shows that the retinal
pattern of activation is displayed on the surface
of the cortex
There is a topographical projection of retinal
activity on the visual cortex of the cat and
monkey.
Tootell, R. B., Silverman, M. S., Switkes, E.,
de Valois, R. L. (1982). Deoxyglucose analysis of
retinotopic organization in primate striate
cortex. Science, 218, 902-904.
40
II. There are problems with drawing conclusions
about mental imagery from such neuroscience data
The bad news

• The capacity for imagery and for vision are
  independent. Also all imagery results are
  observed in the blind as well as in patients with
  no visual cortex.
• Cortical topography is 2-D, but mental images are
  3-D all phenomena (e.g. rotation) occur in
  depth as well as in the plane.
• Patterns in the visual cortex are in retinal
  coordinates whereas images are primarily in
  world-coordinates
• Unless you make a special effort, your image of
  parts of the room stays fixed in the room when
  you move your eyes or turn your head or walk
  around the room

41
III There are problems with drawing conclusions
about mental imagery from such neuroscience data

• Accessing and manipulating information in an
  image is very different from accessing it from
  the perceived world. Order of access from images
  is highly constrained.
• Some have tried to explain this by postulating
  rapid decay of images, but the times involved in
  these demonstrations are not consistent with the
  data (e.g., times for reporting letters are
  comparable to those involving size or scanning).
  
• Conceptual rather than graphical properties are
  relevant to image complexity (e.g., mental
  rotation) so image representations seem to be
  conceptual
• If images consist in patterns on visual cortex
  then they behave differently when the same
  patterns are acquired from vision. For example
  the important Emmerts law applies to retinal and
  cortical images but not to mental images, a fact
  largely unnoticed.

42
There are problems with drawing conclusions about
mental imagery from these neuroscience data

1. The signature properties of vision (e.g.,
   spontaneous 3D interpretation, automatic
   reversals, apparent motion, motion aftereffects,
   and many other phenomena) are absent in images
2. A cortical display account of most imagery
   findings is incompatible with the cognitive
   penetrability of mental imagery phenomena, such
   as scanning and image size effects
3. The fact that the Minds Eye is so much like a
   real eye (e.g., oblique effect, resolution
   fall-off) should serve to warn us that we may be
   studying what observers know about how the world
   looks to them, rather than what form their images
   take (unless the Minds eye is exactly the same
   as the real eye!).

43
But there are problems with drawing conclusions
about mental imagery from neuroscience data

• Many clinical cases cited by image theorists can
  be explained by appeal to tacit knowledge and
  attention
• The tunnel effect found in vision and imagery
  (Farah) is likely due to the patient knowing how
  things looked to her post-surgery
• Hemispatial neglect seems to be an attention
  deficit, which explains the neglect in imagery
  reported by Bisiach. A recent study shows that
  image neglect does not appear if patients have
  their eyes closed (Bartolomeo Chokron, 2002).
  This fits well with the account I have offered in
  which the spatial character of mental images
  derives from concurrently perceived space.

44
A few examples illustrating the type of problems
that neuroscience accounts run into in explaining
behavioral findings with imagery

• In a footnote in Kosslyn, Thompson Ganis (2007)
  the authors cite Ned Block as claiming that one
  does not need an actual 2D cortical display, so
  long as the connections upstream from the cortex
  can decode any pair of neurons in terms of their
  distance in V1, so arguments against literal
  spatial display are not relevant.
• Think of long stretchy axons connecting points on
  a 2D surface (retina or visual cortex) to
  subsequent processes. Now imagine that the V1
  neurons are randomly moved around so they no
  longer constitute a 2D layout. As long as the
  upstream connections remain fixed it will still
  behave as though there was a 2D surface.
• Call this the encrypted 2D layout version of
  literal space

45
Encrypted 2D space example
Imagine this is the way it was initially
Imagine this is the way it is after scrambling
The Associative cortex will still function the
same way even when the layout on V1 is not
isomorphic to the mental image
46
1. The encrypted-spatial layout alternative
(cont)

• By itself the encrypted-layout alternative will
  not do because without referring to the original
  locations, the relation between pairs of neurons
  and scan time is not principled. In the end the
  only principle we have is Timedistance/speed so
  unless the upstream system actually decrypts the
  neuronal fibers into their original 2D surface
  locations, the explanation for the increased
  time with increased imagined distance remains a
  mere stipulation it is not obligatory. It
  stipulates, but does not explain why, when two
  points are further away in the imagined layout it
  takes longer to scan between them or why scanning
  between them requires that one visit
  intermediate locations along the way because in
  the encrypted layout there are no distances nor
  any intermediate locations.
• Once again so long as what we have is just a
  stipulation, as opposed to a general principle,
  we can apply it to any form of representation!

47
2. Image size and the visual cortex

• There is evidence that when imagining large
  objects that overflow ones phenomenal image a
  different pattern of activation in visual cortex
  occurs than when imagining a small object.
• This in itself is not remarkable since all
  scientists accept that a difference in mental
  experience must be accompanied by some difference
  in the neural state this is called materialism,
  or more technically supervenience.
• The activation pattern when imagining a large
  visual pattern is claimed to be similar to the
  activation pattern when perceiving a large visual
  pattern (large on the retina). In vision,
  objects that extend into the parafovia of the
  eye, project onto the more frontal parts of the
  visual cortex. Imagists claim that this is also
  true when imagining a large pattern that fills
  the mental screen.

48
2a. Image size. Continued

• The mere fact that larger images lead to
  activation in different (rather than larger)
  regions of the cortex does not in itself help to
  explain the size effect.
• The explanation of why larger images are
  associated with shorter reaction times, is that
  for a given display resolution, more details can
  be displayed when the image is larger (and a
  neural account for this assumption is given in
  terms of lateral inhibition among neurons in V1).
  The actual size (as well as the resolution) of
  the image display always enters into explanations
  of the size effect.

49
3. The oblique effect

• In vision, when a set of lines is to be
  discriminated (distinguished from a single blur)
  the discrimination is easier when the lines are
  vertical or horizontal than when they are at a
  45 angle. This is called the Oblique Effect.
  It is a low-level effect that occurs in early
  vision.
• Does the Oblique effect occur with mental images?

50
Do images have low-level visual properties?

• Imagine a grating in which the bars are
• Horizontal
• Vertical
• Oblique (45)
• Imagine the bars getting closer and closer
  together. In which of these displays do the bars
  blur together first?
• In vision, the oblique bars blur earlier (called
  oblique effect)
• In imagery, a similar result was found
• Kosslyn, Thompson Ganis (2006) argued that this
  is because there are more vertical- and
  horizontal-tuned cells than oblique-tuned cells
  in visual cortex. So this confirms that images
  are projected onto visual cortex.

51
Neurological explanations for both cases?

• An accepted explanation of the psychophysical
  case (where lines were viewed) is that there are
  more horizontal and vertical-sensitive the visual
  cortex than oblique-sensitive cells in V1. Can
  the same explanation work for the imagined bars?
  Kosslyn et al claim that it does and that this
  provides additional support for the view that
  images are laid out in visual cortex.
• But this argument rests on a misunderstanding of
  how the orientation-specific cells get their
  orientation property they get it from the way
  they are wired to photoreceptive cells on the
  retina. Vertical cells are more often wired to
  columns of cells while horizontal cells are more
  often wired to rows of photocells.
• If patterns of bars were activated on the surface
  of cortex by mental imagery then horizontal cells
  would be no more likely to be activated by
  horizontal patterns than by vertical patterns.
  The only way that images of horizontal bars would
  preferentially activate horizontal cells is if
  the images were on the retina!

52
What happens when horizontal/vertical cells are
activated by means other than retinal patterns?
9 vertical 9 horizontal 5 oblique
The proportion of Vertical, Horizontal Oblique
cells remains the same in all cases they are
random samples!
53
An overarching consideration

• What if colored three-dimensional images were
  found in visual cortex? What would that tell you
  about the role of mental images in reasoning?
• Would this require a homunculus?

54
Should we welcome back the homunculus?

• In the limit if the visual cortex mapped the
  contents of ones conscious experience involving
  mental imagery we would need an interpreter to
  see this display in visual cortex
• But we will never have to face this prospect
  because experiments show that the contents of
  mental images (other than in iconic memory that
  lasts for a fraction of a second) are already
  conceptual (or, as Kosslyn puts it, are
  predigested) and therefore unlike any picture.
• Finally, you can make your image do whatever you
  want, and to have whatever properties you wish.
  There are no known constraints on mental images
  that cannot be attributed to lack of knowledge of
  the imagined situation (e.g., imagining a
  4-dimensional object).

55
Are there any ways of representing spatial
layouts that are not excluded by this analysis?

• Maybe we have been looking in the wrong place for
  things that fall under the formal requirements of
  being spatial. Maybe they are not in the head
  after all.
• I have sketched a way of looking at this problem
  that locates the spatial character of thought in
  the concurrently-perceived world (see Chapter 5
  of my 2007 book, Things and Places). I will end
  with just a hint of this approach. It relies on
  findings from the study of the interaction among
  perceptual modalities and imagery as well as with
  motor actions and also neuroscience findings
  concerned with coordinate transformation
  mechanisms in the brain.

56
Another chapter in the imagery debateThe
interaction of images with vision and motor
control

• One of the more interesting lines of research on
  the spatial character of mental images involves
  studies of the interaction of images with
  perceived spatial layouts and with the motor
  system
• From the beginning it has been clear to me that
  one of the properties of mental images that makes
  them appear spatial is that they connect in
  certain ways not only with vision, but also with
  the motor system
• We can point to things in our image!
• We can project our images onto perceived space
  even space perceived in different modalities.
  I believe that this observation is the key to
  understanding the alleged spatial character of
  images.
• This does not require that a picture be
  projected, only the locations of a small number
  of features. As it happens, in my day job I have
  studied a mechanism I call a FINST that is well
  suited for this task.

57
Projecting imagesShepard Podgorny experiment
Both when the displays are seen and when the F is
imagined, RT to detect whether the dot was on the
F is fastest when the dot is at the vertex of the
F, then when on an arm of the F, then when far
away from the F and slowest when one square off
the F.
58
Both vision and visual imagery have some
connection to the motor system

• Imagery clearly has some connection to motor
  control
• You can point to things in your image.
• This may be why images feel spatial
• You can get Stimulus-Response compatibility
  effects between the location of a stimulus in an
  image and the location of the response button in
  space
• Ronald Finke showed that you could get adaptation
  with imagined hand position that was similar to
  adaptation to displacing prism goggles
• Both these findings provide support for the view
  that the spatial character of images comes from
  something being projected onto a concurrently
  perceived scene.
• This is the main new idea in Chapter 5 of Things
  Places)

59
S-R Compatibility effect with a visual
displayThe Simon effect It is faster to make a
response in the direction of an attended objects
than in another direction
Response for A is faster when YES in on the left
in these displays
60
S-R Compatibility effect with a recalled
(imagined) display
The same RT pattern occurs for a recalled display
as for a perceived one
RT is faster when the A is recalled (imagined)
as being on the left
61
What about mental scanning?
62
Using a concurrently perceived room to anchor
FINSTs tagged with map labels
63
Studies of mental scanningDoes it show that
images have metrical space?
The image scanning effect was shown to be
Cognitively Penetrable.But what allows a smooth
scan across the image is the perceptual display.
Without the perceived map scanning would not be
smooth and continuous and the timing would not be
accurate (Pylyshyn Cohen, 1999).
64
Where do we stand?

• It seems that a literal picture-in-the-brain
  theory is untenable for many reasons including
  the major empirical differences between mental
  images and cortical images. A serious problem
  with any format-based explanation of mental
  imagery is the cognitive penetrability of many of
  the imagery demonstrations.
• So the pictorial quality of images is an illusion
  that arises from the similarity of the experience
  of imaging and of seeing
• So how do we explain the similarity of the
  experience of imagining and of seeing
  the fact that they both seem to involve a
  pictorial panoramic display?
• It is very likely that neither experience
  correctly reveals the form of the representation.

65
Conscious experience and the picture-theory
The picture theory was initially meant to explain
why our perceptual experience is panoramic and
stable while the visual inputs are partial and
changing. This assumes that the content of
experience is represented.

• But the picture theory of vision has been
  thoroughly discredited There is no rich
  panoramic display in vision (e.g., see change
  blindness, superposition studies, )

66
This is what our conscious experience suggests
goes on in vision
67
This is what the demands of explanation suggests
must be going on in vision
68

• For a copy of these slides seehttp//ruccs.rutge
  rs.edu/faculty/pylyshyn/ObjectsPlaces2009

69
You are now here
X
But you are also here
70
END