Imagery slides

1
Imagery slides
2
Imagery and Memory

• Memory Examples Dual Code Theory
• To recall Y you must first recall X
• Windows, doorknob, glasses, other facial
  features, global-to-local
• But Something like the same thing happens in
  recall of alphabet letters and many other
  memorized lists
• Imageability rating are more effective than
  frequency of occurrence or frequency of
  co-occurrence in paired-associates learning.

3
Vision is clearly involved when images are
superimposed onto vision

• Many experiments show that when you project an
  image onto a display the image acts very much
  like a superimposed display
• Shepard Podgorny (paper folding task)
• Interference effects (Brooks)
• Controvercial Perky effect Perception or
  response bias?

4
Project an image onto a perceived form
5
Brooks spatial interference study
Respond by pointing to symbols in a table or by
saying the words left or right
6
Perception or attention effects?

• Many impressive imagery effects can be plausibly
  attributed to attention
• Bisiach widely-cited finding on visual neglect
• Bartolomeo, P., Chokron, S. (2002). Orienting
  of attention in left unilateral neglect.
  Neuroscience and Biobehavioral Reviews, 26(2),
  217-234.
• Dulin, D., Hatwell, Y., Pylyshyn, Z. W.,
  Chokron, S. (2008). Effects of peripheral and
  central visual impairment on mental imagery
  capacity. Neuroscience and Biobehavioral Reviews,
  32(8), 1396-1408.
• Does neglect require vision?
• Chokron, S., Colliot, P., Bartolomeo, P.
  (2004). The role of vision in spatial
  representations. Cortex, 40, 281-290.

7
We can to some extent control our attended region
Is an image being projected onto a percept, or
just a selective attention?
Farah, M. J. (1989). Mechanisms of
imagery-perception interaction. Journal of
Experimental Psychology Human Perception and
Performance, 15, 203-211.
8
Shepard Podgorny experiment
Both when the displays are seen and when the F is
imagined, RT to detect whether the dot was on the
F is fastest when the dot is at the vertex of the
F, then when on an arm of the F, then when far
away from the F and slowest when one square off
the F.
9
Similarities between perception of visual scenes
and perception of mental images

• Judgments from mental images
• Shape comparisons (of states Shepard Metzler)
• Size comparisons (Weber fraction or ratio effect)
• What do they tell us about the format of images?
• But this applies to nonvisual properties (e.g.,
  price, taste)

10
More demonstrations of the relation between
vision, imagery (and later action)

• Images constructed from descriptions
• The D-J example(s)
• Perception or inference/guessing
• But there are even more persuasive
  counterexamples we will see later
• The two-parallelogram example
• Amodal completion
• Reconstruals Slezak

11
Dynamic imagery

• Imagining actions Paper Folding

12
Mental rotation
Time to judge whether (a)-(b) or (b)-(c) are the
same except for orientation. Time increases
linearly with the angle between them (Shepard
Metzler, 1971)
13
What do you do to judge whether these two figures
are the same shape?
Is this how the process looked to you?
When you make it rotate in your mind, does it
seem to retain its rigid 3D shape without
re-computing it?
14
Rotation is neither holistic nor impenetrable
15
Mental rotation the real story

• In mental rotation the phenomenology
  motivates the theory of rotation but what the
  data actually show is that,
• Mental rotation is only found when the comparison
  figures are enantiomorphs or if the difference
  between figure pairs can only be expressed in
  figure-centric coordinates eg. they are 3D
  mirror-images
• No rotation occurs if the figures have landmarks
  that can be used to identify the relations among
  their parts.
• Records of eye movements show that mental
  rotation is done incrementally It is not a
  holistic rotation as often reported. If fact
  even the phenomenology is not of a smooth
  continuous rotation.
• The rate of rotation depends on the conceptual
  complexity of both the figure and comparison task
  so that, at least, is not a result of the
  architecture (Pylyshyn, 1979). There are even
  demonstrations that it depends on how the subject
  interprets the figure (Kosslyn, 1994).

16
Mental Scanning

• Hundreds of experiments have now been done
  demonstrating that it takes longer to scan
  attention between places that are further apart
  in the imagined scene. In fact the relation is
  linear between time and distance.
• These have been reviewed and described in
• Denis, M., Kosslyn, S. M. (1999). Scanning
  visual mental images A window on the mind.
  Cahiers de Psychologie Cognitive / Current
  Psychology of Cognition, 18(4), 409-465.

17
Studies of mental scanningDoes it show that
images have metrical space?
Does this show that images are spatial, or have
spatial properties, or that they preserve
metrical spatial properties? (Kosslyn, S. M., T.
M. Ball, et al. (1978). "Visual images preserve
metric spatial information Evidence from
studies of image scanning." Journal of
Experimental Psychology Human Perception and
Performance 4 46-60.
18
The idea of images being in some sense spatial is
an interesting and important claim

• I will discuss this claim at some length later
  because it reveals a deep and all-consuming error
  that runs through all imagery theorizing by
  psychologists, neuroscientists and philosophers.
  
• This is in addition to the errors I discussed
  earlier The idea that subjects understand the
  task of imagining something to be the task of
  pretending they are seeing it, and the idea that
  certain properties of the world are properties of
  the image (the intentional fallacy)

19
Constructing an image

• What determines what the image is like when it is
  constructed from memory or from knowledge?
• After constructing an image can you see novel
  aspects of the imagined situation?
• Examples

20
Imagine seeing these events unfolding
Examples to probe your intuition and your tacit
knowledge

• You hit a baseball. What shape trajectory does
  it trace? It is coming towards you Where would
  you run to catch it? If you have ever played
  baseball you would have a great deal of tacit
  knowledge of what to do in such (well studied)
  cases.
• You drop a rubber ball on the pavement. Tap a
  button every time it hits the ground and bounces.
  Plot height vs time.
• Drop a heavy steel ball at the same time as you
  drop a light ball (a tennis ball), e.g., from the
  leaning tower of Pisa. Indicate when they hit
  the ground. Repeat for different heights.
• Take a clear glass containing a colored liquid.
  Tilt it 45º to the left (counter-clockwise).
  What is the orientation of the liquid?

What is responsible for the pattern shown here?
21
What color do you see when two color filters
overlap?
?
22
Where would the water go if you poured it over a
full beaker of sugar?
Is there conservation of volume in your image?
If not, why not?
23
Seeing Mental Images

• Do images have size?
• Can we say that one image is larger than another?
• If so, what properties do we expect the
  smaller/larger image to have?

24
Do mental images have size?Imagine a
very small mouse. Can you see its whiskers? Now
imagine a huge mouse. Can you see its whiskers?
25
(No Transcript)
26
Do this imagery exerciseImagine a parallelogram
like this one
Connect each corner of the top parallelogram with
the corresponding corner of the bottom
parallelogram
Now imagine an identical parallelogram directly
below this one
What do you see when you imagine the connections?
Did the imagined shape look (and change) like the
one you see now?
27
Slezak figures
Pick one (or two) of these animals and memorize
what they look like. Now rotate it in your mind
by 90 degrees clockwise and see what it looks
like.
28
Slezak figures rotated 90o
29
P 29
Space
30
Images and the representation of spatial
properties

• We need to understand what it could mean for a
  representation to be spatial.
• At the very least it must mean that there are
  constraints placed on the form of the
  representation that do not apply when the
  representation is not spatial.

31
Studies of mental scanningDoes it show that
images have metrical space?
Does this show that images are spatial or have
spatial properties or that they preserve
metrical spatial properties? (Kosslyn, S. M., T.
M. Ball, et al. (1978). "Visual images preserve
metric spatial information Evidence from
studies of image scanning." Journal of
Experimental Psychology Human Perception and
Performance 4 46-60.
32
The idea that images are in some sense spatial is
an interesting and important claim

• I will return to this claim later because it
  reveals a deep and ubiquitous error that runs
  through most (all?) imagery theorizing by
  psychologists, neuroscientists and philosophers.
  This is the error of mistaking descriptive
  adequacy with explanatory adequacy. Lets call
  this conflating, the missing constraint error.
• This is in addition to the two errors I discussed
  earlier
• Ignoring the fact that the task of imagining
  something is actually the task of pretending you
  are seeing it, and
• The mistaken assumption that certain properties
  of the world are properties of the image (the
  intentional fallacy)

33
Connecting Images and Motor actions

• Images and visual-motor phenomena
• S-R Compatibility / Simon effect
• Finkes imagined wedge-prism goggles
• Harrys subitizing-by-pointing

34
Another chapter in the imagery debateThe
interaction of images with vision and motor
control

• One of the properties of mental images that makes
  them appear spatial is that they connect in
  certain ways not only with vision, but also with
  the motor system
• We can point to things in our image!
• We can project our images onto perceived space
  even space perceived in different modalities.
  I believe that this observation is the key to
  understanding the spatial character of images.
• This projection does not require a picture to be
  projected, only the location of a small number of
  features. Over the past few decades I have been
  studying a mechanism called a visual index, or a
  FINST, that is well suited for this task.

35
Both vision and visual imagery have some
connection to the motor system

• There are a number of experiments showing the
  close connection between images and motor
  control
• You can get Stimulus-Response compatibility
  effects between the location of a stimulus in
  space and the location of the response button in
  space,
• Ronald Finke showed that you could get adaptation
  with the position of the misperceived hand that
  was similar to adaptation to displacing prism
  goggles,
• Both these findings provide support for the view
  that the spatial character of images comes from
  something being projected onto a concurrently
  perceived scene and then functioning much as
  objects of perception.
• This is the main new idea in Chapter 5 of Things
  Places)

an image
imagined
36
S-R Compatibility effect with a visual
displayThe Simon effect It is faster to make a
response in the direction of an attended objects
than in another direction
Response for A is faster when YES in on the left
in these displays
37
S-R Compatibility effect with an imagined display
The same RT pattern occurs for a recalled display
as for a perceived one
RT is faster when the A is recalled (imagined)
as being on the left
38
Recall the studies of mental scanning
Does this result show that images have metrical
properties?
Does this result show that images have spatial
properties?
? We showed that the image scanning effect is
Cognitively Penetrable
But the way we compute the time it takes to scan
across an image is by imagining something moving
across the real perceived display. Without this
display, we could not use our time-to-collision
computation to compute the time to cross various
distances on the image because there are no
actual distances on the image! (Pylyshyn Cohen,
1999)
39
Using a concurrently perceived room to anchor
FINSTs tagged with map labels
40
The Spatial character of images

• What does it mean to say that images are spatial?
  
• It means that certain constraints hold among
  spatial measures (e.g., axioms of geometry and
  measure theory, such as triangle inequality,
  symmetry of distances, Euclidean axioms,
  Pythagoras theorem
• That certain constraints hold among distances,
  that certain relations can be defined among these
  distances (e.g., between, farther than),
  that Newtonian Physics holds between the terms
  that are used in explanations (e.g., distances
  and time).
• That mental images and motor control interact
  with one another to some degree so you can
  point to objects in your image.
• Certain visual-motor reflexes are automatic or
  preconceptual ? They are computed within the
  encapsulated Visual Module
• Preconceptual motor control is not sensitive to
  visual illusions, relative to control that is
  computed by the cognitive (seeing as) system.

41
Evidence for a literal spatial display in the
brain

• I will discuss the proposal that V1 is the
  imagery display in the brain. But since the
  conclusion will be that it is not, lets look at
  other options.
• The problem is to explain such phenomena as the
  scanning effect or the size effect without
  assuming a physical display.
• The main alternative to a spatial display is
  something called a functional space. This
  proposal was introduced by Kosslyn in his
  characterization of the depictive nature of the
  image representation.

42
Mental images as depictive representations

• A depictive representation is a type of
  picture?, which specifies the locations and
  values of configurations of points in a space.
• The space in which the points appear need not be
  physical but can be like an array in a computer,
  which specifies spatial relations purely
  functionally?. That is, the physical locations
  in the computer of each point in an array are not
  themselves arranged in an array it is only by
  virtue of how this information is read and
  processed that it comes to function as if it were
  arranged into an array.
• Depictive representations convey meaning via
  their resemblance to an object?.
• When a depictive representation is used, not only
  is the shape of the represented parts immediately
  available to appropriate processes ?, but so is
  the shape of the empty space and one cannot
  represent a shape in a depictive representation
  without also specifying a size and
  orientation?.

43
Form vs Content of images

• As in earlier discussion, one must be careful in
  distinguishing form from content. We know that
  there is a difference between the content of
  images and the content of other (nonimaginal)
  thought Images concern sensory appearances while
  propositions can express most other contents.
• In attributing a special form of representation
  to images one should ask whether some symbolic
  system (e.g., sentences of LOT) would not do.
  Simplicity (Occams Razor) would then prefer a
  single format over two, especially if the one
  format is essential for representing thoughts and
  inferences Fodor, J. A. and Z. W. Pylyshyn
  (1988). "Connectionism and cognitive
  architecture A critical analysis." Cognition
  28 3-71.
• The most promising contents that might require
  different forms of representation are those that
  essentially represent magnitudes. Of the
  magnitudes most often associated with images are
  spatial ones.

• There has been a long-standing debate in
  Artificial Intelligence concerning the advantages
  of logical formats vs other symbol systems vs
  something completely difference (procedure).

44
What is assumed in imagist explanations of mental
scanning?

• In actual vision, it takes longer to scan a
  longer distance because real distance, real
  motion, and real time is involved, therefore this
  equation holds due to natural law
• Time distance speed
• But what ensures that a corresponding relation
  holds in an image? The obvious answer is
  Because the image is laid out in space!
• But what if that option is closed for empirical
  reasons?
• Imagists appeal to a Functional Space which
  they liken to a matrix data structure in which
  some cells are adjacent to other cells, some are
  closer and others further away, and to move from
  one to another it is natural that you pass
  through intermediate cells
• Question What makes these sorts of properties
  natural in a matrix data structure?

45
Thou shalt not cheat

• There is no natural law that requires the
  representations of time, distance and speed to be
  related according to the motion equation. You
  could just as easily imagine an object moving
  instantly or with constant acceleration or with
  any motion relation you like, since it is your
  image!
• There are two possible reason why the observed
  relation
• Actual Time Representation of distance
  Representation of speed
• typically holds in an image-scanning task
• Because subjects have tacit knowledge that this
  is what would happen if they viewed a real
  display, or
• Because the matrix is taken to be a simulation of
  a real physical display, as it often is in
  computer science.
• Notice that in the second case the explanation
  for the Reaction Time comes from the simulated
  real display and not from the matrix.

46
The missing constraint in appeals to space in
both scanning and mental rotation

• What is assumed about the format or architecture
  of the mental representation in the examples of
  mental rotation?
• According to philosopher Jesse Prinz (2002) p
  118,If visual-image rotation uses a spatial
  medium of the kind Kosslyn envisions, then images
  must traverse intermediate positions when they
  rotate from one position to another. The
  propositional i.e., symbolic system can be
  designed to represent intermediate positions
  during rotation, but that is not obligatory.
• This is a very important observation, but it is
  incomplete. One still needs to answer the
  question What makes it obligatory that the
  object must pass through intermediate positions
  when rotating in functional space, and what
  constitutes an intermediate position? These
  terms apply to the represented world, not to the
  representation!

47
The important distinction between architecture
and represented content

• It is only obligatory that a certain pattern must
  occur if the pattern is caused by fixed
  properties of the architecture as opposed to
  being due to properties of what is represented
  (i.e., what the observer tacitly knows about the
  behavior of what is represented)
• If it is obligatory only because the theorist
  says it is, score that as a free empirical
  parameter that any theory can assume.
• This failure of image theories is quite general
  all picture theories suffer from the same lack of
  principled constraints.

48
The important distinction between descriptive and
explanatory adequacy

• It is important to recognize that if we allow one
  theory to stipulate what is obligatory without
  there being a principle that mandates it, then
  any other theory can stipulate the same thing.
  Such a theories are unconstrained so they can fit
  any possible observation i.e., they are able
  to describe anything but explain nothing.
• A theory that does not explain why some pattern
  is obligatory can still be useful the way an
  organized catalog is useful. It may even list the
  features according to which it is organized. But
  it does not give an account of why it is
  organized that way rather than some other way.
  To do that it needs to appeal to something
  constant such as a law of nature or a fixed
  property of the architecture.

49
The important distinction between descriptive and
explanatory adequacy

• We have come back to the distinction between
  architecture- and representation-governed
  process. Consider the parallel between the
  apparent obligatory nature of mental rotation and
  the apparent obligatory nature of the pattern of
  dots and dashes exhibited by the code box (double
  spike occurs before single spike except after a
  long-short-long-short pattern).
• Here is another way to look at what it means to
  be obligatory. Suppose psychological experiments
  show that reaction time is a linear function of
  number of items being processed. As with the
  code box example, what we dont know is whether
  this pattern is fortuitous or principled. There
  are several ways in which we could show that it
  is principled
• If it can be derived from a basic law of nature
  (very rarely happens). Note that this is about
  the pattern of the representation, not the
  pattern of the world imagined.
• If it follows from independently motivated
  assumptions about the architecture
• If it holds for other relevant inputs (where
  being relevant is theory-dependent)

50
How are these obligatory constraints realized?

• Image properties, such as size and rigidity are
  assumed to be inherent in the architecture (e.g.,
  of the display)
• That raises the question of what kind of
  architecture could possibly enforce rigidity of
  shape?
• Notice that there is nothing about a spatial
  display, let alone a functional space, that makes
  it obligatory that shape be rigidly maintained as
  orientation is changed.
• Such rigidity could not be a necessary property
  of the architecture of an image system because we
  can easily imagine that rigidity does not hold
  (e.g. imagine a rotating snake!).
• There is also evidence that mental rotation is
  incremental, not holistic, and the speed of
  rotation depends on the conceptual complexity of
  the shape and the comparison task.

51
Review Why do theories that appeal to a
functional space not explain imagery behavior

• They fall prey to one of the following errors
• The intentional fallacy confounding properties
  of the representation with properties of the
  represented
• Task demands neglecting the fact that subjects
  are being asked to pretend that they are seeing
  something

52
What makes some properties seem natural in a
matrix but not so natural in a symbolic data
structure?

1. A matrix is generally viewed as a two-dimensional
   structure in which specifying the x and y values
   (rows and columns) specifies the location of any
   cell. But thats just the way it is
   conventionally viewed. Rows, columns and cells
   are not actually spatial locations.
2. In a computer there is no requirement that in
   getting from one cell to another one must pass
   through any other specified cells nor is there
   any requirement that there be empty cells between
   any pairs of cells.

53
What makes some properties natural in a matrix
while not so natural in a symbolic data structure?

• The main reason it is natural to view a matrix as
  having spatial constraints is that one is tacitly
  assuming that it represents some space. Then it
  is the represented space that has the
  constraints, not the matrix.
• Notice the subtle succumbing to the intentional
  fallacy again!
• Any constraints that the functional space
  exhibits are constraints extrinsic to the format.
  Such constraints reside in the external world
  which the functional space represents.
• But such extrinsic constraints can be added to
  any model of scanning, including a propositional
  one.

54
What warrants the obligatory constraint?

• But it is no more obligatory that the relation
  between distance, speed and time hold in
  functional space than in a symbolic
  (propositional) representation. There is no
  natural law or principle that requires it. You
  could imagine an object moving instantly or
  according to any motion relation you like, and
  the functional space would then comply with that
  motion since it has no constraints of its own.
• So why does it seem natural for imagined moving
  objects to traverse a functional space than a
  sequence of symbolic representations of
  locations?
• There are at least two reasons why a functional
  space might seem more natural than a symbolic
  representation of space, and both depend on (1)
  subjective experience and (2) the intentional
  fallacy.

55
Where does the obligatory constraint come from?

• There are at least two reasons why the
  following equation holds in the mental image
  scanning task, even though, unlike in the real
  vision case, it does not follow from a natural
  law.
• Actual Time Representation of distance
  Representation of speed
• Because subjects have tacit knowledge that this
  is what would happen if they viewed a real
  display, and they understand the task to be one
  of reproducing properties of this viewing, or
• Because the matrix is taken to be a simulation of
  real space. In that case the reason that the
  equation holds is that it is supposed to be
  simulating real space and the equation holds in
  real space.
• In that case it is not something about the form
  of the representation that provides the
  principled constraint, its the fact that it is
  supposed to be simulating real space which is
  where the obligation comes from. But the same
  thing can be done for any form of representation.

56
Real and functional space

• What is assumed by picture accounts of mental
  imagery experiments, including those involving
  image scanning, image size and image rotation, is
  that images have the properties of a real spatial
  display as viewed by the minds eye. This is what
  provides a principled explanation.
• But this explanation carries a number of
  assumptions, including that images are 2D
  patterns laid out in real space (presumably on
  visual cortex). Because the evidence does not
  support this assumption, pictorialists appeal to
  a functional space.
• What is a functional space and how does it
  explain the scanning or image size findings?

57
What is functional space?

• Because functional space is cited by almost
  every imagery theorist, it deserves some
  attention.
• The main value of a functional space is that it
  has whatever properties we want to assume for it
  i.e., it can be made to fit any data. We
  stipulate that it takes longer to scan greater
  image distances. The law relating distance,
  time and speed does not apply to image distance
  so we assume it.
• For that reason the functional space assumption
  has no advantage over any other assumption about
  how space is represented the properties we
  assign to functional space can be assigned to any
  theory. So the concept of functional space does
  no explanatory work.
• The assumption that functional space must have
  certain obligatory (as Prinz put it) spatial
  properties, relies on one of several mistakes
  which result in these properties seeming more
  natural in a functional space.

58
Why is it natural to assume that functional
space is like real space?

• There are several reasons why a functional
  space, such as a matrix data structure, appears
  to have natural spatial properties (e.g.,
  distances, size, empty places)
• Because when we think of functional space, such
  as a matrix, we think of how we usually interpret
  it.
• A matrix does not intrinsically have distance,
  empty places, direction or any other such
  property, except in the mind of the person who
  draws it or uses it!
• Moving from one cell to another does not require
  passing through intermediate cells unless we
  stipulate that it does. The same goes for the
  concept of intermediate cell itself.

59
Why is it natural to assume that functional
space is like real space?

• Because when we think of a functional space, such
  as a matrix, we think of it as being a way of
  simulating real space in the model making it
  more convenient to build the model which
  otherwise would require special hardware
• This is why we think of some cells as being
  between others and some being farther away.
  This makes properties like distances seem natural
  because we interpret the matrix as simulating
  real space.
• In that case we are not appealing to a functional
  space in explaining the scanning effect, the size
  effect, etc. The explanatory force of the
  explanation comes from the real space that we are
  simulating.
• This is just another way of assuming a real space
  (in the brain) where representations of objects
  are located in neural space
• All the reasons why the assumption of real brain
  space cannot be sustained in explanations of
  mental imagery phenomena apply to this version of
  functional space.

60
Why is it natural to assume that functional
space is like real space?

• Because what we really want to claim is that
  images are displayed on a real spatial surface
  a blackboard. But to model this we would need to
  build a hardware display. An easier way to do
  this is simply to claim explicitly that there is
  a display or even simulate one using software
  (such as Kosslyn, et al. (1979) claim to have
  done).
• This allows us to view some cells as being
  between others and some being farther away.
  This makes properties like distances seem natural
  because we interpret the matrix as simulating or
  standing in for a real spatial display board or
  screen.
• In that case we are not appealing to a functional
  space in explaining the scanning effect, the size
  effect, etc. The explanatory force of the
  explanation comes from the real space that we are
  claiming and simulating. This is just another way
  of assuming a real space (in the brain) where
  representations of objects are located in neural
  space.

61
Functional space and explanatory power

• There is a notion of explanatory power that needs
  to be kept in mind. It is best illustrated in
  terms of models that contain empirical
  parameters, as in fitting a polynomial curve to
  data.
• The general fact about fitting a model to data is
  that the fewer parameters that need to be
  estimated from the data to be fitted, the more
  powerful the explanation. The most powerful
  explanation is one that does not have to use the
  to-be-fitted data to tune the model.
• In terms of the current example of explaining
  results of experiments involving mental imagery,
  appealing to a functional space leaves open an
  indeterminate number of empirical parameters, so
  it provides a very weak (or vacuous) explanation.
• A literal (brain) space, on the other hand, is
  highly constrained since it must conform to
  Euclidean axioms and Newtonian physics
  otherwise it would not be the space of natural
  science. But that kind of space implies that
  images are displayed on a surface in the brain
  and while that is a logical possibility it is not
  an empirical one

62
Explanation and Description

• Another way to look at what is going on is to
  think about the difference between a description
  and an explanation. The two ways of
  characterizing a set of phenomena appear similar
  they both speak of how things are and how they
  change (think of the Code Box example).
• But a description of a systems behavior can
  apply to many different types of system with
  different mechanisms and different causal
  properties. And the same mechanisms can also
  produce very different behaviors under different
  circumstances. Although a general statement of
  what constitutes scientific explanation and how
  it differs from description has a long and
  controversial history, the simple Code Box
  example will suffice to suggest the distinction I
  have in mind.

63
Cognitive Penetrability again

• A description states the observed generalizations
  (the observed patterns of behavior). The
  explanation goes beyond this. The difference is
  related to the question of how mutable a set of
  generalizations are and what types of effects can
  lead to changes in these generalizations.
• Causal accounts tend to have a longer time scale
  and when they change they tend to change
  according to different sorts of principles than
  those that describe the patterns.
• Notice that we have come back to the criterion of
  cognitive penetrability. According to this way
  of looking at the question of explanatory
  adequacy, an theory meets the criteria of
  explanatory adequacy if it describes the
  architecture of the system and its operation.

64
A note about time scales and types of changes

• Causal accounts tend to have a longer time scale
  and when they change they tend to change
  according to different sorts of principles than
  those that describe the patterns.
• Consider the Code Box example.
• Changes that are not architectural tend to occur
  rapidly different patterns are observed simply
  because different topics or words or even
  languages might be transmitted.
• Changes that are architectural require altering
  which letters or other symbols are transmitted
  (e.g., they may be numerals) or changing whether
  the outputs consist of short and long pulses that
  are interpretable as Morse Code. They require
  what we might think of as rewiring.

65
(No Transcript)
66
A different way of approaching the question of
spatial representation

• I offer a provisional proposal that preserves
  some of the advantages of the global spatial
  display, but assumes that the relevant spatial
  properties are in the perceived world and can be
  accessed if we have the right access mechanisms
  for selecting and indexing objects in the
  perceived world
• Lets call this the Index Projection Hypothesis
  because it suggests that mental objects are
  somehow projected onto and associated with
  perceived objects in real space
• But this proposal is very different from
  image-projection because only a few object-labels
  are projected not the rich visual properties
  suggested by the phenomenology

67
The Image Projection Hypothesis

• This projection hypothesis relies on the
  spatial locations of objects in the concurrently
  perceived world to meet the conditions outlined
  earlier. It rests on two assumptions
• We have a system of pointers (viz, the FINST
  perceptual index mechanism to be described) by
  which a small number (n4) of objects in the
  world can be selected and indexed. Indexes
  provide demonstrative references to individual
  targets qua individuals, that keep referring to
  these objects despite changes in their location
  or any other properties.
• When we perceive a scene that contains indexed
  objects, our perceptual system is able to treat
  those objects as though they were assigned unique
  labels. Thus our perceptual system is able to
  detect configurational properties among the
  indexed objects.

68
The index projection hypothesis (2)

• The hypothesis claims that the subjective
  impression that we have access to a panorama of
  detailed high-resolution perceptual information
  is illusory. What we have access to is only
  information about selected or indexed objects.
• We have the potential to obtain other information
  from more of the scene through the use of our
  system of perceptual indexes. This is the basic
  insight expressed in the world as external
  memory slogan or the situated cognition
  approach.
• In reasoning using mental images we may assign
  indexes to perceived objects based on our memory
  of (or our assumptions about) where certain
  mental objects are located
• But notice that the memory representation is
  itself not used in spatial reasoning and
  therefore need not meet the spatial constraints
  listed earlier it can be in some general LOT

69
Examples of the projection hypothesis

• To illustrate how the projection hypothesis
  works, first consider index-based projection in
  the visual modality, where indexes can convert
  some apparently mental-space phenomena into
  perceived-space phenomena (more on the non-visual
  case later)
• Examples from some mental imagery experiments
• Mental scanning (Kosslyn, 1973)
• Mental image superposition (Podgorny Shepard,
  1978)
• Visual-motor adaptation (Finke, 1979)
• S-R compatibility to imagined locations (Tlauka,
  1998)

70
Studies of mental scanningOften cited to suggest
that spatial representations are literally
spatial and have metrical properties
71
Brain image or index-based projection?

• A way to do this task
• Associate places on the memorized map with
  objects located in the same relative locations in
  the world that you perceive (e.g., the room you
  are in)
• Move your attention or gaze from one place to
  another as they are named

72
Using a perceived room to anchor FINSTs tagged
with map labels
73
Using vision with selected labeled objects

• If you project the pattern of map places by
  indexing objects in the room in front of you that
  correspond to the memorized relative locations,
  then you can scan attention from one such indexed
  object to another. The relation time distance
  ? speed holds because the space you are scanning
  is the real physical space in the room.
• You can also use the indexed objects to infer
  configurational properties you may not have
  noticed, despite memorizing the location of
  objects. e.g.
• Which 3 or more places on the map are collinear?
• Which place on the map is furthest North (or
  South, East, West)?
• Which 3 places form an isosceles triangle?
• Such configurational consequence can be detected
  as opposed to logically inferred, so long as they
  involve only a few places, because the visual
  system can examine the indexed objects in the
  scene

74
Connecting Images and Motor actions

• Images and visual-motor phenomena
• S-R Compatibility / Simon effect
• Finkes imagined wedge goggles
• Harrys subitizing-by-pointing

75
Both vision and visual imagery have some
connection to the motor system

• There are a number of experiments showing the
  close connection between images and motor
  control
• You can get Stimulus-Response compatibility
  effects between the location of a stimulus in
  space and the location of the response button in
  space,
• Ronald Finke showed that you could get adaptation
  with the position of the misperceived hand that
  was similar to adaptation to displacing prism
  goggles,
• Both these findings provide support for the view
  that the spatial character of images comes from
  something being projected onto a concurrently
  perceived scene and then functioning much as
  objects of perception.
• This is the main new idea in Chapter 5 of Things
  Places)

an image
imagined
76
In all these cases you only need indexes to a few
visual objects located in appropriate places

• In all examples we have seen, the results can be
  explained without appealing to a mental display,
  if you assume that
• You can index a few visible objects (including
  texture elements on an apparently plain surface)
  and
• The visual system can treat indexed objects as
  distinct or visually labeled

77
This story is plausible for visual cases, but how
does it work without vision (e.g., in the dark)?

• We must rely on our remarkable capacity to orient
  to (point to, navigate towards, ) perceived or
  recalled objects (including proprioceptive
  objects) in space without vision
• ? Call this general capacity our spatial sense
• How can the projection hypothesis account for
  this apparently world-centered spatial sense
  without assuming a global allocentric frame of
  reference?
• Answer Just as it does with vision, by anchoring
  represented objects to (non-visually) perceived
  objects in the world

78
The spatial sense and the projection hypothesis

• Indexing non-visual objects must exploit
  auditory and proprioceptive signals, and perhaps
  even preparatory motor programs (the
  intentional frame of reference proposed by
  Anderson Bruneo, 2002 Duhamel, Colby
  Goldberg, 1992)
• Is there some special problem about
  proprioceptive inputs that makes them different
  from visual inputs?

79
Is there a problem with proprioceptive inputs
indexing objects the way visual indexes do?

• Unlike visual objects, proprioceptive objects
  are not fixed in an allocentric frame of
  reference or are all objects the same?
• Notice that in vision and audition, even though
  static objects are fixed in an allocentric frame
  of reference, they nonetheless move relative to
  sensors, so their location in an allocentric
  frame must be updated as the proximal pattern
  moves (Andersen, 1999 Stricanne, Anderson
  Mazzoni, 1996)
• The neural implementation of FINST indexes in
  vision requires an active updating process of
  some kind
• Maybe the same updating operation can also yield
  the sense of same location in space for
  proprioceptive objects
• There are good reasons to think that
  proprioceptive signals may also be given in an
  allocentric frame of reference! (Yves Rosetti)

80
What is the real problem of our sense of space?

• In order to solve the problem of how we index
  objects in the world using proprioceptive inputs
  we need to solve the problem of how we recognize
  two such inputs as corresponding to actions
  (e.g., reaching) towards the same object in the
  world
• This is the problem of the equivalence of
  movements, or of proprioceptive inputs,
  corresponding to the same object it is the
  problem that Henri Poincaré recognized as the
  central problem of understanding our sense of
  space (in Poincaré Why space has three
  dimensions Les Dernier Penseés, 1913)
• Solving the equivalence problem would solve the
  problem of coordinating signals across frames of
  reference
• Thats why mechanisms of coordinate
  transformation are of central importance they
  generate the relevant equivalences!

81
Assumption Coordinate transformations are the
basis for the illusory global frame of reference

• A coordinate transformation operation takes a
  representation of an object relative to one
  coordinate system say retinal coordinates and
  produces a representation of that object relative
  to another frame of reference say relative to
  the location of a hand in proprioceptive or
  kinematic coordinates
• Coordinate transformations define equivalence
  classes of proprioceptive inputs that correspond
  to actions (e.g., reaching, eye movements)
  towards the same object in space
• Such transformations are well-known and
  ubiquitous in the brain (especially in posterior
  parietal cortex and superior colliculus)
• A consequence of these mechanisms is that, as
  (Colby Goldberg, 1999) put it, Direct
  sensory-to-motor coordinate transformation
  obviates the need for a single representation of
  space in environmental coordinates (p319)

82
Coordinate transformations need not transform all
points in a given frame of reference

• Coordinate transformations need not transform all
  points (including points in empty space) or all
  sensory objects Only a few selected objects need
  to be transformed at any one time
• The computational complexity of coordinate
  transformations can be made tractable by only
  transforming selected objects (as is done by
  matrix operations in computer graphics)
• This idea is closely related to the
  conversion-on-demand hypothesis of Henriques et
  al. (1998) and Crawford et al. (2004).
• In the Henriques et al COD proposal, visual
  information about object locations is held in a
  gaze-centered frame of reference and objects are
  converted to motor coordinates when needed

83
Coordinate transformations define equivalence
classes of gestures which individuate
proprioceptive objects just the way that FINST
indexes do in vision

• Coordinate transformations compute equivalence
  classes of proprioceptive signals s
  corresponding to distinct motor actions to
  individual objects in real space. The equivalence
  class is given by s s' iff there is a
  coordinate transformation between S ? S'
• As in the visual case, only a few such
  equivalence classes are computed, corresponding
  to a few distal objects that were selected and
  assigned an index, as postulated in FINST Theory
• We can thus bind several objects of thought to
  objects in real space (including sensory
  objects perceived in proprioceptive modalities)
• This can explain the spatial character of
  spatial representations, just the way they did in
  the purely visual cases illustrated earlier

84
Mental imagery and neuroscience

• Neuroanatomical evidence for a retinotopic
  display in the earliest visual area of the brain
  (V1)
• Neural imaging data showing V1 is more active
  during mental imagery than during other forms of
  thought
• The form of activity differs for small vs large
  images in the way that it differs when viewing
  small and large displays
• Transcranial magnetic stimulation of visual areas
  interferes more with imagery than other forms of
  thought
• Clinical cases show that visual and image
  impairment tend to be similar (Bisiach, Farah)
• More recently psychophysical measures of images
  shows parallels with comparable measures of
  vision, and these can be related to the receptive
  cells in V1

85
Status of different types of evidence in the
debate about the form of mental images

• Phenomenology. Is it epiphenominal?
• Neuroscience evidence for
• Role of vision
• Type and location of neural structures
  underlying images
• Are the neural mechanisms for early vision used
  in imagery?
• Does neuroanatomy provide evidence for the nature
  of depictive representations.

86
Neuroscience has shown that the retinal pattern
of activation is displayed on the surface of the
cortex
There is a topographical projection of retinal
activity on the visual cortex of the cat and
monkey.
Tootell, R. B., Silverman, M. S., Switkes, E.,
de Valois, R. L. (1982). Deoxyglucose analysis of
retinotopic organization in primate striate
cortex. Science, 218, 902-904.
87
Problems with drawing conclusions about the
nature of mental images from neuroscience data

• The capacity for imagery and for vision are known
  to be independent. Also all imagery results are
  observed in the blind.
• Cortical topography is 2-D, but mental images are
  3-D all phenomena (e.g. rotation) occur in
  depth as well as in the plane.
• Patterns in the visual cortex are in retinal
  coordinates whereas images are in
  world-coordinates
• Your image stays fixed in the room when you move
  your eyes or turn your head or even walk around
  the room
• Accessing information from an image is very
  different from accessing it from the perceived
  world. Order of access from images is highly
  constrained.
• Conceptual rather than graphical properties are
  relevant to image complexity (e.g., mental
  rotation).

88
Problems with drawing conclusions about mental
images from the neuroscience evidence

1. Retinal and cortical images are subject to
   Emmerts Law, whereas mental images are not
2. The signature properties of vision (e.g.
   spontaneous 3D interpretation, automatic
   reversals, apparent motion, motion aftereffects,
   and many other phenomena) are absent in images
3. A cortical display account of most imagery
   findings is incompatible with the cognitive
   penetrability of mental imagery phenomena, such
   as scanning and image size effects
4. The fact that the Minds Eye is so much like a
   real eye (e.g., oblique effect, resolution
   fall-off) should serve to warn us that we may be
   studying what observers know about how the world
   looks to them, rather than what form their images
   take.

89
Problems with drawing conclusions about mental
images from the neuroscience evidence

• Many clinical cases can be explained by appeal to
  tacit knowledge and attention
• The tunnel effect found in vision and imagery
  (Farah) is likely due to the patient knowing what
  things now looked like to her post-surgery
• Hemispatial neglect seems to be a deficit in
  attention, which also explains the
  representational neglect in imagery reported by
  Bisiach
• A recent study shows that imaginal neglect does
  not appear if patients have their eyes closed.
  This fits well in the account I will offer in
  which the spatial character of a mental images
  derives from concurrently perceived space.
• What if colored three-dimensional images were
  found in visual cortex? What would that tell you
  about the role of mental images in reasoning?
  Would this require a homunculus?

90
Should we welcome back the homunculus?

• In the limit if the visual cortex contained the
  contents of ones conscious experience in imagery
  we would need an interpreter to see this
  display in visual cortex
• But we will never have to face this prospect
  because many experiments (including ones by
  Kosslyn) show that the contents of mental images
  are conceptual (or, as Kosslyn puts it, contain
  predigested information).
• And finally, it is clear to anyone who thinks
  about it for a few seconds that you can make your
  image do whatever you want and to have whatever
  properties you wish.
• There are no known constraints on mental images
  that cannot be attributed to lack of knowledge of
  the imagined situation (e.g., imagining a 4D
  cube).
• All currently claimed properties of mental images
  are cognitively penetrable.

91
Explaining mental scanning, mental rotation and
image size effects in terms of functional space

• When people are faced with the natural conclusion
  that the iconic position entails space (as in
  scanning and size effects) they appeal to
  functional space
• A Matrix in a computer are often cited as an
  example
• Consider a functional space account of scanning
  or of mental rotation
• Why does it take longer to scan a greater
  distance in a functional space?
• Why does it take longer to rotate a mental image
  a greater angle?

92
Why do conscious contents misguide us?

• The contents that appear in our conscious
  experience almost always concern what we are
  thinking about and not what we are thinking with
  with content rather than form.
• The processes that we see unfolding in our mind
  are almost always attributable to what we know
  about how the things we are thinking about would
  enfold, rather than being due to laws that apply
  to our cognitive architecture. cf Code Box
• We should take seriously the possibility that
  (almost) all constraints and law-like behaviors
  of objects of our experience are constraints due
  to our knowledge rather than of the mental
  architecture. Notice the mental rotation example
  and the mistake that Jesse Prinz makes.

93
This is what our conscious experience suggests
goes on in vision
Kliban
94
This is what the demands of explanation suggests
must be going on in vision
95
Imagine this shape rotating slowly
Is this how it looked to you?
When you make it rotate in your mind, does it
retain its rigid 3D shape without re-computing
it? Would you expect to see this kind of
information process? Does the experience in this
case reassure you that the rotation was smooth?
Are you sure something rotated?
96
What about the evidence of conscious experience?
Is it irrelevant?

• I have often been accused of relegating conscious
  experience to the category of epiphenomena
  something that accompanies a process but does not
  itself have a role in its causation.
• But images are not illusory or unnatural, they
  are quite real. The problem is that people have
  theories of the causal or information-processing
  that underlies these phenomena and these theories
  are almost always false because they assume a
  simple and obvious mapping from the experience to
  the computational or brain states so that we can
  see the form of the representation.
• The connection between conscious experience and
  information processing is deeply mysterious (its
  the mind-body problem). But one thing we do know
  is that the sequence of events that unfolds when
  we imagine something does not reveal causal laws
  because there are no causal laws of conscious
  states as conscious states.

97
The important distinction between architecture
and represented content

• It is only obligatory that a certain pattern must
  occur if the pattern is caused by fixed
  properties of the architecture, as opposed to
  being due to properties of what is represented
  (i.e., what the observer tacitly knows about the
  behavior of what is represented)
• If it is obligatory only because the theorist
  says it is, then score that as a free empirical
  parameter (a wild card)
• If we allow one theory to stipulate what is
  obligatory without there being a principle that
  mandates it, then any other theory can stipulate
  the same thing. Such theories are unconstrained
  and explain nothing.
• This failure of image theories is quite general
  all picture theories suffer from the same lack of
  principled constraints

98
The important distinction between architecture
and represented content

• It is only obligatory that a certain pattern must
  occur if the pattern is caused by fixed
  properties of the architecture, as opposed to
  being due to properties of what is represented
  (i.e., what the observer tacitly knows about the
  behavior of what is represented)
• If it is obligatory only because the theorist
  says it is, then score that as a free empirical
  parameter (a wild card)
• If we allow one theory to stipulate what is
  obligatory without there being a principle that
  mandates it, then any other t