Sound Squared Exploring Visual Perception Through Auditory Sensory Substitution

1
Sound Squared?Exploring Visual Perception
Through Auditory Sensory Substitution

• Alex Storer
• May 3, 2006
• CN 730

2
Your (Near) Future

• Introduction to Sensory Substitution
• Using the vOICe
• Training Paradigm
• Testing Paradigm
• Discussion

3
Sensory Substitution
Visual World
Visual System
Cortex
Visual Transduction System
Other Sensory Modality
Processing
4
Examples

• Tongue Stimulation
• Bach-y-Rita, 1970

Auditory stimulation Meijer, 1992
5
But Does It Work?

• Identify Geometric Shapes
• See certain illusions
• Startle Response
• Subjective Feeling of Vision
• Very Limited Resolution
• Limited Mobility
• Difficult to Learn

6
Why is this Important for Science?

• Remove effects of early visual processing

7
Why is this Important for Science?

• Development of a new sense
• Contingencies for consciousness
• Role of plasticity
• Perceptual learning
• Differential processing in early/late/not blind
• And more!

8
The vOICe

• Presents visual information through sound

9
The vOICe From All Angles

• System already implemented and online
• Popular with some late-blind users
• Simple, mobile and comprehensible

• Not space-variant, terrible time resolution
• Steep learning curve!
• Learning a new sense, or learning how to best
  use the system?

10
Training
20 Hrs Using the vOICe Wandering CNS Shape Iden
tification
11
Training in Real Life

• Video

12
Training on Shapes

• Shapes Presented
• Asked to identify location, shape and
  orientation
• Feedback Given
• Lots and lots of debugging
• 5 sessions without orientation, 5 with

13
18
14
(No Transcript)
15
19
16
(No Transcript)
17
20
18
(No Transcript)
19
Results
20
Ms Supernatural Powers

• Recognize and locate simple shapes
• Navigate the world (with difficulty)
• Identify objects in the world (with difficulty)
• Find the walking stick on the ground
• Weakness Space/resolution trade off
• Meta-comment Incredibly top-down

21
My Elegant Experiment

• A task that translates well to naïve users
• Address an interesting question in vision
• Design a task that is covered by the
  supernatural powers
• Solution A same-or-different task involving
  illusory contours

22
Further Elegance
Changes Consistent or Inconsistent Each
pac-man can be randomly rotated The entire
shape is randomly rotated and placed randomly No
Feedback!
Inconsistent Consistent
23
The Naïve Subject
24
Aligned Consistent
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
Random Consistent
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
Aligned Inconsistent
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
Results
37
Interpretation

• No reporting of illusory contour perception
• No significant difference between control and
  trained subjects
• Only one subject in each category
• Everything could be totally random
• But I dont think so!

38
Interpretation

• Recall Natural Image Statistics

Geisler, Perry, Super,and Gallogly 2001
Has the system user learned statistical
properties of the visual world?
39
Further Experiments

• Perform testing while training takes place
• Use a control trained on only shapes or only
  real world
• Similar effects in early or late blind users?
• Or something completely different
• Craik-OBrien-Cornsweet?
• Motion Perception?

40
In the Rear View

• This study can be seen as a proof of concept
• M didnt report any strong visual experience
• Future studies need a robust, effective system
  that is equally simple
• Potentially very powerful way to address basic
  questions

41
Thanks!

• The ever-anonymous M and A
• The ever-eponymous Ennio
• This ever-anomalous flying dog