On crossmodal sensory streams...

1
On cross-modal sensory streams...
Invited Panel Presentation at SIGGRAPH '98,
Orlando, Florida

• The vOICe
• by
• Peter B.L. Meijer

Philips Research Laboratories Peter.B.L.Meijer_at_phi
lips.com
2
The vOICe Seeing with your Ears

• Basic facts
• Blind people cannot see...but usually they can
  hear
• Multimedia computing allowsaudio-visual
  transformations

3
The vOICe Seeing with your Ears

• Basic facts
• Blind people cannot see...but usually they can
  hear
• Multimedia computing allowsaudio-visual
  transformations

4
The vOICe implementations
Java applet
Hardware prototype
Windows 95/98/NT
http//www.seeingwithsound.com/
5
The vOICe image-to-sound mapping
Frequency
Time
Sum of sinusoids
Amplitude
6
The vOICe image-to-sound mapping
Frequency
Spectrographic synthesis
Time
7
The vOICe image-to-sound mapping
Could this really work? Check map sounds
back to images
Yes!(at least technically)
Theory
8
The vOICe image reconstruction
The vOICe
Soundscape
Spectrogram
9
The vOICe image reconstruction
Soundscape
10
The vOICe and human speech
Pronouncing one image per second
11
The vOICe mental imagery?
Representation may dramatically affect
recognition!
12
The vOICe limitations

• Issues
• Frequency-Time uncertainty
• Physiology / Perception
• just-noticeable difference (JND)
• critical bands (human cochlea)

13
The vOICe limitations

• Issues
• Frequency-Time uncertainty
• Physiology / Perception
• just-noticeable difference (JND)
• critical bands (human cochlea)
• auditory streaming/segregation
• neural pathways bandwidth
• neural processing plasticity

14
The vOICe limitations

• Issues
• Frequency-Time uncertainty
• Physiology / Perception
• just-noticeable difference (JND)
• critical bands (human cochlea)
• auditory streaming/segregation
• neural pathways bandwidth
• neural processing plasticity

• Psychology / Education
• sounds (too) unpleasant
• minimum required results
• acceptable training effort
• motivation for learning
• training programs

15
Conclusions

• The vOICe approach
• needs more research
• is cheap technology
• is non-invasive

16
Conclusions

• The vOICe approach
• needs more research
• is cheap technology
• is non-invasive
• is available Now!

http//www.seeingwithsound.com/