Interaction with Sound

1
Interaction with Sound

• Explorations beyond the Frontiers of
• 3D virtual auditory Environments

Niklas Röber, AG GAMES/UISE, ISG Otto-von-Guericke
University Magdeburg, Germany
2
Motivation and Introduction
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• Can one interact with a 3D environment solely
  using sound?
• Auditory vs. visual perception
• Scene sonification
• Scene auralization

3
Research Goals
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• Definition, analysis and classification of
• 3D virtual (augmented) auditory environments
• Development of
• Methods for scene and object sonification
• Interaction and scene exploration techniques
• A 3D audio framework
• Stability of sound rendering
• Quality of 3D sound spatialization
• Achievable degree of realism

4
Auditory Perception
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• Environmental perception
• Location and distance
• 3D sound perception
• Individual per person
• Based on head-related transfer function
• (HRTF / HRIR)
• Auditory perception
• Auditory Gestalt Mach86, Ehrenfels90
• Auditory scene analysis Bregman90
• Selective listening Williams92

5
Research Overview
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• 3D scene and object sonification
• User input and spatial interaction techniques
• Efficient sound rendering
• Example applications

6

• 3D virtual auditory Environments

7
Auditory Displays
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• Convey abstract information using auditory means
• No screens necessary
• High-temporal resolution
• Rapid detection, alerting
• Display classification analogic symbolic
  continuum Kramer 94
• Auditory icons Gaver89, Mynatt92
• Earcons, hearcons Blattner89, Bölke95

• Lack of absolute values
• Lack of orthogonality
• Limited spatial precision

8
3D auditory User Interfaces
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• Spatial interaction and gestures
• Balance function with aesthetics Vickers06
• Related work
• Auditory menu Mynatt92, Blattner92, Brewster92,
  
• Spatial auditory display Cohen91, Wenzel92,
  Walker00, Marentakis05,
• 3D auditory menu Begault94, Crispien96,
  Kobayashi98, Frauenberger07,
• Main questions
• Improving 3D perception with personalized HRTFs
• Task-related cataloguing of sonification
  techniques
• Spatial sonification / interaction for virtual 3D
  environments

9
3D virtual auditory Environments
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• Focusing on 3D virtual audio-only environments
• Sonification (output)
• Orientation, navigation, exploration, and
  manipulation
• Binaural display
• Interaction (input)
• Key element real-time interaction
  sonification
• Natural listening cues
• Head-tracking and spatial interaction

10
Scene Auralization
Introduction Auditory Environments Sound
Simulation Applications Conclusions
(ICAD 2004, CGAIDE 2004)

• 3D scene auralization
• Sounds, speech, music
• Room acoustics
• Not sufficient!
• Non-realistic auditory scene representation
• Non-physically based auralization
• Additional non-object sounds
• Exaggerated / suppressed parameters
• Situation-based display styles
• (aud. landmarks, aud. texture, path
  sonification, menu)

11
Global Sonification / Interaction
Introduction Auditory Environments Sound
Simulation Applications Conclusions
(ICAD 2004, CGAIDE 2004, ICAD 2005)

• Tasks orientation, navigation, and scene
  exploration
• Passive
• Hearcons / auditory icon
• Auditory landmarks
• Interactables (object grouping)
• Guiding systems
• Detail auditory lens
• Tasks orientation, searching
• Focus on a particular direction and / or
  class of objects
• (zoom to objects on desk)

• Active
• Pathway sonification (soundpipes)
• Auditory lens / hear frustum

12
Local Sonification / Interaction
Introduction Auditory Environments Sound
Simulation Applications Conclusions
(ICAD 2004, CGAIDE 2004, ICAD 2005)

• Tasks orientation, object selection and
  manipulation
• Passive
• Hearcons / auditory icon
• Detail auditory textures
• Ext. of Mynatts parameter nesting Mynatt92
• Describes an object and its function using
  different auditory representations
• Interaction dependencies
• (door locked, door opening, door open)

• Active
• Radar / sonar
• 3D pointing / selection (cane)
• Auditory textures
• Dependency modeling

13
Detail Usability Analysis
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• Usability test (14 participants)
• Examination of developed sonification /
  interaction techniques
• Qualitative evaluation
• Tasks
• Orientation and navigation
• Find and select specific objects
• Test setting
• Polhemus FASTRAK (head-tracking, stylus)
• Gamepad
• HiFi Headphones

14
Detail Results
Introduction Auditory Environments Sound
Simulation Applications Conclusions
Auditory Lens 1 very bad 2 bad / worse 3
neither 4 good / better 5 very
good Ring Menu
µ 3.79 µ 3,57 µ
4,00 s 0.8 s 1.0
s 1.0 µ 3,64 µ
4,64 s 1.1 s 0.5

• Orientation with lens
  Navigation with lens
  Assist. of head-tracking
• Overall interaction
  Item localization Gamepad
  vs. gestures

15
Discussion
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• 3D audio framework
• Non-realistic auditory scene representation
• 3D scene / object sonification
• Sound rendering implemented using OpenAL
• Problem!
• OpenAL has several severe restrictions
• Generalized HRTFs
• Basic environmental modeling

16

• Efficient Sound Rendering and Simulation

17
GPU-based Sound Signal Processing
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• Very efficient for impulse response filtering
• GPU-based sound signal processing Whalen05,
  Gallo04
• General sound effects (chorus, reverb, )
• 3D sound rendering convolution / frequency
  weighting (10 bands)

18
GPU-based Room Acoustic Simulation
Introduction Auditory Environments Sound
Simulation Applications Conclusions
(ICMC 2006, DAFx 2007)

• Smith92, VanDuyne93, Bilbao04,
  Campos05
• Wave propagation using time-domain difference
  model
• Computationally very complex
• Applicable to lower frequencies

• Ray/Energy-based Simulation Funkhauser02,
  Tsingos04, Jedrzejewski04
• Approximates sound waves using directional rays
• Applicable to middle and higher frequency ranges

Wave-based Simulation
19
Detail Optimal Sampling
Introduction Auditory Environments Sound
Simulation Applications Conclusions
(ICMC 2006)

• Hexagonal lattices provide a higher packing
  density Conway76
• Optimal sampling BCC lattice
• Unit length increases to
• Sampling efficiency
• 8 neighbors with 4 axes of propagation
• Less pronounced frequency dispersion error
• Update frequency with unit length changes
  to

20
Detail GPU Implementation
Introduction Auditory Environments Sound
Simulation Applications Conclusions
(ICMC 2006)

• Based on 3D textures, fragment shader and 3D
  framebuffer-objects
• Shader samples texture using screen aligned
  slicing quads
• BCC decomposed into two 3D textures
• Two nodes computed in one step

21
Discussion
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• 3D waveguide meshes
• Improved efficiency (20x 60x)
• Improved simulation results for BCC lattice
• Ray acoustic simulation
• Real-time simulation up to 30k models (incl.
  auralization)
• Integration of wave-based effects (diffraction)
• Frequency-based material modeling
• Promising virtual HRIR simulations
• Simulations exhibit all important features
• On the way to personalized HRTFs

HRIR horizontal plane
22
Applications and Case Studies I
Introduction Auditory Environments Sound
Simulation Applications Conclusions
(ICAD 2005, Gamesconference 2005, AudioMostly
2006)

• Audio-only computer games
• Development of games that are played solely
  through listening.
• Three action, one auditory adventure game
• Usability test
• Augmented audio reality (AAR)
• Enhancing a real environment with additional
  auditory information.
• Self developed AAR system
• Usability test

Audiogame Mosquito AAR Game The
hidden Secret
23
Applications and Case Studies II
Introduction Auditory Environments Sound
Simulation Applications Conclusions
(CGAIDE 2004, TIDSE 2006, AudioMostly 2007)

• Interactive audiobooks
• Combining audiobooks with interactive elements
  from computer games.
• Non-linear story graph with variable degree of
  interaction
• Usability tests
• Scene authoring environment
• Extension of audio framework
• Authoring of 3D sound sources, acoustics,
  auditory textures and ring menu systems

Storytree The Pit and the Pendulum Autho
ring Environment
24
Summary and Contribution
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• 3D virtual auditory environments (ICAD 04,
  CGAIDE 04, ICAD 05, AM 08)
• Non-realistic auditory scene representation
• 3D scene and object sonification / interaction
• 3D audio framework
• Graphics-based sound rendering (ICMC 06, DAFx
  07)
• 3D waveguide meshes with optimal sampling
• Ray acoustics simulation with diffraction and
  material modeling
• Virtual HRIR simulations
• Applications and case studies (DIGRA 05, TIDSE
  06, AM 06, AM 07)
• Audiogames and interactive audiobooks
• Augmented audio reality

25
Conclusions
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• 3D auditory environments are as effective as
  visual environments
• Spatial sonification / interaction
• High-quality sound rendering
• Audio-centered design
• Domains
• Audiogames and interactive audiobooks
• Augmented audio reality
• Aiding the visually impaired

26
Future Work
Introduction Auditory Environments Sound
Simulation Applications Conclusions

• User interface design
• Multi-user presentation and interaction
• Advanced gestures
• Perceptual presentation
• Acoustic rendering
• Comparison with real-world measurements (Bell
  Labs Box)
• Experimentation with virtual HRIR measurements
• Augmented audio reality
• Improvement of positioning accuracy / system
  latency

27
Thank you!

• Questions?
• http//x3t.net/thesis.html
• A Thank You to everyone who helped and
  participated in this research,
• especially to all my students!

28
Promotionsfeier
Appendix

• Beginn 18 Uhr
• Kleiner Saal der Festung Mark
• ( Eingang Jakobstrasse, eine Treppe hoch )