RealTime Auralization of Sound in Virtual 3D Environments

1
Real-Time Auralization of Sound in Virtual 3D
Environments

• by Scott McDermott
• sdm1718_at_louisiana.edu

2
Overview Objective

• Develop an adaptive virtual environment that
  simulates real-time generation of 3D sound.
• Design algorithms to efficiently and effectively
  compute realistic 3D sounds in this environment.
• Apply these techniques to various applications,
  including simulations, virtual reality, gaming,
  and modeling.

3
Outline

• Sound Perception
• Digital Sound and Computers
• 3D Sound Approximations
• True 3D Sound

• Surround Sounds (Stereo Expansion) Approach
• Head Response Transfer Function (HRTF) Approach
• Beam Tracing Approach
• The Graphics Analogy

4
Sound Perception

• When we hear a sound, we automatically obtain
  certain information about the source

• Direction
• Distance
• Elevation
• Environmental conditions
• Status of source

5
Sound Perception

• 8 types of cues for sound spatialization 1

• Interaural Delay Time (direction)
• delay between time arrives at each ear (0 to 0.63
  ms)
• Head Shadow (direction and distance)
• difference in volume from one ear to the other
  (up to 9 dB)
• Pinna Response (direction and elevation)
• outer ear filters sound, compare between two ears
• Shoulder Response (elevation and direction)
• reflections off upper body (1-3 kHz)

• Head Motion
• move head to re-evaluate these filters
• Vision
• ignore audio cues if different from visual
• Early Echo Response (distance and direction)
• echos from environment (50 to 100 ms)
• Reverberation (distance and direction)
• late dense echos from environment (gt 100 ms)

6
Sound Perception

• An environment with true 3D sound will need to
  take all of these into account.
• It must also be able to perform calculations and
  apply filters in real-time.
• The result must be convincing to the listener and
  enhance the virtual experience.

7
Sound in the Digital World

• Sound in the physical world exists as waves of
  pressure changes.
• A microphone converts pressure changes to changes
  in voltage.

• An analog to digital convert changes these
  voltage signals to discrete digital signals.
• A computer stores, manipulates, and retransmits
  these abstractions of sound.

• The sounds can be stored in various formats and
  qualities (such as mono or stereo, 8 or 16 bit,
  11 or 44 kHz).
• The reverse of this process allows the computer
  to re-generate the sound.

Blah blah blah
8
3D Sound, The Basics

• In a virtual 3D environment, sound can originate
  from an infinite number of locations relative to
  the observer.
• Ideally, when the observer hears the sound it
  should take into account the environment.
• Specifically

9

• DistanceCauses sound to arrive at different
  times.

10

• Reflection ReverberationCauses copies of
  the sound to arrive at different times.

11

• Diffraction RefractionCauses sound to bend
  around objects or arrive at different times.

12

• Absorption AttenuationCauses the sound to be
  weaker when it arrives.

13
3D Sound Approximations Surround Sound

• Surround sound uses various filters to simulate
  the effects of sound spatialization.
• These filters create effects such as
  reverberation, localization, and attenuation.
• Sound paths are not calculated.
• Used in most theaters and home entertainment
  units.

14
Surround Sound

• The user is situated with a set of speakers
  around him.
• To simulate 3D localization, sound is played
  louder, out of phase, and/or at slightly
  different times from each speaker.
• Comes in a variety speaker placement setups 8

Dolby 5.1
Two Speaker Stereo
Headphones
Quadraphonic
15
3D Sound ApproximationsHead Related Transfer
Functions

• Used in conjunction with surround sound to create
  better 3D approximations.
• Microphones record sound from within the ear of a
  person or a model.
• Differences between original sound and recordings
  are used to create filters.
• These filters are applied to generated sounds to
  create the illusion of dimensionality.

16
Surround Sound Head Related Transfer Functions

• Pros
• Cons

• Relatively cheap.
• Effective.
• Makes sense.
• Many different approaches (non-standard).
• Works only with limited speaker positions.
• Not entirely generic.
• Still not pure 3D sound.

17
True 3D Sound

• 3D graphical environments already exist.
• Light paths traverse the scene and surface
  intensities are calculated.
• Currently, sound paths are at most superficially
  computed.
• Yet, programmers already have a wealth of
  environmental data.
• Various possible approaches

18
True 3D Sound Beam Tracing

• Approach

• Divide the environment into cells or regions.
• Precompute and store beam paths from various
  source locations.

• Lookup, in real-time, reverberation paths from
  the avatar to the source.
• Use these paths to calculate delay and
  attenuation from the original, anechoic, audio
  signal for each of the echoes.

19
Beam Tracing

• Quick and effective (with a good data structure).
• Intuitive.
• Scalable for large environments.
• Needs offline computations.
• Assumes sources are stationary.
• Assumes source locations are finite.

• Pros
• Cons

20
True 3D Sound

• On a basic level, we can determine sound
  propagation similar to how light travels through
  a 3D environment.
• One simple, but computationally intensive method
  would be similar to ray tracing.
• Ray tracing algorithms are generally very
  effective but also extremely slow and prone to
  sampling errors.
• Most real-time algorithms for graphical computers
  make various assumptions

21
True 3D SoundThe Graphics Analogy

• The 3D Graphics Pipeline

• Objects are made from geometric primitives
  composed of points.
• These vertices are transformed to be relative to
  the camera.
• Objects outside of the viewing field are clipped.
• Rays are sent from the camera, through each point
  on the projection plane, and into the scene.
• Corresponding pixel values in the viewport are
  calculated from these rays.

22
True 3D SoundThe Graphics Analogy

• Objects are made from geometric primitives
  (triangles, rectangles) composed of points.
• Light intensities are calculated based on surface
  normals of these points.
• These intensities are fed into the graphics
  pipeline.

23
True 3D Sound The Graphics Analogy

• Many of these computations are forwarded to
  optimized 3D graphics cards.
• Many of these same techniques could be employed
  for generating realistic 3D sounds.
• We would need to develop and design 3D sound
  cards and appropriate algorithms.

24
Conclusion

• 3D graphics and many other components of todays
  computer systems have been almost thoroughly
  developed.
• 3D sound is still in the infancy stage.
• This field has a great deal of research
  potential.

25
References

• 1 Burgress, David, A. Techniques for Low Cost
  Spatial Audio. ACM UIST, pages 53-59, 1992.
• 2 Ellis, Sean. Towards More Realistic Sound in
  VRML. ACM Virtual Reality and Modeling, pages
  95-100, 1998.
• 3 Flaherty, Nick. 3D audio new directions in
  rendering realistic sound. Electronic
  Engineering, pages 49, 52, 55, 56, 1998.
• 4 Funkhouser, Thomas, A. , Patrick Min, and
  Ingrid Carlbom. Real-time Acoustic Modeling for
  Distributed Virtual Environments. SIGGRAPH,
  pages 365-374, 1999.
• 5 Funkhouser, Thomas, A. , Ingrid Carlbom, Gary
  Elko, Gopal Pingali, and Mohan Sondhi. A Beam
  Tracing Approach to Acoustic Modeling for
  Interactive Virtual Environments.
• 6 Funkhouser, Thomas, A. , Ingrid Carlbom, Gary
  Elko, Gopal Pingali, and Mohan Sondhi.
  Interactive Acoustic Modeling of Complex
  Environments. Acoustical Society of America,
  1999.
• 7 Min, Patrick, and Thomas A. Funkhouser.
  Priority-Driven Acoustic Modeling for Virtual
  Environments. EUROGRAPHICS, 2000.
• 8 Tsingos, Nicolas, Thomas A. Funkhouser, Addy
  Ngan, and Ingrid Carlbom. Modeling Acoustics in
  Virtual Environments Using the Uniform Theory of
  Diffraction.
• 9 Hull, Joseph. Surround Sound Past, Present,
  and Future. Dolby Laboratories Inc.
  http//www.dolby.com/tech/.
• 10 Suen, An-Nan, Jhing-Fa Wang, and Jia-Ching
  Wang. VLSI Implementation of 3-D Sound
  Generator. IEEE Transactions on Consumer
  Electronics, pages 679-688, 1997.

26
Real-Time Auralization of Sound in Virtual 3D
Environments

• by Scott McDermott
• sdm1718_at_louisiana.edu