Physically Based Sound

1
Physically Based Sound

• COMP259 Nikunj Raghuvanshi

2
Overview

• Background
• FEM Simulation
• Modal Synthesis (FoleyAutomatic)
• Comparison/Conclusions

3
Motivation

• Sounds could in-principle be produced
  automatically, just like graphics Sound
  Rendering
• Sound Rendering has not received much research
  effort
• Main Goal Automatic generation of non-music,
  non-dialogue sound

4
Sound Production Today

• Movies Foley Artists
• http//www.marblehead.net/foley/index.html
• Games Anyone noticed the huge sound directory
  in Unreal Tournament?

5
PBS Sound Production in Nature

• Collisions/Other interactions lead to surface
  vibrations
• Vibrations create pressure waves in air
• Pressure waves sensed by ear

Vibration
Propagation
Perception
Surface Vibration Pressure Wave Ear
6
Main Aims of PBS

• Physics simulator gives contact/collision
  information
• Assign material properties for sound, Wood,
  concrete, metal etc.
• Sound simulator generates sound using this data
  (in real time?)

7
Challenges

• Sound must be produced at a minimum of 44,000 Hz
• Extremely High Temporal Resolution (timesteps in
  the range of 10-6-10-8 s)
• Stiffness of underlying systems (eg. Metallic
  sounds. K/m108)
• Stability may require even smaller timesteps

8
Two Approaches

• FEM deformable simulationO'Brien, J. F. et. al.,
  Synthesizing Sounds from Physically Based
  Motion. SIGGRAPH 2001.
• FoleyAutomatic (Modal Synthesis)Kees van den
  Doel et. Al., FoleyAutomatic Physically-based
  Sound Effects for Interactive Simulation and
  Animation. SIGGRAPH 2001.

9
Main ideas

• Deformable Simulation (arguably) much more
  physically based
• Foley Automatic Additive Synthesis

Component Sinusoids
Sound Signal
10
Overview

• Background
• FEM Simulation
• Modal Synthesis (FoleyAutomatic)
• Comparison/Conclusions

11
Simulation Requirements

• Temporal Resolution
• Simulate Vibration as well as Propagation
• Vibration Modeling Deformable Model for Objects
• Propagation Modeling Explicit Surface
  Representation
• Physical/Perceptual Realism

12
System Structure
13
Vibration Modelling

• FEM with Tetrahedral Elements
• Linear Basis Functions, greens strain
• Explicit Time Integration
• Typically nodes 500, elements 1500, dt
  10-6-10-7 s

14
Sound Propagation Modelling

• Fluid Dynamic FEM simulation of surrounding air?
  Very expensive. Instead
• Employ Huygens Principle Pressure Wave may be
  seen as sum of pressure wavelets

Receiver
Receiver
Pressure Wave
Pressure Wavelets
15
Surface Vibrations and Sound
Pressure contribution of a patch,
Unit Normal
Velocity
Density of Air
Sound Propagation Speed in Air
Acoustic Impedance of Air
16
Surface Vibrations and Sound

• Approximate differential elements with surface
  triangles
• Apply band pass filters
• Low pass windowed sinc filter
• High pass DC blocking filter
• Result Pressure known for all surface triangles

17
Putting it all together
Pressure/Signal at Receiver Filtered Average
Pressure Area of Triangle Visibility Term
Receiver
Vibration
Approximation of Beam Pattern Distance Falloff
18
Propagation Delay
Accumulation Buffer
1
Receiver Distance from Source
d1
t1 d1/c
2
Source
d2
t2 d2/c
Receiver
t0
Sound Propagation Speed
19
Results Capabilities

• General models
• Generated sounds are accurate
• Stereo Sound
• Dopplers Effect

20
Demo
21
Results Accuracy
22
Results Speed
Scene TimeStep(s) Nodes/Elems
Time/Audio Time Bowl 10-6
387/1081 91.3/4.01 mins Clamped Bar
10-7 125/265 240.4/1.26
mins Vibraphone 10-7 539/1484
1309.7/5.31 mins (1 day) Timings on
a 350MHz SGI Origin MIPS R12K processor
23
Overview

• Background
• FEM Simulation
• Modal Synthesis (FoleyAutomatic)
• Comparison/Conclusions

24
Features

• Modal resonance model of solids
• Location dependent sounds
• Impact, slide, roll excitation models
• Real-time, low latency
• Easy integration with simulation/animation
• Practical
• Do not model propagation of sound from source to
  receiver

25
Synthesis Method
Sound Samples
Force
Vibration
Emission
User
Propagation
Listener
Speakers
26
Vibration
Surface u(x,t) of body responds to external
contact force F(x,t)
u(x,t)
F(x,t)
Strain Functional Speed of Sound
Under suitable boundary conditions, the solution
to the PDE is a sum of sinusoids
27
Emission
Sound pressure s(t) linear functional L of
surface vibration u(x,t)
u(x,t)
L
s(t)
Note that propagation is not modeled in above
28
The Modal Synthesis Model
u(x,t)
F(p,t)
L
s(t)
The response u(x,t) of an arbitrary solid object
to an external force can be described as a
weighted sum of damped sinusoids
Impulse response/modal model
Since L is linear, it implies at s(t) must be a
sum of damped sinusoids too
29
Example A 1D string
a1
ak
a0
1st Mode
2nd Mode
Higher modes
Frequency f0
Frequency f1 2f0
Frequency fk kf0
...

Main Idea Sum contributions of all the modes The
point of impact decides the proportions in which
the modes are to be mixed ak. Therefore, ak is a
function of p, the point of impact The
frequencies and damping parameters are a property
of the object, and independent of how the object
is hit
30
The Modal Synthesis Model
u(x,t)
F(p,t)
L
s(t)
Impulse response, modal model
Kth mode Gain Factor Point Damping
Vibration
of impact Term
Frequency
Parameters measured experimentally
31
Force Modeling
At runtime Find gain parameters given the
location, strength and kind of force. Synthesize
sound from previous equation.

• Impact
• Sliding
• Rolling

Wavetable
Stochastic
32
Impact Forces

• Duration hardness (T)
• Magnitude energy transfer (w)
• Multiple micro-collisions

Example
33
Sliding/Scraping
Micro-collisions lead to noisy audio-force
34
Sliding/Scraping

• Wavetable approach
• Store force parameters
• Modulate amplitude with energy transfer
• Modulate rate with contact speed
• Synthesis Approach
• Fractal noise represents roughness
• Filter through reson filter
• Resonance contact speed
• Width randomness of surface

35
Rolling
No relative surface motion

• Differences with sliding
• Smoother Use low pass
• More damping
• Harder to create
• Less understood
• Essential coupling?

36
Rolling Smooth Surfaces

• Polyhedral objects do not lead to smooth rolling
  forces

• Instead use smooth surfaces directly

37
Rolling Contact Evolution

• Evolve the contact in Reduced coordinates
• q (u,v,s,t, )

c(u,v)
d(s,t)
38
Rolling Contact Evolution

• Piecewise parametric surfaces, loop subdivision
  surfaces
• Explicit integration, no stabilization
• Multiple contacts and conforming contacts are not
  handled
• Used only when multiple contacts in close
  spatio-temporal proximity

39
Demo
40
Dynamic Forces
Pebble-in-Wok Demo
Contact force
Slipping speed
Rolling speed
Impulses
and locations
41
Results

• 0.1 CPU time per mode
• Graceful degradation of quality
• The bell demo is interactive
• Uses a PHANToM for interaction
• Authors do not report any real timings
• State that sound quality is perception-based
  and has no metric as of now

42
Overview

• Background
• FEM Simulation
• Modal Synthesis (FoleyAutomatic)
• Comparison/Conclusions

43
Discussion

• FEM Physically Rigorous and General
• Too slow for interactive applications
• Doesnt scale well
• Inappropriate to apply a 30fps technique to
  44000fps?
• Maybe too general for the problem domain?

44
Discussion

• Modal model exploits the vibrational nature
• Higher Efficiency
• But, not rigorously physically based
• Finding the parameters requires experimentation
  and earballing
• No rigorous correlation between physical and
  perceptual parameters

45
Discussion

• For Realtime Need for a technique to cover the
  middle ground
• Extracting modal parameters in general requires
  solving PDEs
• Not possible to do in an automated manner
• Approximate modal parameters and then use modal
  synthesis?

46
Conclusion

• PBS involves orders of magnitude smaller temporal
  and spatial scales
• Research is sparse, problems are dense
• Main contributions of the two papers besides
  vibration modeling
• FEM Efficient modeling of sound propagation
• FoleyAutomatic Efficient, Approximate models to
  handle surface properties and contact forces

47
References

• O'Brien, J. F., Cook, P. R., Essl G.,
  "Synthesizing Sounds from Physically Based
  Motion." The proceedings of ACM SIGGRAPH 2001,
  Los Angeles, California, August 11-17, pp.
  529-536.
• Kees van den Doel, Paul G. Kry and Dinesh K. Pai,
  FoleyAutomatic Physically-based Sound Effects
  for Interactive Simulation and Animation
  Computer Graphics (ACM SIGGRAPH 01 Conference
  Proceedings), pp. 537-544, 2001.

48
Acknowledgements

• Some images were taken from the referred papers
  and the corresponding SIGGRAPH slides