Multi-Modal Interface for A Real-Time CFD Solver

1
Multi-Modal Interface for A Real-Time CFD Solver

• Maryia Kazakevich, Pierre Boulanger, Walter
  Bischof and Manuel Garcia
• Nov 4, 2006

2
Outline

• Sonification Background
• Psychoacoustics
• Sonification methods
• Example sonification
• Project
• Background
• Specifics
• Example
• Further work

3
Sonification Background

• Sonification is the use of non-speech audio to
  convey information B.N. Walker
• Data -gt to sound
• As an alternative or to complement visual and
  possibly other displays (e.g.haptic)
• Increasing information bandwidth, reinforcement
• Recognition of features not obvious from visual
  displays
• Possibility to concentrate on different
  complementary information by two senses
• Global events through sound, local details
  through visual cues

4
Psychoacoustics

• Possibility to map field data to
• Loudness, Frequency, Envelope, Spatial location
• Sound parameters require a certain percentage of
  change for the change to be noticed
• Minimum audible angle, intensity change, Tone
  duration
• Softer tone is usually masked by a louder tone if
  their frequencies are similar
• Relation between subjective sound traits and
  their physical representations
• Loudness relation to intensity and frequency

5
Sonification Background

• What is being sonified
• General sonification toolboxes
• Specific to data sets
• Time dependent or static data
• How
• Prerecorded sound
• Modifying physical properties of sound, pressure,
  density, particle velocity
• Modifying pitch, envelope, duration, timber, etc
• Sonification in real-time or not

6
Comparison of sonification methods
7
Sonification of Vector Fields E. Klein

• Rectilinear grids of vectors
• A sphere at the listeners position. Random
  samples within that sphere
• Mapping vector direction and magnitude of sampled
  particle
• Vector direction to
• sound location
• Vector magnitude to
• sound level and pitch

8
Sonification of 3D Vector Fields

• Two consecutive vector samples taken at random
  locations within the listeners head volume
• Hermite curve to achieve C1 continuity between
  two sound positions

9
Sonification of 3D Vector Fields

• Vorticity (turbulence) in the sampled area
• All of the samples in the area are
• roughly the same magnitude and
• direction constant and smooth
• sound low vorticity
• Vectors vary widely sound
• appears to shift more, giving
• the impression of higher
• turbulence
• Size of the sample volume in relation to the
  density of the vectors within the field plays an
  important role

10
Project Background

• Input
• Fluid field with velocity vector, pressure, plus
  potentially density, temperature and other data
• Changes with time
• Output
• Sound characterizing the given fluid field
• Ambient global to the whole field
• Local at the point or area of interaction

11
Project sound options

• Global
• Every particle in the field contributes to the
  sound
• The further sound source is from the virtual
  pointer the less contribution it makes, the
  quieter it is
• Local point
• Only the field characteristics at the virtual
  pointer position contribute to the sound
• Local region
• Particles of the specific subset area around the
  pointer contribute to the sound
• Possibility to add zoom factor to expand or
  contract the space of interaction

12
Structure
Max/MSP Program

• Each rendering program is independent of any other

13
Haptic Program

• Read from the haptic device and sends pointer
  info to the sound and visual programs
• Pointer position and orientation (converted to
  the data field dimensions)
• Interaction sphere diameter - local region
• Gives a force feedback
• Virtual walls provides a force disallowing
  movement of the device outside of the data field
  boundary
• Other feedback possible produce a force that is
  proportional to the flow density and its
  direction

14
Visualization program

• Displays vector field, virtual pointer
  (microphone) and interaction sphere
• SGI OpenGL Performer Library for graphical
  representation

15
Max/MSP

• Max/MSP is a graphical programming environment
  for sound manipulation
• Allows you to write your own objects
• Large capability for a very sophisticated program
• Various built in audio signal processing
  objects
• noise - generates white noise
• reson - filters input signal, given center
  frequency and bandwidth
• - product of two inputs, in given case scales
  a signals amplitude by a value

16
Max/MSP object

• Calculates velocity vector at the position of the
  virtual microphone depending on interaction
  sphere radius (using Schaeffers interpolation
  scheme)
• Small from vertices of the grid cell
• Large from all the vertices inside the influence
  sphere
• velocity value angle at that position

17
Max/MSP object

• Two output values for both angle and velocity
• Output value / max value
• Output (value / max value) 5/3
• Relationship between loudness level and
  intensity
• S a3/5 B.Gold
• Thus, a function between values and amplitude
  should be
• a const data value5/3
• to imply S data value

18
Max/MSP program
white band noise is modified in amplitude and
frequency to simulate a wind effect
Frequency , were v -gt 0,1 -gt 500,
1500
Amplitude 5/3, were v5/3 -gt
0,1 and a5/3 -gt 0.5, 1
19
Exploration example

• Shows sound exploration of the wind flow over the
  Mount St Helens

20
Further work

• Refining the program
• Possible other set-ups for Max/MSP sound program
• Using headphones or speakers to convey spatial
  sound
• Experiments
• To study user interaction with given environment
• Short case-tests to determine user ability to
  navigate within the flow using
• Only visual clues
• Only sound clues
• Both visual and sound clues

21
References

• 1 B.N. Walker, J.T. Cothran, July 2003,
  Sonification Sandbox a Graphical Toolkit For
  Auditory Graphs, Proceedings of the 2003
  International Conference on Auditory Display,
  Boston, MA
• 2 H.G. Kaper, S. Tipei, E. Wiebel, 5July 2000,
  Data Sonification and Sound Visualization
• 3 K. Metze, R.L. Adam, N.J. Leite, Cell Music
  The Sonification of Digitalized Fast-Fourier
  Transformed Microscopic Images
• 4 M. Ballora, B. Pennycook, P.C. Ivanov,
  L.Glass, A.L. Goldberger, 2004, Heart Rate
  Sonification A New Approach to Medical
  Diagnosis, LEONARDO, Vol. 37, No. 1, pp. 4146
• 5 M. Noirhomme-Fraiture, O. Schöller, C.
  Demoulin, S. Simoff, Sonification of time
  dependent data

22
References

• 6 Y. Shin, C. Bajaj, 2004, Auralization I
  Vortex Sound Synthesis, Joint EUROGRAPHICS - IEEE
  TCVG Symposium on Visualization
• 7 E. Childs, 2001, The Sonification of
  Numerical Fluid Flow Simulations, Proceedings of
  the 2001 International Conference on Auditory
  Display, Espoo, Finland, July 29-August 1
• 8 E. Klein, O.G. Staadt, 2004, Sonification of
  Three-Dimensional Vector Fields, Proceedings of
  the SCS High Performance Computing Symposium, pp
  8
• 9 G. Kramer, B. Walker, T. Bonebright, P. Cook,
  J. Flowers, N. Miner, J. Neuhoff, R. Bargar, S.
  Barrass, J. Berger, G. Evreinov, W.T. Fitch, M.
  Gröhn, S. Handel, H. Kaper, H. Levkowitz, S.
  Lodha, B. Shinn-Cunningham, M. Simoni, S. Tipei,
  Sonification Report Status of the Field and
  Research Agenda, http//www.icad.org/websiteV2.0/R
  eferences/nsf.html

23
References

• 10 C.Wassgren, C.M. Krousgrill, P. Carmody,
  Development of Java applets for interactive
  demonstration of fundamental concepts in
  mechanical engineering courses,
  http//widget.ecn.purdue.edu/meapplet/java/flowvi
  s/Index.html
• 11 W.A. Yost, 2000, Fundamentals of Hearing An
  Introduction, Forth Edition
• 12 S.A. Gelfand, 2004, Hearing An Introduction
  to Psychological and Physiological Acoustics,
  Forth Edition, Revised and Expanded
• 13 T. Bovermann, T. Hermann, H. Ritter, July
  2005, The Local Heat Exploration Model for
  Interactive Sonification, International
  Conference on Auditory Display, Limerick, Ireland
• 14 B. Gold, N. Morgan, 2000, Speech and Audio
  Signal Processing Processing and Perception of
  Speech and Music