VE displays (II)

1
VE displays (II)

• Doug Bowman
• Edited by Chang Song

2
Audition

• 2nd most studied sense
• 2nd most common VE display
• Stimulus disturbance of molecules in a medium
  (air)
• Perceptions pitch, loudness, location

3
Audition

• Generation and display of spatialized 3D sound
• Localization
• Process of determining the location and direction
  from which a sound emanates.

4
3D Sound Localization Cues

• Binaural Cues
• Spectral and dynamic cues
• Head-related transfer functions (HRTFs)
• Reverberation
• Sound Intensity
• Vision and Environment familiarity

5
Binaural Cues

• Based on
• interaural time differences (ITD)
• The difference in time between the arrival of the
  sound to each ear
• interaural intensity differences (IID)
• The difference in sound intensity between the
  arrival of the sound to each ear
• other more complicated cues

6
Ambiguous Situation

• ITD, IID are important cues for determining a
  sound sources lateral location
• Ambiguous situation 2 different sound-source
  locations provide similar values for ITD and IID
• When the sound source is directly in front,
  behind, above, or below the listener

7
Spectral and Dynamic Cues

• Dynamic movement of the listeners head or sound
  source itself
• If sound source was directly in front of the
  listener, a leftward rotation of the head?
• resulting in the ITD and IID favoring the right
  ear, thus helping to break the ambiguity.
• The Spectral content of the signals reaching the
  ears
• Spectral cues do have their drawbacks in that
  they occur only at relatively high frequencies

8
Head-Related Transfer Functions

• HRTF
• The spatial filters that describe how sound waves
  interact with the listeners torso, shoulders,
  head, and particularly the outer ears.
• HRTFs modify the sound waves from a sound source
  in a location-dependent manner.
• These modified sound waves provide a localization
  cue to the listener.

9
Reverberation

• The collection of reflected waves from various
  surfaces within a space and acts as an important
  acoustical cue for localizing sound source
• ??? ??? ??? sound? ??? ???? ??? ??? ????.
• Not for sound sources direction, but for sound
  distance

10
Sound Intensity

• Primary cue for determining a sound sources
  distance
• Intensity fall-off (1/d2)

11
3D Sound Generation

• 3D sound sampling synthesis
• Auralization

12
3D sound sampling synthesis

• Basic record sound that the listener would hear
  in the 3D application
• Binaural audio recording two small microphones
  are placed inside the users ears.
• Realistic
• Any changes in the sound sources location,
  introduction of new objects would requires new
  recording.
• Alternative by processing a monaural sound
  source with a pair of left- and right-ear HRTFs
• HRTF measurement should be done for all possible
  points in the space -gt interpolation schemes

13
Auralization

• Rendering the sound field of a source
• To simulate the binaural listening experience
• Physical and mathematical models
• Wave-based modeling
• Solve wave equation No analytical solution
• Numerical solutions are required
• Ray-based modeling
• The paths taken by the sound waves as they travel
  from source to listener are found by following
  rays emitted from the source.

14
3D auditory displays

• Two basic approaches for displaying 3D sounds
• Headphone-based
• Headphones also block out real-world noises
• Speaker-based
• Headphone-based
• No crosstalk
• A phenomenon that occurs when the left ear hears
  sound intended for the right ear and vice versa.
• IHL problem Inside-the Head Localization
• False impression that a sound is emanating from
  inside the users head.

15
3D auditory displays

• Usage
• Virtual objects emitting sound
• physical interaction with a virtual object
• Sensory substitution
• The process of substituting sound for another
  sensory modality, such as touch.
• The feel of a button press

16
Haptic Display

• Provide the user with the sense of touch by
  simulating the physical interaction between
  virtual objects and the user
• Haptic
• Force feedback (Joint/muscle )
• Tactile feedback ( skin-based)
• The combination of the two.
• Haptic Rendering Software
• To synthesize forces and tactile sensations
• An active field

17
Haptic Cues

• Two fundamental cues
• Tactile
• Kinesthetic

18
Tactile cues

• perceived by a variety of cutaneous receptors
  located under the surface of the skin
• Mechanoreceptors force, vibration, slip
• Thermo-receptors skin temperature
• Noci-receptors pain

19
Kinesthetic cues

• Perceived by receptors in the muscles, joints,
  and tendons of the body
• Produce information about joint angles and
  muscular length and tension
• Determine the movement, position, and torque of
  different parts of the body, such as limbs.

20
Effects of tactile and kinesthetic cues on haptic
perception

• In more complex tasks, both haptic cues are
  fundamental
• Handshake between two people
• temperature of the other persons hand as well as
  pressure from the handshake
• Information about the position and movement of
  each persons hand, arm, elbow and shoulder and
  any forces that the other person is applying

21
Haptic Display Types

• Categorized based on the types of actuators ( the
  components that generate the force or tactile
  sensations)
• 5 categories
• Ground-referenced
• Body referenced
• Tactile
• Combination
• Passive

22
Grounded Referenced Haptic Devices

• Devices are fixed to the physical environment,
  their range is often limited
• ??
• Force-reflecting joystick or steering wheel
  Game and in-flight simulator
• Pen-based haptic display Stylus, Phantom
  (SensAble Technologies)
• String based feedback devices
• Threadmills, motion platforms and other
  locomotion devices for traveling and etc.
• Robot (Large) articulated arms
• Safety issue
• Large range of motion
• Argonne remote manipulator and SARCOS Dextrous
  Arm Master

23
Body Referenced Haptic Devices

• Devices is grounded to the user rather than to
  the physical environment
• Wight and size limitations
• 2 types
• Arm exoskeleton external force-feedback systems
• Hand-force-feedback
• Figure 3.22 ??

24
Tactile devices

• Different types of tactile actuators
• Inflatable bladders
• Vibrators
• MS X-box, Sony Playstation use vibration to
  simulate collision
• Vibrotactile actuators on the fingertips Fig.
  3.23
• Pin arrangements
• Electrical currents
• Temperature-regulating devices

25
Haptic displays

• Exoskeleton
• Apply forces to the hands, elbow and shoulder
• Robot arms
• Phantom
• Tactile devices

26
Near-field haptics

• Use of props - poor mans haptic display
• Grounding in VE
• Examples
• pen tablet
• GM plastic cup
• hairy spider
• airplane cockpit

27
Displaying to other somatic senses

• Simple, special-purpose displays for
  temperature, air movement, etc.
• Fan
• Heat lamp

28
Olfactory displays

• Sense of smell not studied extensively - use in
  VEs?
• A few academic projects
• Smell synthesis still in the future

29
Scent cannon
VIDEO
30
Vestibular/kinesthetic displays

• Sense of body, self-motion
• Virtual body representation
• Can display to these senses by
• Using motion platforms
• Stimulating the proper parts of the brain

31
Conclusions

• Making VEs multi-sensory is
• becoming easier
• becoming cheaper
• Sensory substitution may make task performance
  more efficient
• Multi-sensory VEs enhance presence