Machines Without Screens

1
Machines Without Screens

• Part of the Topics in Computing Series of
  Lectures
• Dr. D. Fitzpatrick
• Friday, 14 November 2003

2
In this Lecture

• How some of our senses work
• Synthetic Speech and how it works
• Describing Maths using speech
• 3D audio, and Force Feedback

3
(No Transcript)
4
How the Eye Works

• Light rays enter the eye through the cornea.
• The cornea takes widely diverging rays of light
  and bends them through the pupil
• The lens of the eye is located immediately behind
  the pupil. The purpose of the lens is to bring
  the light into focus upon the retina, the
  membrane containing photoreceptor nerve cells
  that lines the inside back wall of the eye.
• The photoreceptor nerve cells of the retina
  change the light rays into electrical impulses
  and send them through the optic nerve to the brain

5
How People Read

• not a linear progression
• use of Sacades and fixations
• consequence? Eye tracks to highlighted, or other
  key portions of the page.

6
(No Transcript)
7
How the Ear Works

• The outer ear or pinna (plural pinnae) leads to
  the middle ears auditory canal or meatus.
• The auditory canal terminates with the ear drum,
  or tympanic membrane.
• Beyond the ear drum is the inner ear, which
  contains the hidden parts of the ear encased in
  bone .

8
How the Ear Works II

• There are semicircular canals, and three
  liquid-filled passages that are associated with
  equilibrium rather than hearing.
• They tell us about the orientation of the head
• cause us to get dizzy when they are
  malfunctioning
• cause some of us to get seasick when the head,
  body and eyes undergo motional disturbances.
• The three little bones of the air-filled middle
  ear which are attached to the eardrum, excite
  vibrations in the cochlea, the liquid-filled
  inner ear.

9
How we hear Sound

• In the cochlea the vibrations of sound are
  converted into nerve impulses which travel along
  the auditory nerve, toward the brain 
• The purpose of the auditory canal is to guide
  sound waves to the ear drum. The pinna acts as a
  collector of sound from the outside world, and
  also acts as a directional filter.
• The intensity of a sound wave in the auditory
  canal is proportional to the intensity of the
  sound wave that approaches the listener.

10
Sound Waves

• We are immersed in an ocean of air.
• The snapping of fingers, speaking, singing,
  plucking a string or blowing a horn set up a
  vibration in the air.
• The sound wave travels outward from
• the source as a spherical wavefront
• It is a longitudinal wave
• In contrast, waves in a stretched string are
  transverse waves

11
How fast does the sound wave travel?

• If the air temperature is 20 degrees Celsius a
  sound wave travels at a velocity of 344 metres or
  1,128 feet a second
• Sound travels in helium almost 3 times as fast as
  in air, and longitudinal sound waves can travel
  through metals and other solids far faster.

12
How Do We Hear?

• The sound waves that travel through the air cause
  components of our ears to vibrate in a manner
  similar to those of the sound source.
• What we hear grows weaker with distance from the
  source, because the area of the spherical wave
  front increases as the square of the distance
  from the source, and power of the source wave is
  spread over that increasing surface.
• What actually reaches our ears is complicated by
  reflections from the ground and other objects.

13
Role of Speech

• Primary mode of communication
• Convey emotional content
• Problems with synthetic speech
• Monotonous basically uninflected speech
• Not possible to convey emotional content
• Consequence? very boring...

14
What is speech?

• speech can be decomposed into three primary
  components
• frequency, amplitude and time.
• Frequency is the term used to describe the
  vibration of air molecules caused by a vibrating
  object...which are set in motion by an egressive
  flow of air during phonatation. measured in
  Hertz (Hz).
• Speech not as simple as other acoustic sounds
  can contain many elements vibrating at different
  frequencies.
• frequency of repetition referred to as the
  fundamental frequency f0.

15
What is Speech? II

• Amplitude The acoustic component which gives
  the perception of loudness.
• the maximal displacement of a particle from its
  place of rest
• measured in decibels
• Duration the third component in the acoustic
  view.
• The measurement, along the time-line of the
  speech signal

16
Introducing Prosody

• Simple description Inflection
• that set of features which lasts longer than a
  single speech sound.
• . . . those auditory components of an utterance
  which remain, once segmental as well as
  non-linguistic as well as paralinguistic vocal
  effects have been removed

17
What will it Sound Like?

• The aim is to discard the monotone
• E.g If emboldened text is found
• 1. Rate will slow (Duration)
• 2. Pitch range will increase (F0)
• 3. Volume will increase (Amplitude)
• Most structural and font information will be
  conveyed by prosody

18
Speaking Text Attributes

• Major headings read as section x.
• Slower rate, they have a lower average pitch, a
  lower baseline fall.
• Minor headings read as x.x,
• Same slower rate, lower average pitch, lower
  baseline fall
• Emphasis increase pitch range, increase accent
  height, minimise smoothness, maximise richness,
  increase amplitude where possible.

19
Speaking Maths

• We intend to use prosodic changes to convey
  equations
• 1. The prosodic system is already familiar
• 2. Prosody is capable of expressing
  mathematical material
• 3. All we have to do is...match the prosody
  to the maths!!

20
Mathematical Prosody

• Equations resemble a tree when broken down
• Nested levels conveyed by
• 1. use of parentheses or brackets
• 2. juxtaposition of symbols vertical
  horizontal

21
Linearity

• Only the most simple math is linear
• a b c
• This is easy to represent in a linear fashion
• Unfortunately, math doesnt stop here, though
  many wish it did?
• Now try
• a b c - d
• Still linear - well sort of!

22
Linearity

• Using implicit hierarchy rules it would be
  understood to be
• a (b c) - d
• But what if we really wanted
• a b (c - d)

23
Linearity

• These simple equations are still considered to be
  linear in nature
• But, linearity has a very short half life when
  learning math
• Math rapidly becomes two-dimensional
• Representing that non-visually becomes difficult

24
Linearity
25
Linearity

• This relatively simple equation could be
  represented
• a sqrt(((x super 2 base) - y) / z)
• Essentially, using parentheses, we can represent
  ANY equation in a linear fashion, BUT???
• Speech is a basically linear representation

26
Designing a Browser?

• What is the goal of a Math Browser?
• To allow users to traverse an equation
• In whole or in parts
• Forwards or backwards
• Upwards or downwards
• Under user control
• To convey structure and semantics

27
3D Audio

• Surround sound has amazed film watchers for years
  immersing the audience in a full experience.
• However, surround sound systems do not provide a
  true 3D reconstruction of a setting.
• Surround sound provides location of sources
  within a single plane, a true reconstruction
  would include all possible planes, the entire
  sound field.

28
3D Audio II

• Through research it has been learned that the
  physical shape of the ears and head affect how we
  perceive sound.
• To perfectly record a sound field, microphones
  must either be placed in the ears, or a model of
  the human head complete with ear canals can be
  created with microphones inside the ear cavities.
• These two audio channels can then be driven to
  headphones creating a very life-like reproduction
  of the 3D sound environment.