Technical Issues: Sound

1
Technical IssuesSound
2
Technical IssuesSound

• While graphics and animation can bring a game
  world to life, it has been said that sound and
  music give the game its soul.
• A game world without audio would seem rather
  hollow and empty in comparison.
• Audio can make its appearance in games in a
  variety of different forms
• Sound effects.
• Voice and speech effects.
• Musical score.

3
Technical IssuesSound
Screen shot from Parappa the Rapper. This was
one of the first music-rhythm based video games,
and was quite popular due to its interesting
andunique gameplay.
4
Technical IssuesSound
Screen shot for Mad Maestro! This interesting
game lets you coordinatean entire orchestra of
musicians.
5
Technical IssuesSound
Movie from Karaoke Revolution. An interesting
game that you actually controlby singing into a
microphone, instead of mashing buttons in rhythm.
6
Sound Requirements for Video Games

• Sound and music are used to provide the audio
  component of representation in video games.
• They are used to give an indication of what the
  game world and its inhabitants sound like, and
  usually much more.
• Unfortunately, this is something many game
  developers often neglect.
• Most new developers ignore sound and music
  entirely in their first game or two.

7
Sound RequirementsSet the Mood

• Perhaps even more so than visual effects, sound
  and music can set the tone and mood for a video
  game.
• It can be anything from light and cheerful to
  dark and foreboding.
• The tempo of music can also influence or reflect
  the pace of action in the game.
• If done properly, audio effects can support
  storytelling and help immerse the player at the
  same time.

8
Sound RequirementsSet the Mood
Screen shot of the Legend of Zelda Ocarina of
Time. The music throughout the game helps to
establish the tone and mood.
9
Sound RequirementsProvide Feedback

• Just like visual effects, sound effects should
  provide some kind of audible feedback for every
  user action.
• Such feedback should indicate the success or
  failure of an action.
• A sound effect for a successful action should
  appear natural for such a completion.
• A sound effect for a failure can either be
  natural, or some kind of beep or buzzer.
• For example, shooting a gun should make a bang if
  you have ammunition, or a click if you do not.

10
Sound Requirements Provide Feedback
Screen shot from Oni. Oni provides good audio
feedback to the user. Everyaction has a audible
result. In this case, for example, a successful
punch hasa smack sound effect, and a missed
punch has a whiff sound effect.
11
Sound RequirementsProvide Cues to Players

• Sound effects should be used to provide cues of
  future events to the player.
• For example, hearing footsteps coming down a
  hall as a prelude to someone entering a room.
• Music can do the same, by changing its tone to
  reflect the mood of events.
• For example, the pace of music can pick upor
  turn foreboding when conflict is about to occur.

12
Sound RequirementsProvide Cues to Players
Screen shot from Splinter Cell. Sound cues are
given throughout the gamethrough ambient sounds,
whispers of overheard conversations, andchanges
in the background music of the game.
13
Sound RequirementsBe Realistic

• Most game audio effects should sound as realistic
  and fitting as possible.
• This is interesting, because graphics, animation,
  and game physics can range from realistic to
  outlandish and still be fitting.
• The audio effects in place, however, must match
  and fit the visual effects or else the game seems
  inconsistent.
• If you cannot generate a suitable effect, then
  sample one from the real world!

14
Sound RequirementsBe Realistic
Screen shot from Doom II. Even though you are
fighting out-of-this-worldcreatures with
out-of-this-world weaponry, the sounds seem
tofit the situation in a realistic fashion.
15
Sound RequirementsPick the Right Talent

• Whether you are picking actors for voice overs or
  musicians to provide the soundtrack for a game,
  make sure you select the right people for the
  job.
• They have the right sound for the game.
• They can do it on time.
• They fit within budget.
• They are reliable enough to stick through the
  entire job from start to finish.
• They will provide you the necessary rights for
  the game when the job is done.

16
Sound RequirementsPick the Right Talent
17
Sound RequirementsPick the Right Talent
Screen shot from Prisoner of War. This allegedly
has some of the worstvoice acting ever heard in
a game. Apparently, those German accents were
really, really bad!
18
Sound RequirementsLocalization

• If you have a game with speech elements (either
  on their own or embedded in other sound effects),
  you need to worry about localization in other
  countries.
• Separate sound tracks in other languages.
• Subtitles and other text support.
• These on their own may not be enough.
• Sometimes, wording and phrasing will need to be
  changed to be made acceptable or sound realistic
  in other parts of the world.

19
Sound RequirementsLocalization
Screen shot from Crouching Tiger, Hidden Dragon.
It has all dialog in Mandarin with English
subtitles. A nice effect to complement the movie.
20
Sound RequirementsCreate a Living World

• It is important to put ambient sound effects into
  a game to help bring life to it.
• This can include many things
• Conversations between non player characters.
• Animal life in the world (even if it is not
  seen).
• Environment effects such as wind and rain.
• Terrain effects such as water.
• Having these audio effects makes the game seem
  considerably richer.
• The player feels they are in a living world.

21
Sound RequirementsCreate a Living World
Screen shot from the Legend of Zelda The Ocarina
of Time. The world is full of life and sound.
Whether it is sound of chickens, a babbling
brook, or creatures at night, it has considerable
ambience.
22
Sound RequirementsGet Timing Right

• The human ear is an incredibly sensitive piece
  of equipment.
• Most people can pick up on problems with audio
  faster than they can with graphics or video.
• Consequently, it is very important to make sure
  that audio is timed correctly.
• Voice must be in sync with characters.
• Sound effects must be in sync with the
  corresponding actions.
• There should no hiccups, pauses, popping, or
  glitches in sound as it is delivered!

23
Sound RequirementsAllow Customization

• You should allow users as much customization as
  possible with game audio effects.
• Audio volume (split into master, effects, voice,
  and music volumes if possible).
• Audio quality (sample rate).
• Number of channels (mono, stereo, or 3D).
• Music tracks to play (either choosing from those
  provided with the game, or user supplied tracks).
  Note that it does not always make sense to allow
  this last one.

24
Sound RequirementsAllow Customization
Screen shot from Grand Theft Auto. Not only
could you tune audio volume, but you could listen
from a variety of game provided radio stationson
CD, or insert a CD of your own favourite music
instead.
25
Audio Techniques

• There are a variety of techniques for providing
  audio effects in a video game.
• Typically, the technique used depends on the type
  of audio effect
• Sound effects.
• Voice and speech effects.
• Musical score.
• In most cases, there are two choices generating
  the effect, or playing back a sampled and stored
  effect.

26
Sound Effects

• All sounds arrive at our ears in the form of some
  kind of wave.
• Thus, for a sound effect in a game, we have
  essentially two choices
• Generate a sound with the appropriate waveform
  that gives the effect that we are looking for.
• Pick a sound from the real world close to what is
  wanted, sample its waveform, and store the result
  for game playback.
• Both approaches can be quite useful, and have
  their advantages and disadvantages.

27
Sound EffectsGeneration

• Generating a sound effect involves selecting from
  basic waveforms, combining them together,
  applying an envelope, and sending the result to
  the sound device.
• There are many waveforms to choose from
• Sine, cosine, tangent, square, triangle,
  sawtooth, various noises, and so on.
• They can be combined in many ways
• Additively, subtractively, consecutively,
  piecewise, and so on.

28
Sound EffectsGeneration
Samples of the basic sound waves. (Top row, left
to right sine, cosine,tangent, and white
noise bottom row, left to right triangle,
squareand sawtooth.)
29
Sound EffectsGeneration

• An envelope is a simple technique used to alter
  the pitch or volume of a sound wave form
• Attack The time from the start of a sound to
  reach its maximum intensity.
• Hold The time the sound is at maximum
  intensity.
• Decay The time the sound drops to a lower
  intensity.
• Sustain The time at this lower intensity until
  release.
• Release When the sound fades to nothing.
• Envelopes can introduce a variety of interesting
  effects to a generated sound.

30
Sound EffectsGeneration
Attack
Hold
Decay
Sustain
Release
A sample envelope.
31
Sound EffectsGeneration

• Advantages
• No storage space required for sound samples.
• Since effects are generated, it is easy to have
  variation in the effects produced.
• Conceptually easy to do.
• Disadvantages
• Can take time to find the desired sound.
• Can be hard to produce realistic noises (though
  the original Nintendo could technically only
  create sine, square, and noise waves, and it
  didnt suffer for it).
• Computation power required to generate the
  effects on the fly this is a problem if
  resources are scarce.

32
Sound EffectsGeneration
Screen shot from WaveGen. A nice and simple
waveform generatorto demonstrate different sound
generation issues.
33
Sound EffectsSampling and Storing

• The basic premise behind this approach is that a
  sound waveform is read at periodic intervals
  this is known as sampling.
• If a sufficient number of samples is taken, the
  collection of readings will closely approximate
  the original waveform.
• So, if these samples are stored in a file of
  some kind, they can be played back in the future
  to reproduce the original waveform and the
  original sound as well.

34
Sound EffectsSampling and Storing
A sample waveform sampled at regular intervals.
35
Sound EffectsSampling and Storing

• Various factors affect the quality of the
  sampling process
• Sampling frequency. The more frequent the
  sampling, the better the approximation you have
  to the original waveform. Values tend to range
  from 8000 Hz (phone grade) to 44100 Hz (CD
  quality).
• Sampling bits. The more bits used to store a
  sample, the higher the quality of individual
  samples. Values tend to be either 8 or 16 bits.
• Number of channels. This is typically either 1
  for a mono recording or 2 for a stereo recording.
  (More channels can be used for environmental or
  surround audio, but this process can also be done
  differently.)

36
Sound EffectsSampling and Storing

• A variety of formats can be used for storing
  sound effects
• WAV Microsofts standard sound format.
• AU Sun Microsystems sound format.
• AIFF Apple/SGI sound format.
• SND NeXT sound format.
• A variety of various raw formats.
• Sometimes, sound effects are stored in MP3, Ogg
  Vorbis, or CD format as well.
• In many cases, these formats use some kind of
  compression to reduce the storage requirements of
  recorded samples.

37
Sound EffectsSampling and Storing

• Car horn
• Cat meow
• Door slam
• Elephant
• Explosion
• Glass breaking
• Machine gun
• Modem

• Phone ring
• Pig squeal
• Pour a drink
• Splash
• Thunder
• Traffic jam
• Typewriter
• Vacuum cleaner

38
Voice Effects

• Voice effects, like all sounds, can be
  represented as waveforms.
• Like sound effects, voice effects can be either
  generated or sampled and stored.
• Sampling and storing voice effects can be done in
  the same way as other sounds.
• Generating realistic sounding voice effects,
  however, is still an incredibly difficult task to
  do properly.

39
Voice EffectsSampling and Storing

• Stored voice effects can be treated essentially
  the same as stored sounds.
• The difference, in this case, is that the sound
  is being created by an actor reading a script,
  and not through some other means.
• For clarity, it is typically a good idea to use
  the best sampling techniques for all game speech.
• Examples
• Humans from Warcraft I II.
• Orcs from Warcraft I II.

40
Voice EffectsGeneration

• With sampled voice effects, a game is ultimately
  limited in its speech options.
• There is only a finite amount of storage space,
  so only a limited number of voice effects can be
  supported.
• If more flexibility is required, with speech
  chosen at run-time, some kind of generation
  technique is required.
• This can be computationally expensive, and
  difficult to obtain realistic results.

41
Voice EffectsGeneration

• To generate a voice effect, the game must do the
  following basic steps
• Decide upon the text to be spoken.
• Select a voice in which the text will be spoken,
  and various voice characteristics.
• Phonetically break the text into its basic
  phonemes, emphases, pauses, and other components.
• Piece together recorded samples for these
  phonetic components and smooth over any seams
  between adjacent samples.
• Output the results.

42
Voice EffectsGeneration

• Modern speech generation has shown many
  improvements.
• Microsoft TTS kits in many languages.
• ATT and Bell Labs research with naturalsounding
  voices in many languages.
• Many, many more this is active research!
• Some examples from over the years

ATT Natural Voices
C64 SAM
Present
1980s
43
Musical Score

• Music, like all sound, can also be represented as
  a waveform.
• Like sound effects, music can be either generated
  or sampled and stored.
• Sampling and storing music can be done in the
  same way as other sounds.
• Generating music can be done without too much
  difficulty with some realistic effects.

44
Musical ScoreSampling and Storing

• Music for video games can be sampled and stored
  much in the same way as sound effects.
• Typically, one of the following formats is used
  for doing so
• CD audio (an audio track on the game CD)
• MP3 (MPEG audio)
• Ogg Vorbis (an open, free alternative)
• WAV (Microsoft audio format)
• An example (from Grand Theft Auto)

45
Musical ScoreGeneration

• There are two well supported approaches to
  sequencing collections of sounds to produce music
  in video games.
• MIDI (Musical Instrument Digital Interface)
• Allows multiple channels of computer generated
  sounds to be composed and played sequentially.
  Sounds tend to be tones from musical instruments.
• MODules
• Allows sequencing a series of sounds to assemble
  them into a tune, with a variety of special
  effects applied along the way. Sounds can be
  anything.

46
Musical ScoreGeneration

• Interesting research is currently investigating
  algorithmic music generation.
• Based on various input parameters, algorithms
  evolve a non-repetitive music score in real-time.
• This allows new music experiences tuned to the
  gamer or the particular events in the game world.
• An example of this is ZenStrings, developed by
  Condition30. Examples of its music
  includeRevenge , Neuralista , Coded
  Reality , PlayaPiano , The Festival ,
  Mutations , Key Largo for Strings ,
  Tranquilitatus , and The Game Begins .

47
A Final WordEnvironmental Audio

• Original games only had mono output to a single
  speaker.
• The next advancement in games was stereo sound to
  two speakers.
• Audio is separated into a left and right channel
  to provide some sense of direction as to where
  the sounds originated from.
• For modern gaming, significant work has been put
  into environmental audio, also known as surround
  sound.

48
A Final WordEnvironmental Audio

• There are two main techniques to environmental
  audio
• Use a variety of techniques to provide depth in
  sound using only two speakers.
• Use four or more speakers and directing audio
  appropriately.
• For best effects, you must take level design and
  game physics into consideration.
• Echoes, reverberations, dampening, and so on.
• Blocking of sounds caused by obstacles or
  terrain.
• And so on you must now render your audio!

49
A Final WordEnvironmental Audio
Speaker placement for a Dolby 5.1 surround sound
system. You get a frontcentre channel, front
left and right channels, rear left and right
channels, and a subwoofer (for non-directional
bass). Newer technologies are emerging