Chapter 9: Multimedia

1
Chapter 9 Multimedia

• Most of the outside information entering our
  brains, enters through our eyes or our ears. They
  are, indeed, input devices through which our
  world communicates with us.

2
Multimedia

• In this chapter
• What is multimedia?
• How are pictures stored within the computer?
• What can be done with digitized pictures?
• How can pictures be created using a computer and
  software?
  
• What is the importance of vocal communication?
• How is speech stored and played back by the
  computer?
  
• What are some problems the computer has when
  analyzing speech?
  
• What areas are computers used in audio
  communication?

3
What is Multimedia?

• Multimedia has become interactive
• Interactive multimedia The use of media such as
  text, graphics, animation, video, and audio in an
  interactive way that allows a participant to
  control it.
• To understand the value of interactive
  multimedia, we will examine visual and audio
  concepts in detail.

4
Visual MediaManipulating Images

• Previously created images (Modifying or
  processing images that already exist)
  
• Three types of techniques used to manipulate
  images
  
• Minor Processing Techniques
• Enhancement
• Restoration

5
Visual MediaManipulating Images

• Common example of digitized images Satellite
  weather maps.
  
• GEOS-7 satellite picture taken on Aug. 24, 1992.
• Outlines created by a computer program.
• Program color coded the picture to give
  information about the hurricanes wind speed.

6
Visual MediaManipulating Images

• Landsat 4 Satellite
• Uses 3 of the 7 frequencies available from the
  Thematic Mapper of the satellite.
  
• Like having a camera with different filters, each
  letting a different frequency through (including
  infrared).

7
Visual MediaManipulating Images

• Image enhancement
• A type of digital image processing whose goal is
  to highlight or enhance particular aspects of an
  image or change an images structure.
  
• Common example of image enhancement
  False-coloring An image enhancement technique
  that consists of changing the colors of an image
  or assigning colors to various parts of an image.

8
Visual MediaManipulating Images

• Image restoration
• A type of digital image processing whose goal is
  to eliminate known, but unwanted image flaws or
  degradations.
  

9
Visual MediaManipulating Images

• Image enhancement example
• It makes you wonder if photographs can be trusted.

10
Visual MediaCreating Images

• Artists follow rules (or intentionally dont
  follow rules) to create original images
  
• Hiding parts of things that wouldnt normally be
  visible.
  
• Sources of light that illuminate the objects.
• Creating images using the computer
• Line images.
• Solid forms to 3D.
• Animation and video.

11
Visual Media Creating Images

• Creating Line Images
• What are some uses of line drawings?
• Map making.
• Architectural drawings.
• Business graphs.
• Plans for building machinery, aircraft,
  furniture-in-a-box.
  
• City population density maps.
• How can line images be created?
• Using drawing programs such as Adobe
  Illustrator.
  
• Have drawing related features including geometric
  shapes, free-hand tools, fills, patterns, text.

12
Visual Media Creating Images

• Bit-mapped or Raster graphical images
• Constructed of individual pixels.
• Only crude manipulation can be done (modification
  of each pixel).
  
• Retains bit-mapped appearance.

• Object-oriented or Vector graphic images
• Stored in the computer as lines, curves, or
  geometric shapes.
  
• Formulas are used to draw (circles, twice the
  size, are still circles!)
  
• Objects can be moved or modified easily.

13
Visual Media Creating Images

• Designing 3-dimensional pictures drawn in
  2-dimensions
  
• Curved surfaces
• Color
• Texture
• Shading

14
Visual Media Creating Images

• One of the classic problems in computer
  graphics
  
• Hidden-line problem Concerns itself with how to
  hide the outlines or surfaces of a solid object
  that shouldnt be seen from the direction of the
  observation.
• Example Studies of an airline pilot in
  action.
  

15
Visual Media Creating Images

• Perspective The quality that allows
  3-dimensional images to be drawn on 2-dimensional
  surfaces and yet retain the look of a
  3-dimensional image.

16
Visual Media Creating Images

• Shading A technique used to give the appearance
  of illumination by some combination of light
  sources.
  

The architecture program used to draw this
building has a 3D description of the building It
can produce pictures from any angle.
17
Visual Media Creating Images

• Shading flat surfaces of the head (as in b) then
  blending the surfaces to give a more natural look
  (as in c).

a
b
c
18
Visual Media Creating Images

• Texture A property of a surface. It is observed
  and identified through the reflection of the
  light off the surface.

19
Visual MediaAnimation and Video

• Animation
• Early animation started with pictures flipping
  hand-cranked players.
  
• Animation 30 years ago
• Cartoons had to have the figures drawn by
  artists.
  
• The animator would make them move while taking
  pictures, frame by frame.
  
• Animation now
• Computers are used to create full length feature
  cartoons.
  
• Toy Story
• Individual frames stored on the computer.
• Special player programs can be used to view them
  on the computer screen or recorded on videotape
  using special hardware and software.

20
Visual MediaAnimation and Video

• Jurassic Park dinosaurs
• Wire frames were created in the computer to
  depict dinosaurs.
  
• Each wire frame is moved by the computer which
  calculates positions of each part viewed from a
  certain angle and distance.
  
• The computer covered the surfaces of the wire
  frames with a texture layer.
  
• Computer takes into account sources of light
  illuminating the figure.
  

21
Visual MediaAnimation and Video

• Morphing A graphics technique used in animation
• A beginning image will distort and change in a
  predetermined number of frames into a final image.

22
Visual MediaAnimation and Video

• Digitizing and Manipulating Video
• Digital video hardware systems Systems capable
  of capturing and manipulating video.
  
• Analog video uses standards developed over 50
  years ago.
  
• Each differs in the number of horizontal lines
  that make up a single video frame and in the
  number of frames per second that are displayed on
  the TV set.
• European broadcast standards (PAL).
• United States broadcast standards (NTSC).
• 30 frames per second.
• VHS.

23
Visual MediaAnimation and Video

• Video capture The process of digitizing an
  analog TV signal.
  
• Done using hardware that produces a digital form
  of the analog TV signal.
  
• Uses the VCR to feed the video signal into a
  digital hardware card that plugs into the
  computer.
  
• Each frame is converted into a bitmapped image.
• The digital video signal is compressed.
• Eliminates data that our eye doesnt see or that
  is hardly noticed.
  
• Codec A scheme for coding decoding large
  amounts of data. (The data is economical and
  uncomplicated to store.)

24
Visual MediaAnimation and Video

• Common codecs
• Motion Picture Experts Group (MPEG)
• MPEG-1
• Developed to handle slower media such as
  videoCDs.
  
• Deteriorates when high-speed action occurs.
• Roughly the quality of VHS.
• 70 minutes with sound fits on a CD-ROM disc.
• MPEG-2
• Full-screen codec standard that brings twice the
  resolution of VHS.
  
• Four times the resolution of MPEG-1.
• Optimized for higher demands of broadcast,
  satellite, and DVD.

25
Visual MediaAnimation and Video

• Common codecs (continued)
• Motion Picture Experts Group (MPEG)
• M-JPEG
• Requires more memory space than MPEG-2
• Each frame is digitized separately using JPEG
  compression.
  
• Makes it easier to edit.
• As we move from analog to digital broadcast,
  coding and decoding will become unnecessary.
  (Everything will already be digital!)
  

26
Visual MediaAnimation and Video

• DVD The Digital Versatile Disc
• DVD An optical disk storage technology that
  looks like the standard CD, but is faster and has
  a much larger storage capacity.
  
• Two forms
• DVD-Video Discs Intended as a replacement of
  VHS.
  
• DVD-ROM Intended as a computer storage medium.
• Digital Video Players
• Software products are available that will play
  back video coded with one of these codec
  systems.
  
• QuickTime, ActiveMovie

27
Visual MediaAnimation and Video

• Streaming media Information in one of many
  different visual or audio forms that is sent from
  a server in packets to the requesting computer.
  
• Contains parts of the medium that may be a
  recording or an actual live event.
  
• The information is played back in the order sent,
  starting soon after a few packets arrive, but
  before all of them have been sent.

28
Audio MediaHuman Speech

• The importance of vocal communication
• It is the fastest method of communication.
• Most communication is vocal.
• Researchers believe that human intelligence is
  closely linked to the development of vocal
  communication and language.

29
Audio MediaHuman Speech

• Creating Speech Recorded Speech
• Recording and digitizing individual words and
  phrases.
  
• Common uses
• Talking cash registers.
• Automated national telephone information
  service.
  
• Problems with recorded speech
• Creating whole sentences of human speech would
  take a huge amount of computer memory.
  
• Several versions of each word would need to be
  digitized to account for pitch and volume levels
  (for punctuation and variations of the same word).

30
Audio MediaHuman Speech

• Creating Speech Speech Synthesis
• Speech synthesis The electronic production of
  sounds and sound patterns that closely resemble
  human speech.
  
• Uses phonemes The fundamental sounds of any
  given human language.
  
• Three additional factors that have an affect on
  how a word or phrase sounds
  
• Inflection - The rising and falling pattern of
  pitch.
  
• Duration - The time spent on or the time between
  individual phonemes.
  
• Elision - Splicing phonemes together so that when
  one ends, the other begins. The connection will
  sound natural.

31
Audio MediaHuman Speech

• Creating Speech Speech Synthesis
• The sounds produced are electronically
  generated.
  
• The sounds simulate those produced by the human
  vocal tract.
  
• Can produce both major types of human sounds
• Voiced sounds Sounds produced by the vibration
  of vocal cords in conjunction with specific
  positioning of teeth, tongue, and lips. Include
  all vowels and some consonants.(d, g, b)
• Voiceless sounds Characterized by the lack of
  vibration of the vocal cords. (s, k, t)

32
Audio MediaHuman Speech

• Recognizing Spoken Words
• More difficult for the computer than speech
  synthesis.
  
• Voiceprint A visual plot of frequency versus
  time of sound produced by a human speaker.
  
• No two voiceprints are completely identical.
• Human voice is determined at least by
• Pitch The number of cycles per second of a
  particular sounds vibration.
  
• Resonance The reverberation or amplification of
  the voice in the cavities of the vocal tract.
  
• Problem Because each individual has uniquely
  shaped vocal cords and resonating chambers, each
  has a uniquely individual voice.

33
Audio MediaHuman Speech

• Recognizing Speech
• Disjointed speech Words spoken one at a time
  with silence between words.
  
• Software programs are available for home use.
• Each person goes through a recording session to
  train it.
  
• Total vocabulary to fewer than 1000 words spoken
  by an individual.
  
• Problems
• Speaking in a disjointed manner is unnatural.
• Any changes in the persons voice (head cold,
  sinus infection) may make the speech
  unrecognizable.
  

34
Audio MediaHuman Speech

• Recognizing Speech
• Continuous speech Words spoken in a continuous
  stream of sounds, usually with no pauses between
  words.
  
• More difficult for the computer to understand.
  Computer has difficulty when one word ends and
  next begins.

35
Audio MediaMusic

• The Computer and Recorded Music
• Compact Disc Technology
• CDs dont wear out like records and magnetic
  tape.
  
• After numbers are recorded, they dont change.
• Scratching the surface can affect quality of
  sound.
  
• Compact discs can be produced from a master disc
  very inexpensively.

36
Audio MediaMusic

• The Computer and Recorded Music
• DAT (Digital Audio Tape) technology
• Advantages of DAT
• Binary information is recorded onto magnetic
  tape.
  
• DAT can easily record sound information.
• Perfect duplication of CDs can be made onto
  tape.
  

37
Audio MediaMusic

• Recorded Audio Files and Formats
• Individual standards have been developed for
  special purposes or for specific computers.
  
• Sound files can be huge. Sound quality affects
  size.
  
• The four most common sound file formats
• WAV Originally developed with Windows 3.1
• AU Developed by SunAudio for UNIX systems but
  now supported by Windows and Macintosh.
  
• AIFF Originated on the Macintosh computer.
• MP3 (MPEG - level 3) Can be played on Windows,
  Macintosh, and UNIX systems.

38
Audio MediaMusic

• Hardware and Software Needs
• Playing back Web sound
• Need a sound card.
• Newer multimedia computers have sound cards
  included.
  
• Need external speakers.
• Receiving sound files from the Web.
• Must be a type of file your browser recognizes.
• Most new browsers handle Web sound.

39
Audio MediaMusic

• MIDI A Revolution in Music
• MIDI (Musical Instrument Digital Interface)
  Musical standard created in 1982 that makes
  possible communication of audio information
  
• Between keyboards, drum pads, etc.
• To a computer-controlled mixer.
• Each instrument connected to a MIDI bus usually
  has a small special-purpose computer with some
  memory.
  
• MIDI bus The main communications channel for
  MIDI information that can consist of signals sent
  to as many as 16 different instruments or groups
  of instruments.

40
Audio MediaMusic

• All MIDI instruments have three-pronged
  connectors that are used to connect other MIDI
  instruments or computers.
  
• Thru port Sends MIDI signals obtained from the
  IN port on to other MIDI instruments.
  
• OUT port Sends out MIDI signals.
• IN port Receives MIDI signals.
• The three connectors necessary for a MIDI
  interface
  

41
Audio MediaMusic

• Several ways to use a MIDI instrument
• By itself.
• A MIDI synthesizer can connect to an amplifier
  with speakers.
  
• Using one MIDI instrument as a master to a
  second, acting as a slave.
  
• Can not only play its own notes, but can also
  control the slaves notes.
  
• In conjunction with a MIDI sequencer.
• MIDI sequencer A device, usually a computer,
  that records and plays back MIDI signals.
  
• Can be either a computer with sequencing software
  or a dedicated sequencer.
  

42
Internet Music and Audio Net

• Three techniques for both downloading sound to
  your computer and listening to live audio over
  the WWW.
  
• 1. Streaming audio A request to hear music is
  made by clicking on a button. The Website sends
  back the sound which you hear immediately. You
  dont receive a copy of the sound. (You may need
  to add a plug-in to hear the sound.)
• Plug-in A piece of software that is added to
  your browser to give it desired capability.
  
• 2. Sound file is sent over the WWW. It allows you
  to store the sound file on your computer.
  
• 3. MIDI files Created by a performer on a MIDI
  instrument. Needs a MIDI player (available on the
  Internet.)