Assessing Audio Visual Quality

1
Assessing Audio Visual Quality

• P905 - AQUAVIT
• Assessment of Quality for audio-visual signals
  over Internet and UMTS
• Geoff Morrison, BT Labs

2
Contents

• Why we are doing this
• What we want to achieve
• How we are going about it
• Who is doing it
• BT, CSELT, Berkom
• When are we doing it
• February 1999 to December 2000
• The Results

3
Rationale

• Coding algorithms, such as MPEG-4 and H.263, can
  compress audio and video signals to bit rateslt1
  Mbit/s as available over mobile and IP networks.

• The impact of impairments, from compression and
  transmission, on users' opinion must be
  understood.
• Audiovisual communication must be evaluated under
  representative conditions and real tasks, such
  as conversational services and AV database
  retrieval.
• Network performance must be monitored to
  guarantee a minimum QoS. Need single-ended models
  to check audio and video transmission quality in
  real time.

4
Compression Impairments

• Digital TV in studios is 216 Mbit/s
• Lossless compression typically only halves the
  bit rate
• To reach very low rates requires lossy audio and
  video compression, introducing significant
  distortions

Is the resulting application usable for
purpose? How best to assign a given total bit
rate between video and audio?
5
Transmission Impairments

• AV bits ltgt data
• Audiovisual content has a timeliness aspect
• Critical in conversational applications
• Important in database retrieval applications
• Error detection and retransmission protocols not
  ideal for audiovisual content
• variable delay
• variable throughput
• Compressed AV is very sensitive to transmission
  errors, packet loss
• more compression ? more sensitivity

6
Error Handling

• Error resilience tools for encoder
• structure compressed data so that damage is
  localised
• optimise encoder options for the network
  performance
• Forward Error Correction
• improve apparent network performance seen by
  decoder
• Error concealment at decoder
• reconstruct approximations for damaged parts of
  pictures and sound
• Combinations of above

How well do these work in practice?
7
Project Objectives

• Understand the potential of audio, video and AV
  communication at bit rates lt 1Mbit/s for UMTS and
  IP.
• Develop tools and techniques for simulation of
  complete audiovisual transmission systems.
• Develop quality evaluation methods for complete
  audiovisual transmission systems.
• Investigate the relationship between system
  parameters (network and terminal) and subjective
  quality.
• Provide guidelines for the system parameters to
  be used for new applications.
• Encourage the use of audiovisual communication on
  these dominant networks of the future

8
Work Items

• Define the tests to be performed
• Develop/implement a UMTS test bed
• Develop/implement an IP test-bed
• Conduct subjective tests
• Make objective measurements and relate them to
  subjective results
• Disseminate the findings

9
Definition of tests

• Literature survey for available test methods
• Select appropriate methods

10
UMTS Test Bed

• Video codec (MPEG-4 and/or H.263)
• Audio codec (MPEG AAC, scalable and non-scalable)
• Error resilience tools (MPEG-4)
• MPEG-4 multiplex
• Transmission errors - simulated and perhaps
  actual
• Error recovery
• Demux
• Optimisation of parameters
• Tools to make objective measurements
• Production of material for subjective and
  objective tests

11
UMTS Test Bed

• Baseline non real-time test bed
• MPEG-4 baseline video codec, AAC and G.723.1
  audio, AL layer and MPEG DMIF
• Enhanced non real-time test bed
• Error resilience tools, layered audio

12
UMTS Test Bed
13
IP Test Bed

• Video codecs (H.263 and/or MPEG-4)
• Audio codecs (G.723.1, MPEG AAC scalable and n/s)
• Error resilience tools (ITU-T, MPEG, IETF)
• Packetisation
• Transmission errors - simulated, measured
  profiles and actual
• Error recovery
• Optimisation of parameters
• Tools to make objective data measurements
• Production of material for subjective and
  objective tests

14
IP Test Bed

• Baseline non real-time test bed
• G.723.1, H.263, simplified RTP, simple network
  impairment simulator
• Enhanced non real-time test bed
• more codecs including MPEG, error resilience,
  complete RTP, error concealment, enhanced network
  simulator
• Enhanced real-time test bed
• real time image capture, encoding and decoding

15
IP Test Bed
16
Subjective tests

• Conventional subjective tests covering a
  suitably wide range of variables
• the codecs
• bit rate, video frame rate, audio bandwidth
• Error conditions (packet loss, delay variability,
  )
• Protocols (CBR, VBR, ABR, RSVP, ..)
• Application (one-way, conversational)
• Users may accept lower quality in return for
  mobility. Testing will seek an estimation of the
  threshold of acceptability of audio and video
  quality in mobile applications

17
Objective measurements

• Objective assessment of audio quality
• TOSQA, PAMS, ...
• Objective assessment of video quality
• model of human vision
• other algorithms to objectively estimate users
  opinion
• Compare subjective and objective assessments
• Investigate relationship between parameter values
  and subjective audio, video and AV quality
  assessments

18
Double Ended Model

• Measuring instrument needs both original input to
  encoder and delivered version at decoder output

19
Single Ended Model

• Measuring instrument needs only decoder output
• more suitable for live performance monitoring

20
Project Resources

• 20 man-months

21
Results

• Reports due in Q4 of 2000.
• The findings should be of benefit to at least 3
  groups
• Network Providers. Results such as minimum
  desirable bit rates and error characteristics
  should influence the planning and operation of IP
  and UMTS networks intended to carry audiovisual
  services.
• Application Providers. The results will provide
  reliable guidance about which applications and
  operating parameters will satisfactorily meet
  customers' expectations and those which will not.
• Researchers in the field of "human factors". It
  is expected that the project will advance the
  state of the art in perceptual modelling and in
  the understanding of cross-modal effects.