Media%20Scaling%20/%20Content%20Adaptation

1
Media Scaling /Content Adaptation

• Ketan Mayer-Patel

2
Media Scaling Overview

• Change encoding of the media to accommodate
  changes in available bandwidth.
• Components of the problem
• Estimating bandwidth.
• What can be adjusted.
• How to adjust.
• Cross media issues.
• Transcoding pre-encoded media.

3
Estimating Bandwidth

• Requires feedback.
• How often?
• In what form?
• TCP-friendliness
• Current thought act like TCP.
• Padhyes equation

4
Feedback

• How often?
• Design of TFRC says you want to limit this to
  once per RTT.
• Otherwise, rate of feedback is a design decision.
• Tradeoff responsiveness vs. feedback bandwidth
• Should be related to when the source can use the
  info.

5
Feedback

• In what form?
• Media independent information
• Loss rate and delay.
• Media specific information
• Quality of reconstructed media.
• Not necessarily well related to loss and delay.
• User preferences
• For example Regions of interest.
• Media specific feedback very application
  dependent.

6
Scaling Dimensions

• Different media can be scaled along different
  dimensions.
• Two major considerations
• Encoding support.
• The media representation often restricts the
  range and type of scalability.
• Perceptual quality.
• Non-linear subjective response to changes.
• Think of frame size changes.

7
Video

• Temporal scaling
• Reduce the resolution of the stream by reducing
  the frame rate
• Spatial scaling
• Reduce the number of pixels in an image
• Frequency scaling
• Reduce the number of DCT coefficients used in
  compression
• Amplitude scaling
• Reduce the color depth of each pixel in the image
• Color space scaling
• Reduce the number of colors available for
  displaying the image

8
Perceptual Impact and MPEG
Perceptual Impact
Effectiveness
Representable
Temporal Scaling Spatial Scaling Frequency
Scaling Amplitude Scaling Color Scaling
Medium High Medium Low Low
Highly Highly Very Not very Not very
Possible Unsupported Supported Unsupported Uns
upported
9
Scaling Dynamics

• Given some adjustment parameter p
• How can we adjust p given target rate r?
• Depends on how p is related to r.
• 3 basis situations
• Well known in advance.
• Well known for future.
• Indeterminate

10
Scaling Dynamics

• Two types of well known relationships.
• Immediate relationship.
• Given target bitrate r and media lifespan,
  parameter can be perfectly adjusted to give
  desired number of bits.
• This is rare.
• Future relationship.
• Parameter is adjusted after the fact to conform
  to target rate next time media is produced.
• Example frame rate.

11
Scaling Dynamics

• If relationship is indeterminate, what can we do?
• Incremental adjustments in appropriate direction.
• Basic algorithm
• At time ti Encode using pi
• Calculate rate mismatch di
• pi1 adjustment(pi, di)
• Issues
• Accounting for errors between target and achieved
  rates.
• Designing a good adjustment function.

12
Bitrate Accounting

• Cumulative accounting.
• Spectrum of accounting choices
• No carry.
• Deficit/surplus simply informs the adjustment
  function.
• Indefinite carry.
• In the limit, overall average rate matches target
  exactly.
• In between, a variety of decay and window-based
  possibilities.

13
Adjustment Functions

• Basic issues
• Shape
• Stability
• Lots of control theory here.
• Informal rules of thumb
• Damp small changes.
• Keep history (feedback)
• Take advantage of domain-specific knowledge.

14
Quality Example

• Quality/rate tradeoff for video generally looks
  like this

The problem is that if we dont know what
the curve is.
Useful quality adjustment range
15
Quality Example

• Possible strategies?
• Multipass solutions.
• Generally take too long.
• Content analysis
• Complex and only work in restricted domains.
• Model the tradeoff curve.
• Simple
• Only works well is model is close to reality.
• Also known as a rate-distortion framework

16
Rate-Distortion

• Ideal
• bits vs. quality as a closed form, invertible
  formula
• Strategies
• Choose a proxy for quality
• SNR measures
• Playback characteristics
• Example recovered frames per second
• User feedback
• Choose a proxy for rate
• Q Factor
• Number of non-zero coefficients
• Framerate

17
Rate-Distortion contd

• Fit curve
• Choice of function family based on representative
  sampling of content domain.
• Derive parameters
• Static for content domain.
• Adaptively based on recent ADUs.
• Analysis of sub-sampled ADU.

18
Cross Media Issues

• Aggregate behavior of multiple related streams
  may require different per stream behavior.
• Example 64 kbs audio with 300 kbs video.
• Suppose available bandwidth drops to 182 kbs
• 32 kbs audio, 150 kbs video
• 64 kbs audio, 118 kbs video.
• Integrate the adaptation process
• Cross inform the adaptation process

19
Transcoding

• Have assumed that media encoded on the fly so
  far. What about pre-recorded material?
• Transcoding issues
• Computational complexity
• Compressed domain processing.
• Scalability and availability of the service.
• More common solution
• Preset, multiple representations.

20
InfoPyramid

• Holistic approach to multimedia documents.
• Simultaneous adaptation of all related media.
• Utility curves defined for each media type along
  its adaptive dimension.
• Utility curve for document is summation of
  components.
• Document adaptation done as a resource allocation
  problem.