Current Work on Video Coding

1
Current Work on Video Coding

• Raj Kumar

2
Overview

• Fine-Grained Scalability (FGS)
• What is it and why do we need it
• What are its drawbacks
• Our new scheme (FGS)
• B-Frame only scheme
• All-Frame Scheme
• Future Directions

3
Fine Grained Scalability

• What is it
• What are the pros and cons

4
Traditional Vs Fine-grained Encoding
5
Temporal FGS
6
Problem with Temporal FGS

• Reference Frames are poor quality
• Doesnt use motion-prediction well
• Compression ratio suffers compared to traditional
  encoders (approx 2-2.5 DB)

7
Insight

• FGS provides two degrees of freedom
• SNR
• Temporal
• Is there an optimal path that will maximize
  overall quality.
• How do we determine it
• How do we take advantage of it

8
Idea

• Can we use a part of the enhancement layer, in
  addition to the base layer, to predict motion?
• We can if we know how much of the base layer will
  be present at the decoder when the temporal
  layers are introduced
• So if we build in rules about when temporal
  layers are introduced relative to spatial
  quality, we can!

9
Implementation

• Increase SNR quality to a predetermined point
• Then improve temporal quality
• Further improve SNR quality at the new frame-rate
• And so on.

10
FGS Scheme (B-frame only)
11
Two issues

• When (at what SNR quality) do we introduce new
  temporal-frames?
• How much extended-reference do we use

12
Introducing Temporal Frames

• We conducted a study where users chose preferred
  frame-rate at different bit-rates
• As the bit-rate goes up, people prefer better
  frame-rates
• High-motion videos (Stefan, Coastguard) require a
  quicker transition

13
Determining the size of the reference
 

• Plot SNR performance as a function of enhancement
  at various bit-rates
• Choose enhancement corresponding to best quality

14
Performance

• Improvement over FGS varies from 0.19 dB to 1.28
  dB
• At low bit-rates simple videos benefit
  (Coastguard)
• At high bit-rates complex videos benefit (Mobile)

15
FGS for All-Frames

• We can improve performance further by using
  enhancement for base-layer P-frames also
• But P-frames are present at all bit-rates. How
  much enhancement do we use?

16
FGS Scheme (All-frames)
Temporal Enhancement Layer
Spatial Enhancement Layer
I
P
Base Layer
17
Improvement in Performance

• At 3 bit-planes
• Performance varies from 0.03 dB to 0.59 dB
• Performance will degrade at low bit-rates, due to
  incomplete references

18
Conclusion

• FGS uses a spatio-temporal notion of
  video-quality
• Improves performance of traditional FGS from 0.16
  to 1.61 dB

19
Current Issues

• Spatio-Temporal preference curves vary for
  different videos
• How do we choose the correct curve automatically
• Use Motion-Vectors, Image-Complexity

20
Future Directions

• Factors other than video-characteristics do
  affect spatio-temporal preferences
• Environment (resolution, display-type, lighting)
• User preferences (moods, some prefer
  spatial-quality always)
• How do we account for these other dimensions ?
• Large dimensional space requires many (thousands)
  of sample points

21
Future Directions

• Build online experiment
• Interactive web-based tool that lets user watch
  many videos
• Allow user to change spatio-temporal settings
• Record data corresponding to changes
  (video-characteristics, display resolution and
  type, user information)