Overview of Multiview Video Coding

1
Overview of Multi-view Video Coding

• Yo-Sung Ho Kwan-Jung OhSystems, Signals and
  Image Processing, 2007 and 6th EURASIP Conference
  focused on Speech and Image Processing,
  Multimedia Communications and Services. 14th
  International Workshop on

2
Outline

• Introduction
• Applications of Multi-view Video
• Requirements for Multi-view Video Coding
• Test Data Sets and Test Conditions
• Joint Multi-view Video Model (JMVM)
• Conclusion

3
Introduction
4
Introduction

• Multimedia Demands
• holography
• two-view stereoscopic system with special glasses
• multi-view video
• Multi-view Video
• FVV, FVT, 3DTV
• What is multi-view video?
• Why we need multi-view video coding (MVC)?
• MVC has been studied in the past.
• MVP, MCP, DCP, MPEG4 MAC, H.263/4

5
Introduction

• MPEG and ad hoc group on 3DAV
• 4 exploration experiments on 3DAV
• EE1 on omni-directional video
• EE2 on FTV
• EE3 on coding of stereoscopic video using
  multiple auxiliary components (MAC)
• EE4 on depth/disparity coding for 3DTV and
  intermediate view interpolation
• Companies have interests for FTV 3DTV
• JVT MVC group of MPEG VCEG
• JMVM as an extension of the H.264/MPEG4 AVC

6
Application of MVC
7
Application of MVC

• Free Viewpoint Television (FTV)
• Three-dimensional TV (3DTV)
• Immersive Teleconference

8
FTV

• What is FTV (Free Viewpoint Television)?
• Application of FTV
• Entertainment
• Education
• Sightseeing
• Surveillance
• Archive

9
FTV
10
3DTV

• What is 3DTV?
• Interaction may not be required
• To broadcast on 3DTV

11
3DTV
12
3DTV

• Capture by various types of multiple cameras
• 1D parallel
• 2D parallel
• 1D arc
• etc.
• Intermediate view reconstruction (IVR)

13
Immersive Teleconference

• What is immersive teleconference?
• Interaction

14
Requirements for Multi-view Video Coding
15
Requirements for Multi-view Video Coding

• Requirements for multi-view video coding
• Compression related requirements
• System support related requirements

16
Compression Related Requirements

• Compression efficiency
• View scalability
• Free viewpoint scalability
• Spatial/Temporal/SNR scalability
• Backward compatibility
• Resource consumption
• Low delay
• Robustness

17
Compression Related Requirements

• Resolution, bit depth, chroma sampling format
• Picture quality among views
• Temporal random access
• View random access
• Spatial random access
• Resource management
• Parallel processing

18
System Support Related Requirements

• Synchronization
• View generation
• Non-planar imaging and display systems
• Camera parameters

19
Test Data Sets and Test Conditions
20
Test Data Sets and Test Conditions
21
Test Data Sets and Test Conditions
22
Joint Multi-view Video Model
23
Joint Multi-view Video Model
24
Random Access

• GGOP contains
• frames.
• For accessing any frame within a GGOP, we have to
  decode maximum number of frames.
• b4(S5/T7), following 18 referencing frames.

25
Time-first coding order
26
Encoder Complexity

• Minimum decoded picture buffer (DPB)
• EX GOP_length16, number_of_views8, the DPB
  size 42
• MVC codec will have the same coding delay as
  single view video coding since time-first coding
  is mandated.

27
GOP structures and view prediction structure
28
GOP structures and view prediction structure
29
GOP structures and view prediction structure
30
Illumination compensation

• ICA MC illumination change-adaptive motion
  compensation
• Macroblocks(MB) mode in h.264/MPEG-4 AVC Inter
  1616 mode, Direct 1616 mode (include B_Skip),
  and P_Skip mode
• DVIC difference value of illumination change
• ICA ME illumination change-adaptive motion
  estimation

31
(No Transcript)
32

• SAD calculation for the motion estimation of ST
  blocks
• In order to compensate the illumination change

33

• 1
• 1
• Illumination compensated residual signal

34
Other Technical Issues

• View-temporal prediction structure
• Single video v.s. Multi-view video
• Three main coding structures
• Encode multiple video sequences separately
• Utilizes inter-view correlation only
• Utilizes both temporal and inter-view correlation
• View interpolation prediction
• Decoder side disparity estimation
• Computing depth at encoder side and transmitting
  this to the decoder
• Motion/Disparity vector coding
• Highly correlated each other

35
Conclusion
36
Conclusion

• The multi-view video includes multi-viewpoint
  video sequence captured by several cameras at the
  same time.
• Compress multi-view video efficiently
• MPEG and JVT are leading the standardization of
  MVC.