Viewpoint Switching in Multiview Video Streaming

1
Viewpoint Switching in Multiview Video Streaming
Circuits and Systems, 2005. ISCAS 2005. IEEE
International Symposium on

• Xun Guo, Yan Lu, Wen Gao, Qingming Huang

2006.11.23. kimsungsoo
2
Multiview video

• ??? ?? ??? multiple cameras? capture
• Video sequences
• ??? ???? ???? ???? ??? ? ?? ?? ??
• ???? ?? ?? ? ?? ??? ??
• Multiview video streaming?? ??? ?? ???? ??,
  ??? ????? ????? ?? ????

3
Multiview video streaming
4
Switching between two views with GME
5
H.264 Encoder Block diagram

• P
• Prediction macroblock
• Intra mode Based on a reconstructed frame
• Inter mode Motion-compensated prediction from
  one or more reference frames
• Dn
• Residual or difference macroblock
• X
• Quantized transform coefficients
• Reordered and entropy encoded
• De-blocking filter
• Reduces the effects of blocking distortion

6
Proposed encoding process(1/3)

• 1.Intra frame
• encoded with the traditional coding scheme
• GMEs are performed between the current frame at
  the frame buffer and the corresponding decoded
  frames at its adjacent views.
• The estimated global motion parameters are
  represented with three pairs of motion vectors.
• GME results after inverse can be shared by the
  adjacent views.

7
Proposed encoding process(2/3)

• 2.Inter frame
• global motion vectors are encoded into the
  bitstream with the method developed in MPEG-4.
• motion vectors
• useful only when the switch happens
• transmitted only when the switching happens
• the current frame is decoded, and the GMEs with
  the frames at its adjacent views are performed as
  well

8
Proposed encoding process(3/3)

• global parameters can efficiently predict the
  motion between two adjacent pictures.
• When switch happens
• the difference between P1 and P2 can be
  efficiently limited by warping P1 with the global
  parameters
• the drifting error taken by switch can be reduced
  efficiently.

9
The proposed encoder for viewpoint switching
10
H.264 Decoder Block diagram

• X
• Quantized coefficients
• Entropy decoded and reordered
• Dn
• Residual or difference macroblock
• Rescaled and inverse transformed
• P
• Prediction macroblock
• Fn
• Filtered values of uFn PDn

11
Proposed decoding process

• The decoding method is almost the same as the
  original H.264
• except that sometimes the preprocessing has to be
  done on the reference frames.
• When there is no switch
• the original decoding process is performed
• the decoded frame is saved in the decoder buffer
• when the switch happens
• global motion vectors corresponding to the
  reference frame are received and decoded
• reference frame in the buffer is warped according
  to the global motion parameters prior to motion
  compensation.

12
The proposed decoder for viewpoint switching
13
intra/inter coding mode selection

• due to the displacement between the views
• there are always some regions that cannot be
  compensated with GME
• (e. g. the border regions that only exist in one
  frame)
• the drifting problem may still exist.
• mode decision? Lagrangian function? ????

14
Experimental results (1/3)

• H.264 reference software JM 8.2
• http//iphome.hhi.de/suehring/tml/download/old_jm/
  jm82.zip
• race1 and golf2 provided by KDDI Lab
• ftp//ftp.ne.jp/21/KDDI/multiview/Race1/
• race1_0.yuvrace1_7.yuv
• 3000 frames (YUV 420, 320x240)
• parameters are set as follows
• RD optimization on
• Hadamard transform on
• Reference number 1
• Entropy coding CABAC

15
Experimental results (2/3)
16
Experimental results (3/3)
17
CONCLUSIONS

• Global motion estimation model? ??? ??? ?? ????
  ????? ??
• New RDO??? ??? inter/intra coding mode? ???? ??
  ??
• ?? ? ???? drifting error? ????? ??
• Switch directly(H.264 coding scheme)? ???? ?
• ??? ??? ?? ? ?? ??? ????? (2dB)? ??