Static Sprite Generation

1
Static Sprite Generation

• Prof? David, Lin
• Student? Jang-Ta, Jiang
• 2006.02.16

2
Outline

• Introduction
• Sprite-Generation Architecture
  GME?Segmentation?Extraction?Blending
• Application in Frame-Based Video Coding
• Experimental Results
• Conclusions

3
Reference

• 1 Y. Lu, W. Gao, and F. Wu, "Sprite generation
  for frame-based video coding," in Proc. IEEE
  International Conference on Image Processing
  (ICIP), 1, pp. 473--476, 2001.
• 2 Y. Lu, W. Gao, and F. Wu, " High efficient
  sprite coding with directional spatial prediction
  , " ICIP (1) 2002 201-204
• 3 Y. Lu, W. Gao, and F. Wu, " Efficient
  background video coding with static sprite
  generation and arbitrary-shape spatial prediction
  techniques, " IEEE Trans. Circuits Syst. Video
  Techn. 13(5) 394-405 (2003)

4
Introduction (1/4)

• A sprite is an image composed of pixels belonging
  to a video object visible throughout a video
  segment
• For instance, sprite generated from a panning
  sequence will contain all the visible pixels of
  the background object throughout the sequence
• Portions of this background may not be visible in
  certain frames due to the occlusion of the
  foreground objects or the camera motion.
• Thus, the sprite contains all parts of the
  background that were at least visible once

5
Introduction (2/4)

• The sprite encoding syntax can be utilized for
  the transmission of any still image to the
  decoder since a sprite is essentially just a
  still image
• Static sprites are those that are directly copied
  (including appropriate warping and cropping) to
  generate a particular rendition of the sprite at
  a particular time instant
• Improves the coding efficiency for video
  sequences with lots of revisiting backgrounds

6
Introduction (3/4)

• The main idea of static sprite coding technique
  is to generate the reconstructed VOPs by directly
  warping the quantized sprite using specified
  motion parameters
• Residual error between the original VOP and the
  warped sprite is not added to the warped sprite

7
Introduction (4/4)
8
Motivation

• The motion of foreground video objects not only
  disturbs the accuracy of motion estimation but
  also blurs the generated sprite image
• We presents a novel technique for the generation
  of background sprite image with improved Global
  Motion Estimation (GME) and automatic
  segmentation
• The proposed technique is used to construct high
  quality sprite directly from raw video sequence

9
Outline

• Introduction
• Sprite-Generation Architecture
  GME?Segmentation?Extraction?Blending
• Application in Frame-Based Video Coding
• Experimental Results
• Conclusions

10
Sprite-Generation Architecture

• The first frame is blended into the blank sprite
  to obtain the initial sprite.

11
Sprite-Generation Architecture (cont.)

• First, the global motion between the current
  frame and the sprite is estimated.
• Secondly, the reliability map is extracted based
  on the segmentation information.
• Third, both the current frame and the extracted
  reliability map are warped toward the sprite
  image.
• Finally, the warped frame is blended with the
  sprite using three different strategies in terms
  of the warped reliability map.

12
Global Motion Estimation (GME)

• GME aims at finding the global motion parameters
  of the current frame relative to the sprite
  image.
• A traditional hierarchical GME based on gradient
  decent algorithm is employed
• However, the difference is that a hybrid scheme
  jointing the short-term and long-term motion
  estimation is developed
• Traditionally, GME either performs the short-term
  motion estimation and then concatenates the
  parameters or performs the long-term motion
  estimation directly

13
Global Motion Estimation (GME)

• Instead of estimating the global motion of the
  current image directly from the previous sprite,
  the described algorithm first warps the previous
  sprite and then calculates the global motion
  referencing the warped sprite image at the top
  level of the hierarchical GME
• The proposed hybrid GME scheme can tackle the
  long-term motion influence with the assistance of
  the previously estimated motion, and meanwhile
  avoid error accumulation by directly using the
  original sprite as reference image in the final
  step of GME

14
Intermediate global motion
Initial global motion
15
Rough Image Segmentation

• 1. Motion occlusion zones detection
• detected from the difference image
• the zones can be detected using a threshold
• morphological filtering is developed

16
Rough Image Segmentation

• 2. Segmentation model initialization
• Canny operator is employed to detect all the edge
  pixels in the current frame
• the initial segmentation model is extracted by
  selecting foreground edge pixels in term of the
  motion occlusion zones

17
Rough Image Segmentation

• 3. Moving object extraction
• utilizing a filling-in technique to the initial
  segmentation model
• the foreground objects are segmented from the
  mask

18
Pixel Reliability Extraction

• Segment the current frame into three regions
• Pixels not belonging to the background consist of
  the undefined region (UN)
• Pixels belonging to the background are segmented
  into reliable region (RR) and unreliable region
  (UR)
• UR is defined by extracting some pixels along the
  borders of the background object,i.e.

19
Pixel Reliability Extraction

• (a)
  (b) (c)
• (a) extracted reliability map
• (b) original segmentation map
• (c) reliability map warped toward the sprite

20
Reliability-Based Image Blending

• Reliable pixels in the input frame can be
  averagely blended with reliable pixels in the
  sprite, or replace unreliable pixels in the
  sprite and then mark them as reliable.
• Unreliable pixels in the input frame cannot be
  used to update reliable pixels in the sprite, but
  it can be averagely blended with the unreliable
  pixel, or fill the empty region in the sprite and
  then set it as unreliable.
• Undefined pixels in the input frame do not
  contribute to the sprite updating.

21
Reliability-Based Image Blending

• After processing all frames of the video
  sequence, the reliable and unreliable regions in
  the sprite are merged together to obtain the
  final opaque areas of the sprite image
• Two advantages?
• prevent the unreliable pixels from constantly
  contributing to the sprite updating
• unreliable region tackles the aperture problem of
  the sprite generation that might happen in the
  place where the reliable pixel never appears

22
Some Background Sprite
ForemanCoastguard
23
Outline

• Introduction
• Sprite-Generation Architecture
  GME?Segmentation?Extraction?Blending
• Application in Frame-Based Video Coding
• Experimental Results
• Conclusions

24
Application in Frame-Based Video Coding

• Sprite coding is usually used in object-based
  video coding, whose implementation premise is
  that precise segmentation mask should be
  available in advance
• A frame-based video coding scheme incorporating
  sprite coding techniques is developed
• Rough segmentation is enough for the purpose of
  sprite generation

25
Application in Frame-Based Video Coding

• The coding mode can be selected automatically in
  two step
• Firstly, the selection is decided between SPRITE
  and INTRA/INTER mode
• If SPRITE mode is selected, error signals are
  encoded using MPEG-4 INTER coding method
  otherwise the traditional block-based motion
  compensation is performed and then INTRA or INTER
  mode is secondly selected

26
Application in Frame-Based Video Coding

• The raw MBs in INTRA mode and the error signal
  MBs in INTER mode are encoded using MPEG-4 coding
  scheme, respectively
• The proposed coding method is more efficient
  compared with traditional frame-based techniques,
  because the sprite motion model can effectively
  describe the motion of camera, and therefore the
  bits can be greatly saved

27
Outline

• Introduction
• Sprite-Generation Architecture
  GME?Segmentation?Extraction?Blending
• Application in Frame-Based Video Coding
• Experimental Results
• Conclusions

28
Experimental Results (1/5)

• (1) Original frame (2) auxiliary
  segmentation (3) rough segmentation

29
Experimental Results (2/5)
Stefan sequence, 300 frames in CIF at frame rate
30fps (auxiliary segmentation)
(a) reliability-based blending (average PSNR
23.1dB)(b) average blending(average PSNR
22.4dB)
30
Experimental Results (3/5)

• (c)
  (d) (e)
• (c) original image
• (d) image reconstructed from the sprite in (a)
• (e) image reconstructed from the sprite in (b)

31
Experimental Results (4/5)
Stefan sequence, 300 frames in CIF at frame rate
30fps (rough segmentation)
(f) reliability-based blending (average PSNR
22.9dB)(g) average blending(average PSNR
22.0dB)
32
Experimental Results (5/5)

• (h)
  (i)
  (j)
• (h) original image
• (i) image reconstructed from the sprite in (f)
• (j) image reconstructed from the sprite in (g)

33
Outline

• Introduction
• Sprite-Generation Architecture
  GME?Segmentation?Extraction?Blending
• Application in Frame-Based Video Coding
• Experimental Results
• Conclusions

34
Conclusions

• Considering that accurate segmentation is very
  hard to obtain, we proposes to utilize the
  reliability-based blending in the sprite
  generation
• Experiments have proven that the
  reliability-based blending scheme can effectively
  eliminate the blurs caused by moving foreground
  objects due to the inaccurate segmentation
• The proposed technique can produce the sprite
  directly from the raw video sequence by using the
  rough image segmentation and the
  reliability-based blending scheme

35
Thank You