Natural and Synthetic Video in MPEG4

1
Natural and Synthetic Video in MPEG4

• Written By Jorn Ostermann and Atul Puri

2
Overview

• Coding of Natural Video
• Shape Coding
• Motion Coding
• Texture Coding

• Bitstream Issues
• Scalable Video Coding
• Robust Video Coding

• Coding of Synthetic Video
• Object Mesh Coding
• Face Animation
• Still Texture Coding

3
Basics of Video Coding

• MPEG-4 uses the concept of Visual Objects (VOs)
• Video Object Planes (VOPs) are instances of VOs
  in a video sequence
• MPEG-4 uses I-VOP, P-VOP, B-VOP

4
Shape Coding

• Binary alpha maps
• Rectangular bounding box covers the VOP
• Divided into 16X16 pels called binary alpha
  blocks(BAB)
• Each pel is coded within a context, depending
  what mode it is in

5
Shape Coding - 2

• Grayscale alpha maps
• Represents degree of the transparency of the
  pixel
• 0 is transparent, 255 is opaque
• Can be used in hybrid video

6
Motion Coding

• Motion Estimation
• Estimates where a macroblock moves to
• P-VOPs estimate using current VOP and previous
  VOP
• B-VOPs estimate using current VOP and
  neighbouring VOPs
• Motion Block Predictor
• Generates motion vector difference signal for 8X8
  or 16X16 pel blocks

7
Motion Coding - 2

• Motion Compensation
• Padding of the texture
• Synthesis of the predicted VOP

8
Texture Coding

• Encodes the luminance and chrominance information
• Boundary 8X8 blocks are padded
• Encoded using DCT and then quantized
• Intra block tools
• Non-linear scaler for DCT

9
Texture Coding - 2

• DC Prediction
• AC Prediction

10
Texture Coding - 3

• Adaptive DC scanning

11
Scalable Video Coding

• Spatial Scalability
• Temporal Scalability

12
Sprite Coding

• Sprites are graphic images that move or are
  animated individually
• MPEG-4 sprites are static sprites
• High coding efficiency sprite with warping

13
Sprite Coding - 2
14
Sprite Coding - 3

• Sprite is built, so an image is created
• Sprite and warping parameters are sent
• Only warping parameters are needed for subsequent
  frames
• Sprite Generation
• Consists of 3 steps, Global motion estimation,
  warping and blending

15
Sprite Coding - 4

• Global Estimation
• Estimate the motion of the image with reference
  to the sprite
• Constrained by 8 parameters

16
Sprite Coding - 5

• Pyramid is used to reduce computational
  complexity
• Search algorithm is used at top level
• Warping and Blending
• Once motion is found, image is aligned with the
  sprite
• Blend the image by averaging the edges

17
Sprite Coding - 6

• Sprite decoding
• The sprite is sent once, while for each frame up
  to 4 reference points are sent
• Differential coding is used from frame to frame
• Sprite is warped to the image using the reference
  points
• Sprite is sent in pieces or coarsely

18
Robust Video Coding

• Resynchronization
• Resynchronization markers are put between rows of
  macroblocks (GOBs)
• Position of markers are arbitrary, but usually at
  uniform intervals
• No data dependencies between video packets

19
Robust Video Coding - 2

• Data Partitioning
• Localize the error inside a video packet
• The motion and DCT data are separated
• Within the motion and DCT, data is partitioned

20
Robust Video Coding -3

• Reversible Variable Length Codes
• VLC, once error is detected, rest of video packet
  is discarded
• RLC, once error is detected, decoder reads
  backward from next marker until error is found
• Prefix property in both directions

21
Object Mesh Coding

• Planar graph that partitions an image into
  traingles
• Only node points are tracked
• Warping used to create new pictures

22
Object Mesh Coding - 2

• I-plane coding
• All node information is stored (or parameters if
  regular mesh)
• First node is coded, rest are differential

23
Object Mesh Coding - 3

• P-plane coding
• For each vertex, the motion is defined
• To find the vertex, object tracking is used
• Motion vector estimation and tracking
• Form block around centre and minimize error
• Hexagonal matching

24
Face Animation

• Facial animation parameters (FAPs) define a face
• 66 base level FAPs, 68 FAPs total

25
Face Animation - 2
26
Face Animation - 3

• Visemes and Expressions
• Visemes are preset combinations of FAPs
• Visemes used for speech
• Expressions used to show emotions

27
Still Texture Coding

• Wavelets are used because it is more scalable
• Usually used in synthetic videos