Detecting video frames in AVC_H.264 compressed bitstreams

1
Detecting video frames in AVC/H.264 compressed
bitstreams
2
AVC/H.264 video overview

• At a high level, video consists of a sequence of
  individual images or video frames
• Uncompressed video requires a substantial amount
  of storage space
• 1 second of 19201080/50p video requires 1920
  1080 1.5 (bytes per pixel in I420) 50
  155.52 MB
• Video compression is used to reduce storage and
  bitrate demands
• AVC/H.264, initially released in 2003, remains
  one of the most popular video compression
  standards worldwide
• We will use Virinext Bitstream Analyzer for
  analyzing AVC/H.264 files.

3
High level AVC/H.264 bitstream overview

• At a high level, AVC/H.264 video is a sequence of
  NAL Units (Network Abstraction Layer Units)

4
AVC/H.264 Slice NAL Unit

• Encoded video frames are stored within Slice NAL
  units
• Slice can contain either a full frame or just
  part of a frame.
• A special slice type, called an IDR-slice
  (Instantaneous Decoding Refresh), signals a reset
  in frame referencing

5
AVC/H.264 Slice Types

• The following slice types are used
• I-slice Contains only intra-predicted data
  (spatial prediction within the slice)
• P-slice Contains inter-predicted data
  referencing one prior slice
• B-slice Contains inter-predicted data
  referencing up to two different frames.
• I-slice only exploit spatial redundancy only.
• P- and B-slices use both intra-prediction and
  inter-prediction.

6
AVC/H.264 Slice Types
7
AVC/H.264 Slice and error propagation

• Slice NAL units are stored in bitstream
  independently
• Error in a bitstream within one slice(data loss
  or corruption) do not affect directly other
  slices
• Errors in decoding in some slice can lead in
  errors in another frames due to to incorrect
  predictions.

8
AVC/H.264 Slice and multithreading

• Slices are stored independently on bitstream
  level
• They allows for multithreaded processing on both
  the encoding and decoding sides
• When frame consists of multiple independent
  slices, this slices can be encoded and decoded in
  parallel.

9
AVC/H.264 frame boundary detection AUD

• Access Unit Delimiter (AUD) NAL Unit is used to
  indicate the start of a new video frame
• AUD is an optional, and it may not appear in the
  bitstream at all.

10
AVC/H.264 frame boundary detection

• If any of these conditions are met, the new slice
  belongs to a new frame otherwise, both the last
  and new slices are part of the same frame.
• 1. lastSlicepic_parameter_set_id !
  newSlicepic_parameter_set_id
• 2. lastSliceframe_num ! newSliceframe_num
• 3. lastSlicefield_pic_flag !
  newSlicefield_pic_flag
• 4. lastSlicefield_pic_flag 1
  newSlicefield_pic_flag 1
  lastSlicebottom_field_flag !
  newSlicebottom_field_flag
• 5. lastSlicenal_ref_idc ! newSlicenal_ref_idc
  and (lastSlicenal_ref_idc 0
  newSlicenal_ref_idc 0)
• 6. (lastSlicem_nalUnitType
  AVCNAL_SLICE_IDR) ! (newSlicem_nalUnitType
  AVCNAL_SLICE_IDR)

11
AVC/H.264 frame boundary detection

• If any of these conditions are met, the new slice
  belongs to a new frame otherwise, both the last
  and new slices are part of the same frame.
• 7. (lastSlicem_nalUnitType
  AVCNAL_SLICE_IDR) (newSlicem_nalUnitType
  AVCNAL_SLICE_IDR) (lastSliceidr_pic_id
  ! newSliceidr_pic_id)
• For cases when SPSpic_order_cnt_type 0
• 8.1. lastSlicepic_order_cnt_lsb !
  newSlicepic_order_cnt_lsb
• 8.2. lastSlicedelta_pic_order_cnt_bottom !
  newSlicedelta_pic_order_cnt_bottom
• For cases when SPSpic_order_cnt_type 1
• 9.1. lastSlicedelta_pic_order_cnt0 !
  newSlicedelta_pic_order_cnt0
• 9.2. lastSlicedelta_pic_order_cnt1 !
  newSlicedelta_pic_order_cnt1

12
AV1 video analysis with Virinext Bitstream
Analyzer

• Screenshot of frame preview, syntax viewer and
  GOP widgets of AVC video file opened in Virinext
  Bitstream Analyzer

13
References

• Virinext Bitsteam Analyzer - GUI tool for
  displaying and analyzing coded video and audio
  streams
• Detecting video frames in AVC/H.264 compressed
  bitstreams
• The structure of AVC (H264) video
• H.264 Advanced video coding for generic
  audiovisual services
• Advanced Video Coding