Performance Characterization of Video-Shot-Change Detection Methods

1
Performance Characterization of Video-Shot-Change
Detection Methods

• U. Gargi, R. Kasturi, S. Strayer
• Presented by Isaac Gerg

2
What is a Shot?

• The process of identifying changes in the scene
  content of a video sequence so that alternate
  representations may be derived for the purposes
  of browsing and retrieval. Quoted directly
• Shot A sequence of frames shot from the same
  camera.
• Shot-Change examples cuts, transitions, wipes,
  etc.

3
Why Do We Care?

• Indexing video retrieval
• Compression (e.g. MPEG) determining key frames.
• Removing commercials! (TiVo)

4
Preview

• Create a method for measuring the performance of
  a shot-change algorithm.
• Measure both false detections missed
  detections.
• Measure performance of both cut detection
  gradual transition detection.
• Apply shot-change algorithms to ground truth
  video sequence.
• Perform measurements and throughput analysis.
• Compare the results.

5
Ground Truth Video Sequence

• 640x480 _at_ 30 frames/s.
• 75 minutes in length
• M-JPEG format
• Human volunteers used to establish ground truth.
• Custom software used to notate shot-change.

6
Defining a Detection

• Algorithm detection must occur within so many
  frames of ground truth detection.
• Mapping Range RM
• Cut changes RM 3Gradual Transition RM10

7
Detection Performance Measurements

• Where MD is Missed Detections FA is False
  Alarms.

8
Desirable Characteristics

• 90-95 recall with 70-75 precision.
• Robust.
• Automatic thresholds.
• High throughput.
• Perform well on both cuts and gradual
  transitions.

9
Algorithms Evaluated

• Color Histograms
• RGB, HSV, YIQ, XYZ, Lab, Luv, Munsell,
  Opponent
• Frame Difference Measurements
• Bin-to-bin Differences (B2B), Chi-square test,
  Histogram intersection, Average Color
• Dimensionality 1D, 2D, 3D
• MPEG Algorithms A, B, C, D, E, F
• Block Matching Methods A, B, C

10
Best Methods - Cut

• Histogram intersection
• 1D and 3D methods.

11
Best Methods - Cut

• MTM colorspace (many flops).
• LAB appeared as good compromise when considering
  throughput.
• Opponent (OPP) almost as good as LAB, but needs
  only integer computations.

image Hall, E. L. . Computer Image Processing
and Recognition. Academic Press, New York
12
Best Methods - Cut

• Best recall MPEG-A, 97 with 6 precision.
  Uses only I frames.
• Best precision MPEG-D, 88 with 79 recall.
  Uses I, B, P frames.

13
Worst Methods - Cut

• Chi-square test histogram difference
• Average color of a frame.
• 2D methods.Indicates luminance is important.
• YYY colorspace. Indicates color content is
  important
• All the block-matching methods.

14
Best Methods - Transition

• Only MPEG algorithms evaluated.
• MPEG-D Uses all frames (I, P, B). Uses
  multiframe differences to detect gradual
  transitions. Uses 11 parameters.
• MPEG-F Uses color information (Y, Cr, Cb). Uses
  order statistics to detect gradual transitions.
  Needs 7 parameters.

15
Worst Methods - Transition

• MPEG-A was the worst. Only contains I frames.
• Most performed poorly as they expected a
  particular transition curve.

16
MPEG - Source Effects

• Desirable to have a good MPEG method independent
  of encoder.
• Authors found dependence on algorithm performance
  and MPEG encoder used.
• MPEG does not specify encoding method, only
  syntax of encoded bitstream.
• Different error estimates or DCT matrices may be
  used during encoding.
• MPEG-F appeared to be the most robust.

17
Conclusions

• Need accurate model of color.
• Color luminance information combined yield best
  results.
• MPEG shot detection gradual transition methods
  have a long way to go. Encoding too variable.
• Gradual transitions not detected well by an of
  the MPEG methods.

18
Questions?