Video summarization by video structure analysis and graph optimization

1
Video summarization by video structure analysis
and graph optimization

• M. Phil 2nd Term Presentation
• Lu Shi
• Dec 5, 2003

2
Outline

• Motivation
• Video structure
• Video skim length distribution
• Spatial-temporal graph modeling
• Optimization based video shot selection
• Experimental results

3
Motivation

• Huge volume of video data are distributed over
  the Web
• Browsing and management in the huge video
  database are time consuming
• Help the user to quickly grasp the content of a
  video
• Two kinds of applications
• Video skimming (dynamic)
• Video static summary (static)

4
Goals

• Conciseness
• Content coverage
• Spatial and temporal
• Coherency
• Not too jumpy

5
Flowchart
6
Video structure

• Video narrates a story just like an article does
• Video (story)
• Video scenes (paragraph)
• Video shot groups
• Video shots (sentence)
• Video frames

7
Video structure

• Graphical example

8
Video structure

• Can be built up in a bottom-up manner
• Video shot detection
• Video shot grouping
• Video scene formation

9
Video structure

• Video shot detection
• Video slice image 1
• Column - pairwise distance
• Filtering and thresholding
• 
  

10
Video structure

• Video shot grouping
• Window-sweeping algorithm 2
• Spatial similarity
• Temporal distance
• Intersected video shot groups form loop scenes

11
Video structure

• Summarize each video scene respectively
• Loop scenes and progressive scenes
• Loop scenes depict an event happened at a place
• Progressive scenes transition between events
  or dynamic events

12
Video structure

• Scene importance length and complexity
• Content entropy for loop scenes
• Measure the complexity for a loop scene

13
Video structure

• Determine each video scenes target skim length
• Determine each progressive scenes skim length
• If , discard it, else
  
• Determine each loop scenes skim length
• If
  ,discard it
• Redistribute to remaining scenes

14
Graph modeling

• Spatial-temporal dissimilarity function
• Linear with visual dissimilarity
• Exponential with temporal distance

15
Graph modeling

• The spatial temporal relation graph
• Each vertex corresponds to a video shot
• Each edge corresponds to the dissimilarity
  function between shots
• Directional and complete

16
Skim generation

• The goal of video summarization
• Conciseness given the target skim length
• Content coverage
• The spatial temporal dissimilarity function
• The spatial temporal relation graph
• A path corresponds to a series of video shots
• Vertex weight summation
• Path length is the summation of the dissimilarity
  between consecutive shot pairs

17
Skim generation

• Objectives
• Search for a path in the graph such that
• Maximize the path length (dissimilarity
  summation)
• Vertex weight summation should be close to
  but not exceed it
• The objective function

18
Skim generation

• Global optimal solution
• Let denote the paths begin with ,
  whose vertex weight summation is upper bounded by
• The optimal path is denoted by
• The target is

19
Skim generation

• Optimal substructure
• Dynamic programming
• Effective way to compute the global optimal
  solution
• Trace back to find the optimal path
• Time complexity , space complexity

20
Experiments

• Key frames of selected video shots

21
Experiments

• There is no ground truth so that it is hard to
  objectively evaluate a video skim
• Subjective experiment
• Parameters

22
Conclusion

• Video structure analysis
• Scene boundaries, sub-skim length determination
• Graph scene modeling
• Optimization based sub skim generation
• Generate a video skim

23
Reference

• 1 C. W. Ngo, Analysis of spatial temporal
  slices for video content representation, Ph. D
  thesis, HKUST, Aug.2000
• 2 Y. Rui, T.S. Huang, and S. Mehrotra,
  Constructing table-of content for videos, ACM
  Multimedia Systems Journal, Special Issue
  Multimedia Systems on Video Libraries, vol. 7,
  no.5, pp. 359368, Sept 1999.

24
Q A

• Thank you!!