Video summarization by graph optimization

1
Video summarization by graph optimization
Lu Shi Oct. 7, 2003
2
Outline

• Introduction
• Goals
• Stage I Candidate video shot selection
• Video segmentation
• Video feature detection
• Candidate video shots
• Stage II Graph based video summary generation
• Dissimilarity function
• Spatial-temporal relation graph
• Optimization
• Experiments and Results
• Conclusion Future Work

3
Introduction

• Motivation
• Huge volume of video data are distributed over
  the Web
  
• How to help the user to grasp the content of the
  video quickly
  
• When the bandwidth is narrow, how to present the
  video to the user
  
• Applications
• Video skimming (dynamic)
• Static story board (static)

4
Goals

• Criterion for video summary
• Conciseness.
• The video skimming should not exceed the given
  target length
  
• Comprehensive coverage
• Both the visual diversity and temporal
  distribution of the original video should be
  covered.
  
• Visual coherence.
• The video skimming should not be too jumpy

5
Stage I Candidate shot selection

• Video segmentation
• A video shot is an unbroken sequence of images
  recorded continuously by a camera.
  
• The content of a video shot can be represented by
  key frames(e.g first and last)
  
• A video sequence is formed by a series of video
  shots
  
• Video shots can be detected by various video
  segmentation methods.

6
Stage I Candidate shot selection

• Video segmentation
• Middle slice image (Concatenated by video frame
  center lines)
  
• Calculate minimal pixel difference between rows
• Filtering and thresholding

7
Stage I Candidate shot selection

• Video feature detection
• Face detection
• Voice, noise detection
• Audio volume
• Specific color (fire,etc)
• Text caption
• Features indicate interesting content that should
  be considered putting into the summary
  

8
Stage I Candidate shot selection

• Select candidate shots
• With interesting features extracted
• Any combination of extracted features
• Adjacent candidate shots can be merged into
  video shot clusters to increase the visual
  coherence

9
Stage II Graph modeling

• Video shot pairwise dissimilarity function
• Visual(spatial) similarity Histogram
  correlation between key frames
  
• Temporal distance the distance between shot
  center points
  
• Definition

10
Stage II Graph modeling

• Video shot pairwise dissimilarity function
• Linear with visual dissimilarity
• Exponential with temporal distance to
  approximate the users memory (k 400 in the
  experiment)
  
• Definition
• Similar definition for video clusters

11
Stage II Graph modeling

• Video shot cluster pairwise dissimilarity
  function
  
• Between one video shot and one video shot
  cluster
  
• Between two shot clusters

12
Stage II Graph modeling

• Model the candidate shot set as a directional
  graph
  
• G(V,E), conveys both the spatial and the temporal
  property of the video
  
• A vertex vi corresponds to a video shot, the
  weight on the vertex is the shots length
  
• An edge eij corresponds to the dissimilarity
  between video shot i and shot j
  

13
Stage II Graph modeling

• The real shot/cluster pairwise dissimilarity
  function
  

14
Stage II Graph based video summary generation

• Video skimming generation
• Given a target video skimming length
  SummaryLength
  
• A path in the spatial-temporal relation graph
  corresponds to a set of video shots
  
• The object function is the length of the path
• Find the longest path, with the constraint that
  the vertex weight summation of the path is within
  Summarylength-threshold, SummaryLength
  

15
Stage II Graph based video summary generation

• Optimal substructure
• We denote the state as (ThisShot, LeftSize)
• The optimal substructure is
• If LeftSize is too small then opt(ThisShot,
  LeftSize) 0
  
• And then we can use dynamic programming to find
  the best solution.
  

16
Stage II Graph based video summary generation

• Dynamic programming
• Set opt(LastShot, 0..threshold) to 0
• Set opt(LastShot, threshold1SummaryLength) to
  -X
  
• Calculate the opt(ThisShot, LeftSize) with the
  optimal substructure equation, ThisShot from
  LastShot-1 to 0,
  
• Get opt(0,SummaryLength), which is the longest
  paths length. Then trace back to find the path.
  
• The time complexity
• The spatial complexity

17
Stage II Graph based video summary generation

• Video skimming generation
• The generated video skimming based on video shots
  and video shot clusters is shown below (
  SummaryLength 1500, Video Length 11479).
  

18
Stage II Graph based video summary generation

• Static video story board generation
• The static video story board is generated with
  the key frames of the skimming video shots.
  

19
Stage II Graph based video summary generation

• Evaluation
• The generated video skimming has grasped both the
  visual diversity and temporal coverage
  
• Massive subjective test not carried out yet (Does
  it make sense?)
  
• Quantitative objective evaluation is a big
  problem
  

20
Future work

• Combine with video structure
• V-Toc (Video table of contents)
• Video shot groups
• Video scenes

21
Future work

• Video structure
• Video shot group and video scene

22
Q A

• Thank you!