MINMAX Optimal Video Summarization with Frame Skip Constraint

1
MINMAX Optimal Video Summarization with Frame
Skip Constraint

• 1,2Zhu Li
• 3Guido Schuster
• 1Aggelos K. Katsaggelos
• 2Bhavan Gandhi
• 1Department of ECE, Northwestern University,
  Evanston, USA
• 2Motorola Labs, Schaumburg, USA
• 3Hochschule fur Technik Rapperswil (HSR),
  Switzerland

2
Outline
Introduction Definitions and Assumptions Formulati
ons Optimal and Heuristic Solutions Solution to
the Dual Problem Frame Distortion
Metric Simulation Results
3
Introduction

• Why Video Summary
• View time constraint, a shorter version is more
  desirable in some applications, for eg, security,
  mil and entertainment apps
• Storage, Bandwidth and Energy Constraint, a
  shorter version with better SNR quality conveys
  more useful information.

• Solution
• Video Shot Segmentation,
• Key frames selection within a video shot
• Previous works clustering on visual features,
  curve simplification, utility maximization, etc.

4
An application scenario
2G/2.5G data channel 6kpbs 15kpbs

• Operating at voice rate
• Reasonable visual quality
• Synchronized with audio if so wish

5
Definitions and assumptions
6
Rate-Distortion Formulation

• Summarization as a rate-distortion optimization
  problem
• MDOS formulation
• MROS formulation
• Frame skip constrained

7
Dynamic Programming Solution

• Dynamic Programming
• Exhaustive search not practical.
• Segment distortion state and rate definitions

MROS summary frame selections l0, l1, lm-1,
l00.
8
The Algorithm

• MROS algorithm
• The recursion
• The initial condition

9
The DP Trellis
MROS algorithm
n6, no skip constraint
n6, max skip3
10
An Example Solution

• Start from R10
• Dmax and Kmax Constrained state transition
• Stop when the final virtual frame fn is reached.
• Multiple optimal solutions
• Sf0, f4, f7, f0, f4, f6 f0, f2, f5

f r a m e k
epoch t
11
Frame Distortion Metric

• An elusive problem
• Application specific
• PCA analysis to find the subspace spanned by a
  large set of video frames
• Weighted Euclidean distance in PCA space as
  frame distortion metric
• Works well with subjective perception.

12
Simulation results
MROS foreman sequence, frames 150299 n150,
Dmax 6.4, Kmax no constraint Results m25
13
Simulation results skip constrained
MROS foreman sequence, frames 150299 n150,
Dmax 6.4, Kmax 10 Results m32.
14
A Heuristic Solution
Distortion Constrained Skip (DCS) algorithm Set
L0, add fL to the summary S FOR k1 TO n IF
d(fL, fk) gt Dmax Lk, add fL to the
summary S END END
DCS is the optimal solution if
15
Solution to the MDOS formulation
Bi-Section searching on the operational R-D
function

• The ORD is non-increasing (lemma 1.)
• Bi-section search on the distortion, and solve
  for each distortion with MROS solution.
• Will converge to the optimal solution D.

16
Conclusion and Future Work

• The solution is rate-distortion optimal
• The heuristic DCS algorithm is near optimal most
  of time and quite efficient
• Summaries operates at voice rate suitable for 2G
  and 2.5G deployment (demo)
• bond sequence at 13.2kpbs, Dmax6.0
• bond sequence at 11.7kpbs, Dmax8.0
• bond sequence at 9.7kpbs, Dmax12.0
• foreman sequence at 10.8kpbs, Dmax 6.0
• foreman sequence at 9.4kpbs, Dmax8.0
• foreman sequence at 8.4kpbs, Dmax12.0
• Future work bit constrained MINMAX summarization.

17
Questions ?
? .
?brigado !