Multimodal

1
Video Summarization
Video Highlights Extraction
Video Browsing
Video Retrieval
Video Representation
Multimodal Analysis
2
Video Summarization
Highlights based Summarization
Video Browsing
Table of Contents based Summarization
Top-down
Multimodal Analysis
Video Representation
Video Content
Bottom-up
Video Retrieval
3
Highlight Groups
Grouping
Highlight Candidates
Audio-visual markers association
A highlight
Audio-Visual Markers
Visual Marker
Audio Marker
Key audio-visual object detection
Play / Break
Play
Break
Feature extraction segmentation
Video with Audio Track
4
Browsing
Highlights based Summarization
ToC based Summarization
Retrieval
ToC
Highlights
Index
Highlight Groups
Scenes
Visual
Groups
Semantic
Highlight Candidates
Shots
Audio
Audio-Visual Markers
Key frames
Camera motion
Play/Break
5
(No Transcript)
6


7
Audio Class Recognition (GMM)
Classify the input audio into one of 5 Audio
Markers using GMM
Input Audio
Applause
Feature Extraction
Comparing Likelihoods
Cheering
Music
MFCC Coefficients.
Speech
Excited Speech
Training Audio Clips
Audio Class
Feature Extraction
Training the GMM Classifiers using MFCC features
and BIC.
8
Audio Class Recognition (GMM)
Classify the input audio into one of 2 Audio
Markers using GMM
Input Audio
Excited Speech
Feature Extraction
Comparing Likelihoods
Other (Applause,Cheering, Music, Speech)
MFCC Coefficients.
Task Sports Highlights
Training Audio Clips
Audio Class
Feature Extraction
Training the GMM Classifiers using MFCC features
and CV
9
(No Transcript)
10
Audio Class Recognition (GMM)
Classify the input audio into one of 5 Audio
Markers using GMM
Input Audio
Applause
Feature Extraction
Comparing Likelihoods
Cheering
Music
MFCC Coefficients.
Speech
Excited Speech
Training Audio Clips
Audio Class
Feature Extraction
Training the GMM Classifiers using MFCC features
and BIC.
11
Importance Level
MDCTs
Input Audio
Class Label
Importance Level Calculation
Feature Extraction
Audio Classifier
Task
12
Generic Audio Classification
Applause
Compare Likelihoods
Cheering
Class Label
Music
MDCTs
Speech
Excited Speech
Training Audio Clips
Feature Extraction
Training Data for GMMs
13
Importance Level
MDCTs
Input Audio
Class Label
Importance Level Calculation
Feature Extraction
Audio Classifier
Task
14
Task Specific Audio Classification
Compare Likelihoods
Excited Speech
Other (Applause,Cheering, Music, Speech)
Class Label
MDCTs
Training Audio Clips
Feature Extraction
Task Sports Highlights
15
(No Transcript)
16
(No Transcript)