Multimedia Search and Retrieval

1
Multimedia Search and Retrieval
2
Agenda

• Introduction
• Browsing the Structure Hierarchy
• Query by Example
• Semantic-Level Classification
• Meta Search Engines

3
Introduction

• Applications
• Large-scale Multimedia Search Engines
• Media Asset Management Systems
• Audio-Visual Broadcast Servers
• Personal Media Servers (for Consumers)
• Different Requirements
• Browsing
• Specific Search

4
Agenda

• Introduction
• Browsing the Structure Hierarchy
• Query by Example
• Semantic-Level Classification
• Meta Search Engines

5
Structure Hierarchy

• Segmenting
• Mostly based on Shot extraction
• Grouping
• Generating a table of contents
• Extracting story and summary

6
News Program Browser
Example Broadcast News Program
7
Representation Browsing
Extracted Semantic Structures
8
Representation Browsing (2)
Stories about an event
9
Representation Browsing (3)
News Summary of the day
10
Agenda

• Introduction
• Browsing the Structure Hierarchy
• Query by Example
• Semantic-Level Classification
• Meta Search Engines

11
Query by Example

• Based on similarity between a query video and the
  target video
• Index to Salient Objects
• Search based on
• Example, Feature, and Sketch
• Problems
• Difficulties in finding an example
• Various Distance functions

12
VideoQ Search System
13
Agenda

• Introduction
• Browsing the Structure Hierarchy
• Query by Example
• Semantic-Level Classification
• Meta Search Engines

14
Semantic-Level Content Classification

• Bridge the gap between low-level features and
  high-level semantics
• Two approaches for Automatic Classification
• Content Modeling Using Probabilistic Graphic
  Models
• Indexing with Semantic Templates

15
Content Modeling

• Multijects Probabilistic Multimedia Objects
• Semantic Label
• Summarizing a time-sequence of low-level features
• Representing the probability
• Three categories
• Objects, Events, and Sites

16
Content Modeling (2)

• Multinet Multiject Network
• Represent higher-level probabilistic dependencies
  between multijects

17
Content Modeling (3)

• Building a Multiject
• Extracting features
• Modeling Modalities with a HMM
• Combining the features from multiple media
• Hierarchical Hidden Markov Model
• Event-Coupled Hidden Markov Model

18
Hidden Markov Model
19
Hierarchical HMM

• Independent HMMs are trained for audio and video
  features
• Building a Supervisor HMM
• Supervisor HMM encodes correlation of two
  modalities

20
Event-coupled HMM

• Models temporal constraints explicitly
• Models Time difference of the onset of events
• tA, tV Times that the event begins in audio and
  video

21
Semantic Templates

• The gap between the content-based retrieval
  results and the user need
• Non-intuitive query procedures
• Features devoid of semantics
• Principles
• Two-way Learning
• Intuitive Models
• Synthesizing New Concepts

22
Semantic Templates (2)

• A set of icons or example scene/objects that
  represent the associated semantic
• Represented by their feature vectors

23
Semantic Templates (3)

• Generating Templates
• User templates
• Objects with spatio-temporal constraints
• Object features and their weigth
• A high dimensional global space
• Enlarging the query point to a boundary space
• Mapping all the videos in to this global space

24
Semantic Templates (4)
25
Agenda

• Introduction
• Browsing the Structure Hierarchy
• Query by Example
• Semantic-Level Classification
• Meta Search Engines

26
Meta Search Engines

• Links users transparently to multiple search
  engines based on their performance on different
  types of queries
• Three main components
• Query dispatcher
• Query translator
• Display Interface
• Benefits from MPEG-7

27
References

• S.F. Chang, Q. Huang, T. Huang, A. Puri, B.
  Shahraray, Multimedia Search and Retrieval, in
  Multimedia Systems, Standards, and Networks, A.
  Puri and T. Chen (eds.). New York Marcel Dekker,
  559-584, 2000.
• T.T. Kristjansson, B.J. Frey, and T.S. Huang,
  Event-coupled Hidden Markov Models, submitted
  to Advanced in Neural Information Processing
  Systems, 1998.
• M.R. Naphade, T.T. Kristjansson, B.J. Frey, and
  T.S. Huang, Probabilistic Multimedia Objects
  (Multijects) A Novel Approach to Video Indexing
  and Retrieval in Multimedia Systems, IEEE
  International Conference on Image Processing,
  Oct. 1998, Chicago, IL.
• S.-F. Chang, W. Chen, and H. Sundaram, "Semantic
  Visual Templates - Linking Visual Features to
  Semantics," IEEE Intern. Conference on Image
  Processing, Chicago IL, October 1998.
• L. R. Rabiner, A Tutorial on Hidden Markov
  Models and Selected Applications in Speech
  Recognition, Proc. of IEEE, Vol. 77, Feb 1989

28

• Presented for
• Multimedia Systems Course
• Prof. Ze-Nian Li
• School of Computing Science
• Simon Fraser University
• July 2002

Most of the pictures or their basic ideas are
taken from the referenced papers.