Video search using a visual dictionary

1
Video search using a visual dictionary

• Emilie DUMONT and Bernard MERIALDO
• dumont, merialdo_at_eurecom.fr
• Institut Eurécom
• Département Communications Multimédia
• Sophia-Antipolis - FRANCE

2
Multimedia Information Retrieval

• Information retrieval looks easy
• Google image Dog

3
Multimedia Information Retrieval

• Information retrieval looks easy
• Google image Dog

? How we can search a shot without
text-annotation?
4
Generalization of text-document retrieval

• Text-document is represented by a vector of words
• Each entry represents the importance of a word
• A user types words to compose a query
• The system returns a ranked list of
  text-documents

5
Generalization of text-document retrieval

• Text-document is represented by a vector of words
• Each entry represents the importance of a word
• A user types words to compose a query
• The system returns a ranked list of
  text-documents
• ? Generalization in the image and video domain
• Video-document is represented by a vector of
  visual element
• Each entry represents the importance of a visual
  element
• A user types visual elements to compose a query
• The system returns a ranked list of
  video-documents

6
Visual dictionary

• Visual elements represent semantic concepts
• Each image is divided into blocks, call visual
  elements
• Blocks are repesented by two vectors color and
  texture
• Construction of Global Visual Dictionary (GVD)
• By clustering blocks using K-Means
• By selecting medoids of clusters
• Construction of Query Visual Dictionary (QVD)
• By selecting the most discriminative elements

7
Artificial Search for Video Shot (ASVS)

• Optimize the visual dictionary for an artificial
  search task with automatic evaluation

Query Visual Dictionary
rank r
query
result list
keyframe
8
Experimental protocol

• Dataset BBC Rushes task of TrecVid06
• 40 hours of video
• Several parameters
• Different block sizes 1210, 108, 85, 54,
  42
• Different sizes of Global Visual Dictionary 25
  to 1500
• Different sizes of Query Visual Dictionary 25,
  50, 75, 100, 200
• Different number of block query 1 to 10
• Different features

9
Global Visual Dictionary size
10
Query Visual Dictionary size
11
Conclusion and future work

• Visual dictionary indexes shots easily and
  rapidly.
• Evaluation is automatic.
• Searching a shot is efficient.
• Relevance of visual elements

12
Thank you for your attention.
Multimedia Information Group of Eurécom Institute