MPEG7 Video Retrieval using Bayesian Networks

1
MPEG-7 Video Retrieval using Bayesian Networks

• Luis M. de Campos
• Juan M. Fernández-Luna
• Juan F. Guadix

Departamento de Ciencias de la Computación e
Inteligencia Artificial E.T.S.I.
Informática Universidad de Granada
2
Introduction

• Brief overview about our work on the design of a
  
• search engine based on Bayesian Networks
• to retrieve MPEG-7 videos
• using their text annotations.

3
Overview

• Preliminaries introduction to
• Information Retrieval
• MPEG-7 standard
• Bayesian Networks
• MPEG-7 video retrieval models based on Bayesian
  networks.

4
Preliminaries (I) Information Retrieval

• Information Retrieval is concerned with the
  representation, storage, organisation and
  accessing of information items.
• Indexing Querying Retrieval

5
Preliminaries (II) MPEG-7

• Multimedia Content Description Interface
• Standard to describe multimedia content using
  metadata
• The content of a multimedia file concepts,
  objects in movement, who is speaking, ...
• Aspects related to the management of the content,
  i.e., duration, structure, format and size of the
  file, number of frames per shot,...
• Tools
• Descriptors, Schemes, Data Definition Language.

6
Preliminaries (III) MPEG-7

• Descriptors elements, data representation.
• (Time to represent a duration, histogram to
  represent a colour or a string to represent a
  title)
• Schemes structure and semantic of the
  relationships among elements.
• (A film divided into scenes and shots, including
  textual description in the scene level and
  description about colour, movement and audio
  amplitude in the shot level)
• DDL (Data Definition Language) Language to
  extend or modify the previous set of tools. It is
  a variety of XML Schema. Therefore, descriptions
  files are XML files.

7
Preliminaries (IV) MPEG-7




T0000000F3000
0
PT16M33S11772N30000F
overlap"false"


T0000000F30000
PT3S22
112N30000F


A tv presenter is reporting
information about a meeting of the security
council in UN.



T00
000322112F30000
PT9S18288N30000Fn

Collin Powell is
speaking about the USA position in the Iraq
crisis.




8
Preliminaries (V) MPEG-7
From the point of view of IR, the structure of a
video is seen conceptually
Vídeo
Scene 1
Scene 2
Scene 3
Shot 1
Shot 2
Shot 3
Shot 4
Shot 5
Shot 6
Frame
9
Preliminaries (VI) Bayesian networks

• Graphical models able to represent and
  efficiently manipulate n-dimensional probability
  distributions.
• The knowledge obtained from a problem is encoded
  in a Belief network by means of the quantitative
  and qualitative componets

10
Preliminaries (VII) Bayesian networks

• Qualitative part Directed Acyclic Graph G(V,E)
• V (Nodes) ? Random variables, and
• E (Arcs) ? (In)dependence relationships.

• Quantitative partA set of conditional
  distributions
• Drawn from the graph structure,
• representing the strength of the relationships,
• stored in each node.

11
MPEG-7 Video Retrieval Models based on Bayesian
Networks (I)

• Taking advantage of the structure of an MPEG-7
  video
• Video, Scenes, Shots, Frames
• And of free text annotation tags in the .xml file

12
MPEG-7 Video Retrieval Models based on Bayesian
Networks (II)
V
13
MPEG-7 Video Retrieval Models based on Bayesian
Networks (III)

• Assesment of probability distributions
• Prior probability in term nodes p(ti)1/M.
• Probability distributions in the rest of nodes
  P(U pa(U)).
• Problem Great number of parents.
• Solution Probability functions.

14
MPEG-7 Video Retrieval Models based on Bayesian
Networks (IV)

• Query term instantiation.
• Run a propagation algorithm p(u Q),?U.
• Generate a ranking.
• Problem
• Great number of nodes in the graph.
• Complex topology.
• Solution
• Evaluation of probability functions in each
  layer.

15
MPEG-7 Video Retrieval Models based on Bayesian
Networks (V)
In shots
In Scenes and Videos
where vij and wij
Exact propagation
16
MPEG-7 Video Retrieval Models based on Bayesian
Networks (VI)

• Once a relevance probability has been assigned to
  each unit,
• Which units are offered to the user?
• Those which present an accurate context, wider
  enaugh to be a good response to the query.
• How?
• Transforming the Bayesian Network into a
• Influence Diagram

17
MPEG-7 Video Retrieval Models based on Bayesian
Networks (VII)
Sh4
Sh5
Sh6
D1
D3
D2
S4
S3
D5
D4
V2
18
MPEG-7 Video Retrieval Models based on Bayesian
Networks (VIII)

• Integrated Tool
• Video capture from tv.
• Automatic annotations form subtitles.
• Manual annotations based on ontologies.
• Querying and obtaining the best units.
• Automatic generation of a video with the results.

19
MPEG-7 Video Retrieval Models based on Bayesian
Networks (IX)
Lalmas and Graves model
MediaFormatDS
20
The end...

• Thank you very much