The MPEG-7 Standard - A Brief Tutorial -

1
The MPEG-7 Standard- A Brief Tutorial -

• Ali Tabatabai
• Sony US Research Laboratories
• February 27, 2001

2
Outline

• Objectives of the MPEG-7 Standard
• Main Elements of MPEG-7
• Scope of MPEG-7
• MPEG-7 Application Areas
• MPEG 7s relation with other standards

3

• Why do we need MPEG-7 ?

• Support for Advanced Query

• Visual
• Audio
• Sketch

• N e e d

4
MPEG A Brief History (1)

• MPEG Moving Picture Experts Group

• ISO / IEC/JTC1/SC29/WG11

5
MPEG A Brief History (2)

• MPEG-1 Interactive CD and MP3 11 / 1992
• MPEG-2 DTV, STB, DVD 11 / 1994
• MPEG-4 Web and Mobility ver1 09
  /1998 ver2 11 /1999
• MPEG-7 ??? 08 / 2001
• MPEG-21 Multimedia Framework 11 / 2001

6
MPEG-7 What Is It ?
THE MPEG 7 STANDARD

• Types of Audio Visual Information
• Audio, Speech
• Moving video, still pictures, graphics
• Information on how objects are combined in
  scenes

7
MPEG-7 Application Areas

• Storage and retrieval of audiovisual databases
  (image, film, radio archives)
• Broadcast media selection (radio, TV programs)
• Surveillance (traffic control, surface
  transportation, production chains)
• E-commerce and Tele-shopping (searching for
  clothes / patterns)
• Remote sensing (cartography, ecology, natural
  resources management)
• Entertainment (searching for a game, for a
  karaoke)
• Cultural services (museums, art galleries)
• Journalism (searching for events, persons)
• Personalized news service on Internet (push media
  filtering)
• Intelligent multimedia presentations
• Educational applications
• Bio-medical applications

8
MPEG-7Description Scope for AV Content

• Description Granularity
• Low-level
• High-level
• Form
• Access
• Classification
• Link
• Context

9
MPEG-7 Main Elements

• Descriptors (D)
• syntax and semantics of each feature
  representation
• Description Schemes (DS)
• structure and semantics of the relationships
  between components
• Description Definition Language (DDL)
• creation of new DSs
• modification/extension of existing DSs

10
MPEG-7 Major Functionalities

• Systems (ISO / IEC 15938 - 1)
• Description Definition Language (ISO / IEC 15938
  - 2)
• Visual (ISO / IEC 15938 - 3)
• Audio (ISO / IEC 15938 - 4)
• Multimedia Description Schemes (ISO / IEC 15938 -
  5)
• Reference Software (ISO / IEC 15938 - 6)

11

• MPEG-7 Main Elements (2)

• DS1

• DS3

12
MPEG-7 Systems

• It defines tools to

• provide for efficient storage and transport

• synchronize between content and description

• manage and protect intellectual property

13
MPEG-7 DDL and its Components

• Description Definition Language
• Creation of the Ds and DSs XML Schema MPEG-7
  Extensions
• Instantiation of XML
• XML Schema
• Data types
• Simple and Complex types
• Elements, attributes
• Inheritance, Abstract types
• MPEG-7 extensions
• Array and Matrix data type

14
MPEG-7 Audio

• Sound Effects
• Music Instrument Timbre
• Spoken Content
• Melody Contour

15
MPEG-7 Visual (1)

• Color
• quantization, dominant, scalable,
  color-structure, layout, GoF/GoP
• Texture
• Shape
• region-based, contour-based, 3D
• Motion
• camera motion, motion trajectory, parametric
  motion, motion activity

16
MPEG-7 Visual (2)

• Localization
• spatio temporal
• Others
• face recognition

17
MPEG-7 Basic Visual Structures

• Grid Layout
• 2D-3D Multiple View
• Time Series
• Spatial 2D Coordinates
• Temporal Interpolation

18
Low level Audio Visual descriptors

• Color
• Shape
• Position
• Texture

• Color
• Camera motion
• Motion activity
• Mosaic

• Color
• Motion trajectory
• Parametric motion
• Spatio-temporal shape

• Spoken content
• Spectral characterization
• Music timbre, melody

19
MPEG-7 MMDS Basic Elements
Root, Top-level elements, Packages
Time, Duration, Medialocators
Language Annotation,Person, Place
20
MPEG 7 Content Management and Description
Title, Creator, Creation location date,
Purpose, Classification, Genre, Review, Parental
guidance, etc. (Author generated)
Format, Coding, Instances, Identification,
Transcoding Hint, etc. (Several instances)
Rights holder, Access rights, Usage Record,
Financial aspects, etc. (Evolution)

• Viewpoint of the structure Segments
• Spatial / temporal structure
• Audio, video low-level Ds
• Elementary semantic information.

• Viewpoint of conceptual notions
• Events, objects, abstract concepts, and their
  relation

21
Example of Segment trees
Foreground
Background
22
Time
Axis
Segment Tree
Shot1 Shot2
Shot3
Segment 1
Sub-segment 1
Sub-segment 2
Sub-segment 3
Sub-segment 4
segment 2
Segment 3
Segment 4
Segment 5
Segment 6
Segment 7
23
MPEG 7 Navigation and Access
Efficient support of discovery, browsing,
navigation, visualization
Substitution of the original content Adaptation
to terminal, network, or user preferences
24
MPEG 7 Hierarchical summary
25
MPEG 7 Sequential summary
26
MPEG 7 Variation
Universal Multimedia Access Adapt delivery to
network and terminal characteristics (QoS)
27
MPEG-7 Content Organization
Collection
Classification
Description and organization of collection of
documents
28
MPEG-7 Collection
29
MPEG 7 User Interaction
Collection
Classification
User Interaction
User identification and preferences Filtering,
search and browsing
30
MPEG-7 Its Relation with other standards

• AHG on Metadata harmonization
• SMPTE Metadata dictionary, KLV encoding
• Dublin Core Metadata Initiative
• European Broadcast Union
• AHG on TV AnyTime Application
• Large number of Liaisons
• SMPTE
• Dublin Core
• W3C (XML Schema)
• etc.

31
MPEG-7 TimeLine - The Work Plan

• Divergence

• Competition
• Individual work
• Definition
• scope and r

• Convergence

2001
2000
1999
1998
1996
Call for proposals
Working draft
Committee draft
International standard
Draft international standard
Final committee draft
32
Conclusions on AV Content Description and MPEG-7

• MPEG-7
• AV content description for interoperable
  application
• Description Definition Language
• XML Schema (flexibility) Binary version
  (efficiency)
• Description Schemes
• Library of description tools
• Covers a wide range of generic needs