Outline

1
Outline

• MPEG-7 and
• MPEG-21

2
MPEG-7 ISO 15938

• Multimedia Content Description Interface

3
How Much Information?

• The worlds total yearly production of print,
  film, optical, and magnetic content would require
  roughly 1.5 billion GB (1.5EB) of storage.
• This is equivalent to 250MB per person for every
  man, woman, and child on earth.

4
Digital Information

• Increasingly, individuals produce their own
  content
• Of all information produced in the world
• 93 is stored in digital form
• HD in stand-alone PCs account for 55 of total
  storage shipped each year
• Over 80 billion photographs are taken annually
• gt400 petabytes
• gt 80 million times storage required for text

Peta 1015
5
Information Individuals

• ITEM AMOUNT TERABYTES
• Photos 80 billion images 410,000
• Home Video 1.4 billion tapes 300,000
• X-Rays 2 billion images 17,200
• Hard disks 200 million installed 13,760
• TOTAL 740,960

Tera 1012
6
MPEG Family of Standards (1)

• MPEG-1(1992) for the storage and retrieval of
  moving pictures and audio on storage media.
• MPEG-2 (1995) for digital television, the
  response for the satellite broadcasting and cable
  television industries in their transition from
  analog to digital formats.

7
MPEG Family of Standards (2)

• MPEG-4 (1998 v.1, 1999 v.2)
• First real multimedia representation standard
• Encodes content as independent objects
• Enables those objects to be manipulated
  individually or collectively on an audio visual
  scene
• Allows interactivity

8
Extension in Purpose

• MPEG-1, -2, and -4
• Make content available
• MPEG-7
• Lets you find the content you need
• MPEG-21
• Describes big picture across wide range of
  networks and devices

9
MPEG-3, -5, and 6 ???

• MPEG-3 existed to enable HDTV
• Accomplished with tools of MPEG-2
• Work item abandoned
• -1, -2, -4, -5 or -8 ???
• MPEG decided NOT to follow either logical
  expansion
• Chose number 7 instead

10
MPEG-21

• Comprehensive and flexible framework for the 21st
  Century
• Quality of Service
• Rights Management
• E-Commerce
• Efficient multimedia resource use across networks
  and devices
• Key concern is processor loading in network
  terminals

11
MPEG-7 ISO 15938
Description consumption
Description generation
Description
A standard for describing features of multimedia
content.
12
MPEG-7 ISO 15938
MPEG-7 Scope Description Schemes
(DSs) Descriptors (Ds) Language (DDL) Ref MPEG-7
Concepts
Search Engine Searching filtering Classificatio
n Manipulation Summarization Indexing
Feature Extraction Content analysis (D,
DS) Feature extraction (D, DS) Annotation tools
(DS) Authoring (DS)
13
Goals and Objectives

• Provide interoperability among systems and
  applications used in generation, management,
  distribution and consumption of audio-visual
  content descriptions.
• Help users or applications to identify, retrieve,
  or filter audiovisual information with
  descriptions of streamed or stored media.

14
MPEG-7 Context

• Audiovisual information used to be consumed
  directly by human beings
• Increasingly created, exchanged, retrieved,
  re-used by computational systems
• Representations that allow some degree of
  interpretation of the informations meaning can
  be accessed and processed by computer

15
MPEG-7 Constituent Components

• ISO/IEC 15938-1 MPEG-7 Systems
• ISO/IEC 15938-2 MPEG-7 DDL (Description
  Definition Language)
• ISO/IEC 15938-3 MPEG-7 Visual
• ISO/IEC 15938-4 MPEG-7 Audio
• ISO/IEC 15938-5 MPEG-7 MDS (Multimedia
  Description Schemes)
• ISO/IEC 15938-6 MPEG-7 Reference Software
• ISO/IEC 15938-7 MPEG-7 Conformance

16
Comprehensive AV Descriptions

• Catalog
• Title, Creator, Rights
• Semantics
• Who, what, when, where of objects and events
• Structural features of AV content
• Color of image, timbre of sound
• Leverage AV data representations
• MPEG-1, -2, -4

17
Interoperability

• Uses XML Schema for content description
• Over 100 XML industry Standard Groups
• XML Repository at www.xml.org
• Groups with similar Objectives to MPEG-7
• Society of Motion Picture and Television
  Engineers (SMPTE) Metadata Dictionary
• European Broadcasting Union (EBU) P/Meta
• Dublin Core
• Digital Imaging Group (DIG)
• TV-Anytime
• Ohio Online Computer Center / Research Libraries
  Group (OCLC/RLG)
• Similar approaches with notable divergence from
  MPEG-7

18
MPEG-7 Standardized Tools

• Enable detailed structural description
• Descriptors
• Description schemes
• Language
• Different Granularity
• Region, Image, Video Segment, Collection
• Different Areas
• Content description, management, organization,
  navigation

19
MPEG-7 Applications

• Support and facilitate
• Media portals
• Content broadcasting
• Ubiquitous multimedia
• Multimedia processing important to end user
• Multimedia processing important to providers of
  service and content

20
MPEG-7 Data Applications (1)

• Play a few notes on a keyboard and retrieve a
  list of musical pieces similar to the required
  tune, or images matching the notes in a certain
  way, e.g. in terms of emotions.
• Draw a few lines on a screen and find a set of
  images containing similar graphics, logos,
  ideograms,...
• Define objects, including color patches or
  textures and retrieve examples among which you
  select the interesting objects to compose your
  design.

21
MPEG-7 Data Applications (2)

• On a given set of multimedia objects, describe
  movements and relations between objects and so
  search for animations fulfilling the described
  temporal and spatial relations.
• Describe actions and get a list of scenarios
  containing such actions.
• Using an excerpt of Pavarottis voice, obtaining
  a list of Pavarottis records, video clips where
  Pavarotti is singing and photographic material
  portraying Pavarotti.

22
Some Application Domainswith Applications

• Digital Libraries
• Image catalog, musical dictionary, biomedical
  imaging
• Multimedia editing
• Media authoring, personal electronic news service
• Cultural Services
• History museums, art galleries
• Multimedia directory services
• Yellow pages, tourist geographical information
  services
• Broadcast media selection
• Radio channel, TV channel

23
The Ds of MPEG-7

• Audio-Visual Descriptor (D)
• Description Schemes (DSs)
• Description Definition Language (DDL)

24
Relation Between the Different MPEG-7 Elements
25
MPEG-7 Terminology Data

• Audio-visual information described using MPEG-7
  without regard to storage, coding, display,
  transmission, medium or technology
• Intended to be sufficiently broad to encompass
  graphics, still images, video, film, music,
  speech, sounds, text,

26
Data Examples

• MPEG-4 stream
• Video tape
• CD containing music
• Sound or speech
• Picture printed on paper
• Interactive multimedia installation on the web

27
MPEG-7 Terminology Feature

• Distinctive characteristic of data signifying
  something to someone
• Cannot be compared without meaningful feature
  representation (descriptor) and its instantiation
  (descriptor value)

28
Feature Examples

• Color of an image
• Pitch of a speech segment
• Rhythm of an audio segment
• Camera motion in a video
• Style of a video
• Title of a movie
• Actors in a movie

29
MPEG-7 Terminology Descriptor (D)

• Representation of a Feature
• Defines syntax and semantics of the Feature
  representation
• Allows evaluation of corresponding feature by
  means of the Descriptor Value
• Several Descriptors may represent a single
  feature by addressing different relevant
  requirements

30
Descriptor Example

• Color Feature
• Color histogram
• Average of frequency components
• Motion field
• Text of the title

31
(No Transcript)
32
Shape Descriptors

• Contour shape
• Region shape

33
Motion Descriptors
34
MPEG-7 Terminology Descriptor Value

• Instantiation of a Descriptor for a given data
  set, or subset of that data set
• Descriptor Values are combined using a
  Description Scheme to form a Description

35
Motion Activity

• Need to capture pace or Intensity of activity
• High Action chase scenes segments
• Low Action talking heads segments
• Use Gross Motion Characteristics
• avoiding object segmentation, tracking etc.

36
INTENSITY

• Expresses pace or Intensity of Action
• Uses scale of very low - low - medium - high -
  very high
• Extracted by suitably quantizing variance of
  motion vector magnitude

37
SPATIAL DISTRIBUTION

• Captures the size and number of moving regions in
  the shot on a frame by frame basis
• Enables distinction between shots with one large
  region in the middle ( e.g.,talking heads) and
  shots with multiple small moving regions
  (e.g.,aerial soccer shots)

38
TEMPORAL DISTRIBUTION

• Expresses fraction of the duration of each level
  of activity in the total duration of the shot
• Straightforward extension of the intensity of
  motion activity to the temporal dimension
• A talking head, typically exclusively low
  activity, would have zero entries for all levels
  except one

39
DIRECTION

• Expresses dominant direction if definable as one
  of a set of eight equally spaced directions
• Extracted by using averages of angle (direction)
  of each motion vector
• Useful where there is strong directional motion

40
MPEG-7 Terminology Description Scheme

• Specifies structure and semantics of
  relationships between its components
• Components may be both Descriptors and
  Description Schemes
• A Descriptor contains only basic data types,
  provided by the Description Definition Language
• A Descriptor does not refer to another Descriptor

41
Description Scheme Example

• Movie, temporally structured as scenes and shots
• Including textual descriptors at the scene level
• Including color, motion and audio descriptors at
  the shot level

42
Description Schemes in MPEG-7

• Creation and Production
• Title, creator, classification, purpose of
  creation
• Usage
• Rights holders, access rights, publication,
  financial info
• Media
• Storage format, AV content encoding, media
  identification
• Structural Aspects
• Color, texture, shape, motion, audio
• Conceptual Aspects
• AV conceptual notions
• Basic Elements
• Data types, math structures, schema tools

43
MPEG-7 Terminology Description

• Consists of a Description Scheme and the set of
  Descriptor Values (instantiations) that describe
  the Data
• The Description Scheme may not be fully
  instantiated, depending upon completeness of the
  Descriptor Values set

44
MPEG-7 Terminology Description Definition
Language (DDL)

• Language that enables creation of new Description
  Schemes and Descriptors
• Enables extension and modification of existing
  Description Schemes
• Expresses relations, object orientation,
  composition, partial instantiation

45
DDL Logical Components

• XML Schema structural language components
• XML Schema structural datatype components
• MPEG-7 specific extensions
• Datatypes for matrices and arrays
• Datatypes for time point and duration
• Data value propagation (HeaderType)

46
MPEG-7 Systems

• Specifies functionalities such as preparation of
  MPEG-7 Descriptions
• Efficient transport/storage
• Synchronization of content and description
• Development of conformant decoders
• Mechanism for providing multimedia content is
  considered part of a complete application and
  lies outside the scope of the standard

47
MPEG-7 Terminal

• Obtains MPEG-7 data from transport
• Extracts elementary streams from delivery layer
• Undo transport/storage specific
  framing/multiplexing
• Retain synchronization timing
• Forwards elementary streams of individual access
  units to compression layer
• Decodes
• Schema streams describing data structure
• Full or partial content description streams
• Generates user requested multimedia streams
• Feeds back via delivery layer for
  transmission/storage

48
MPEG-7 Terminal
49
MPEG-7 DDL

• With extensions, XML meets key requirements
• Datatype definition
• D and DS declaration
• Attribute declaration
• Typed reference
• Content model
• Inheritance/subclassing mechanism
• Abstract D and DS
• DS inclusion

50
MPEG-7 Visual

• Specifies set of standardized Ds and DSs
• Mainly address specific features
• Color, texture, motion
• Often requires other low-level Ds or support
  elements
• Structure grid layout, spatial coordinates
• Viewpoint multiple view
• Localization region locator
• Temporal time series, temporal interpolation

51
MPEG-7 VisualStandardized Descriptors

• Color
• Color Space, Color Quantization, Dominant Color,
  Scalable Color Color Layout,Color Structure,
  Group of Picture Color
• Texture
• Homogeneous Texture, Texture Browsing, Edge
  histogram
• Shape
• Region Shape, Contour Shape, Shape 3D
• Motion
• Camera Motion, Motion Trajectory, Parametric
  Motion, Motion Activity
• Face Recognition, others

52
MPEG-7 Audio

• Specifies set of standardized Ds and DSs
• Addresses four classes of audio
• Pure music, Pure speech, Pure sound effects,
  Arbitrary soundtracks
• May address audio features
• Silence, Spoken content, Timbre Sound effects,
  Melody, etc
• Often requires other low-level Descriptor
  categories
• Scalable Series ScalableSeries,
  SeriesofScalarType, etc
• Audio Description Framework AudioSampledType,
  AudioWaveformEnvelopeType

53
MPEG-7 AudioStandardized Descriptors

• Silence
• SilenceType
• Spoken content (from speech recognition)
• SpokenContentSpeakerType
• Timbre (perceptual features of instrument sounds)
• InstrumentTimbreType, HarmonicInstrumentTimbreType
  , PercussiveInstrumentTimbreType
• Sound effects
• AudioSpectrumBasisType, SoundEffectFeatureType
• Melody Contour
• CountourType, MeterType, BeatType
• Description Schemes utilizing these Descriptors
  are also defined

54
MPEG-7 Multimedia Description Schemes (MDS)

• Specifies high-level framework for generic
  descriptions of all kinds of multimedia
• Contrasts with specific descriptions addressed by
  Visual and Audio.
• Levels
• Basic elements
• Content management Content description
• Creation and production viewpoint
• Media
• Usage
• Structural Aspects,
• Conceptual Aspects

55
MDS Hierarchy Levels and Relationships
56
MPEG-7 Reference Software

• Reference implementation of relevant MPEG-7
  Standard
• Experimentation software (XM)
• Creation of of D and DSs bitstreams with
  normative syntax rather than tool performance
• Four categories of components
• DDL parses and DDL validation parser
• Visual Descriptors
• Audio Descriptors
• Multimedia Description Schemes (MDS)

57
MPEG-7 Conformance

• Guidelines and procedures for testing
  implementations for conformance

58
Possible MPEG-7 ApplicationsAbstract
Representation
59
Standard Eigenfaces

• The eigenfaces for this database were
  approximated using a principal components
  analysis on a representative sample of 128 faces.
  Recognition and matching was subsequently
  performed using the first 20 eigenvectors. In
  addition, each image was then annotated (by hand)
  as to sex, race, approximate age, facial
  expression, and other salient features. Almost
  every person has at least two images in the
  database several people have many images with
  varying expressions, headwear, facial hair, etc.

http//whitechapel.media.mit.edu/vismod/demos/face
rec/basic.html
60
Face Recognition
http//whitechapel.media.mit.edu/vismod/demos/face
rec/system.html
The system diagram above shows a fully automatic
system for detection, recognition and model-based
coding of faces for potential applications such
as video telephony, database image
compression, and automatic face recognition. The
system consists of a two-stage object detection
and alignment stage, a contrast
normalization stage, and a Karhunen-Loeve
(eigenspace) based feature extraction stage whose
output is used for both recognition and
coding. This leads to a compact representation of
the face that can be used for both recognition as
well as image compression. Good-quality
facial images are automatically generated using
approximately 100-bytes worth of encoded data.
The system has been successfully tested on a
database of nearly 2000 facial photographs from
the ARPA FERET database with a detection rate of
97. Recognition rates as high as 99 have
been obtained on a subset of the FERET database
consisting of 2 frontal views of 155 individuals.
61
Photobook http//wasi.www.media.mit.edu/people/tp
minka/photobook/

• Tool for performing queries on image databases
  based on image content.
• Works by comparing features associated with
  images, not the images themselves.
• Features are parameter values of particular
  models fitted to each image.

62
http//whitechapel.media.mit.edu/people/tpminka/ph
otobook/foureyes/seg.html
63
Texture Modeling
64
This is an example of a Photobook search based on
shape. The query image is in the upper left the
images in a small tools database are displayed in
raster scan order of similarity
65
Content Retrieval using Image as the Query
66
Movie Tool
67
References

• ICCE 2001 MPEG-7 Tutorial Session, 6/17/2001,
  Smith, Manjunath, Day
• MPEG 7 Main Page http//www.darmstadt.gmd.de/mobil
  e/MPEG7/
• IEEE Transactions on Circuit and Systems for
  Video Technology, Vol. 11, No. 6, Special Issue
  on MPEG-7
• Special Thanks to Dr. Manjunath of UCSB for
  providing a copy of his ICCE foils for use at our
  IEEE meeting

68
Practice in MPEG-21

• MPEG achievements
• MPEG-21 the open Multimedia Framework
• Contribution to MPEG-21 Digital Item Adaptation
• Resource Adaptation Engine based on the Generic
  Bitstream Syntax Description Language
• MPEG-21 based MM Communication System
• Two Internet Application Use Cases
• Reference SW and Management Tools

69
MPEG-21 Multimedia Framework ISO/IEC 21000

• Situation No complete solutions exist that allow
  different communities (content, financial,
  communications etc.), each with their own models,
  rules, procedures and content formats to interact
  efficiently.
• The multimedia content delivery chain encompasses
  different players (content creation,
  production, delivery etc.).
• To support this, the content has to be
  identified, described, managed and protected.
• Purpose MPEG-21 MM-Framework shall enable
  interoperability in this situation.

70
The basic elements of the MPEG-21 framework

• What
• A Digital Item is a structured digital object
  with a standard representation, identification,
  meta-data and resource within the MPEG-21
  framework.
• Who
• A User is any entity that interacts in the
  MPEG-21 environment or makes use of a Digital
  Item.

Transaction / Use / Relationship ? Digital Item
? ? Authorization / Value Exchange ?
User A
User B
71
The Multimedia Framework
Example Container Item Resource
Digital Item Declaration
Example Unique Identifiers
Digital Item Identification and Description
Example Storage Management
Content Management and Usage
Transaction / Use / Relationship ? Digital Item
? ? Authorization / Value Exchange ?
Event Reporting Metrics Interfaces
Event Reporting Metrics Interfaces
Intellectual Property Management and Protection
Example Encryption Authentification Watermarking
Example Resource Adaptation
Terminals and Networks
Content Representation
Example Natural and Synthetic Scalability
72
Example
The Digital Item
showing the hierarchical structure of the Digital
Item Declaration Model.
ExampleDigital Item Declaration (DID)
73
MM-Communication System Delivering a DI

• Digital Items (including their resources)
  are delivered to a variety of terminals over
  dynamic network conditions.

Mobile phone
3G terminal
Laptop/PDA
Digital Item Provider
Desktop
Users with different
access and capabilities
Broadcasting
? We need generic Management Tools that enable
users to access (get delivered) rich multimedia
content.
74
Resource Adaptation (1) - Introduction
Multimedia Communication Systems Two extreme
cases of delivery Best-effort scheduling and
worst-case reservation of resources Both of them
are not well suited to cope with large-scale,
dynamic MM-systems Adaptation the capability of
a system to dynamically change its behavior in
order to keep the QoS above a certain level
75
Resource Adaptation (1) Introduction (cont)
Where? by the various components on the delivery
path to the users servers, proxies, routers, or
the user site we need co-operative rather than
competitive components of the adaptation process
76
Resource Adaptation (2) Examples for Videos

• Temporal scaling
• Reduce number of VOPs (Video Object Plane) in the
  video resource
• Spatial scaling
• Reduce the number of pixels in an image

77
Resource Adaptation (2) Examples for Videos

• Frequency scaling
• Reduce the number of DCT coefficients
• Color Scaling
• Reduce the number of colors available
• Modality Translation
• conversion of the modality (e.g., image, text,
  audio, graphics) of the source resource

78
Resource Adaptation (3) static vs. dynamic

• Static Resource Adaptation
• M3box
• Multimedia Message Center
• Siemens CT

• Dynamic Resource Adaptation
• delivered video is susceptible to uncontrolled
  degradation
• what is required is a controlled and graceful
  quality adaptation.

MPEG-21 generalizes resource adaptation
79
MPEG-21 Digital Item Adaptation (Part 7)
CDI
CDI (Content Digital Item)
Digital Item
Digital Item
CDI
Adaptation Engine
MPEG-21 DII
Adapted
MPEG
MPEG-21 IPMP/REL
Resource
Resource
Adaptation Engine
Adaptation Engine
Digital Item
Descriptor
Descriptor
MPEG-21 DIA Descr.
Description
Description
Adaptation Engine
Adaptation Engine
Resource
DIA Tools
80
Our Contribution to MPEG-21 DIA
Generic Bistream Syntax Description Language
81
Multimedia Bitstream Syntax Description

• We introduced the gBSDL
• generic MM Bitstream Syntax Description
  Language
• It is NOT an alternative format, but an
  additional layer describing the high-level
  structure of a multimedia resource.

82
Multimedia Bitstream Syntax Description

• It is based on XML-Schema.
• It may be transmitted along with DI
• DIA Description in the DID
• gBSD may be embedded
• in a readable format
• in binary format (BiM encoded)
• It is normative.

MPEG-21 DII
MPEG
MPEG-21 IPMP/REL
Descriptor
Descriptor
MPEG-21 gBSD
Resource
CDI
83
gBSDL- Schema Hierarchy

• BSDL-1 built-in datatypes derived from
  XML-Schema
• BSDL-2 language constructs defining new types of
  constraints on XML documents for describing the
  structure of the multimedia resource
• gBS imports the Schema for BSDL-1 Extensions
  and provide the following functionalities
• codec independence
• semantic marking
• hierarchical descriptions

84
gBSD-Example for an MPEG-4 ES
Classification identifying the codec

Markers for Adaptation
Classification identifying the syntactic meaning
of the unit
Address Information
85
Management Tools 2 Use Cases
Users
Servers
Heterogeneous Network
Webserver
LAN
Stationary MM User
ActiveRouter
. . . . .
Mobile MM User
Active Cache(Proxy)
86
Use Case 1 Adaptation due to Congestions
Look-up bandwidth requirements for
coastguard.mp4, e.g., from the MPEG-7
MediaProfile DS
User Request for a CDI containing coastguard.mp4
Proxy Cache
Check outgoing bandwidth
Match outgoing bandwidth against MediaProfile DS
of the CDI
Adaptation necessary
Adaptation not necessary
Deliver CDI with coastguard.mp4
Consult XDI (Context Digital Item) for the best
Adaptation Alternative
Apply Adaptation based on gBSDL
Deliver CDI with coastguard.mp4
87
gBSDL Experiments on MPEG-4 VESs (1)
MPEG-4 Visual ES Specific BSDL Schema
DI Provider Pre-Processing
.mpg4
.xml
BintoXML
BSDLtogBSDL(XSLT)
_gbsdl.xml
Media Resource Adaptation Sequence
_adapt_01.mpg4
Remove ½ B-VOPs
Proxy First Adaptation
_gbsdl_adapt_01.xml
_adapt_02.mpg4
Remove all B-VOPs
Client Second Adaptation
_gbsdl_adapt_02.xml
88
gBSDL Experiments on MPEG-4 VESs (2)
lt?xml version"1.0"?gt ltBitstream
xmlbase"Content/coastguard.mpg4" xmlnsxsi"htt
p//www.w3.org/2001/XMLSchema-instance" xmlns"MP
EG4" xmlnsmp4"MPEG4" xsischemaLocation"MPEG4
../Schemas/MPEG4.xsd"gt ltVOgt ltStartCodegt00000100
lt/StartCodegt ltIsVOIdentifiergt0lt/IsVOIdentifiergt
ltStuffinggt0lt/Stuffinggt ltPayloadgt5-17lt/Payloadgt
lt/VOgt ltVOPgt ltStartCodegt000001B6lt/StartCodegt
ltTypegt0lt/Typegt ltStuffinggt16lt/Stuffinggt ltPaylo
adgt23-4658lt/Payloadgt lt/VOPgt lt!--... and so on
...--gt lt/Bitstreamgt
89
gBSDL Experiments on MPEG-4 VESs (3)
lt?xml version"1.0" encoding"UTF-8"?gt ltgBSDL xsi
schemaLocation"g-BSDL ../Schemas/gBSDL.xsd" xml
ns"g-BSDL" xmlnsxsi"http//www.w3.org/1999/XML
Schema-instance"gt ltHeadergt ltClassificationAlias
alias"MV4" href"urnmpegmpeg4videocssyntact
icalLabels"/gt ltDefaultValues unit"byte" addr
essMode"Absolute" globalAddress"Content/coast
guard.mpg4"/gt lt/Headergt ltgBSDUnit
syntacticalLabel"MV4VO" start"0"
length"18"/gt ltgBSDUnit syntacticalLabel"MV4I_
VOP" start"18" length"4641"/gt ltgBSDUnit
syntacticalLabel"MV4P_VOP" start"4659"
length"98" marker"violent"/gt ltgBSDUnit
syntacticalLabel"MV4B_VOP" start"4757"
length"16" marker"violent"/gt ltgBSDUnit
syntacticalLabel"MV4B_VOP" start"4773"
length23" marker"violent"/gt ltgBSDUnit
syntacticalLabel"MV4P_VOP" start"4796"
length"178" marker"violent"/gt lt!--... and so
on ...--gt
90
gBSDL Experiments on MPEG-4 VESs (4)
lt?xml version"1.0" encoding"UTF-8"?gt ltgBSDL xsi
schemaLocation"g-BSDL ../Schemas/gBSDL.xsd" xml
ns"g-BSDL" xmlnsxsi"http//www.w3.org/1999/XML
Schema-instance"gt ltHeadergt ltClassificationAlias
alias"MV4" href"urnmpegmpeg4videocssyntact
icalLabels"/gt ltDefaultValues unit"byte" addr
essModeRelative" globalAddress"Content/coast
guard.mpg4"/gt lt/Headergt ltgBSDUnit
syntacticalLabel"MV4VO" start"0"
length"18"/gt ltgBSDUnit syntacticalLabel"MV4I_
VOP" start"18" length"4641"/gt ltgBSDUnit
syntacticalLabel"MV4P_VOP" start"4659"
length"98" marker"violent"/gt ltgBSDUnit
syntacticalLabel"MV4B_VOP" start"4773"
length23" marker"violent"/gt ltgBSDUnit
syntacticalLabel"MV4P_VOP" start"4796"
length"178" marker"violent"/gt lt!--... and so
on ...--gt
91
gBSDL Experiments on MPEG-4 VESs (5)
lt?xml version"1.0" encoding"UTF-8"?gt ltgBSDL xsi
schemaLocation"g-BSDL ../Schemas/gBSDL.xsd" xml
ns"g-BSDL" xmlnsxsi"http//www.w3.org/1999/XML
Schema-instance"gt ltHeadergt ltClassificationAlias
alias"MV4" href"urnmpegmpeg4videocssyntact
icalLabels"/gt ltDefaultValues unit"byte" addr
essModeRelative" globalAddress"Content/coast
guard.mpg4"/gt lt/Headergt ltgBSDUnit
syntacticalLabel"MV4VO" start"0"
length"18"/gt ltgBSDUnit syntacticalLabel"MV4I_
VOP" start"18" length"4641"/gt ltgBSDUnit
syntacticalLabel"MV4P_VOP" start"4659"
length"98" marker"violent"/gt ltgBSDUnit
syntacticalLabel"MV4P_VOP" start"4780"
length"178" marker"violent"/gt lt!--... and so
on ...--gt
92
gBSDL Experiments on MPEG-4 VESs (6)
Proxy Cache
CDI
gbsdl
_gbsdl.xml
DI processor
REL,DID,etc
.mpg4
coastguard.mpg4
_adapt_01.mpg4
93
gBSDL Experiments on MPEG-4 VESs (7)
Compactness of Binarized gBSDL Descriptions
First and Second Adaptation Step Remove all
B-Frames
foreman
94
Use Case 2 Remove Violency (1)
95
Use Case 2 Remove Violency (2)
lt?xml version"1.0" encoding"UTF-8"?gt ltgBSDL xsi
schemaLocation"g-BSDL ../Schemas/gBSDL.xsd" xml
ns"g-BSDL" xmlnsxsi"http//www.w3.org/1999/XML
Schema-instance"gt ltHeadergt ltClassificationAlias
alias"MV4" href"urnmpegmpeg4videocssyntact
icalLabels"/gt ltDefaultValues unit"byte" addr
essModeRelative" globalAddress"Content/coast
guard.mpg4"/gt lt/Headergt ltgBSDUnit
syntacticalLabel"MV4VO" start"0"
length"18"/gt ltgBSDUnit syntacticalLabel"MV4I_
VOP" start"18" length"4641"/gt lt!--... and so
on ...--gt
MPEG-4 Visual ES Specific BSDL Schema
DI Provider Pre-Processing
.mpg4
.xml
BintoXML
BSDLtogBSDL(XSLT)
_gbsdl.xml
lt?xml version"1.0" encoding"UTF-8"?gt ltgBSDL xsi
schemaLocation"g-BSDL ../Schemas/gBSDL.xsd" xml
ns"g-BSDL" xmlnsxsi"http//www.w3.org/1999/XML
Schema-instance"gt ltHeadergt ltClassificationAlias
alias"MV4" href"urnmpegmpeg4videocssyntact
icalLabels"/gt ltDefaultValues unit"byte" addr
essMode"Absolute" globalAddress"Content/coast
guard.mpg4"/gt lt/Headergt ltgBSDUnit
syntacticalLabel"MV4VO" start"0"
length"18"/gt ltgBSDUnit syntacticalLabel"MV4I_
VOP" start"18" length"4641"/gt ltgBSDUnit
syntacticalLabel"MV4P_VOP" start"4659"
length"98" marker"violent"/gt ltgBSDUnit
syntacticalLabel"MV4B_VOP" start"4757"
length"16" marker"violent"/gt lt!--... and so on
...--gt
Remove violency
_adapt_01.mpg4
First Adaptation
_gbsdl_adapt_02.xml
96
Visions

• Short term
• Long Term
• Distributed Multimedia Lab

97
FP6 Future and Emerging Technologies (FET)

• Networked Audiovisual systems and home platforms
• Objective
• To develop end-to-end networked audio-visual
  systems and applications with full interactivity
  capacity.
• Peer-to-Peer Digital Item Adaptation
• Peer-to-Peer Mobile Agents and Digital Items

98
Interactive Multimedia
99
Calm Technology and Multimedia Adaptation
By placing a multimedia stream in the periphery
we are able to attune to many more things than we
could if everything had to be at the center.
By moving a multimedia stream to center we are
empowered to act.
100
Context-Aware Multimedia Adaptation

• Context-Aware Multimedia Adaptation
• Listen to Content Changes in Stream
• Adapt when critical event detected, e.g.,
  change to higher resolution, but smoothly
• Combine with voluntary priority change
• Example

101
Distributed Multimedia Lab

• Open framework, as such a realization of MPEG-21
• Co-operative Components and IPMP aware
• Object-coding based (synthetic, natural,
  animated)
• Meta-Data driven (MPEG-7/21, Dublin Core, etc.)
• Adaptive (to resource, user, )
• Mobile
• Service-Oriented (e.g., towards highly
  interactive applications)