Multimedia Databases

1
Multimedia Databases

• Tylor Sampson

2
What is a Relational Database

• Manages a collection of data.
• Tables (relations) consist of rows and columns.
• Each row is called a tuple.
• Tables have relationships with each other.
• These tuple components, called attribute values,
  are identified and referenced by names.
• Queries and integrity constraints are expressed
  declaratively, using operators based on the
  relational algebra and relation comparisons.

3
What we want to do, RDB can not

• Find media objects by features or content.
• Objects within the Media Instance.
• Color, Shape, Texture.
• Search using media objects (similarity,
  proximity).
• Correctly return (report) media objects.

4
Types of Retrieval

• Information Retrieval
• Typically metadata , keyword, etc.
• Content Based Retrieval
• Features are identified and matched
• Pattern recognition
• Similarity, proximity

5
Middleware

• Union of MM and IR
• Connects Multimedia Objects to traditional
  Relation DBMS
• Allows existing structure to be extended
• Relational Databases are being retrofitted with
  some Multimedia support
• Binary Field objects can be stored inside a
  field in a relational database.
• Full Text Search

6
One Model of a media Object
DB_Attribute
IR_Feature
CBR_Feature
Collection of Tables
OID
Media Object
Locator
Type
7
One Model of an Object

• OID Identifier of an object
• Type The type of object
• DB_Attribute Structured data to describe the
  attributes of the object
• IR_Feature IR Related features (keywords) these
  are extracted from the object
• CBR_Feature Content Features, like color,
  texture, shape, objects within the media.
• Locator A pointer to locate the object

8
A Media Instance of a video clip

• MI (OID,ST,FE,R,F)
• OID Id of the Object
• ST Set of all possible frames
• FE Feature
• R Relations, connections between states
• F Feature Relations

9
Sample Frames
Frame 1
Frame 2
10
FE Feature

• Tank
• Neo
• Trinity
• Morpheus
• Nebuchadnezzar
• Electronics
• Wires

11
R Relations between States

• Relations are formed and automatically generated
  from previously found content or features.
• Delete(Morpheus,F1,F2)
• Frames are the same except for Morpheus
• In_Vehicle(Nebuchadnezzar,F1,F2)
• Both frames take place in the same ship.

12
F - Feature Relations

• Based on one particular frame.
• Left_Of(Tank, Neo,F1)
• Left_Of(Tank, Trinity,F1)
• In_Front_Of(Morpeus,Neo,F1)
• Talking_To(Tank,Morpeus,F2)

13
Algorithms

• Features
• Content
• Indexing
• Searching

• Segmentation / Tessellation
• Similarity
• Proximity / Spatial
• Blobs
• Quadtrees

14
Image Segmentation / Tessellation
Square Tessellations
Hexagonal Tessellations
Arbitrary or Jigsaw Tessellations
Quad Tree Tessellations
15
Assigning metadata to subsets

• Keywords to areas (Annotation)
• Descriptors for a range of frames

16
Blobs - Blobworld Picts
17
Semantics

• Assertions are used to describe content in a
  higher (than pixel) level.
• Semantics are usually stored as metadata.
• Image OID contains 3 law enforcement officers.
• Image OID contains 3 syrup bottles.
• Image OID 2 individuals are drinking.

18
Semantic Objects of a Picture
19
Heuristics / AI

• Weighted Results Depending on how relevant a
  media object is, determine if object is a close
  match.
• Thresholds Can be adjusted if results do not
  correctly identify wants wanted.
• Ratings Determines which results to display.

20
Video Segmentation
21
How to Search a MMDB

• Search for similar Multimedia objects.
• Traditional keyword searches
• Content Based Search
• Proximity
• Example
• Select media type From MMDB where Set of
  Features, Content, Relations

22
How to Return Results

• Based on Ratings
• Provide jitter free results
• OK, frames Range of Video1.MPG match criteria.
• Or, system can display the media object
  immediately.

23
Commercial Products

• Mirror
• Multimedia Information Retrieval Reducing
  information OveRload
• MediaLand
• Blobworld
• QBIC
• IBMs Query by Image Content
• SoundFisher

24
A sample audio object
Voice 3
Voice 2
Voice 1
Sounds
Street Noise
Time
25
Soundfisher Pict 2
26
A few Real Life uses

• Face Recognition / Thumb Print /OCR
• Feature match, similarity, etc.
• GIS
• Spatial Database Engine

27
Questions