Video Data Management Systems: Metadata and Architecture

1
?????Video Data Management Systems Metadata
and Architecture

• Chapter 9 of
• Multimedia Data Management
• Using Metadata to Integrate and Apply Digital
  Media

2
????

• ??Video Data Management System?????,?????Digital
  Library????????,???????DL???????
• Good understanding of digital media
• Typical applications of digital media
• Types of queries

3
????

• Introduction
• Video Data Management System (VDMS)
• Application of Video
• Classification of Video Queries
• ViMOD The Video Data Model

4
Introduction

• Video data management system (VDMS)
• Storage of video on computer systems
• Content based retrieval
• Real-time synchronized delivery of video
• Content based retrieval
• Data modeling
• Automatic extraction of data models
• Query and retrieval mechanisms

5
Video Data Management System (VDMS)
6
What is a VDMS

• A software system which provides
• Content based access to video data
• Audiovisual content of video
• Semantic content of video
• Facilities
• Facilities provided by standard DBMS (insertion,
  deletion, schema definition)
• User interface
• Predefined set of query classes and an associated
  query interface
• Tools for navigation and manipulation video data

7
Example Scenario Sporting Event VDMS (I)

• Purpose
• Postgame analysis
• Plan strategies for future games
• Analyze game strategies of opposing teams

• Scenario 1
• Remember the OSU game from last fall?
• Retrieve ltGamefootballgt ltSchoolOSUgt ltYear1994)
• The video is cued to the beginning of the OSU
  game of 1994

8
Example Scenario Sporting Event VDMS (II)

• Scenario 2
• Didnt OSU score a field goal in the 3rd quarter
  of the game?
• Locate ltQuarter3gt ltPlayfield-goalgt ltTeamOSUgt
• The retrieved video is marked with the time
  points of all field goal attempts

• Scenario 3
• Can we see a close up shot of this kick?
• Retrieve ltPlayfield-goalgtltShotClose upgt
• The database is searched for a close up shot and
  the video is cued if the search is successful

9
Example Scenario Sporting Event VDMS (III)

• Scenario 4
• Lets look at the track of the kickers foot
• Tracking Mode. Using the interface, a bounding
  box is placed around the kickers foot to
  indicate the object to be tracked.
• The system tracks the kickers foot through the
  shot, and displays a track of the foot

• Scenario 5
• Lets see other kickers with similar kicks in
  last years NCAA football
• Similarity Search. ltYEAR1993gtltGameNCAA-footgtltPla
  y field goalgt ltMatch-CriteriaIntra video object
  location based matchinggt
• Compare the kickers tracks for attempts. Ranked
  set

10
Content of Video
11
Content of Video

• Semantic content
• Message of information conveyed
• Audiovisual content
• Video clips and audio signals
• Distinction Amount of contextual information and
  knowledge required to extract contents

12
Semantic Content

• Content extraction
• Need background knowledge
• Complex, manually
• Example
• Emotion, Classification
• Similar to manage textual information
• Access Finer grain
• scenes, shot

13
Audiovisual Content

• Content extraction
• No Need background knowledge
• (Semi-)automatically
• Example
• Object recognition, object tracking over time,
  temporal events recognition, word and sentence
  recognition, unusual sound events
• Camera and object motion, color and texture
  properties, audio properties

14
Application of Video
15
Feature Films

• Film viewer
• List films with TitleX, ActorsY, DirectorsZ,
• List films with GenreWestern
• Film critics
• Find scene where ActorX Emotioncry
• Find shot with camerastationary, Lens
  actionsZoom in
• Find scene with Special EffectMorphing
• Film Database Managers
• Number of rentals for TitleX, ActorY
• Average number of movies per customer per week

16
News Video

• News Browser
• Retrieve hockey events occurred between 1994 and
  1995
• Retrieve results of 1992 elections
• News Producers and Reporters
• News reuse
• Nomination of a new presidential candidate
• Highlight the persons life beginning from birth

17
Sporting Event Videos

• Casual Viewer
• Locating game videos (like film viewers)
• Sports Coaches, Trainers
• Coaching teams, analyzing player performance,
  game strategies
• Example Queries

18
Classification of Video Queries
19
Content Type

• Semantic Query
• Require high level semantic recognition and
  interpretation of the video content
• Require metadata generated manually
• Find scene with ActorX EmotionCrying
• Audiovisual Query
• Require metadata generated automatically or
  semi-automatically
• Find shot with CameraStationary, Lens
  ActionsZoom in

20
Matching Required

• Exact match query
• Find scene with ActorX
• Similarity match query
• Find all triple axles by female skaters with
  similar launching patterns

21
Function

• Location queries
• Locate video information
• Find scene with ActorX
• Point to the beginning of matched videos
• Tracking queries
• Track visual quantities
• Track the ball through this shot
• Location of the ball in each of the frames in the
  shot

22
Temporal Unit Type

• Unit query
• Complete units of video
• Find films with ActorX
• Subunit Query
• Subunits of video
• Find scenes with ActorX

23
Requirement Summary for Video Data Model

• A notion of time
• A segmented representation for time intervals
• A relationship between time intervals
• A set of descriptions associated with each time
  interval

24
ViMOD The Video Data Model
25
Video Data Model

• V
• Video Interval tb, te
• Temporal Relations R
• R((r1,v1), (r2,v2), , (rk,vk))
• Feature Count n
• Feature Type (w0, w1,, wn)
• Feature (F1, F2, F3,, Fn)

26
Segmentation Criteria (I)

• The basis on which a particular interval of the
  video can be chosen
• Grouping of criteria
• Syntactic segmentation criteria
• Domain independent
• Semantic segmentation criteria
• Domain specific

27
Segmentation Criteria (II)
28
(No Transcript)
29
Video Features and Video Feature Type-- Metadata
30
Feature Classification Criteria (I)

• Content Dependence
• Independent the feature is not directly
  available from the video data
• Meta features
• e.g. Budget of a video
• Dependent
• Data features
• e.g. Story
• Temporal Extent Video or Image

31
Feature Classification Criteria (II)

• Labeling
• Domain model based labels
• Qualitative features (Q-features)
• Low-level domain independent models
• Raw features (R-features)

32
Type of Video Features
33
Meta Features

• In general, apply to a complete video
• Examples

34
Video Q-Features

• Has a value belonging to a finite set of labels
• Low level property
• Cinematographic properties
• Higher level properties
• Time frame, point of view

35
Video Q-Feature Examples
36
Video R-Features
37
Image Q-Features
38
Image R-Features
39
ViMOD Architecture
40
ViMOD Architecture

• Video server
• Database interface
• Metadata store
• Query processor
• Insertion module
• User interface

41
Block Interactions

• Data insertion operation
• Database Interface
• Metadata store
• Insertion module
• User interface

• Data retrieval operation
• Query processor
• User interface
• Database interface
• Metadata store

42
(No Transcript)
43
??

• ???????DL,??
• ????????
• ?????????
• ???????????(Query)
• ???????????????????????