Digital Video Library Network

1
Digital Video Library Network

• Supervisor Prof. Michael Lyu
• Student Ma Chak Kei, Jacky

2
Introduction

• Overview
• System Architecture
• Video Server
• Indexing Server
• Query Server
• Client Applications
• Related Technology

3
Overview

• Make large video library to be searchable
  information resources
• Video
• Captures the experience of society
• News, TV, Movieetc
• Search and Discovery
• Automated extraction of knowledge from video
• Integration of speech, image, and natural
  language understanding for library creation and
  exploration

4
Information Retrieval

• Given a large collection of multimedia records,
  find similar/interesting things
• Allow fast, approximate queries
• Find rules/patterns
• Similarity search
• Find pairs of documents that are similar
• Find medical cases similar to Smiths
• Find pairs of stocks that move in sync

5
Application Areas

• Education and training
• Consumer and business access to news and
  information of interest
• Entertainment
• Interactive television
• Meeting/corporate memory
• Video conferences

6
Diverse Technologies

• Image Understanding
• Scene Understanding
• Speech Recognition
• Metadata/Entity Extraction
• Natural Language Processing
• More
• Database, Network, User Interface...

7
System Architecture

• Component Based
• High Extensibility
• High Availability
• High Performance
• Workstation or Distributed Systems over Internet

8
System Architecture
9
System Architecture
10
Video Server

• Specialized in capturing, storing, and delivery
  videos
• Dual with different video sources
• Features
• Video Storage
• Meta-Media Attributes
• Video Delivery

11
Video Storage

• Store segmented video in digital formats
• Video segmentation
• Using low-level visual features
• Using multimedia cues
• Semantic segmentation
• Using audio, visual, textual signals at different
  stages
• For Example use audio feature to separate speech
  and commercials then use text analysis to do
  story-level segmentation
• Require knowledge on the video source

12
Meta-Media Attributes

• For information
• related to but not within the video
• impossible to be extracted from the video
• Five baisc types
• Production feature
• Media feature
• Text description
• Intellectual property information
• References

13
Video Delivery

• Main concern
• number of current clients
• quality of services
• Streaming protocol
• reduce the latency for starting the video
• exploit the error tolerance nature of video
• QoS
• User perspective
• Application perspective
• Transmission perspective

14
QoS Perspectives
15
QoS Processing Model
16
Indexing Server

• Specialized in indexing the video for retrieval
  use
• Features to be indexed
• Textual Information
• Physical Features
• Semantic Features
• Advanced indexing on
• Video caption
• Company logo
• Face recognition

17
Textual Information

• Includes
• Provided meta-media attributes
• Generated script by automatic speech recognition
• Tradition information retrieval for text
  documents
• Lexical analysis
• Removal of stopwords
• Stemming
• Selection of index terms
• Construction of term categorization structures

18
Speech Recognition
19
Physical Features

• Low-level objects and associated features
• Features indexed
• Color
• Texture
• Shape
• Motion
• Spatiotemporal structures

20
Extract Physical Features

• Segment the video into separate shots
• Consistent background scene
• Extract salient video regions and video objects
• Index video objects with features mentioned
• Advanced video object extraction in MPEG-4

21
Semantic Features

• More intuitive and direct then physical features
• Probabilistic graphic model
• By Hidden Markov Model (HMM) to investigate the
  combination of input features that represent an
  object
• Identify events, objects, and sites
• Using multimedia training data
• Limit the lifetime of objects to the shots
  duration
• Compute probabilities ofP(car AND road segment
  of multimedia data)
• Higher level HMM between different
  objects(Markov chain Monte Carlo method)

22
Complexity of Features
23
Query Server

• Transform user query to formal queries
• Natural language processing
• Ranking of results
• Different IR Models
• Boolean Model
• Vector Model
• Probabilistic Model
• Have knowledge of individual Indexing Servers
• Multimedia Portals!

24
Client Applications

• Basic functionality
• Query
• Presentation of Results
• Video Playback
• Additional functionality
• Linkage to external database
• Manipulation of video

25
MPEG4

• Standard to address multimedia contents
• Represent units of aural, visual or audiovisual
  content as media objects
• Natural or synthetic origin
• Compose the scene by description of media objects
• Support QoS in a media-object level
• Indexing of media-object become easy

26
MPEG7

• Standard to describe the multimedia content data
  with some degree of interpretation of the
  semantics
• Act as the interface for multimedia applications
• e.g. Between Video Server and Indexing Server

27
Conclusion

• Challenges
• Multilingual Processing
• Cognitive Processing
• Library Interoperability
• Intellectual Property
• Security Issues

28
Thank you