Multimedia Information Systems

1
Multimedia Information Systems

• Vijay Atluri
• atluri_at_andromeda
• Office 200R Ackerson Hall
• Phone [phone]
• Office Hours 2 hours after the class and by
  appointment

2
What are the most known media?

• Images
• 2-d color images, gray scale medical images 2-d
  (X-ray) or 3-d (MRI scans)
• Captured images
• Captured images need to be analyzed
• Synthesized images / visualizations (Artificially
  created)
• Synthetic images do not need to be analyzed (
  featured information is already available).
• Text
• sequential vs. hypertext
• semistructured
• Organized into chapters, paragraphs etc. or may
  be indexed using HTML/SGML/XML

3
What are the most known media?

• Video
• video clips, movies
• captured
• interactive
• Audio
• digitized voice or music
• 1-d time-series
• financial, marketing, production time series data
  such as stock prices, sales numbers
• Handwritten
• electronic notes
• Traditional data
• Scientific data
• collections of sensor data, e.g.,
  ltx,y,z,t,pressure,temperature,resistivitygt

4
MM Applications

• Travel Industry
• intelligent travel agent, multimedia
  presentation,
• Entertainment Industry
• Film clip database, video-on-demand,
  pay-per-view, interactive TV, in-flight
  entertainment, video game database, video dating
  services
• Users will be able to select using a mix of
  query/retrieval and browsing capabilities
• Education and training
• classroom without walls distance learning,
  teleclassrooms, interactive training, self
  education, employee reeducation
• multimedia training is 40 more effective,
  retention rate is 30 higher, learning curve is
  30 shorter (study conducted by DoD)
• approximately US spends 56.6 billion every year
  (Marketing research by Training Magazine)

5
MM Applications

• Expert Advice
• Auto repair, medical advice, ..
• Home Shopping
• multimedia presentation of goods sold, sales
  information
• Medical databases
• Text and photograph archives
• Digital Libraries
• Office automation
• Electronic encyclopedia
• DNA databases
• Geographic Information Systems

6
MMDBMS Requirements

• Ability to uniformly query multimedia data
• query and seamlessly integrate data contained in
  different databases (that may possibly use
  different schema), flat files, object-oriented
  databases, spatial databases, arbitrary legacy
  sources
• elicit the content of the media data (a challenge
  by itself) which is highly dependent on the media
  type and storage format
• merge results from different data sources and
  media types
• Ability to retrieve media objects from a local
  storage device in a smooth, jitter-free manner
• considering large storage requirements, highly
  compressed format, secondary and tertiary storage
  devices, mix of storage devices with different
  performance characteristics

7
MMDBMS Requirements

• Ability to develop a presentation in audiovisual
  media from the answer generated by the query
• Ability to deliver the presentation that
  satisfies various quality of service requirements
• synchronization, fidelity, temporal constraints
• does not suffer from jitter and hiccups
• limited buffer availability and bandwidth (output
  devices may reside at distributed network nodes)

8
What kind of queries can users ask?

• Find all images which are created by J. Smith
• Find all images with the same color, shape and
  texture
• Find all images which look like this image
• Find all images which look like this sketch
• Find all images with the same color distribution
  like a sunset photograph
• Find all images which contains a part which looks
  like this image or sketch
• Find companies whose stock prices move similarly
  
• Find other companies that have similar sales
  patterns with our company
• Find cases in the past that resemble last months
  sales pattern of our product

9
What kind of queries can users ask?

• Find past days in which the solar magnetic wind
  showed patterns similar to todays pattern
• Find similar music scores or video clips
• Find all images of sunny days (we are getting
  into semantics)
• Find all images which contain a car
• Find all images which contain a car and a man who
  look like this
• Find all image pairs which contain similar
  objects. (data mining)

10
Sample Multimedia Scenario (from the text)

• Consider a police investigation of a large-scale
  drug operation
• Video data captured by surveillance cameras that
  record the activities taking place at various
  locations
• Audio data captured by legally authorized
  telephone wiretaps
• Image data consisting of still photographs taken
  by investigators
• Document data seized by the police when raiding
  one or more places
• Structured relational data containing background
  information, bank records, etc., of the suspects
  involved
• Geographic information systems data containing
  geographic data relevant to the drug
  investigation being conducted

11
Example of Queries for the MM Scenario

• A police officer, Tom, has a photograph in front
  of him. He wants to find the identity of the
  person in picture.
• Q1 Retrieve all images from the image library
  in which the person appearing in the photograph
  appears
• Tom wants to examine pictures of a suspect Dick.
• Q2 Retrieve all images from the image library
  in which Dick appears
• Two types of queries
• Image-based
• Keyword-based

12
Example of Queries for the MM Scenario

• Q1 input is an image, output is a ranked list of
  images that are similar to the query image
• Need to know what
• similarity means
• ranking means
• Need to efficiently support these two operations
• Q2 input is a keyword, output is an image whose
  name attribute is Dick
• Need to know how to associate different
  attributes with images
• Need to know how to effectively index and
  retrieve images based on such attributes

13
Example of Queries for the MM Scenario

• Tom is listening to an audio surveillance tape.It
  contains a conversation between two individuals A
  and B
• Q3 Find the identity of B, given that A is Dick
• Tom wants to review all audio logs that Dick
  participated during some specified time period
• Q4 Find all audio tapes in which Dick was a
  participant
• Tom is browsing an archive of text documents (old
  newspaper archives, police dept files on old
  unsolved murder cases, witness statements)
• Q5 Find all documents that deal with financial
  transactions with ABC corporation
• Similar queries may be posed on video data

14
Example of Queries for the MM Scenario

• MM Query
• Q6 Find all individuals who have been
  photographed with Dick and who have been
  convicted of attempted murder in North America
  and who have recently had electronic fund
  transfers made into their bank account from ABC
  Corp.
• Need to access heterogeneous database systems
• Need to access several MM databases
• Mugshot database containing the pictures and
  names of individuals
• Surveillance photograph database of still images
• Surveillance video database
• Image processing algorithms to determine who is
  present in which video or still photograph

15
MM Research Issues

• Queries
• Need a single language
• with which MM data of different types can be
  accessed
• with which one should be able to specify
  operations to combine different media types (just
  like join, union, intersection, difference,
  Cartesian product)
• that must be able to access metadata as well as
  raw data
• that must be able to merge, manipulate, and join
  together results from different media sources
• After devising such a language, we need
  techniques to
• optimize a single query
• develop servers that can optimize processing of a
  set of queries

16
MM Research Issues

• Content
• What is meant by content?
• Under what conditions can it be described
  textually?
• Under what conditions it must be described
  directly through the original media type?
• How can we extract the content of
• an image, a video-clip, an audio-clip, a
  free/structured text document?
• How should we index the results of the extracted
  content?
• What is retrieval similarity?
• What algorithms can be used to efficiently
  retrieve media data on the basis of similarity?

17
MM Research Issues

• Storage
• How well disks, CD-ROM, tape systems and tape
  libraries work?
• How do we design disk/CD-ROM/tape servers so as
  to optimally satisfy different clients
  concurrently when they execute
• playback, rewind, fast forward, pause, etc.

18
MM Research Issues

• Presentation and Delivery
• How do we specify the content of MM
  presentations?
• How do we specify the form (temporal/spatial
  layout, fidelity) of this content?
• How do we create a presentation schedule that
  satisfies these temporal, spatial and fidelity
  requirements?
• How can we deliver MM presentation to users when
  there is
• a need to interact with other remote servers to
  assemble the presentation
• a bound on the buffer, bandwidth, load, and other
  resources
• a mismatch between the host servers capabilities
  and the customers machine capabilities,
  preferences, etc.?
• How can such presentations optimize Quality of
  Service?