Multimedia Projects Our Experience

1
Multimedia ProjectsOur Experience

• Research Projects
• NSF Multimedia Laboratory at Florida Atlantic
  University
• (1995-2001)
• Director Dr. Borko Furht

2
Projects

• Content-Based Image Retrieval Using Relevance
  Feedback (Oge Marques)
• IP Simulcast - An Innovative Video and Audio
  Broadcasting Technique Over the Internet (Ray
  Westwater and Jeff Ice)
• XYZ Video Encoding Technique (Ray Westwater
  Joshua Greenberg)
• An Innovative Motion Estimation Algorithm for
  MPEG Codec (Joshua Greenberg)
• A Fast Content-Based Multimedia Retrieval
  Technique (P. Saksobhavivat)
• Interactive Progressive Encoding System (Joe
  Celi)

3
Internet Broadcasting or
Webcasting

• Broadcasting multimedia data (audio and video)
  over the Internet - from a server (sender) to a
  large number of clients (receivers)
• Applications include radio and television
  broadcasting real-time
  broadcasting of critical data
  distance learning
  videoconferencing
  database replication
  electronic
  software distribution

4
Broadcast Pyramid Applied in IP
Simulcast
5
IP Simulcast - An Innovative Technique
for Internet Broadcasting

• IP Simulcast reduces the server (or sender)
  overhead by distributing the load to each client
  (or receiver)
• Each receiver becomes a repeater, which
  rebroadcasts its received content to two child
  receivers
• The needed network bandwidth for the server is
  significantly reduced

6
Characteristics of IP Simulcast

• It is a radically different model of digital
  broadcast, referred to as repeater-server model
• The server manages and controls the
  interconnection of repeaters
• Each repeater not only plays back the data
  stream, but also transmits the data to two other
  repeaters
• IP Simulcast provides guaranteed delivery of
  packets, which is not the case with IP Multicast

7
Product AllCastwww.allcast.com
8
Broadcasting Tree
Once the AllCast Broadcaster is configured, many
users can connect to hear/view content. The
bandwidth usage is distributed across the
participants, as illustrated by the dynamic,
self-healing dissemination tree shown in the
AllCast main window.
9
Microsoft Media Player with AllCast Plug-in

• Users can connect to a broadcast using the
  Microsoft Windows Media Player together with a
  small, seamlessly integrated AllCast Plug-in.
• The plug-in enables the Windows Media Player to
  participate in peer-to-peer broadcasts.

10
XYZ - New Video Compression Technique

• The XYZ video compression algorithm is based on
  3D Discrete Cosine Transform (DCT)
• It provides very high compression ratios and
  excellent video quality
• It is very suitable for real-time video
  compression

11
Forming Video Cube for
XYZ Compression
12
Block Diagram of the XYZ
Compression
13
Key Encoding Equations

• Both encoder and decoder are symmetrical, which
  makes the algorithm suitable for VLSI
  implementation

14
XYZ Versus MPEG
15
Complexity of Video Compression
Techniques
16
XYZ Versus MPEG
MPEG Cr11,NRMSE0.08
MPEG, Motion Est. only Cr27, NRMSE0.14
Original
XYZ Cr45, NRMSE0.079
17
Examples of XYZ Compression
Original
XYZ-compressed Cr51
18
Examples of XYZ Compression
Original
XYZ-compressed Cr110
19
Sensitivity of the XYZ Algorithm to
Various Video Effects
20
Characteristics of XYZ Video
Compression

• XYZ gives significantly better compression ratios
  than MPEG for the same quality of video
• For similar compression ratios, XYZ gives much
  better quality than MPEG
• XYZ is faster than MPEG (lower complexity)
• XYZ is simple for implementation

21
Applications of the XYZ

• Interactive TV and TV set-top boxes
• TV phone
• Video broadcasting on the Internet
• Video-on-demand applications
• Videoconferencing
• Wireless video

22
TV Phone

• Videophone is a box on the top of TV with a small
  camera, modem, and video/audio codec.

23
Design of the TV Phone
24
A Fast Content-Based Multimedia
Retrieval Technique

• Two main approaches in indexing and retrieval of
  images and videos
• Keyword-based indexing and retrieval
• Content-based indexing and retrieval

25
Keyword-Based Retrieval and Indexing

• Uses keywords or descriptive text, which is
  stored together with images and videos in the
  database
• Retrieval is performed by matching the query,
  given in the form of keywords, with the stored
  keywords
• This approach is not satisfactory - the
  text-based description is incomplete, imprecise,
  and inconsistent in specifying visual information

26
New Algorithm for Similarity-Based Retrieval of
Images

• Images in the database are stored as
  JPEG-compressed images
• The user submits a request for search-by-similarit
  y by presenting the desired image.
• The algorithm calculates the DC coefficients of
  this image and creates the histogram of DC
  coefficients.
• The algorithm compares the DC histogram of the
  submitted image with the DC histograms of the
  stored images.

27
Histogram of DC Coefficients for the Image
Elephant
28
Comparison of Histograms of DC Coefficients
29
Example of Similarity-Based Retrieval Using the
DC Histograms
30
Similarity-Based Retrieval of Compressed Video

• Partitioning video into clips - video
  segmentation
• Key frame extraction
• Indexing and retrieval of key frames

31
DC Histogram Technique Applied for Video
Partitioning
32
Example of Similarity-Based Retrieval of Key
Frames Using DC Histograms
33
Interactive Progressive Encoding System

• Users submit requests for imagery to the image
  database via a graphical user interface
• Upon an initial request, a DCT image (version of
  the image based on DC coefficients only) is
  transmitted and reconstructed at the user site.
• The user can then isolate specific regions of
  interests within the image and request additional
  levels of details.

34
Band Transmission in Interactive JPEG System
Based on Spectral Selection
35
Prototype System - IPES and Experimental Results

• Original image Airport

36
Interactive Progressive Transmission in Four Scans
37
Selection of Two Regions
38
Cumulative Number of Transmitted Bits
39
Extracted Images From a Group of Images
40
Applications

• Retrieval and transmission of complex images over
  low bandwidth communication channels (image
  transmission over the Internet, real-time
  transmission of medical images)
• Archiving and browsing visually lossless image
  databases (medical imaginary, space exploration
  and military applications)

41
Content-Based Retrieval

• Large, complex, and ever growing, distributed,
  mostly unstructured, multimedia repositories
• Three ways of retrieving multimedia information
• Free browsing (inefficient, time-consuming,
  doesnt scale well)
• Text-based retrieval (relies on metadata,
  time-consuming, subjective)
• Content-based retrieval (requires intelligent
  interpretation of the contents)

42
Design of MUSE System
User
Image Archive

• GUI
• Image selection
• Result viewing

Image Analysis
Interactive learning Display update

• Image Feature Extraction
• Color
• Shape
• Texture

Probability recalculation candidate ranking
Feature Extraction Similarity comparison
Image Representation Feature Organization
Off-line
Online
43
Query By Example
Best result
Similarity Score 0,1
Example image
44
Relevance Feedback
Good
Bad
Neither
45
Relevance Feedback - Next
46
Technology Behind the MUSE System Feature
extraction

• Extraction of relevant image features impacts the
  overall performance of the system.
• MUSE uses
• color-related features (color histograms, color
  space partitioning and/or quantization, color
  moments, color coherence vectors)
• texture-related features (Multiresolution
  Simultaneous Autoregressive Model - MSAR)
• frequency-related features (DFT, DCT)

47
Technology Behind the MUSE System Bayesian
formulation

• MUSE is based on a Bayesian framework for
  relevance feedback.
• During each iteration of a MUSE session, the
  system displays a subset of images from its
  database, and the user takes an action in
  response, which the system observes.
• Based on the users actions, the probability
  distribution over possible targets is refined.
  (Most systems refine the users query)
• The best candidates are then displayed back.