SAPIR Search in Audiovisual content using P2p IR

1
SAPIR Search in Audio-visual content using P2p IR

• Yosi Mass, Raul Santos

2
Why SAPIR?

• Searchable space created by the growing amounts
  of existing video and multimedia files may
  greatly exceed the area searched by major
  engines.
• Traditional search engines are limited to
  searching in the associated text and meta-data of
  the multimedia content. If content providers
  don't clearly or accurately describe their
  multimedia files, or use inaccurate tags, the
  current method falls short.
• Current internet search is geared mainly to
  relatively powerful desktop machines and accessed
  via regular web browsers, not lightweight mobile
  devices with their connectivity and interactivity
  limitations.

3
SAPIR Objectives

• Develop cutting-edge technology to index and
  search large scale audio-visual information by
  content.
• Make information available on many devices,
  enhanced by social networking while keeping
  privacy and preventing fraud
• Support new trends in MM content production
  personal producer VS professional producers

4
SAPIR challenges

• Dimensions of the search problem
• Efficiency (scalability is the key issue)
• Effectiveness (quality measures of results)
• Efficiency challenges
• Scale in collection size
• Scale in number of users
• Effectiveness challenges
• New search paradigm combining text audio-visual
  content
• Usability challenges

5
SAPIR Consortium
6
SAPIR approach-P2P Architecture
7
Search using the Query by Example Paradigm

• Search for information about a physical object by
  taking an image of it with a mobile phone or find
  a song by humming the melody.
• Support similarity search for metric spaces

8
Feature extraction
ltSapirMMObjectgt lttitlegtwhen waves
collidelt/titlegt ltMpeg7gt ltVisualDescriptor
typeScalableColorTypegt ltVisualDescriptor
typeColorStructureTypegt
ltVisualDescriptor typeColorLayoutTypegt
ltVisualDescriptor typeEdgeHistogramTypegt
ltVisualDescriptor typeHomogeneousTextureTypegt
lt/Mpeg7gt ltcommentsgt ltcomment id"
author"gtbeautifullt/commentgt ltcomment
...gtvery powerfullt/commentgt lt/commentsgt lttagsgt
lttag id"254" author12_at_N00"gtwaveslt/taggt
lttag gtVictoria beachlt/taggt lt/tagsgt lt/SapirM
MObjectgt
9
Indexing

• ltSapirMMObjectgt
• lttitlegtwhen waves collidelt/titlegt
• ltMpeg7gt
• ltVisualDescriptor typeScalableColorTypegt
• ltVisualDescriptor typeColorStructureTypegt
  
• ltVisualDescriptor typeColorLayoutTypegt
• ltVisualDescriptor typeEdgeHistogramTypegt
• ltVisualDescriptor typeHomogeneousTextureTy
  pegt
• lt/Mpeg7gt
• ltcommentsgt
• ltcomment id" author"gtbeautifullt/comme
  ntgt
• ltcomment ...gtvery powerfullt/commentgt
• lt/commentsgt
• lttagsgt
• lttag id"254" author12_at_N00"gtwaveslt/taggt
• lttag gtVictoria beachlt/taggt
• lt/tagsgt
• lt/SapirMMObjectgt

10
Querying

• ltMpeg7Query weight1gt
• ltVisualDescriptor typeScalableColorTypegt
• ltVisualDescriptor typeColorStructureTypegt
• ltVisualDescriptor typeColorStructureTypegt
• lt/Mpeg7Querygt
• lt/Mpeg7Query weight0.5gt
• lttaggtwaveslt/taggt
• lt/Mpeg7Querygt

11
Project status for Apr 2008

• A scalable, extensible and versatile architecture
  for P2P was defined. APIs for P2P content
  management, indexing and search were defined and
  implemented
• Several Scenarios were defined and tested in
  Focus groups
• Definition of a common schema for feature
  representation using MPEG-7 was defined.
• A demo for Indexing and search in 10M Flickr
  files using a combination of content based image
  search combined with text and metadata was
  implemented using the SAPIR APIs.
• Testbed of 50M Flickr files crawled by the EGEE
  grid aiming at 100M towards the Year End. This
  testbed collection will be available for
  scientific experiments (CoPhir
  http//cophir.isti.cnr.it site)
• Next demo (due Nov 08) will include search in
  music, video and speech as well as some scenario
  integration.

12
Tests

• P2P architecture for search in Audio-Visual
  content
• Efficiency Some initial results
• 1M FlickrXML files 500msec per query 50
  peers (8CPU, 16Gb)
• 10M FlickrXML files - 500msec per query 500
  peers (16CPU, 64Gb)
• Effectiveness
• Text image improves over text or image only

13
WP9 Dissemination and exploitation

• Public website
• http//www.sapir.eu
• Dissemination
• First DUP was published
• Participate in Chorus meetings and road map
• Workshops SIGIR07, ECIR08, SAC08
• Demos
• Publications
• More than 20 SAPIR related publications so far
• Contacts with Standards Bodies
• MPEG-21, MPEG-A, MPEG-7
• Exploitation

14
WP9 Dissemination and exploitation

• Proposed contribution to standards
• Extension to MPEG-7 for music and speech.
• Proposals for MPQF (MPEG-7 Query Format)
• A DRM implementation for P2P based on Chillout
• Propose a call for MPEG-21 Query Format

15
Thank You!

• For more info visit http//www.sapir.eu

16
Results (Jan 2007 Mar 2008)

• WP1 Scenarios and a complete guideline for
  usability and user interface design
• WP2 Architecture for P2P and APIs
• WP3 - Definition of a common schema for feature
  representation using MPEG-7.
• WP4, WP5 Demo of indexing and search in 10M
  Flickr files combining text and low level visual
  descriptors
• WP6 Work on interoperable DRM solution
  (Chillout) for P2P networks
• WP7 initial design of Social networking and
  support for mobile devices