Pharos

1
Pharos

• Alessandro Bozzon - WebModels

2
Outline

• PHAROS in brief
• PHAROS in context Challenges for businesses,
  users, technology
• The search problem
• Example application video search
• Example framework Content analytics
• Project organization

3
Outline

• PHAROS in brief
• PHAROS in context Challenges for businesses,
  users, technology
• The search problem
• Example application video search
• Example framework Content analytics
• Project organization

4
PHAROS mission

• To advance audiovisual search from a
  point-solution search engine paradigm to an
  integrated search platform paradigm.
• The PHAROS platform will be built on an
  innovative, open, and distributed architecture
  that enables consumers, businesses and
  organisations to unlock the values found in
  audiovisual content.

5
Objectives expected outcome

• Develop SOA-compliant, open and distributed
  technology platform for development of arbitrary
  information access solutions for AV content,
  taking into account future user and search
  requirements.
• Scalable, pluggable, and separable frameworks for
  
• Content analytics
• Search query analysis
• Platform for innovation
• Framework for integration of technologies
• Quickly adapt to new user and application
  scenarios
• Transform ideas into real solutions
• Evaluate with real users

6
Partners and project key data

• Industry
• Engineering Spa
• Fast Search Transfer AS
• Sail Labs Technology AG
• Webmodels Srl

• Academia
• L3S Research Centre
• Fraunhofer IDMT
• EPF Lausanne
• Open University, KMI
• University Pompeu Fabra
• VTT Research Centre

• Content and Service Providers
• France Telecom
• Circom Regional

• Timescale and budget
• Start January 2007
• Duration 36 months
• Total budget 14.2m
• EC contribution 8.5m

12 partners from 9 countries
7
Outline

• PHAROS in brief
• PHAROS in context Challenges for businesses,
  users, technology
• The search problem
• Example application video search
• Example framework Content analytics
• Project organization

8
Some business challenges

• PHAROS takes future business and user
  requirements as key design principles
• Amount of data is exploding
• Storage and access challenges
• Give users contextually relevant information
• Technology is changing quickly
• Quick innovation cycles - solutions must
  integrate new and existing technology
• ICT infrastructure is being commoditised
• Free broadband, wireless access in cities

9
Some more business challenges

• Multimedia search portals are the new media
  channels
• Blend content from multiple sources
• Scale to handle famous and long tail content
• Deliver content tailored to each device
• Crowded Online Landscape
• Many diverse technologies required for operating
  search solutions
• Obstacle for diversity / dominance of few players

10
Some user challenges

• Precision contextual relevancy
• aware of rights, user and information contexts
• personalization and recommendation
• Search must support multiple interaction patterns
• active searching, monitoring, browsing and "being
  aware
• Trust and spam
• Ubiquity of access

11
Some technology challenges

• Scalability and performance
• Keep pace with data proliferation
• Opaqueness of rich media / semantic gap
• Openness and extensibility

12
Outline

• PHAROS in brief
• PHAROS in context Challenges for businesses,
  users, technology
• The search problem
• Example application video search
• Example framework Content analytics
• Project organization

13
The search problem High-level architecture
Data volume ingestion rate
Query rate complexity
C O N T E N T
U S E R S
Generic search-driven application
Search query analysis
Query result interaction
Content analytics

q
q
Feature pattern extraction
Query refinement
Indexing
r
r
Content refinement
Contextual Matching
Result analytics and aggregation
PHAROS aims at developing a technology platform
that can be used to build (almost) arbitrary AV
search applications
Storage and Access
14
The search problemExample query modalities
search types
\\wherecontains(amsterdam) and \\topiccontai
ns(building)
52.37N 4.89 E
amsterdam
Long/Lat
XPath
JPG
Keywords
Query analysis Federation
Text search Inverted index
Image search Similarity index
Geo search R-tree index
XML search Semantic index
15
The search problem example queries

• Find video shots where Mr. Sarkozy is speaking
  about
• Combines shot detection and speech analysis
• Uses structural information about shot boundaries
  and semantic information about identified
  speakers
• Find videos with music similar to this one, with
  scenes of sunsets
• Combines music analysis, shot detection and image
  classification
• Uses content-based music features, structural
  information about shot boundaries and information
  from image classification

16
The search problem state-of-the-art

• What is missing in state-of-the-art systems?
• Flexible architecture across content and queries
• Content analytics
• Flexible architecture which extends to support
  new information stores, processing steps and
  content types
• Scalable in traffic and volume of content as well
  as diversity and freshness of content
• Easy integration of latest content analysis
  technology

17
The search problem state-of-the-art

• What is missing in state-of-the-art systems?
• Search engine
• Scalable support for query using structure and
  semantics of multimedia
• Integration of textual, structural and
  content-based search
• Personalised ranking and recommendation for rich
  media
• Query and result analysis
• Advanced query federation and result analytics
  (aggregation)
• Support for query formulation
• Device adaptability

18
Outline

• PHAROS in brief
• PHAROS in context Challenges for businesses,
  users, technology
• The search problem
• Example application video search
• Example framework Content analytics
• Project organization

19
Query scenario I

• Theme Combination of automatic speech
  recognition and visual image analysis
• Goal Explore the documentaries videos collection
  to find information about tourism

20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
Query scenario II

• Theme Leveraging advanced speech technology
• Goal Explore news videos collection for quotes
  on the majority after the French presidential
  elections with quotes of politicians

26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
Outline

• PHAROS in brief
• PHAROS in context Challenges for businesses,
  users, technology
• The search problem
• Example application video search
• Example framework Content analytics
• Project organization

32
Content integration and analytics
Search query analysis
Query result interaction
Content analytics
q
q
Query refinement
Indexing
Feature pattern extraction
r
r
Contextual Matching
Result analytics and aggregation
Content refinement
Search Infrastructure
33
Content Integration and AnalyticsFAST Content
Integration Studio

• Connect, Combine and Transform any content type
• Connect to databases, web-crawls, intranets and
  document systems
• Combine structured and unstructured information
  using fuzzy algorithms or exact matches
• Feed content to indexes, creating the search
  experience
• Enrich operational systems or data warehouses
• Perform linguistic analysis on unstructured and
  structured data

34
Rich Media Natively Supported
AV-RSS
35
AV-RSS Rich content based media annotation
36
Outline

• PHAROS in brief
• PHAROS in context Challenges for businesses,
  users, technology
• The search problem
• Example application video search
• Example framework Content analytics
• Project organization

37
Organization of PHAROS
Management structure
Streams and activity domains
38
Understanding functional requirements

• Capturing future business and user requirements
  as key design principles!
• Industry and user partners in the consortium
• Numerous customers in the media and entertainment
  domain
• PHAROS federation members
• Consultation on the national and international
  level

39

• THANK YOU...
• Contact bozzon_at_elet.polimi.it

40
Innovation with research partners

• Example Multimedia search in PHAROS
• End-to-end search across XML and audiovisual
  content federating search across different
  content based retrieval modes
• Robust end-to-end processing and indexing through
  Image, Music, Audio, Speech, Faces,
• Validated in a range of content scenarios across
  news, documentary, music videos

41
Native rich media processing in FAST CIS
Transcoding preview generation
Face detection analysis
Visual analysis annotation
Speech analysis transcription
42
Example application Video search