ITICERTH

1
ITI-CERTH
Centre for Research and Technology Hellas
(CERTH) Informatics and Telematics Institute
(ITI) (http//www.iti.gr) Director Prof. Michael
Strintzis Multimedia Knowledge Group
(http//mkg.iti.gr) Head Dr. Yiannis
Kompatsiaris Thessaloniki, GREECE

• Amsterdam, Multimedia Semantics XG, 10-11 July
  2006
• Vasileios Papastathis, Research Fellow

2
Presentation Overview

• ITI Multimedia Knowledge Group
• Towards a Common Multimedia Ontology Framework
• IST BOEMIE Project

3
ITI Multimedia Knowledge Group

• Semantic Multimedia Analysis
• Multimedia Indexing and Retrieval
• Multimedia and the Semantic Web
• Knowledge Structures, Languages and Tools for
  Multimedia
• Reasoning and Personalization for Multimedia
  Applications
• Ontology structures in OWL for knowledge-assisted
  multimedia analysis and reasoning
• Rules in SWRL including multimedia features for
  analysis and reasoning
• MPEG-7 Visual part representation in RDF
• Graphical tool for ontology-based annotation of
  images (M-Ontomat)

4
RD Projects

• aceMedia Integrating knowledge, semantics and
  content for user centred intelligent media
  services, IP 2004-2007
• KnowledgeWeb Realizing the Semantic Web,
  funded by the DG XIII, NoE 2004-2007
• MESH Multimedia Semantic Syndication for
  Enhanced News Services, IST IP, 2006-2008
• K-Space Knowledge Space of Semantic Inference
  for Automatic Annotation and Retrieval of
  Multimedia Content, IST - NoE, 2006-2009
• X-Media Knowledge Sharing and Reuse Across
  Media, IST IP, 2006-2009
• BOEMIE Bootstrapping Ontology Evolution with
  Multimedia Information Extraction, IST STREP,
  2006-2008
• PATExpert Advanced Patent Document Processing
  Techniques, IST-STREP, 2006-2008 (collaboration
  with AUTH)

5
Towards a common Multimedia Ontology Framework

• Requirements and
• Harmonization approaches

6
Multimedia Ontology Framework

• Framework for collecting existing multimedia
  knowledge representation
• Define approach for harmonization
• Initiated by aceMedia in EWIMT 2005 but not
  directly connected to aceMedia work
• http//www.acemedia.org/aceMedia/reference/multime
  dia_ontology/index.html

7
List of contributors

• National Technical University of Athens
• Istituto per la Matematica Applicata e le
  Tecnologie Informatiche, Consiglio Nazionale
  delle Ricerche, AIM_at_SHAPE
• University Rennes
• Centre for Digital Music, Queen Mary, University
  of London
• Lehrgebiet Multimedia und Internetanwendungen,
  Informatikzentrum
• University of Southampton, UK
• Institute of Informatics Telecommunications,
  N.C.S.R. "Demokritos"
• Institute of Information Systems Information
  Management, JOANNEUM RESEARCH Forschungsgesellscha
  ft mbH

8
List of contributors (2)

• ISWeb, University of Koblenz
• Freie Universitaet,Netzbasierte
  Informationssysteme, Institut fuer Informatik
• 'Signals and Images' Laboratory, Institute of
  Information Science and Technologies "A. Faedo"
  (ISTICNR), Italian National Research Council
  (CNR)
• Image Bioinformatics Research Group, Department
  of Zoology, University of Oxford
• Technical University of Crete,
• CWI Amsterdam, Maryland Information and Network
  Dynamics Lab, National Technical University of
  Athens
• University of Twente

9
Requirements for MM ontologies

• Scope and Usage
• Media Description-Description of the information
  object
• Representation of Content Structure
• Representation of Content Description
• Knowledge Representation Formalism
• Reasoning Support
• Following an established methodology

10
Requirements (2)

• Scope and Usage
• Annotation
• Analysis
• Retrieval
• Reasoning
• Personalized filtering
• Meta-Modeling

• Media description
• Image
• Video
• 3D Graphics
• Audio
• AudioVisual
• Multimedia presentation

11
Requirements (3)

• Representation of content structure
• Multimedia structure representation
• Spatiotemporal relation representation
• Localization

• Representation of content description
• Low-level description
• Multiple audiovisual descriptors
• Alignment to existing metadata standards
• Basic and complex data types

12
Requirements (4)

• Knowledge representation formalism
• High complexity of multimedia modeling
• Compliance to W3C standards
• Inferencing support
• OWL

• Reasoning support
• DL-based reasoning
• Rules
• Uncertainty
• Fuzziness
• Established methodology
• OntoKnowledge
• OntoClean

13
Harmonization approaches (1)

• Modularization
• Clear, independent ontology modules
• Examples Visual Descriptors Ontology (VDO),
  Multimedia Structure Ontology (MSO),
  Spatio-Temporal Ontology (STO)
• Ontology cross-domain linking
• Alignment techniques for multimedia ontologies
  linking with domain-specific ontologies
• Merging techniques for multimedia ontology
  coordination
• Compliance to standards
• Semantic Web, W3C, MPEG-7, etc

14
Harmonization approaches (2)

• Harmonization of core ontologies
• Core ontologies cannot be simply mapped but need
  to be harmonized
• Need to apply ontological modeling principles
  during harmonization
• Development of a common upper ontology
• Starting point for new ontology construction
• Bridge between existing ontologies
• Act as the ground basis for the establishment of
  an integrated ontology framework
• Back up the common upper ontology

15
BOEMIE

• Multimedia annotation tool and
• A proposed knowledge framework

16
M-OntoMat Annotizer
Semantic labeling
Region merging functionality
Mid-Level Concepts
Descriptor extraction
Descriptor extraction
Automatic segmentation
17
XML output
18
Knowledge framework
Core Ontology-Holistic Representation
Domain Ontology
Spatiotemporal Constraints
Modality Elements
Multimedia Content Ontology
Features/ Multimedia Descriptor Ontology
Classifiers/ Algorithms
19
An indicative example
Athletics_Event_Inst_1
Horizontal_Bar_Inst_1
Foam_Mats_Inst_1
Image_Inst1
Horizontal_Bar_ve_Inst1
Still_region_Inst1
Foam_Mats_ve_Inst1
Still_region_Inst2