Controlled Ontology Evolution Through Semioticbased Ontology Evaluation

1
Controlled Ontology Evolution Through
Semiotic-based Ontology Evaluation

• International Workshop on Ontology Dynamics
• IWOD 2008
• Renata Dividino
• dividino_at_uni-koblenz.de
• Daniel Sonntag
• sontag_at_dfki.de

2
Overview

• Motivation
• Semiotic-based Ontology Evaluation
• Controlled Ontology Evolution through Evaluation
• Semiotic-based Ontology Evaluation Tool
• Evaluation Results
• Conclusion

3
Motivation Reason for changes!
not good for my intend of use? Apply changes!
not good for my application? Apply changes!
Inconsistent? Apply changes!
not aligned to dependent ontologies? Apply
changes!
4
Motivation Evaluation criterion!
Is the ontology still good (or better) for my
intend of use?
Which version is the best one for my app?
Is the ontology consistent?
Can I apply these changes without affecting the
dependent ontologies?
5
Motivation

• Explicit Requirements
• Reasons for changes Evaluation criterion
• Ontology changes captured by ontology evaluation
  process

Support controlled ontology evolution by using
evaluation methods and theories
6
Overview

• Motivation
• Semiotic-based Ontology Evaluation
• Semiotics
• Semiotics Ontology
• Semiotics Ontology Evaluation
• Controlled Ontology Evolution through Evaluation
• Semiotic-based Ontology Evaluation Tool
• Evaluation Results
• Conclusion

7
The Meaning Triangle (Ogden and Richards 1923)
Semiotics is composed of three fundamental
components (Moris, 1938)
Concept
Syntax
Semantics
Pragmatics
Yojo
Symbol
Object
8
Language
Ontology
Semiotic Object
Syntax
Syntax
Semantics
Semantics
Pragmatics
Pragmatics
(Niles Pease, 2001)
9
Concept Communication Context
Rep
Concept
The ontologys representation is interpretable by
some agent
graph-like structures containing terms and their
inter-relationships
represent an intended conceptualization.
Symbol
Object
Object Intended Conceptualization
Symbol Ontology Graph
(Gangemi at al, 2006)
10
Semiotic-based Ontology Evaluation
Syntax
Semantics
Pragmatics
Defining Criterion

Structural Evaluation
Functional Evaluation
Pragmatics Evaluation
11
Semiotic-based Ontology Evaluation

• The quality of ontologies is critical for the
  success of their improvements, cost reduction,
  and maintenance!
• Semiotic-based Evaluation
• semiotic levels inter-dependencies
• interactive evaluation steps incremental
  improvements

Defining Criterion

Structural Evaluation
Functional Evaluation
Pragmatics Evaluation
12
Semiotic Measures
Assesing the ontology syntax and formal semantics
Assesing the ontology cognitive semantics
Assesing the ontology pragmatics
13
Overview

• Motivation
• Semiotic-based Ontology Evaluation
• Controlled Ontology Evolution through Evaluation
• Semiotic-based Ontology Evaluation Tool
• Evaluation Results
• Conclusion

14
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation

• applying quality assessment at each evolution
  step (incrementally changes)
• continual improvements (continual changes)

Capturing
Validation
Representation
Semantics of change
Propagation
Implementation
15
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation
Capturing
Validation
Representation
Semantics of change
Propagation
Implementation
(Stojanovic, 2004)
Semiotic Ontology Evaluation Process
16
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation
Capturing
Validation
Representation
Semantics of change
Propagation
Implementation
Explicit Requirements Reasons for changes
Evaluation Criteria
Semiotic Ontology Evaluation Process
17
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation
Capturing
Validation
Representation
Semantics of change
Propagation
Implementation

Structural Evaluation
18
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation
Capturing
Validation
Representation
Semantics of change
Propagation
Implementation
Making the ontology changes visible in a form of
an adequate representation
Ontology changes need to be managed such that
the ontology remains consistent

Structural Evaluation
19
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation
Capturing
Validation
Representation
Semantics of change
Propagation
Implementation

Structural Evaluation
Functional Evaluation
20
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation
Capturing
Validation
Representation
Semantics of change
Propagation
Implementation
Verify consistency of dependent ontologies
Ontology changes need to be managed such that
the ontology remains consistent

Structural Evaluation
Functional Evaluation
21
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation
Capturing
Validation
Representation
Semantics of change
Propagation
Implementation

Structural Evaluation
Functional Evaluation
Pragmatics Evaluation
22
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation
Capturing
Validation
Representation
Semantics of change
Propagation
Implementation
User is able to approve the changes applied or
to reverse them

Structural Evaluation
Functional Evaluation
Pragmatics Evaluation
23
Controlled Ontology Evolution through Evaluation
Controlled Ontology Evolution through Evaluation
Capturing
Validation
Representation
Semantics of change
Propagation
Implementation

Structural Evaluation
Functional Evaluation
Pragmatics Evaluation
24
Overview

• Motivation
• Semiotic-based Ontology Evaluation
• Controlled Ontology Evolution through Evaluation
• Semiotic-based Ontology Evaluation Tool
• Implementation
• Evaluation Results
• Conclusion

25
Semiotic-based Ontology Evaluation Tool

Structural Evaluation
Functional Evaluation
Pragmatics Evaluation
26
Semiotic-based Ontology Evaluation Tool

Structural Evaluation
Functional Evaluation
Pragmatics Evaluation
User recognition (documentation, versioning)
Consistent intended use, context or usage
(domain coverage, performance, etc)
Adequate representation form (syntactic
correct) Formal consistent
27
Semiotic-based Ontology Evaluation Tool
Syntax
Semantics
Pragmatics
28
Semiotic-based Ontology Evaluation Tool
Validation
Representation
Semantics of change
Propagation
Implementation
29
Semiotic-based Ontology Evaluation Tool
Semiotic Measures
Implementation

• Formal Semantics
• Consistency Checking

Structural Measures
Functional Measures

• Cognitive Semantics
• Task-based Approach

• Pragmatics
• Annotation Analysis

30
Semiotic-based Ontology Evaluation Tool
Semiotic Measures
Implementation

• Consistency Checking
• onto changes remains consistent

Representation
Semantics of change

• Task-based Approach
• onto. changes max. performance

Semantics of change
Propagation

• Annotation Analysis
• changes reported for versioning

Validation
31
Consistency Checking
a logical theory is consistent if it does not
contain a contradiction, or, more precisely, for
no proposition f are both f and f provable.
Person
Seal
Shark (primitive class) Animal and eats some
(Person and Seal)
Disjoint (Person, Seal)
Inconsistent

• Use of the reasoners RACER System and Pellet.

32
Task-based Approach

• How effective a given ontology is in the light of
  a well-defined task (Porzel, 2004 Maedche
  Staab,2002)

Task
Application
Ontology
Performance Results
Compare with Gold Standard Answers
Improvements ?
33
Task-based Approach
Is my evolved ontology still good (or better) for
my intend of use?

• Evaluate different ontology versions!

Ontology V0.1
changes
Ontology V0.2
max. performance for a specific task!
34
Task-based Approach

• How efficient is the system to answers questions
  using just ontologies

Question-Answering
SmartWeb
SWIntO V0.2
Performance Results
V0.1
Compare to Gold Standard Answers
Improvements
35
Task-based Approach
1. Plug the Gold Standard Ontology into the system
2. Query the system
3. Check Time Performace
4. Gold Standard Answers Generation
36
Task-based Approach
1. Plug the Evolved Ontology
2. Query the system
3. Check Time Performance

• Lexicon
• Taxonomy
• Semantic Relations

4. Compare with GS Answers
5. Make Report
6. Apply changes!
37
Task-based Approach

• Comparing Ontologies (Maedche Staab,2002
  Dellschaft Staab, 2006)

root
root
accomodation
area
hotel
area
located-at
hotel
youth hostel
wellness hotel
city
city
located-at
Lexicon(hotel, hotel) 1 Taxonomy(hotel)
hotel, accomodationnhotel,wellness hotel
? Semantic(located-at)vhotel,
accomodationnhotel city,areanarea
v ½ ½ ¼
38
Usability-Related Evaluation

• Annotation Analysis Quantitative analysis of the
  amount of metadata linked to the tag
  rdfcomments

ltowlAnnotationProperty rdfIDstructural/gt ltowl
AnnotationProperty rdfIDfunctional/gt ltowlAnn
otationProperty rdfIDuser-oriented/gt ltowlAnn
otationProperty rdfIDconsistencygt
ltrdfssubPropertyOf rdfresourcestructural/gt lt
/owlAnnotationPropertygt ltowlAnnotationProperty
rdfIDtask-assessmentgt ltrdfssubPropertyOf
rdfresourcefunctional/gt lt/owlAnnotationPrope
rtygt
ltowlClass rdfID"Teacher"gt
ltrdfscommentgtTeacher Class
lt/rdfscommentgt ltrdfssubClassOf
rdfresource"Person"/gt lt/owlClassgt
39
Overview

• Motivation
• Semiotic-based Ontology Evaluation
• Controlled Ontology Evolution through Evaluation
• Semiotic-based Ontology Evaluation Tool
• Evaluation Results
• Conclusion

40
Evaluation Results

• SWIntO Ontology (SmartWeb Project)
• Foundational (DOLCE) and general (SUMO) knowledge
• Domain- and task-specific knowledge
• Football (soccer) entities and events
• Navigation
• Linguistic information
• Discourse
• Multimedia
• http//www.smartweb-project.org/

SmartDOLCEEntity


SmartSUMOAttribute
SmartSUMOSocialRole

SportEventFootballPlayer
SportEventFootballOrganizationPerson




41
Consistency Checking
42
Functional Evaluation I
43
Functional Evaluation II
SWIntO V.0.3.2
SWIntO V.0.3.3
Q1Which matches took place in the semifinals in
1954?
Time-performance31,10 ms GS-performance26,23
ms Vocabulary Overlap 100 Hierarchy Overlap
87 Relation Overlap 45
Evaluated Relation GS Relation
Q1Which matches took place in the semifinals in
1954?
Time-performance31,10 ms GS-performance26,23
ms Vocabulary Overlap 100 Hierarchy Overlap
87 Relation Overlap 45 List of Overlap
Descriptions
Evaluated Relation GS Relation
Q2Who was the world champion in 1990?
Q2Who was the world champion in 1990?
44
Functional Evaluation III
45
Annotation Analysis
46
Conclusion

• Evaluation framework to support and control
  ontology evolution
• Apply changes to an ontology keeping its quality
  with respect to the purpose of the ontology (or
  the purpose of the ontology changes)
• Controlled evolution by assessing the quality of
  the ontology with respect to all semiotic
  dimensions -gt Ontology changes captured by
  ontology evaluation process
• Implementation choose three measures which are
  essential in any ontology evolution/evaluation
  process
• Structural Dimension Consistency Checking
• Functional Dimension Task-based Evaluation
• Usability Dimension Annotation Analysis
• Future Work
• level of granularity integration

47

• Thank you for your attention!

48
References

• RACER System, Renamed abox and concept expression
  reasoner . http//www.racer-system.com
• R. Porzel and R. Malaka, A task-based approach
  for ontology evaluation, 2004.
• M. Ciaramita J. Lehmann A. Gangemi, C. Catenacci,
  Modelling ontology evaluation and validation.
• V. Sugumaran A. Burton-Jones, V. C. Storey and P.
  Ahluwalia, A semiotic metrics suite for assessing
  the quality of ontologies, 2004.
• Janez Brank, Marko Grobelnik and Dunja Mladenic,
  A Survey of Ontology Evaluation Techniques.
• W. Wahlster, Smartweb Mobile applications of the
  semantic web, In Proceedings of Informatik.
  2004s.
• P. Buitelaar and A. Frank,Ontology-driven
  Predicate-Argument Structure Analysis for Event
  Annotation
• John F. Sowa, Ontology, Metadata, and Semiotics,
  2000.
• P. Cimiano,Text Analysis and Ontologies, 2006.
• C.W. Morris, Foundations of a theory of signs.
  In International Encyclopedia of Unified Science
  (O. Neurath, R. Carnap C. Morris, eds), Chicago
  University Press, Chicago, pp. 77-138. 1938.
• I. Niles and A. Pease .Towards a Standard Upper
  Ontology, 2001.
• Y. Sure and R. Studer. On-to-knowledgemethodology
  - final version. Technical Report Deliverable 18,
  Institute AIFB, University of Karlsruhe, 2002.
• A. Gangemi, C. Catenacci, M. Ciaramita, and J.
  Lehmann. Qood grid A metaontology-based
  framework for ontology evaluation and selection.
  In Proceedings of EON2006, 2006.
• N. Guarino.. Towards a Formal Evaluation of
  Ontology Quality, pages 15411672. IEEE
  Intelligent Systems, 2004.
• D.Orbele et al. Dolce ergo sumo On foundational
  and domain models in swinto (smartweb integrated
  ontology), 2006..
• N. Noy. Evaluation by ontology consumers. pages
  15411672. IEEE Intelligent Systems, 2004.
• K. Dellschaft and S. Staab. On How to Perform a
  Gold Standard Based Evaluation of Ontology
  Learning, 2006
• A. Maedche and S. Staab.Measuring Similarity
  between Ontologies, 2002.