Treestructured Conditional Random Fields for Semantic Annotation

1
Tree-structured Conditional Random Fields for
Semantic Annotation

• Jie Tang, Mingcai Hong, Juanzi Li, and Bangyong
  Liang
• Knowledge Engineering Group (KEG)
• Department of Computer Science and Technology
• Tsinghua University
• Nov. 5, 2006

2
Outline

• Motivation and Problem Description
• Related Work
• Our Approach
• Experimental Results
• Future work Summary

3
Introduction

• Semantic web requires annotating existing web
  content according to particular ontologies
• Application of semantic annotation
• Personal profile annotation
• Product information annotation
• Image annotation
• Company annual report annotation

4
Example of Semantic Annotation

• Identifying target entities relations
• Populating the ontology base

Task
October 14, 2002, 400 a.m. PT For years,
Microsoft Corporation CEO Bill Gates railed
against the economic philosophy of open-source
software with Orwellian fervor, denouncing its
communal licensing as a "cancer" that stifled
technological innovation. Today, Microsoft
claims to "love" the open-source concept, by
which software code is made public to encourage
improvement and development by outside
programmers. Gates himself says Microsoft will
gladly disclose its crown jewels--the coveted
code behind the Windows operating system--to
select customers. "We can be open source. We
love the concept of shared source," said Bill
Veghte, a Microsoft VP. "That's a super-important
shift for us in terms of code access. Richard
Stallman, founder of the Free Software
Foundation, countered saying
Metadata
Person
Organization
workIn
Name
Title
Name
Instance
Person2
Person1
CEO
Bill Gates
Founder
Richard Stallman
workIn
workIn
Organization1
Organization2
workIn
Person3
Microsoft
Free Software Foundation
Bill Beghte
VP
5
Challenges2D Dependency

• Information is often dependent
• E.g. Two-dimensionally laid-out

2D Dependencies, Zhu (2005) (Horizonal and
Vertical )
6
Hierarchical Semantic Annotation

• Hierarchical dependency

3. Company Directorate Info Company
directorate secretary Haokui Zhou
Representative of directorate He Zhang
Address No. 583-14, Road Linling, Shanghai,
China Zipcode 200030 Email
ajcoob_at_mail2.online.sh.cn Phone
021-64396600 Fax [phone]. Company
Registration Info Company registration
address No. 838, Road Zhang Yang, Shanghai,
China Zipcode 200122 Company office
address No. 583-14, Road Linling, Shanghai,
China Zipcode 200030 Email
ajcorp_at_online.sh.cn Phone [phone]
Dependency
Metadata
Company Basic Info
has_directorate_info
has_registration_info
Company Directorate Info
Company Registration Info
secretary
Dependency
address
reg_address
Email
representative
zipcode
reg_zipcode
phone
Phone
Email
office_address
Fax
office_zipcode
- How to make use of the dependencies in
annotation? - How to formalize a unified model?
7
Outline

• Motivation and Problem Description
• Related Work
• Our Approach
• Experimental Results
• Future work Summary

8
Related WorkSemantic Annotation

• Annotation using Rule Learning
• Learning annotation rules
• E.g. Ciravegna (2001), Handschuh et al. (2002),
  and Popov et al. (2003)
• Annotation using Classification
• Formalizing the annotation problem as that of
  classification
• E.g. Hammond, Sheth, and Kochut (2002)
• Annotation using Sequential Labeling
• Sequential labeling can describe dependencies
  between targeted entities
• E.g. Reeve (2004)

9
Related WorkInformation Extraction

• Classification Models
• E.g. Cortes and Vapnik (1995), Collions (2002),
  and Finn (2004)
• Dependent Models
• E.g. Ghahramani and Jordan (1997), McCallum et
  al. (2000), and Lafferty et al. (2001)
• Non-linear Dependent Models
• E.g. Sutton et al. (2004), Zhu et al. (2005), and
  Bunescu and Mooney (2004)

10
Outline

• Motivation and Problem Description
• Related Work
• Our Approach
• Experimental Results
• Future work Summary

11
Our Approach

• Hierarchical Semantic Annotation
• Information is organized as a tree structure
• E.g. HTML, XML
• Tree-structured Conditional Random Field (TCRF)
• Modeling hierarchical dependencies in a cyclable
  tree
• Performing parameter estimation by maximizing the
  log-likelihood objective function
• Using TRP algorithm to do the inference in the
  parameter estimation

12
Linear Conditional Random Fields
3. Company Directorate Info Company
directorate secretary Haokui Zhou
Representative of directorate He Zhang
Address No. 583-14, Road Linling, Shanghai,
China Zipcode 200030 Email
ajcoob_at_mail2.online.sh.cn Phone
021-64396600 Fax [phone]. Company
Registration Info Company registration
address No. 838, Road Zhang Yang, Shanghai,
China Zipcode 200122 Company office
address No. 583-14, Road Linling, Shanghai,
China Zipcode 200030 Email
ajcorp_at_online.sh.cn Phone [phone]
13
Linear Conditional Random Fields
O
Z1
O
E1
O
Z2
O
E2


Zipcode
200030
Email
ajcoob_at_...
Zipcode
200030
Email
ajcoob_at_...
ajcorp_at_...
14
Tree-structured CRFs (TCRFs)

• In TCRF, the dependencies are organized as a tree
  structure

15
Modeling with TCRFs
A
R
D
O
Z1
O
E1
O
Z2
O
E2


Zipcode
200030
Email
Zipcode
200030
Email
ajcoob_at_...
ajcorp_at_...
3. Company Directorate Info
4. Company Registration Info
16
TCRF Model
Model
Annotation
How to estimate the parameters?
17
Parameter Estimation
(1) With training data D(x(i), y(i)), the
log-likelihood objective function
p(yx) p(yp, ycx) p(yc, ypx) p(ys, ysx)
where T?1, ?2, µk, µk1,
(2) Derivative of the objective function with
respect to a ?j
here - f denotes both the edge feature t and
the vertex feature s - c (clique) denotes
both edge e and vertex v -? denotes the two
kinds of parameters ? and µ.
(3) With the objective function and the
derivative function, we can use any
gradient-based methods (e.g. L-BFGS) to solve the
optimization problem so as to do the parameter
estimation
18
Calculating the Marginal Probabilities

• Tree-based Reparameterization (TRP)
• TRP is based on the fact that any exact algorithm
  for optimal inference on trees actually computes
  marginal distributions for pairs of neighboring
  nodes.
• TRP Algorithm
• Step 1 Initialization
• Step 2 Updates
• Generating a spanning tree
• Propagation on the spanning tree
• Stop if terminations are met

19
TRPStep 1 Initialization
X1
X2
X3
T01kexp(s(x1, y1))
T02kexp(s(x2, y2))
T03kexp(s(x3, y3))
y1
y2
y3
T023kexp(t(x, y2, y3))
T012kexp(t(x, y1, y2))
T014kexp(t(x, y1, y4))
T036kexp(t(x, y3, y6))
T025kexp(t(x, y2, y5))
T056kexp(t(x, y5, y6))
T045kexp(t(x, y4, y5))
y4
y6
y5
T04kexp(s(x4, y4))
T06kexp(s(x6, y6))
T05kexp(s(x5, y5))
X4
X5
X6
20
TRPStep2 a) Generating spanning tree

• Methods Edge cutting and edge adding

X1
X2
X3
y1
y2
y3
y4
y6
y5
X4
X5
X6
21
TRPStep2 b) Propagation
X1
X2
X3
y1
y2
y3
y4
y6
y5
X4
X5
X6
22
Updating T in TRP
23
After The First Iteration
X1
X2
X3
T11
T12
T13
y1
y2
y3
T014
T036
y4
y6
y5
T14
T15
T16
X4
X5
X6
24
Annotation
?
?
?
?
?
?
?
?
?
?
?


Zipcode
200030
Email
Zipcode
200030
Email
ajcoob_at_...
ajcorp_at_...
3. Company Directorat Info
4. Company Registration Info
25
Annotation (cont.)

• Viterbi algorithm

26
Outline

• Motivation and Problem Description
• Related Work
• Our Approach
• Experimental Results
• Future work Summary

27
Experimental Setup
Baselines
Data Sets
(1) SVM
(2) Linear-CRF
Ontology
Features
28
Annotation Results on Synthetic Data
29
Annotation Results on Real Data
30
Time Complexity
Tested on a computer with two 2.8G P4-CPUs and 3G
memory
31
Outline

• Motivation and Problem Description
• Related Work
• Our Approach
• Experimental Results
• Future work Summary

32
Questions

• How to reduce the computational cost?
• Parallelization
• Incorporation of constraints from ontologies
• How to incorporate the other types of
  dependencies into the CRF model?
• E.g. Multiple dimensions
• Long distant dependencies
• How to identify entities relations in a unified
  model?

33
Summary

• Investigated the problem of hierarchical semantic
  annotation
• Proposed a Tree-structured Conditional Random
  Fields for incorporating the hierarchical
  dependencies
• Employed Tree-based Reparameterization (TRP) to
  perform the parameter estimation
• Our approach significantly outperforms the
  baseline methods (SVM and CRF)

34
Thanks!

• HP http//keg.cs.tsinghua.edu.cn/persons/tj/

35
Challenges

• Information of almost all types exist. SA is
  different in different domains