Learning Entity Specific Models

1
Learning Entity Specific Models

• Stefan Niculescu
• Carnegie Mellon University
• November, 2003

2
Outline

• Introduction
• Learning Entity Specific Models from complete
  data
• Inference in Entity Specific models
• EM for learning Entity Specific models from
  incomplete data
• Learning in presence of simple Domain Knowledge
• Summary / Future Work

3
Entities and Examples

• Today, huge databases track the evolution of
  users, companies or other entities/objects across
  time
• Multiple examples are collected per entity
• Hospitals (Entities) treat many patients
  (Examples)
• Users (Entities) are observed when handling
  various Emails (Examples)
• In fMRI experiments, a Subject (Entity) is
  observed across few tens of Trials (Examples)

4
Entity Specific and General Knowledge

• Each Entity has its own particularities
• Different number of attributes may be available
  for each entity
• Different Hospitals may perform different tests
  on their patients to diagnose a given disease
• A certain User may have some software installed
  while others may not
• Even two attributes that are present in two
  entities may relate in a complete different ways
• However, there are things that are common across
  entities
• Treatments for a disease are the same across
  hospitals

5
Goals and Approaches

• GOAL Make inference about new examples
• From available entities
• From a new entity
• TWO EXTREME APPROACHES
• Learn a General Model by combining all the
  examples available
• May have different attributes for different
  entities
• Entity Specific params will be an Average
  across all Entities
• BAD for making inference about existing Entities
  
• Learn a separate Model for each Entity
• May not have enough data to learn some
  dependencies accurately
• Cannot be used when a new example comes from a
  previously unseen Entity

6
Outline

• Introduction
• Learning Entity Specific Models from complete
  data
• Inference in Entity Specific models
• EM for learning Entity Specific models from
  incomplete data
• Learning in presence of simple Domain Knowledge
• Summary / Future Work

7
Entity Specific Models

• Proposed approach build a model that is
  somewhere between the two extremes
• Takes advantage of multiple Entities to better
  learn General Knowledge
• Also adapts itself to whatever is specific to
  each Entity
• Bayes Nets will be adapted to deal with the two
  issues

8
Assumptions

• Examples independent given the parameters of the
  distribution
• There is no uncertainty about the entity
• This is the case in our studies
• Data is fully observable (no missing values)
• Entities have the same sets of attributes
• Model Bayes Net
• Structure of the Bayes Net is the same for all
  entities
• Parameters of the Bayes Net may vary from entity
  to entity
• It is known which parameters are General and
  which Entity Specific

9
Notations
10
Conditional Probability Tables
CPT for Entity e1
CPT for Entity e2
11
Maximum Data Likelihood setting

• Find the hypothesis that maximizes the likelihood
  of the data
• Assumes that all hypotheses have equal priors

• Constraints for all entities, each column of
  their CP tables should sum up to 1 !!!

12
Optimizing using Lagrange Multipliers

• Can split in a set of independent optimization
  problems

• Apply Lagrange Multipliers Theory
• Solution of Pik is among solutions of
• where

13
Optimizing using Lagrange Multipliers

• Sanity Check If all parameters are general (no
  entity specific params), then this is equivalent
  to a normal Bayes Net
• Sanity Check If all parameters are entity
  specific, first fraction cancels and we have a
  collection of independent Bayes Nets

14
Outline

• Introduction
• Learning Entity Specific Models from complete
  data
• Inference in Entity Specific models
• EM for learning Entity Specific models from
  incomplete data
• Learning in presence of simple Domain Knowledge
• Summary / Future Work

15
Inference in Entity Specific models

• TWO CASES
• For a new (partial) example coming from a
  previously SEEN Entity
• Build the Bayes Network corresponding to that
  Entity, then use any existing BN inference
  algorithm
• For a new (partial) example coming from a
  previously UNSEEN Entity
• Average / Weight Average the entity specific
  parameters corresponding to all previously seen
  entities into a General BN
• OR
• Train a General BN based on all seen entities
  gives some prior over all parameters
• Apply the General BN to make inference about the
  new example

16
Outline

• Introduction
• Learning Entity Specific Models from complete
  data
• Inference in Entity Specific models
• EM for learning Entity Specific models from
  incomplete data
• Learning in presence of simple Domain Knowledge
• Summary / Future Work

17
EM for maximizing data likelihood
18
EM for learning Entity Specific models from
incomplete data

• E STEP
• Compute expected counts under current estimated
  parameters
• Because of incomplete data, the counts are
  random variables
• M STEP Reestimate model parameters
• Maximize likelihood under observed expected
  counts

19
Outline

• Introduction
• Learning Entity Specific Models from complete
  data
• Inference in Entity Specific models
• EM for learning Entity Specific models from
  incomplete data
• Learning in presence of simple Domain Knowledge
• Summary / Future Work

20
Conditional Probability Tables

• Given
• Not given Estimate!

• Given for Entity e
• May be unknown for other entities
• Not given for Entity e Estimate!
• May be given for other entities

CPT for Entity e
21
Maximum Data Likelihood setting

• Find the hypothesis that maximizes the likelihood
  of the data
• Assumes that all hypotheses have equal priors

• Constraints for all entities, each column of
  their CP tables should sum up to 1 !!!

22
Optimizing using Lagrange Multipliers

• Can split in a set of independent optimization
  problems

• Apply Lagrange Multipliers Theory
• Solution of Pik is among solutions of
• where

23
Optimizing using Lagrange Multipliers

• THEOREM The ML estimators exist and they are
  unique. In addition, they can be accurately
  approximated by a bisection method.
• Proof (Sketch)
• Given parameters are treated as constants
• Do not differentiate with respect to them !
• Differentiating to unknown parameters we obtain
  
• 
  

24
Proof sketch
25
Proof sketch

• Substituting in the constraints, we easily
  obtain that A is the solution of a polynomial
  equation

• where
• U is strictly increasing on the domain of
  admissible values
• U takes both negative and positive values
• Therefore A exists, is unique and it can be
  determined by a bisection method
• Once A is known, it is trivial to find Be values
  by substituting A in the constraints
• OBSERVATION There is no closed form for the ML
  estimators, but they can be approximated
  arbitrarily close!

26
Outline

• Introduction
• Learning Entity Specific Models from complete
  data
• Inference in Entity Specific models
• EM for learning Entity Specific models from
  incomplete data
• Learning in presence of simple Domain Knowledge
• Summary / Future Work

27
Summary

• Derived ML estimators for Entity Specific Bayes
  Nets
• Modified EM to deal with learning in presence of
  multiple entities
• Proved how simple Domain Knowledge can be
  incorporated in the learning algorithm

28
Future Work

• Test the Entity Specific Bayes Net model on
  artificially generated data
• Incorporate uncertainty about the entity in the
  model
• Modify the model to be able to have different
  network topologies for different entities
• Improve the representation power of the domain
  knowledge that can be incorporated in learning
  maybe probabilistic rules