Learning From Measurements in Exponential Families

1
Learning From Measurements in Exponential Families

• Percy Liang, Michael I. Jordan and Dan Klein
• ICML 2009

Presented by Haojun Chen
Images in these slides are from Percy Liangs
paper and slides
2
Motivation

• Problem
• Given a set of unlabeled examples and a
  model, one can either label some examples or
  impose general constraints to provide information
  to learn the predicator of the model.
• Example Craigslist advertisement
• Measurements is introduced to provide a unified
  framework for integrating both labels and
  constraints in a coherent manner.

3
Measurements

• a sequence of input
• corresponding hidden
  output
• Measurement values

4
Measurement Examples

• Fully-labeled example
• To represent the output of , let the
  components of include
• Example
• Labeled predicate
• For sequence labeling tasks, if input is
  , we want to know the frequency of some label
  overall positions. The measurements are
• 
  where is the length of the sequence.
• Example

5
From Measurements to Model

• Goal learn a predictor based on observed
  measurements
• Predictor conditional exponential families
• e.g., linear regression, logistic regression
  and conditional random field
• Exponential families definition
• Bayesian Model

6
Approximate Inference

• Variational formulation
• where
  
• Objective function
• Algorithm
• Take alternating stochastic gradient steps

7
Craigslist Results

• Data 1000 advertisements, 11 possible labels
• Measurements
• Fully-labeled examples
• Label predicate
• Model
• Linear-chain conditional random field (CRF)

8
Active Measurement Selection

• Utility of measurement
• Best subsequent measurement

where
9
Active Learning Algorithm

• Define
• Algorithm

, the full algorithm does come with a
significant computational cost,
10
Toy Data Results

• Input space
• Output space
• Measurements
• Fully-labeled examples
• Label predicate
• Start with 100 unlabeled data and test on 1000
  data

11
Part-of-speech Tagging Results

• Data 1000 sentences, 45 possible labels
• Measurements
• Fully-labeled examples
• Label predicate
• Model
• Independent logistic regression