Maximum Likelihood Estimation for Information Thresholding

1
Maximum Likelihood Estimation for Information
Thresholding

• Yi Zhang Jamie Callan
• Carnegie Mellon University
• yiz,callan_at_cs.cmu.edu

2
Overview

• Adaptive filtering definition and challenges
• Threshold based on score distribution and the
  sampling bias problem
• Maximum likelihood estimation for score
  distribution parameters
• Results of Experiments
• Conclusion

3
Adaptive Filtering
Given an initial description of information
needs, a filtering system sifts through a stream
of documents,and delivers relevant documents to a
user as soon as the document arrives. Relevance
feedback maybe available for some of the
delivered documents, thus user profiles can be
updated adaptively.
?
4
Adaptive Filtering

• Three major problems
• Learning corpus statistics, such as idf
• Learning user profile, such as adding or deleting
  key words and adjusting term weights. (Scoring
  method)
• Learning delivery threshold. (Binary judgment)
• Evaluation Measures
• Linear utility r1RRr2NRr3RNr4NN
• Optimizing linear utility gt Finding
  P(relevantdocument)
• In one dimension P(relevantdocument)
  P(relevantscore)
• F measure

5
A Model of Score Distribution Assumptions and
Empirical Justification

• Relevant
• Non-relevant
• According to other researchers, this is generally
  true for various statistical searching systems
  (scoring methods, Manmathas paper, Arampatziss
  paper)

Figure 1. Density of document scores TREC9 OHSU
Topic 3 and Topic 5
6
Optimize for Linear Utility Measure from Score
Distribution to Probability of Relevancy

• p p(r) ratio of relevant documents

7
Optimize for F Measure From Score Distribution
to Precision and Recall
If set threshold at ?
8
What We Have Now?

• A model for score distribution
• Algorithms to find the optimal threshold for
  different evaluation measures given the model
• Learning task find the parameters for the model?

9
Bias Problem for Parameter Estimation while
Filtering

• We only receive feedback for documents delivered
• Parameter estimation based on random sampling
  assumption is biased
• Sampling criteria depends on threshold, which
  changes over time
• Solution maximum likelihood principle, which is
  guaranteed to be unbiased

Figure Estimation of parameters for relevant
document scores of TREC9 OHSU Topic 3 with a
fixed dissemination threshold 0.4435
10
Unbiased Estimation of Parameters Based on
Maximum Likelihood Principle (1)
ML the best estimation of parameters is the one
that maximizes the probability of training data
11
Unbiased Estimation of Parameters Based on
Maximum Likelihood Principle (2)
For each item inside the sum operation of the
previous formula
12
Unbiased Estimation of Parameters Based on
Maximum Likelihood Principle (3)
Calculating the denominator
13
Unbiased Estimation of Parameters Based on
Maximum Likelihood Principle (4)

• For a relevant document delivered

• For a non-relevant document delivered

14
Relationship to Arampatziss Estimation

• If no threshold exists
• The previous formula becomes

• For a relevant document delivered

• For a non-relevant document delivered

Corresponding result will be the same as
Arampatziss
15
Unbiased Estimation of Parameters Based on
Maximum Likelihood Principle (5)

• Optimization using conjugate gradient descent
  algorithm
• Smoothing using conjugate prior
• Prior for p beta distribution
• Prior for variance
• Set

16
Experimental Methodology (1)

• Optimization goal (similar to the measure used by
  TREC9)
• T9U2Relevant_Retrieved-Non_Relevant_Retrieved
  2RR-NR
• Corresponding rule deliver if
• Dataset
• OHSUMED data (348566 articles from 1887 to 1991.
  63 OHSUMED queries and 500 MeSH headings to
  simulate user profiles)
• FT data (210158 articles from Financial Times
  1991 to 1994. TREC topics 351-400 to simulate
  user profiles)
• Each profile begins with 2 relevant documents and
  an initial user profile
• No profile updating for simplicity.

17
Experimental Methodology (2)

• Four runs for each profile
• Run1 biased estimation of parameters because
  sampling bias was not considered
• Run3 maximum likelihood estimation.
• Both runs will stop delivering documents if the
  threshold is set too high, especially in the
  early stages of filtering. We introduced a
  minimum delivery ratio If a profile has not
  achieved the minimum delivery ratio, its
  threshold will be decreased automatically
• Run 2 biased estimation minimum delivery ratio
• Run 4 maximum likelihood estimation minimum
  delivery ratio
• Time 21 minutes for the whole process of 63 OHSU
  topics on 4 years of OHSUMED data (ML algorithm)

18
Results OHSUMED Data
19
Results Financial Times
Run 1 Biased estimation Run 2 Biased estimation min. delivery ratio Run 3 Unbiased estimation Run 4 Unbiased estimation min. delivery ratio
T9U utility 1.44 -0.209 0.65 0.84
Avg. docs. Delivered per profile 9.58 10.44 9.05 12.27
Precision 0.20 0.17 0.22 0.26
Recall 0.161 0.167 0.15 0.193
20
Result Analysis Difference Between Run 4 and Run
2 on TREC9 OHSU Topics
Utility ML - Biased Docs
deliveredML -Biased
Topics
Topics

• For most of the topics, ML (Run 4) delivered
  more documents than Run 2

• For some of the topics , ML (run 4) has a much
  higher utility than Run 2, while they are similar
  in most of the other topics

21
Conclusion

• Score density distribution
• Relevant documents normal distribution
• Non-relevant documents exponential distribution
• Bias problem due to non-random sampling can be
  solved based on the maximum likelihood principle
• Significant improvement in the TREC-9 filtering
  task.
• Future work
• Thresholding while updating profiles
• Non-random sampling problem in other task