Relevance Feedback

1
Relevance Feedback

• User tells system whether returned/disseminated
  documents are relevant to query/information need
  or not
• Feedback
• usually positive
• sometimes negative
• always incomplete
• Hypothesis relevant docs should be more like
  each other than like non-relevant docs

2
Relevance Feedback Purpose

• Augment keyword retrieval Query Reformulation
• give user opportunity to refine their query
• tailored to individual
• exemplar based different type of information
  from the query
• Iterative, subjective improvement
• Evaluation!

3
Relevance Feedback Examples

• Image Retrieval
• http//www.cs.bu.edu/groups/ivc/ImageRover/
• http//nayana.ece.ucsb.edu/imsearch/imsearch.html
• http//www.mmdb.ece.ucsb.edu/demo/corelacm/

4
Relevance Feedback Early Usage by Rocchio

• Modify original keyword query
• strengthen terms in relevant docs
• weaken terms in non-relevant docs
• modify original query by weighting based on
  amount of feedback

5
Relevance Feedback Early Results

• Evaluation
• how much feedback needed
• how did recall/precision change
• Conclusion
• improved recall precision over even 1 iteration
  and return of up to 20 non-relevant docs
• Promising technique

6
Query Reformulation

• User does not know enough about document set to
  construct optimal query initially.
• Querying is iterative learning process repeating
  two steps
• expand original query with new terms (query
  expansion)
• assign weights to the query terms (term
  reweighting)

7
Query Reformulation Approaches

• Relevance feedback based
• vector model (Rocchio )
• probabilistic model (Robertson Sparck Jones,
  Croft)
• Cluster based
• Local analysis derive information from retrieved
  document set
• Global analysis derive information from corpus

8
Vector Based Reformulation

• Rocchio (1965)with adjustable weights
• Ide Dec Hi (1968) counts only the most similar
  non-relevant document

9
Probabilistic Reformulation

• Recall from earlier
• still need to estimate probabilities
• do so using relevance feedback!

10
Estimating Probabilities by Accumulating
Statistics

• Dr is set of relevant docs
• Dr,i is set of relevant docs with term ki
• ni is number of docs in corpus containing term ki

11
Computing Similarity (Term Reweighting)

• assume term independence and binary document
  indexing
• Cons no term weighting, no query expansion,
  ignores previous weights

12
Croft Extensions

• include within document frequency weights
• initial search variant

Last term is normalized within-document
frequency. C and K are adjustable parameters.
13
Query Reformulation Summary so far

• Relevance feedback can produce dramatic
  improvements.
• However, must be careful that previously judged
  documents are not part of improvement and
  techniques have limitations.
• Next round of improvements requires clustering

14
Croft Feedback Searches

• Use probability updates as in Robertson

15
Assumptions

• Initial query was a good approximation.
• Ideal query is approximated by shared terms in
  relevant documents.

16
Assumptions

• Initial query was a good approximation.
• polysemy? synonyms?
• slang? concept drift?
• Ideal query is approximated by shared terms in
  relevant documents.