Modern Information Retrieval: A Brief Overview

1
Modern Information Retrieval A Brief Overview

• By
• Amit Singhal
• Ranjan Dash

2
Layout

• History
• Models Implementations
• Evaluation
• Key Techniques
• Term Weighting
• Query Modification
• Other Techniques and Applications
• Conclusion

3
History

• Starts from 3000BC with Sumerians
• The major IR developments starts in 1950s and
  1960s
• 1950s Vannevar Bush, Luhn
• 1960s
• SMART system Gerald Salton
• Cranfield Evaluation Cyril Cleverdon
• 1970s 1980s
• Various models for document retrieval on small
  text collection
• 1992
• TREC Text Retrieval Conference
• Other fields like retrieval of spoken
  information, non-English language retrieval, info
  filtering,
• Modern Textual IR WWW search 1996 - 1998

4
Models Implementations

• IR systems
• Boolean systems
• Ranked Retrieval Systems
• Models
• Vector space model
• Probabilistic Model
• Inference Network Model
• Implementation

5
Models Implementations..

• Vector space model
• Every word in vocabulary as independent dimension
• Document or query as vectors in this high
  dimensional space
• Positive quadrant of vector space
• Numeric similarity between query vector and
  document vector cosine of the angle between
  them.

6
Models Implementations..

• Probabilistic Model Probabilistic Ranking
  Principle(PRP)
• Ranked by decreasing probability of their
  relevance to a query
• Maron and Kuhn - 1960
• Probability of relevance for doc D

P(RD)


7
Models Implementations..
Assumptions
8
Models Implementations..

• Inference Network Model
• Inference process in an inference network
• A document instantiates a term with a certain
  strength and credit from multiple terms is
  accumulated
• Strength of instantiation of a term weight
• Document ranking for this model Vector space or
  probabilistic models

9
Models Implementations..

• Implementation
• Inverted list
• Stop words

• Stemming little effective for English,
  effective for language with many word inflections
  German
• Multiword phrases
• Techniques to generate list of phrases
  linguistic, statistical

10
Evaluation

• Objective evaluation
• Cranfield Tests
• Characteristics for search effectiveness
• Recall proportion of relevant documents
  retrieved by the system
• Precision proportion of the retrieved documents
  that are relevant
• Average Precision averaging precisions at
  different recall points

11
Key Techniques

• Term weight
• Term frequency
• Raw tf non optimal
• Dampened tf ( logarithmic tf) better one
• Okapi weighting
• Pivoted normalization weighting
• Document frequency
• Document length
• Query modification/expansion via relevance
  feedback

12
Key Techniques

• Query modification/expansion
• Adding synonyms lack of query context
• Relevance feedback Rocchio in 1965
• User judgment to modify the query
• Quite effective
• Pseudo-feedback for short user query
• Top few docs retrieved by initial user query are
  relevant and does relevance feedback to
  generate a new query

13
Other Techniques and Applications

• Cluster Hypothesis Documents that cluster
  together have similar relevance profile for a
  query
• Natural Language Processing ( NLP )
• Not so effective for IR
• Other IR fields besides doc ranking
• Information Filtering (IF), Topic Detection and
  Tracking ( TDT), Speech Retrieval, Cross-language
  retrieval

14
Conclusion

• 40 yrs of experience for IR
• Statistical techniques are the BEST