WIDIT at TREC2005 HARD Track

1
WIDIT at TREC-2005 HARD Track

• Kiduk Yang, Ning Yu, Hui Zhang, Ivan Record,
  Shahrier Akram
• WIDIT Laboratory
• School of Library Information Science
• Indiana University at Bloomington

2
Background

• WIDIT Laboratory
• http//elvis.slis.indiana.edu/
• TREC research group
• http//elvis.slis.indiana.edu/TREC/index.html
• Text REtrieval Conference (TREC)
• http//trec.nist.gov/
• HARD track
• http//ciir.cs.umass.edu/research/hard/guidelines.
  html

3
HARD Track Overview

• Test Collection
• AQUAINT corpus
• English news (1M docs, 3 GB text)
• AP newswire (1998-2000)
• NY Times (1998-2000)
• Xinhua News Agency (1996-2000)
• 50 difficult topics
• Not too many relevant documents
• Low scores in previous experiments
• Task
• Baseline Run
• Retrieve 1000 ranked documents per each topic
• Clarification Forms (CF)
• Create user feedback form to be filled out by
  TREC assessors
• Final Run
• Leverage CF data to improve the baseline result

4
Research Questions

• Baseline Run
• How can IR system handle difficult queries?
• Why are HARD topics difficult? (Harmon Buckley,
  2004)
• lack of good terms
• add good terms
• misdirection by non-pivotal terms or partial
  concept
• identify important terms phrases
• Clarification Form
• What information to get from user?
• How can user help with difficult queries?
• identify good/important terms
• identify relevant documents
• Final Run
• How to apply CF data to improve search results?
• CF-term expanded query
• Rank boosting
• Relevance Feedback

5
WIDIT Strategy

• Baseline Run
• Automatic Query Expansion
• add related terms
• synonym identification, definition term
  extraction, Web query expansion
• identify important query terms
• noun phrase extraction, keyword extraction by
  overlapping sliding window
• Fusion
• Clarification Form
• User Feedback
• identify relevant terms
• identify relevant documents
• Final Run
• Manual Query Expansion
• Post-retrieval Reranking
• Relevance Feedback
• Fusion

6
WIDIT HARD System Architecture
7
QE Overlapping Sliding Window (OSW)

• Function
• identify important phrases
• Assumption
• phrases appearing in multiple fields/sources tend
  to be important
• Algorithm
• Set window size and the number or maximum words
  allowed between words.
• Slide window from left to right in a
  field/source. For each of the phrase it catches,
  look for the same/similar phrase in other
  fields/sources.
• Output the OSW phrase when match is found.
• Change source field/source and repeat step 1 to 3
  till all the fields/sources have been used.
• Application
• Topic fields
• title, description, narrative
• Definition Source
• WordIQ, Google, Dictionary.com, Answers.com

8
QE NLP1 NLP2

• NLP1
• Expand Acronyms/Abbreviations
• uses Web-harvested acronym/abbreviation list
• Identify nouns noun phrases
• uses Brill tagger
• Find synonyms
• queries WordNet
• Find definitions
• queries the Web (WordIQ, Google, Dictionary.com,
  Answers.com )
• NLP2
• Refine noun phrase identification
• uses multiple taggers
• Identify best synset based on term context
• uses sense disambiguation module by NLP group at
  UM
• Identify important terms
• uses OSW on topic fields definitions

9
QE Noun Phrase Identification
AND relation
Minipar
POS tagging
Proper noun phrase
Brills Tagger
Topics
Collins Parser
Noun phrase
Simple phrase
Dictionary phrase
Complex phrase
10
QE Web Query Expansion

• Basic Idea
• Use the Web as a type of thesaurus to find
  related terms (Grunfeld et al., 2004 Kwok et
  al., 2005)
• Method
• Web Query Construction
• construct web query by selecting 5 most salient
  terms from HARD topic
• uses NLP-based techniques and rotating window to
  identify salient terms
• Web Search
• query Google with the Web query
• Result Parsing Term Selection
• parse the top 100 search results (snippets
  document texts)
• extract up to 60 best terms
• uses PIRC algorithm to rank the terms (Grunfeld
  et al., 2004 Kwok et al., 2005)

Web Queries
Google
Web Query Generator
Processed Topics
Selected expansion terms
Search Results
Term Selector
Google Parser
11
QE WebX by Rotating Window
SA

• Rationale
• NLP-based identification of salient/important
  term does not always work
• Related terms to salient/important query terms
  are likely to appear frequently in search results
• Method
• Rotate a 5-word window across HARD topic
  description
• generates m queries for a description of m terms
  (mgt5)
• Query Google
• Merge all the results
• Rank the documents based on their frequency in m
  result lists.
• Select 60 terms with highest weight
  (length-normalized frequency) from top 100
  documents

12
Fusion Baseline Run

• Fusion Pool
• Query Formulation results
• combination of topic fields (title, description,
  narrative)
• stemming (simple plural stemmer, combo stemmer)
• term weights (okapi, SMART)
• Query Expansion results
• NLP, OSW, WQX
• Fusion Formula
• Result merging by Weighted Sum
• FSws ?(wiNSi) where wi is the weight of
  system i (relative contribution of each
  system) NSi is the normalized score of a
  document by system i NSi (Si Smin) / (Smax
  Smin)
• Fusion Optimization
• Training data
• 2004 Robust test collection
• Automatic Fusion Optimization by Category

13
Fusion Overview

• Assumption
• Individual Weakness
• single data source/method/system possesses
  weakness
• Complementary Strengths
• the whole is better than sum of its parts
• Strategy
• Combine a variety of diverse data
  source/method/systems
• Question
• What to combine?
• How to combine?

14
Fusion Optimization

• Conventional Fusion Optimization approaches
• Exhaustive parameter combination
• Step-wise search of the whole solution space
• Computationally demanding when the number of
  parameter is large
• Parameter combination based on past evidence
• Targeted search of restricted solution
  spacei.e., parameter ranges are estimated based
  on training data
• Next-Generation Fusion Optimization approaches
• Non-linear Transformation of fusion component
  scores
• e.g. log transformation to compensate for the
  power law distribution of PageRank
• Hybrid Fusion Optimization
• Semi-automatic Dynamic Tuning (Yang Yu, in
  press)
• Automatic Fusion Optimization by Category (Yang
  et al., in press)

15
Automatic Fusion Optimization
SA
Category 1 Top 10 systems
Category n
Category 2 Top system in each query length
Automatic fusion optimization
performance gaingt threshold?
No
Fetching result sets For different categories
optimized fusion formula
Yes
Results pool
16
Clarification Form

• Objective
• Collect information from the user that can be
  used to improve the baseline retrieval result
• Strategy
• Ask the user to identify and add relevant terms
• validation/filtering of system QE results
• nouns, synonyms, OSW NLP phrases
• manual QE terms that system missed
• free text box
• Ask the user to identify relevant documents
• Problem
• HARD topics tend to retrieve non-relevant
  documents in top ranks
• 3 minute time limit for each CF
• Solution
• cluster top 200 results and select best sentence
  from each cluser
• select best n sentences from top 200 results
• select best sentence from every kth document

17
Clarification Form
SA
18
Final Run
SA

• How to make use of CF data?
• Search with CF-term expanded query
• Boost the ranks of documents
• with CF-terms phrases, BoolAND terms
• that are CF-relevant documents
• Relevance Feeback
• apply Rocchio RF algorithm using CF-relevant
  documents
• OSW Phrases
• Post-retrieval rank boosting
• boost the rank of OSW phrases
• Fusion
• Rerun automatic fusion optimization

19
Results Evaluation Measures

• Mean Average Precision (MAP)
• Average Precision (AP) averaged over queries
• AP sum of precision where relevant item is
  retrieved / number of relevant items in
  the collection
• Single valued measure that reflects performance
  over all relevant documents
• rewards the system that retrieves relevant
  documents at high ranks
• Precision (P)
• P number of relevant items retrieved / total
  number of items retrieved
• A measure of the systems ability to present only
  the relevant items
• R-Precision
• RP precision at rank R, where R number of
  relevant items in the collection
• De-emphasizes the exact ranking of documents

20
Results Overall
21
(No Transcript)
22
References

• Grunfeld, L., Kwok, K.L., Dinstl, N., Deng, P.
  (2004). TREC 2003 Robust, HARD, and QA track
  experiments using PIRCS. Proceedings of the 12th
  Text Retrieval Conference, 510-521.
• Harman, D. Buckley, C. (2004). The NRRC
  Reliable Information Access (RIA) workshop.
  Proceedings of the 27th Annual International ACM
  SIGIR Conference, 528-529.
• Kwok, K. L., Grunfeld, L., Sun, H. L., Deng, P.
  (2005). TREC2004 robust track experiments using
  PIRCS. Proceedings of the 13th Text REtrieval
  Conference.
• Yang, K., Yu, N. (in press).  WIDIT
  Fusion-based Approach to Web Search
  Optimization.  Asian Information Retrieval
  Symposium 2005.
• Yang, K., Yu, N., George, N., Loehrlen, A.,
  MaCaulay, D., Zhang, H., Akram, S., Mei, J.,
  Record, I. (in press). WIDIT in TREC2005 HARD,
  Robust, and SPAM tracks. Proceedings of the 14th
  Text Retrieval Conference.