CSE484 Introduction to Information Retrieval - PowerPoint PPT Presentation

About This Presentation
Title:

CSE484 Introduction to Information Retrieval

Description:

Title: Linear Model (III) Author: rongjin Last modified by: Rong Created Date: 1/27/2004 1:40:44 AM Document presentation format: On-screen Show (4:3) – PowerPoint PPT presentation

Number of Views:180
Avg rating:3.0/5.0
Slides: 41
Provided by: rong7
Learn more at: http://www.cse.msu.edu
Category:

less

Transcript and Presenter's Notes

Title: CSE484 Introduction to Information Retrieval


1
CSE484Introduction to Information Retrieval
  • Rong Jin

2
Why Information Retrieval?Information Overload
  • The amount of data generated on the Internet
    every minute
  • YouTube users upload 48 hours of video,
  • Facebook users share 684,478 pieces of content,
  • Instagram users share 3,600 new photos,
  • The global Internet population is 2.1 billion
    people

http//www.visualnews.com/2012/06/19/how-much-data
-created-every-minute/
3
What is Information Retrieval
  • Information Retrieval (IR) is the study of
    unstructured data
  • usually text, but also audio and images
  • Data is considered unstructured when
  • the structure is unknown, and
  • the semantics of each component are unknown
  • e.g., bank, nut, sun, white house
  • IR systems exploit statistical regularities in
    the data
  • without trying to understand what they mean
  • Contrasting approaches
  • RDBMS systems deal with structured data
  • NLP systems try to find the meaning (semantics)
    in unstructured text

4
Grading Policy
  • Homework (70)
  • Project/Competition (30)

5
Homework (70)
  • Problem (9)
  • demonstrate knowledge of specific techniques
  • implement components within existing search
    engines
  • Late policy (no excuse and no mercy!)
  • 90 credits after one day
  • 50 credits after two days
  • 25 credits afterwards

6
Project (30)
  • Purpose
  • Hands-on experience on the real applications
  • Topic
  • Nearly Duplicate Image Retrieval

7
Project (30)
b2
b1
b5
b8
b3
b7
b6
b4
b1 b2 b3 b4
Bag of Word Representation
Clustering
8
Project (30)
  • Each team with no more than 2 students

Timetable
11/02/2013 Issue the package for image processing and image datasets
12/06/2013 Presentations evaluation (evaluate by our classmates)
9
Support System
  • Textbook
  • Search Engines Information Retrieval in Practice
    (SEIR), by W. Bruce Croft, Donald Metzler, and
    Trevor Strohman, Addision Wesley, 2010
  • Other readings see the course web page
  • Course web page
  • http//www.cse.msu.edu/cse484
  • Syllabus, homework assignments, lecture notes,
    readings, etc.
  • Office hours
  • Myself Mon/Wed 430pm-530pm
  • Feel free to contact me by rongjin_at_cse.msu.edu

10
Support System
  • Ask Questions!!!

11
Why is IR Important?
  • Most communication between humans is unstructured
    information
  • text, images, audio
  • It is becoming common to store information in
    electronic form
  • word processing systems have been common for
    20-30 years
  • storage devices (e.g., disks) have become very
    inexpensive
  • The Internet provides easy access to information
    worldwide, but
  • the information may be disorganized, or organized
    for other uses
  • the information you need may be in a language you
    dont speak

12
What is a Document?
  • Examples
  • web pages, email, books, news stories, scholarly
    papers, text messages, Word, Powerpoint, PDF,
    forum postings, patents, etc.
  • Common properties
  • Significant text content
  • Some structure (e.g., title, author, date for
    papers subject, sender, destination for email)

13
Documents vs. Database Records
  • Database records (or tuples in relational
    databases) are typically made up of well-defined
    fields (or attributes)
  • e.g., bank records with account numbers,
    balances, names, addresses, social security
    numbers, dates of birth, etc.
  • Easy to compare fields with well-defined
    semantics to queries in order to find matches
  • Text is more difficult

14
Documents vs. Records
  • Example bank database query
  • Find records with balance gt 50,000 in branches
    located in Amherst, MA.
  • Matches easily found by comparison with field
    values of records
  • Example search engine query
  • bank scandals
  • This text must be compared to the text of entire
    news stories

15
Comparing Text
  • Comparing the query text to the document text and
    determining what is a good match is the core
    issue of information retrieval
  • Exact matching of words is not enough
  • Many different ways to write the same thing in a
    natural language like English
  • e.g., does a news story containing the text bank
    director steals funds match the query?
  • Some stories will be better matches than others

16
Big Issues in IR
  • Relevance
  • What is it?
  • Simple (and simplistic) definition A relevant
    document contains the information that a person
    was looking for when they submitted a query to
    the search engine
  • Many factors influence a persons decision about
    what is relevant e.g., task, context, novelty,
    style
  • Topical relevance (same topic) vs. user relevance
    (everything else)

17
Big Issues in IR
  • Relevance
  • Retrieval models define a view of relevance
  • Most models describe statistical properties of
    text rather than linguistic
  • i.e. counting simple text features such as words
    instead of parsing and analyzing the sentences
  • Statistical approach to text processing started
    with Luhn in the 50s
  • Linguistic features can be part of a statistical
    model

18
Big Issues in IR
  • Evaluation
  • Experimental procedures and measures for
    comparing system output with user expectations
  • Originated in Cranfield experiments in the 60s
  • Typically use test collection of documents,
    queries, and relevance judgments
  • Most commonly used are TREC collections
  • Recall and precision are two examples of
    effectiveness measures

19
Big Issues in IR
  • Information Needs
  • Search evaluation is user-centered
  • Keyword queries are often poor descriptions of
    actual information needs
  • Interaction and context are important for
    understanding user intent
  • Query refinement techniques such as query
    expansion, query suggestion, relevance feedback
    improve ranking

20
IR and Search Engines
  • A search engine is the practical application of
    information retrieval techniques to large scale
    text collections
  • Web search engines are best-known examples, but
    many others
  • Open source search engines are important for
    research and development
  • e.g., Lucene, Lemur/Indri, Galago
  • Big issues include main IR issues but also some
    others

21
Search Engine Issues
  • Performance
  • Measuring and improving the efficiency of search
  • e.g., reducing response time, increasing query
    throughput, increasing indexing speed
  • Indexes are data structures designed to improve
    search efficiency
  • designing and implementing them are major issues
    for search engines

22
Search Engine Issues
  • Dynamic data
  • The collection for most real applications is
    constantly changing in terms of updates,
    additions, deletions
  • e.g., web pages
  • Acquiring or crawling the documents is a major
    task
  • Typical measures are coverage (how much has been
    indexed) and freshness (how recently was it
    indexed)
  • Updating the indexes while processing queries is
    also a design issue

23
Search Engine Issues
  • Scalability
  • Making everything work with millions of users
    every day, and many terabytes of documents
  • Distributed processing is essential
  • Adaptability
  • Changing and tuning search engine components such
    as ranking algorithm, indexing strategy,
    interface for different applications

24
Spam
  • For Web search, spam in all its forms is one of
    the major issues
  • Affects the efficiency of search engines and,
    more seriously, the effectiveness of the results
  • Many types of spam
  • e.g. spamdexing or term spam, link spam,
    optimization
  • New subfield called adversarial IR, since
    spammers are adversaries with different goals

25
Dimensions of IR
  • IR is more than just text, and more than just web
    search
  • although these are central
  • People doing IR work with different media,
    different types of search applications, and
    different tasks
  • E.g., video, photos, music, speech

26
Dimension of IR
From the Jamie Callans lecture slide
27
Dimensions of IR
Content Applications Techniques
Text Web search Ad hoc search
Images Vertical search Filtering
Video Enterprise search Classification
Scanned docs Desktop search Question answering
Audio Forum search
Music P2P search
Literature search
28
Web Search
29
Text Categorization
30
Text Categorization
  • Open directory project
  • the largest human-edited directory of the Web
  • Manual classification
  • 5,247,409 sites, 99,075 editors, and 1,019,403
    categories
  • Need to automate the classification process

31
Question Answering
32
Image Retrieval
33
Image Retrieval
34
Image Retrieval
35
Image Retrieval using Texts
36
Image Retrieval using Texts (Flickr)
37
Document Summarization
38
Document Summarization
39
Recommendation Systems
40
One More Reason for IR
1,000,000 award
Write a Comment
User Comments (0)
About PowerShow.com