Jaime Teevan - PowerPoint PPT Presentation

1 / 37
About This Presentation
Title:

Jaime Teevan

Description:

The Re:Search Engine Jaime Teevan MIT, CSAIL People Forget a Lot Change Blindness Change Blindness Re:Search Engine Merge Old and New Results Overview Memorability ... – PowerPoint PPT presentation

Number of Views:39
Avg rating:3.0/5.0
Slides: 38
Provided by: peopleCsa8
Category:
Tags: jaime | teevan

less

Transcript and Presenter's Notes

Title: Jaime Teevan


1
  • Amazing

The
ReSearch Engine
  • Jaime Teevan
  • MIT, CSAIL

2
Pick a card, any card.
3
  • Abracadabra!

Case 1 Case 2 Case 3 Case 4 Case 5 Case 6
4
Your Card is GONE!
5
People Forget a Lot
6
Change Blindness
http//www.usd.edu/psyc301/ChangeBlindness.htm
7
Change Blindness
http//www.usd.edu/psyc301/ChangeBlindness.htm
8
ReSearch Engine
?
9
Merge Old and New Results
Old
Merged
New
10
We still need magic!
11
Overview
  • Memorability study
  • Recognition study
  • Assumptions
  • Implementation issues
  • Evaluation issues
  • Choose your own adventure

12
Memorability Study
  • Participants issued self-selected query
  • After an hour, asked to fill out a survey
  • 129 people remembered something

13
Data Analysis
  • Probability of being remembered
  • Anything? of words? of fields?
  • Features
  • Result features clicked, not clicked, last
    clicked, rank, dwell time, frequency of visit,
    etc.
  • Query features query type, query length, of
    search in session, elapsed time, etc.
  • Remembered rank v. real rank
  • Map remembered rank to real rank

14
Memorability
15
Remembered Results Ranked High
16
Recognition Study
  • Same set-up as Memorability Study
  • Follow-up survey Results the same?
  • Case 1 New results
  • Case 2 Random 4 same
  • Case 3 Clicked to top
  • Case 4 Same results
  • Case 5 Intelligent merging
  • 165 people completed both steps

19
38
41
66
81
17
Assumptions
  • Re-search v. search
  • Memorable v. relevant
  • Results change v. stay the same
  • Hide change v. show change
  • Forget v. remember as forgettable
  • Merge v. identify old or new

Why?
How to test?
What if Im wrong?
18
Implementation Issues
  • Page of cached result may disappear
  • Multiple result pages
  • Identifying repeat queries
  • Exact query may be forgotten
  • User identified
  • Search sessions are not repeat queries

19
Evaluation Issues
  • Various goals to test
  • Does a merged list look like the original?
  • Does merging make re-finding easier?
  • Is search improved overall?
  • Lab study
  • How to set up re-finding task?
  • Timing differences significant enough?
  • Longitudinal study What to measure?
  • What are good baselines?

20
Choose Your Own Adventure
  • Re-search v. search
  • Memorable v. relevant
  • Results change v. stay the same
  • Hide change v. show change
  • Forget v. remember as forgettable
  • Merge v. identify old or new
  • Implementation issues
  • Evaluation issues

21
Choose Your Own Adventure
  • Re-search v. search
  • Memorable v. relevant
  • Results change v. stay the same
  • Hide change v. show change
  • Forget v. remember as forgettable
  • Merge v. identify old or new
  • Implementation issues
  • Evaluation issues

(Done)
22
Hide Change v. Show Change
  • Why I think change should be hidden
  • Example dynamic menus
  • How to prove
  • New results better, called the same or worse
  • Baseline for testing 2 lists, change explicit
  • What if we should show change?
  • Memorability suggests changes to highlight
  • Other applications where want to hide change

(Done)
23
Memorable v. Relevant
  • Why I think memorability is important
  • Relevance at a future date is what matters
  • Necessary to hide change
  • How to prove
  • Baseline for lab study with target first
  • What if relevance is whats important?
  • Mapping between memorable and relevant
  • Useful related work on implicit feedback

(Done)
24
Re-search v. Search
  • Why I think people repeat searches
  • Information seeking literature
  • Re-finding consistently reported as a problem
  • How to prove
  • Study shows prefer to follow known paths
  • Search log analysis
  • What if people just want to search?
  • Memorable results ranked first
  • Other domains where list consistency matters

(Done)
25
Merge v. Identify Old and New
  • Why I think results should be merged
  • Information need not necessarily one or other
  • People dont like to do extra work
  • How to prove
  • Search log analysis
  • Look at what people do in longitudinal study
  • Lab study timing becomes an issue
  • What if people want to identify query type?
  • Other applications where merging is useful

(Done)
26
Results Change v. Stay the Same
  • Why I think results change
  • How search engines work
  • Personalization and dynamic content
  • How to prove
  • Track query results
  • What if results dont change?
  • Probably will in future applications
  • Existing applications where lists change

(Done)
27
Forget v. Remember as Forgettable
  • Why I think people forget
  • Visual analogy
  • How to prove
  • Lab study Do people find new information?
  • Longitudinal study Ever click on new result?
  • What if remember as forgettable?
  • Build better model of memorability
  • Highlight important changes

(Done)
28
Implementation Issues
  • Page of cached result may disappear
  • Multiple result pages
  • Identifying repeat queries
  • User identified
  • Search sessions are not repeat queries
  • Exact query may be forgotten

(Done)
29
Evaluation Issues
  • Various goals to test
  • Does a merged list look like the original?
  • Does merging make re-finding easier?
  • Is search improved overall?
  • Lab study
  • How to set up re-finding task?
  • Timing differences significant enough?
  • Longitudinal study What to measure?
  • What are good baselines?

(Done)
30
  • Thank you!
  • Jaime Teevan
  • teevan_at_mit.edu

31
Strategies for Finding
Teleporting
Orienteering
32
Why Do People Orienteer?
  • Easier than saying what you want
  • You know where you are
  • You know what you find
  • The tools dont work

33
Structural Consistency Important
All must be the same to re-find the information!
New name
34
Absolute Consistency Unnecessary
New name
Focus on search result lists
35
Query Changes
  • Most changes are simple
  • Capitalization
  • Phrasing
  • Word ordering
  • Word form
  • New queries shorter
  • What about longer time horizons?
  • Recognition v. recall

36
Result List Changes
  • Tracked 10 queries on Google for a year
  • 1.18 of top 10 disappear each week
  • Rate of change likely to increase, e.g.
  • Raw personalization
  • Relevance feedback
  • People forget their queries
  • 28 of queries forgotten within an hour

37
Example neon signs
Write a Comment
User Comments (0)
About PowerShow.com