Title: Jaime Teevan
1The
ReSearch Engine
2Pick a card, any card.
3Case 1 Case 2 Case 3 Case 4 Case 5 Case 6
4Your Card is GONE!
5People Forget a Lot
6Change Blindness
http//www.usd.edu/psyc301/ChangeBlindness.htm
7Change Blindness
http//www.usd.edu/psyc301/ChangeBlindness.htm
8ReSearch Engine
?
9Merge Old and New Results
Old
Merged
New
10We still need magic!
11Overview
- Memorability study
- Recognition study
- Assumptions
- Implementation issues
- Evaluation issues
- Choose your own adventure
12Memorability Study
- Participants issued self-selected query
- After an hour, asked to fill out a survey
- 129 people remembered something
13Data Analysis
- Probability of being remembered
- Anything? of words? of fields?
- Features
- Result features clicked, not clicked, last
clicked, rank, dwell time, frequency of visit,
etc. - Query features query type, query length, of
search in session, elapsed time, etc. - Remembered rank v. real rank
- Map remembered rank to real rank
14Memorability
15Remembered Results Ranked High
16Recognition Study
- Same set-up as Memorability Study
- Follow-up survey Results the same?
- Case 1 New results
- Case 2 Random 4 same
- Case 3 Clicked to top
- Case 4 Same results
- Case 5 Intelligent merging
- 165 people completed both steps
19
38
41
66
81
17Assumptions
- Re-search v. search
- Memorable v. relevant
- Results change v. stay the same
- Hide change v. show change
- Forget v. remember as forgettable
- Merge v. identify old or new
Why?
How to test?
What if Im wrong?
18Implementation Issues
- Page of cached result may disappear
- Multiple result pages
- Identifying repeat queries
- Exact query may be forgotten
- User identified
- Search sessions are not repeat queries
19Evaluation Issues
- Various goals to test
- Does a merged list look like the original?
- Does merging make re-finding easier?
- Is search improved overall?
- Lab study
- How to set up re-finding task?
- Timing differences significant enough?
- Longitudinal study What to measure?
- What are good baselines?
20Choose Your Own Adventure
- Re-search v. search
- Memorable v. relevant
- Results change v. stay the same
- Hide change v. show change
- Forget v. remember as forgettable
- Merge v. identify old or new
- Implementation issues
- Evaluation issues
21Choose Your Own Adventure
- Re-search v. search
- Memorable v. relevant
- Results change v. stay the same
- Hide change v. show change
- Forget v. remember as forgettable
- Merge v. identify old or new
- Implementation issues
- Evaluation issues
(Done)
22Hide Change v. Show Change
- Why I think change should be hidden
- Example dynamic menus
- How to prove
- New results better, called the same or worse
- Baseline for testing 2 lists, change explicit
- What if we should show change?
- Memorability suggests changes to highlight
- Other applications where want to hide change
(Done)
23Memorable v. Relevant
- Why I think memorability is important
- Relevance at a future date is what matters
- Necessary to hide change
- How to prove
- Baseline for lab study with target first
- What if relevance is whats important?
- Mapping between memorable and relevant
- Useful related work on implicit feedback
(Done)
24Re-search v. Search
- Why I think people repeat searches
- Information seeking literature
- Re-finding consistently reported as a problem
- How to prove
- Study shows prefer to follow known paths
- Search log analysis
- What if people just want to search?
- Memorable results ranked first
- Other domains where list consistency matters
(Done)
25Merge v. Identify Old and New
- Why I think results should be merged
- Information need not necessarily one or other
- People dont like to do extra work
- How to prove
- Search log analysis
- Look at what people do in longitudinal study
- Lab study timing becomes an issue
- What if people want to identify query type?
- Other applications where merging is useful
(Done)
26Results Change v. Stay the Same
- Why I think results change
- How search engines work
- Personalization and dynamic content
- How to prove
- Track query results
- What if results dont change?
- Probably will in future applications
- Existing applications where lists change
(Done)
27Forget v. Remember as Forgettable
- Why I think people forget
- Visual analogy
- How to prove
- Lab study Do people find new information?
- Longitudinal study Ever click on new result?
- What if remember as forgettable?
- Build better model of memorability
- Highlight important changes
(Done)
28Implementation Issues
- Page of cached result may disappear
- Multiple result pages
- Identifying repeat queries
- User identified
- Search sessions are not repeat queries
- Exact query may be forgotten
(Done)
29Evaluation Issues
- Various goals to test
- Does a merged list look like the original?
- Does merging make re-finding easier?
- Is search improved overall?
- Lab study
- How to set up re-finding task?
- Timing differences significant enough?
- Longitudinal study What to measure?
- What are good baselines?
(Done)
30- Jaime Teevan
- teevan_at_mit.edu
31Strategies for Finding
Teleporting
Orienteering
32Why Do People Orienteer?
- Easier than saying what you want
- You know where you are
- You know what you find
33Structural Consistency Important
All must be the same to re-find the information!
New name
34Absolute Consistency Unnecessary
New name
Focus on search result lists
35Query Changes
- Most changes are simple
- Capitalization
- Phrasing
- Word ordering
- Word form
- New queries shorter
- What about longer time horizons?
- Recognition v. recall
36Result List Changes
- Tracked 10 queries on Google for a year
- 1.18 of top 10 disappear each week
- Rate of change likely to increase, e.g.
- Raw personalization
- Relevance feedback
- People forget their queries
- 28 of queries forgotten within an hour
37Example neon signs