User Feedback and all that

1
User Feedback and all that

• John Tait

2
Overview

• Anomalous States of Knowledge
• Ranked Retrieval
• Relevance Feedback
• Other Issues with User Feedback

3
Anomalous States of Knowledge

• What are they and Why are they important ?

4
Typical Web Retrieval Process
Link Following
Need
KeywordQuery
More Like this
5
Problem One

• The fact that the user is looking for some
  information means they dont know what they are
  looking for (otherwise they wouldnt be looking)
• Therefore they may not be using the correct word
  to search for what they want
• Known as Vocabulary Problem

6
Anomalous State of Knowledge

• There is by definition some gap or error in the
  users state of knowledge so they are in an
  Anomalous State of Knowledge
• idea due to Belkin and others

7
Exercise

• Work wth a partner to identify three occasions
  when you have been in an anomalous state of
  knowledge when searching the internet or using
  the library

8
Problem Two

• Partial or even incorrect search results can
  alter the Anomalous State of Knowledge and thus
  change the users perception of their need.

9
Retrieval Process II
Browsing
Need
Analytic Query
More Like This
10
Ranked Retrieval

• How can we present the best item to the user
  first

11
What are we trying to do in IR

• Find the Document which is most similar to the
  query
• Ranking Interpretation
• show the most similar document first
• then the next most similar document
• and so on

12
Reprise

• Vector Model of IR
• Bag of Words Model of Text for IR

13
Bag of Words Model of Text

• Ignore the order of words in the document
• Just record whether a word appears in a document
• OR
• Just count the number of occurrences of each word

14
Similarity Measures

• Sum of Products
• Similarity(query, document)
• ?(query-termidocument-termI)
• Cosine Formula
• Various Others
• See Kowalski Chapter 7

15
Similarity as Ranking

• Use the Similarity Measure to rank the documents

16
Issues

• Most Web Searches are of Length One
• Mentioning the queried item many times does not
  necessarily make the document the most relevant
• Essentially returns the whole web as the result
  needs modification to work in practice

17
Relevance Feedback

• More Like this done properly

18
Observation

• The user is probably in the best position to
  judge the relevance of a document
• Likewise the user is probably in the best
  position to judge which returned (highly ranked)
  documents are irrelevant

19
Retrieval Process
No More Like This
Need
Analytic Query
More Like this
20
Relevance Feedback in Nutshell

• Perform an initial retrieval
• Ask the user to indicate which documents are
  relevant/irrelevant
• Add all terms from relevant documents
• Remove all terms from irrelevant documents
• requery

21
Variants

• Using Ranking and Weighting
• Pseudo relevance feedback
• use terms from all (highly ranked) retrieved
  documents
• very helpful if very few documents retrieved
• perpetuates errors/misunderstandings from
  original query

22
Exercise 2

• What are advantages of positive feedback ?
• What are advantages of negative feedback ?
• Whis is best ?

23
Relevance Feedback Conclusion

• Consistently proven an effective way to improve
  retrieval
• Biggest problem is getting users to engage in the
  interaction, especially if no highly relevant
  documents are in the initially retrieved set

24
More User Feedback
25
Other Uses for User Feedback

• What sort of information is required
• general interest vs academic study
• child vs adult
• veracity
• High recall vs high precision searches
• what was Sunderlands score on Saturday
• how many web pages mention Sunderland AFC

26
Key Issue

• User Interface must be
• easy to use
• attractive to use
• instant reward

27
Conclusions

• Anomalous States of Knowledge
• vocabulary problem
• Ranked Retrieval
• similarity matching
• Relevance Feedback
• positive and negative feedback
• Other forms of User Feedback