Evaluations in information retrieval

1
Evaluations in information retrieval
2
Evaluations in information retrievalsummary

• The following gives an overview of approaches
  that are applied to assess the quality of
• information retrieval systems, and more
  concretely of search systems
• the resulting set of records obtained after
  performing a query in an information retrieval
  system
• Note This should not be confused with assessing
  the quality and value of the content of an
  information source.

3
Evaluations in information retrievalintroduction

• The quality of the results, the outcome of any
  search using any retrieval system depends on many
  components / factors.
• These components can be evaluated and modified to
  increase the quality of the results more or less
  independently.

4
Evaluations in information retrievalimportant
factors

• The information retrieval system ( contents
  system)
• The user of the retrieval system and the search
  strategy applied to the system

Result of a search
5
Evaluations in information retrievalwhy? (Part
1)

• To study the differences in outcome/results when
  a component of a retrieval system is changed,
  such as
• the user interface
• the retrieval algorithm
• addition by the database of uncontrolled, natural
  language keywords versus keywords selected from a
  more rigid, controlled vocabulary

6
Evaluations in information retrievalwhy? (Part
2)

• To study the differences in outcome/results when
  a search strategy is changed.
• To study the differences in outcome/results when
  searches are performed by different groups of
  users, such as
• children versus adults
• inexperienced users versus more experienced,
  professional information intermediaries/professio
  nals

7
Evaluations in information retrieval the simple
Boolean model

• Boolean model items in database items
  selected items not selected
• Items selected
• relevant items irrelevant items

Relevant Yes 1 In
Irrelevant No 0 Out
8
Relevant items in a database scheme
Relevant items! (In most cases the small
subset) Irrelevant / NOT relevant items (In
most cases the large subset)
Dependent on the aims, independent of the search
strategy
9
Selecting relevant items by searching a database
scheme
Dependent on the aims and dependent on the
search strategy
Selected and relevant! Selectedbut not
relevant
Not selected but relevant Not selectedand
not relevant
Dependent on the aims, independent of the search
strategy
10
Recall definition and meaning

• Definition
• Of selected
  relevant items
• Recall -------------------------------------
  ------------ 100
• Total of relevant items
  in database
• Aim high recall
• Problem in most practical cases, the total of
  relevant items in a database cannot be measured.

11
Selecting relevant items recall
Selected and relevant! Selectedbut not
relevant
Not selected but relevant Not selectedand
not relevant
12
!? Question !? Task !? Problem !?
How to use of the concept recall, when you do
not know the total number of relevant items in
the database ?
13
Recall how to use the concept of recall

• Using the same database, variations in recall
  express the effect of search variations
• Variations in search terms
• Use of a classification scheme
• Use of a thesaurus
• ...

14
!? Question !? Task !? Problem !?
How can you change your search strategy to
increase the recall?
15
Precision definition and meaning

• Definition Of
  selected relevant itemsPrecision
  --------------------------------------- 100
  Total of selected
  items
• Aim high precision

16
Selecting relevant items precision
Selected and relevant! Selectedbut not
relevant
Not selected but relevant Not selectedand
not relevant
17
!? Question !? Task !? Problem !?
How can you change your search strategy to
increase the precision?
18
!? Question !? Task !? Problem !?
When you change your search strategy to increase
the precision, which consequence do you expect
for the recall, in most cases?
19
Relation between recall and precision of searches
Ideal Impossibleto reach in most systems
100 Recall 0
0 Precision 100
Search (results)
20
!? Question !? Task !? Problem !?
Indicate on the figure that a user improves a
search.
21
!? Question !? Task !? Problem !?
Indicate on the figure that a database producer
and / or the retrieval system improves the
retrieval quality.
22
!? Question !? Task !? Problem !?
Indicate the relation between the recall and
precision in a classical information retrieval
system in the form of a figure. Indicate in
that figure a good and a bad search.
23
Recall and precision should be considered
together

• Examples
• Increase in retrieved number of relevant items
  may be accompanied by an impractical decrease in
  precision.
• Precision of a search close to 100 may NOT be
  ideal, because the recall of the search may be
  too low. Make search / query broader to increase
  recall !
• Poor (low) precision is more noticeable than bad
  (low) recall.

24
Evaluation in the case of systems offering
relevance ranking

• Many modern information retrieval systems offer
  output with relevance ranking.
• This is more complicated than simple Boolean
  retrieval, and the simple concepts of recall and
  precision cannot be applied.
• To compare retrieval systems or search
  strategies, decide to consider for comparison a
  particular number of items ranked highest in each
  output.This brings us to for instance first-20
  precision.

25
!? Question !? Task !? Problem !?
Give examples of retrieval systems that offer
relevance ranking.