Experimental Components for the Evaluation of Interactive Information Retrieval Systems

1

• Experimental Components for the Evaluation of
  Interactive Information Retrieval Systems
• Pia Borlund

Dawn Filan3/30/04610551
2
The Goal

• To evaluate IR systems in a way that is as close
  to actual information seeking process as
  possible, while still being in a controlled
  environment.

3
Research Questions

• Can simulated information needs be substituted
  for real information needs?
• What makes a good simulated situation with
  reference to semantic openness and types of
  topics of the simulated situations?

4
Hybrid Evaluation Model

• Increased demand
• Relevance revolution
• Cognitive revolution
• Interactive revolution
• Combine two main approaches
• System-driven approach (controlled)
• Cognitive user-centered approach (realism)

5
The Experimental Setting

• 3 components
• The involvement of potential users as test
  persons
• The application of dynamic and individual
  information needs
• Use of dynamic relevance judgements

6
Ideal IIR Setting

• Real users who state personal information needs
  to the system and judge the relevance of the
  retrieved documents under controlled
  circumstances.
• Use of simulated work task
• Must be under controlled circumstances so that
  results can be compared across systems and user
  groups.

7
Simulated Work Task

• Triggers and develops a simulated information
  need by allowing for user interpretations of the
  situation.
• Platform against which situational relevance is
  measured.
• 2 variants applied
• Complete need applied (sim 1)
• Only situation applied (sim 2)

8
Situational Relevance

• User-centered, realistic, and dynamic measure of
  relevance
• Judgements are not based on the request or query,
  but rather relate to the persons requirements
  and mental state at the time of the retrieval
• Assessed continuously and interactively during
  the session

9
Relevance (Schamber, Eisenberg, and Nilan)

• Multidimensional cognitive concept whose meaning
  is dependant in users perceptions of information
  and their information needs
• Dynamic concept that depends on users judgements
  of quality of the relationship between
  information and the information need
• Complex but systematic and measurable concept if
  approached conceptually and operationally from
  the users perspective

10
Meta-Evaluation

• Should simulated work tasks be recommended as a
  component of the experimental setting for
  evaluating IIR systems?

11
Meta-Evaluation Questions

• Possibility of substituting real information
  needs with simulated information needs through
  the application of simulated work task
  situations.
• Whether the variants of the simulated task makes
  any difference to the test persons treatment of
  the information need
• What characterizes a good simulated work task in
  terms of how tailored the task should be to the
  user

12
Test Setting

• Full-text online system applying TREC data and
  probabilistic-based retrieval engine
• Search activity and relevance scores were logged
• 24 users from various academic backgrounds and
  education levels
• Asked to prepare a personal information need

13
Testing Procedure

• Brief questionnaire
• Introduction
• Explanation of the test persons role
• Demo of the system
• Execution of 6 search tasks (training, real, 4
  simulated tasks)
• Post-search interview

14
Conclusions

• One can substitute real information needs with
  simulated information through the application of
  simulated work tasks
• One can mix simulated and real information needs
• Treatment of the information need did not differ
  between the group that received the work task and
  request, and those who received just the work
  task.