DAFFODIL Strategic Support Evaluated

1
DAFFODILStrategic Support Evaluated

• Claus-Peter Klas
• Norbert Fuhr
• Andre Schaefer
• University of Duisburg-Essen

2
Outline

• Introduction to DAFFODIL
• Evaluation
• Procedure
• Hypotheses
• Main Results
• Outlook

3
DAFFOFIL is

• Virtual Digital library with strategic support
  given by high-level search functions
• an agent-based system with currently over 35
  different agents providing the low-level and
  high-level functions
• connects 15 data sources for searching and
  browsing from the computer science area

4
(No Transcript)
5
DAFFODIL Concepts

• Strategic Support
• Scientific Workflow
• Personalization
• Personal library
• Collaboration
• Awareness Annotation
• Proactivity

6
Strategic Support

• is provided by high-level functions like
• metadata search
• duplication elemination
• citation search
• author network based strategems
• journal and conference run
• in a integrated way

7
Scientific workflow
Collaborate
8
Personalization Personal Library

• Structured folders
• document metadata
• fulltext
• authors
• conferences
• journals
• websites
• queries
• Collaboration through group folders

9
Personalization Awareness Annotation

• Long term queries
• Query-Profiles interval processing
• Authors new publications
• Journals, conferences new issues, proceedings
• See whats new
• Message via Email, Fax, SMS, ...
• Annotation for scientific discurs

10
Proactivity

• Contextual monitoring of users
• system
• Direct action
• Spellchecking
• Authorname completion
• Related terms
• Recommend next step
• based on current stage
• based on collaborative usage

11
Evaluation framework

• Formal bases given by SaracevicCovi
• Why to evaluate ?
• What to evaluate ?
• How to evaluate ?

12
DAFFODIL Evaluation Scheme

• Hypo I More effective IR through strategic
  support
• Hypo II More effective IR through (pro)-active
  system
• Hypo III More effective IR through
  personalisation and collaboration
• Criteria
• Effectiveness
• Efficiency
• Measure
• Time
• Relevance
• User Satisfaction

13
Evaluation Phases

• 1st Phase
• 14 students in parallel sessions
• System efficiency bug hunting
• 2nd Phase
• 8 researchers
• Focus on user Efficiency and Effectiveness
• 3th Phase (expert phase)
• 3 librarians, 3 professors
• Comments, critics, hints

14
Gathering Evaluation Data

• Tasks on questionnaire
• 1st phase protocolls
• Other phases video capturing of monitor and face
• store usage
• see reaction
• store comments
• 11h video data

15
Real world task classified

• 5 Tasks (4 presented) on 4 levels, classified by
  a information need typology matrix
• work task
• information need

no knowledge
4L
2L
3L
1L
full knowledge
heterogenous set of objects
one object
16
Result (1st Phase)

• Phase 1 14 students
• Error hunting, system efficiency
• Outcome
• speed up 5-7 times
• Find a lot of bugs
• Comment Even so slow when searching, i want to
  use it, since it saves much time, handling the
  found knowledge

17
Main Hypotheses

• more effective and efficient IR through strategic
  support given with DAFFODIL compared to other
  (DL) information system
• more effective, because user can cognitivly
  process and interpret information more easily
• more efficient, because user needs to invest less
  time

18
Result (2nd phase) Task 1 Known item
instantiation (L1)

• Find document by given title (precise)
• Search strategies in content-based image
  retrieval

19
Task 2 Known item instantiation (L2)

• Find document by given title (not precise)
• Methods of automated reasoning

20
ResultTask 3 A real search (L3)

• Find articles by author X in the area of digital
  libraries within the last 4 years.
• Results
• All participants found with Daffodil at least
  12-19 relevant articles.
• With Google only two persons found articles any
  articles, the other gave up.
• Actually, user said, it will be hard with google
  prior to search.

21
Result Task 5 New area (L4)

• find literature about a specific subject
• interactive information retrievalinformation
  searching information seeking
• not known by any participant prior to the
  evaluation
• Explorative Search
• Scanning result lists related terms feature
• Results
• Participants at least found a couple of entry
  points, to further search
• Got very soon an idea, what they dont want.

22
Result (3th phase)Experts Free Search

• Three librarians and three professors
• Free Search with own theme
• Results
• input for improving
• satisfied
• Comments
• librarian
• rich functionality
• D. mediates the knowledge about the tactics to
  the user
• professor
• D. seems in a state that it can be used
• I will propose it to my assistents

23
Summary of evaluation

• Evaluation with 28 participants showed, that a
  user is more effective when searching for
  scientific literature with the strategic support,
  here given by the DAFFODIL system, but not always
  more efficient (task1).
• But, as the tasks become more complex,
• the hypotheses hold.

24
Outlook

• Continue Evaluation
• other systems
• Long-Term Evaluation
• One task from start to end
• new paper for ECDL 2005
• Participants are welcome
• Try it out
• http//www.daffodil.de