Evaluating electronic textbooks: a methodology

1
Evaluating electronic textbooks a
methodology Ruth Wilson and Monica Landoni ECDL
2001 5th September 2001
2
Outline

• The Visual Book
• The WEB Book
• EBONIs methodology for evaluating electronic
  textbooks

3
The Visual Book

• Studied the application of the paper book
  metaphor to the design and production of
  electronic books
• Found that users were able to rely on their
  experience with paper books to interact with the
  electronic book.

4
The WEB Book

• Focused on the impact of appearance on the
  usability of textbooks on the Web.
• Comparison of two versions of a textbook chapter
• Original version very plain, one page per
  chapter, basic use of hyperlinks.
• Revised, scannable version chapter divided
  between 10 pages, heavier use of hyperlinks,
  extra headings, indented lists, coloured
  keywords.
• Revised version was 92 more usable.

5
(No Transcript)
6
EBONI aims

• EBONI aims to
• Evaluate the different approaches to the design
  of electronic textbooks
• Identify and report on the individual
  requirements of students and academics in
  learning and teaching using electronic textbooks
• Compile and promote a set of best practice
  guidelines for the publication of electronic
  textbooks

7
Experimentspsychology Web books

• Peter Kaiser. The Joy of Visual Perception.
  http//www.yorku.ca/eye/thejoy.htm
• Eric Chudler. Neuroscience for Kids.
  http//faculty.washington.edu/chudler/neurok.html
• C. Robin Timmons and Leonard W. Hamilton. Drugs,
  Brains and Behaviour. http//www.rci.rutgers.edu/
  lwh/drugs/

8
ExperimentsHypertext in Context

• Cliff McKnight, Andrew Dillon and John
  Richardson. Hypertext in Context.
• Print
• Original electronic version
• Revised electronic version

9
Experimentscommercial formats

• Rosa Ainley (ed.). New Frontiers of Space
• MobiPocket Reader
• Adobe Acrobat eBook Reader
• Microsoft Reader
• Paperback

10
Experimentse-encyclopaedias

• Britannica
• Columbia
• Encarta

11
Experimentsportable ebooks

• HP Jornada
• Franklin eBookMan
• PalmPilot
• Rocket eBook
• Softbook

12
Ebook evaluation model

• Selection of material
• Selection of participants
• Selection of tasks
• Scavenger hunt, memory tasks, high cognitive
  skill tasks
• Selection of evaluation techniques
• Questionnaires, observation, think-aloud,
  interviews

13
Selection of material (1)

• Electronic books offer a diverse array of
• material for evaluation
• Hardware devices, e.g. REBs, eBookMan, goReader
• Ebook reader software, e.g. Microsoft Reader,
  Adobe Acrobat Ebook Reader
• Web books

14
Selection of material (2)

• EBONI aims to identify design techniques which
  maximise information intake by users, in order to
  provide guidance to creators of digitised
  educational material.
• Texts can be compared according to three
  parameters
• Format
• Content
• Medium

15
Selection of actors

• Four main roles can be distinguished
• Participant
• Evaluator
• Task developer
• Task assessor

16
Selection of tasks

• Tasks are a way of bringing together participants
  and the test material in a structured manner.
  Three types are suggested
• Scavenger Hunts, or retrieval tasks
• Memory tasks
• High cognitive skill tasks

17
Selection of evaluation techniques

• Qualitative feedback can be obtained via
• Subjective satisfaction questionnaires
• Behaviour observation
• Think-alouds
• Interviews

18
Technique complexity
Study X
Interviews
Think-aloud
Covert observation
WEB Book
Questionnaires
Task complexity
Scavenger hunt
Memory tasks
High cognitive skill tasks
19
Guidelines

• Sophisticated experiments can be mapped to
  simple experiments.
• Results of all evaluations will be comparable at
  some level.
• Results will feed into a set of best practice
  guidelines for producing electronic textbooks.

20
Contact details

• Ruth Wilson ruth.m.wilson_at_strath.ac.uk
• EBONI Web site http//eboni.cdlr.strath.ac.uk/
• To join the EBONI mailing list, follow the
  instructions at http//www.jiscmail.ac.uk/lists/o
  pen-eboni.html