eMapScholar: Evaluation

1
e-MapScholar Evaluation
Canan Blake and the Evaluation Team at The Open
University
2
Evaluation Framework

• CIAO
• Context
• Interaction
• Attitudes and Outcomes

3
Context

• wider - framing the teaching
• finer - components of the resource
• how learners use the information
• technology
• rationale

4
(Tracking the )Interaction

• Look in detail at students use of the
• software by analysing their interactions
• qualitative data about the learning process
• difficulties experienced

5
Attitudes and Outcomes

• Investigate whether students have learnt
• the subject matter that the program is
• designed to teach
• But also consider effective outcomes, e.g.
• changes in learners perceptions and
• attitudes

6
Evaluation Activities

• Previous Digimap Evaluations, Needs Assessment,
  Workshops
• Formative
• expert reviews
• student observations
• Summative
• in universities with voluntary students
• online questionnaires

7
Formative Evaluations

• Method and participants
• observation
• educational technology experts
• students
• Findings
• usability issues - the use of graphics and text
• interactivity
• assessment questions
• ...

8
Summative Evaluation

• Aims to assess the effectiveness of the learning
• material as a teaching/learning resource and
• identify how they can be successfully integrated
• into a teaching programme
• November 2002
• Staff and students in two institutions
• Students accessed the resources and sent feedback
  via online questionnaires

9
What are the things you liked about this resource?

• Good and relevant use of the diagrams,
• animated diagrams, clarity of diagrams,
• Mars example (more interesting), summary
• and the reflection
• The ability for the user to interact with the
• resource and the visual aisd throughout
• the session

10
Things disliked

• Inconsistent use of bold to highlight words
• some of the visual aids are sometimes
• thought as interactive but they are not
• In the long view version, it need a period of
• time to download the diagrams or
• pictures

11
CMS Evaluation

• Cognitive walkthrough on the prototype
• Representative tasks that would be carried out by
  potential users (defined from the prototype)
• Will users see the relevant control(button menu)
  for the required action?
• Once users find the control, will they recognise
  that it produces the effect that they want?
• After the action is taken, will users understand
  the feedback that is provided so that they can
  then go on to complete the task?

12
Issues

• What can be evaluated and when?
• Development team involvement - distance
• Practical constraints and timing
• Pedagogical context
• re-usable Learning Objects approach
• customising to provide context relevant
  information and images
• A framework that allows different approaches to
  be combined