Evaluating Learning Objects

1
Evaluating Learning Objects

• Athabasca University
• March 14, 2003

2
Rationale for Developing Effective Learning
Object Evaluation Systems

• Ratings and qualitative assessments aid
  individual users in searching and selecting
  objects.
• Evaluations can provide guidance on how best to
  use an object.
• Quality can be increased by formative evaluation
  throughout the design and development stages.
• Evaluation standards can drive the practices of
  designers and developers.
• Participation in evaluation activities can
  contribute to the professional development of
  those who work with learning objects.
• Evaluation activities can build and support
  communities of practice in relation to learning
  objects.
• Positive evaluations can promote social
  recognition of skilled designers and developers.
• A trusted evaluation system is an essential step
  toward the development of a workable business
  model for the economic exchange of learning
  objects.

3
Criteria for Learning Object Evaluation

• Learning Object Rating Instrument (LORI) utilizes
  a 5-point scale for each of nine dimensions
• Content Quality Veracity, accuracy, balanced
  presentation of ideas, and appropriate level of
  detail
• Learning Goal Alignment Alignment among learning
  goals, activities, assessments, and learner
  characteristics
• Feedback and Adaptation Adaptive content or
  feedback driven by differential learner input or
  learner modeling
• Motivation Ability to motivate, and stimulate
  the interest or curiosity of, an identified
  population of learners
• Presentation Design Design of visual and
  auditory information for enhanced learning and
  efficient mental processing
• Interaction Usability Ease of navigation,
  predictability of the user interface, and the
  quality of UI help features
• Accessibility Support for learners with
  disabilities
• Reusability Ability to port between different
  courses or learning contexts without modification
• Standards Compliance Adherence to international
  standards and specifications

4
Evaluation Models

• Consumer-oriented evaluations
• Expertise-oriented evaluations
• Objectives-oriented evaluations
• Participant-oriented evaluations
• Website evaluations

5
Consumer-oriented evaluations

• Consumer-oriented evaluations of educational
  materials are conducted by governmental
  organizations and non-profit associations that
  train evaluators, often teachers, to apply
  standard criteria, checklists or rating scales,
  to examine the materials and produce reviews in a
  highly structured format.
• This cost-effective method follows structured
  reporting formats to also facilitate comparisons
  among objects.
• The method is criticized for not being
  comprehensive, understandable, and easy to use.

6
Expertise-oriented evaluations

• Expertise-oriented evaluations are conducted by
  recognized experts, either individually or in
  panels.
• This method tends to be more costly than
  consumer-oriented approaches as it lacks the
  efficiency advantages of a fixed procedure
  repeatedly applied by a trained evaluator.
• The method is criticized as being especially
  vulnerable to the subjective biases of the
  evaluators.

7
Objectives-oriented evaluations

• Objectives-oriented approaches couple detailed
  analysis and definition of goals with empirical,
  quantitative studies using pre-post or
  comparative designs that test the extent to which
  the goals have been attained.
• A major drawback of objectives-oriented
  approaches is the cost associated with running an
  empirical study.

8
Participant-oriented evaluations

• Participant-oriented approaches to educational
  evaluation explicitly acknowledge that learning
  is a social process dependent on social context.
• Data is gathered by prolonged and persistent
  observation, informal interviews, and document
  analysis.
• Extensive collection and analysis of detailed
  qualitative data is costly.
• The method is criticized for a lack of
  quantitative ratings, making comparison of
  learning objects difficult.

9
Website evaluations

• Websites presenting consumer-authored reviews of
  items, similar to book reviews available on
  Amazon.com, are becoming more widespread.
  Similarly, the website Epinions supports
  meta-evaluation of both the review and reviewer.
• The method is criticized for a lack of
  opportunity for dialogue among reviewers and
  other users, or for collaborative reviews.
• The individual user reviews made possible by the
  Internet offer huge cost advantages over the more
  formal evaluation approaches previously
  discussed.

10
Third Party Quality Ratings

• Learning Object evaluation by third-party
  reviewers is a key element in promoting
  reusability because the availability of quality
  ratings can have an immediate and compelling
  effect on the outcome of repository searches.
• Quality ratings are already being used to order
  search results in MERLOT.
• The e-learning research community bears the
  significant responsibility to develop
  demonstrably valid evaluation methods.
• It is a challenge to develop effective evaluation
  systems, as they must optimize two opposing
  variables the quantity of objects that can be
  evaluated versus the quality of the obtained
  evaluations.

11
MERLOT

• Learning Object evaluation in MERLOT is largely
  modeled on the academic peer review process for
  scholarly research and publication.
• The MERLOT web site currently supports 14
  discipline-specific communities each with an
  editorial board that guides peer review policies
  and practices.
• Learning Object evaluation is based on a five
  point scale for strongly agree (5) to strongly
  disagree (1) for criteria in three categories.
• Quality of content
• Ease of use
• Potential effectiveness as a teaching-learning
  tool

12

• Quality of Content both the educational
  significance of the content and its accuracy or
  validity.
• Is clear and concise
• Demonstrates a core concept
• Is relevant
• Provides accurate information
• Is flexible and reusable
• Includes adequate amount of material
• Summarizes the concept well
• Quality of content very high

13

• Ease of Use usability for first-time users,
  aesthetic value, and provision of feedback to
  user responses.
• Is easy to use
• Has very clear instructions
• Is engaging
• Is visually appealing
• Is interactive
• Is of high design quality

14

• Potential Effectiveness as a Teaching-Learning
  Tool pedagogically appropriate use of media and
  interactivity, and clarity of learning goals. It
  is important for both authors and reviewers to be
  clear about learning goals.
• Identifies learning objectives
• Identifies prerequisite knowledge
• Reinforces concepts progressively
• Builds on prior concepts
• Demonstrates relationships between concepts
• Is very efficient (one can learn a lot in a short
  period of time)

15
Convergent Participation Model

• Convergent participation model is a panel
  evaluation process designed to obtain better
  outcomes than the peer review model without
  resorting to expensive field studies.
• Evaluation is conducted by a panel drawn from
  different stakeholder groups.
• During the first phase, the evaluators
  independently and asynchronously assess the
  object.
• During the second phase, the evaluators compare
  and explain their earlier assessments, adjusting
  their individual assessments as their judgment
  shifts in response to the panels discussion.
• In the convergent participation model evaluators
  use online tools that enable fully distributed
  participation (synchronous or asynchronous).

16

• Convergent participation is a boundary-crossing
  activity that should assist in establishing
  common evaluation standards.
• This activity has great educative value for all
  concerned and forms ideal conditions for
  professional development.
• The strengths of the convergent participation
  model are that it brings together representatives
  of stakeholder groups, efficiently focuses their
  attention on the points that may be in greatest
  need of resolution, and produces a review that
  concisely presents areas of agreement and dissent
  among the evaluators.
• However, the model is likely too costly to fully
  cover a large repository containing many
  thousands of learning objects.