How is e-learning quality assessed in Finland:

1
How is e-learning quality assessed in
Finland FVUs Referee service
Petra Rutanen University of Oulu Dept. of
Electrical and Information Engineering E-XCELLEN
CE seminar May 20, 2009
2
Referee service a part of the service
operations of the Finnish Virtual University

• Peer review of the quality of online learning
  materials
• The goals of the service are
• To provide evaluation criteria and a service
  model that can be used to carry out quality
  assessment of online material
• To give recognition for e-learning material
  producers (and evaluators)
• To increase the common usability of high-quality
  online learning material produced in
  universities.

3
Referee service a part of the service
operations of the Finnish Virtual University

• A portal site where
• Any e-learning material producer can request an
  evaluation for their material
• Users can search for evaluated materials
• Material producers can test their own material
• Experts can apply to be evaluators
• Guide and introduction for the evaluation
  process
• The referee service defines online learning
  material as an educational entity with
  contentual requirements and goals set for
  learning.The service focuses on examining online
  learning material that its producer wants to
  bring into common use and wants a referee
  evaluation quality mark for.

4
The referee service in refreepalvelu.fi

• The producers of online learning material get
• recognised evaluation of the online learning
  material they have produced
• ideas for further development of the learning
  material
• recognition for the quality of their work
• publicity (and users for the produced learning
  material)
• The users of online learning material get
• support for choosing and using high-quality
  material
• (ideas for developing their own teaching methods)
• examples of high-quality online material
• The evaluators of online learning material get
• information on factors that affect the quality of
  online learning material, and examples of
  high-quality learning material
• Orientation for the process
• evaluation merit and a certificate.

5
The Referee project 2007-2008

• There was a need for
• quality assurance
• develop an evaluation procedure that meets the
  challenges of the production process
• give merit to e learning material producers
• The aim was to create an assessment model that is
  extensive but does not excessively burden the
  referees.
• The service was developed in 2007-2008 as a joint
  Referee service project between the coordinator
  Lappeenranta University of Technology, Tampere
  University of Technology, University of Oulu,
  University of Joensuu and the FVU Service Unit.

6
The evaluation criteria for online learning
material

• The Referee service evaluation criteria are based
  on various national and international evaluation
  criteria for online learning material
• Four categories common usability and
  pedagogical, content, and instrumental criteria
• E-learning material is reviewed criterion by
  criterion under the main named categories
• In every main criterion category there are 2 to 3
  criteria which are evaluated
• The evaluation focuses on a total of 27 criteria
  every division has an supporting description
• Every criteria is evaluated with accuracy by
  0,25, 0,5 or 1 point.
• The maximum points are 7 p for each main
  category, 28 p totally

7
Criteria for common use
1.1 Availability A. Stability of use B.
Accessibility C. Description of the user
rights 1.2 Descriptional information A.
Findability of the descriptional information B.
Meta data 1.3 Transferability A. Usability in
other teaching contexts B. Instructions for
other teachers
8
Pedagogical criteria
2.1 Goals of learning A. Description of the
goals of learning B. Correspondence between the
goals of learning and the online material 2.2
Target group A. Description of the target
group B. Consideration of the target group 2.3
Supporting learning A. Directiveness of the
learning process B. Overall learning experience
9
Contentual criteria
3.1 Relevance A. Suitability of the
contents 3.2 Reliability and up-to-dateness of
the factual contents A. Author information B.
Use of information sources C. Reliability D.
Up-to-dateness 3.3 Clarity and diversity of the
presentation method of the contents A. Clarity
of the contents B. Diversity of the contents
10
Instrumental criteria
4.1 Usability A. Ease of use B.
Manageability C. Graphical/visual
functionality 4.2 Barrier free A. Consideration
of different user environments B. Consideration
of different users in technical
implementation C. Technical requirements D.
Technical reliability
11
The Referee project

• 16 e-learning materials were evaluated during
  2007
• After feedback,11 materials were evaluated during
  2008
• Evaluation groups consisted of Referee project
  actors, experts on e-Learning and each
  quality criteria, and also student members
• Every material passed the assessment
  (qualification limit 10 points)
• The points converted into stars (1-5 stars)
• The producers of e-learning material got detailed
  reports of the evaluation
• Feedback was collected both from evaluators and
  the material makers

12
www.refereepalvelu.fi

• Feedback results
• Evaluators
• The process was described as interesting, smooth
  and educational
• Fuctionality of the process got a rating of 4,33
  (1-5)
• The use of time was suitable, the whole process
  took on average of 11 hours of time/evaluator
• Producers
• Satisfied with comprehensive evaluating
• Valuable ideas for developing the material
• Development as an employee
• The mean value of satisfiction with the
  evaluation report was 3,90.

13
Thank you! petra.rutanen_at_ee.oulu.fi For
details Mr. Totti Tuhkanen (SVY) at
tottituh_at_utu.fi www.refereepalvelu.fi