Quality Matters: Peer Review of Online Courses

1
Using the Quality Matters Rubric to Improve
Online Coursesed bowen, Executive Dean Distance
Learning, Dallas TeleCollege
This session will address how faculty and
administrators could use the Quality Matters
rubric to improve their online course(s).
Participants will become familiar with the rubric
tool developed by the nationally recognized,
FIPSE-funded Quality Matters (QM) project. The QM
rubric provides a research-supported framework
with annotations and examples for applying
quality practices to specific course design
standards.
2
Why Adopt
What's Involved
Implementation Strategies
3
Quality Matters Is a Process A Set of Standards
4
Quality Matters Peer Course Review Process
Faculty Course Developers
Institutions
National Standards Research Literature
Course
Rubric
Faculty Reviewers
Training
Peer Course Review
Feedback
Blueprint for improvement
Instructional Designers
5
Underlying Principles of QM

• The QM toolset and process are
• based in national standards of best practice, the
  research literature and instructional design
  principles
• designed to promote student learning
• integral to a continuous quality improvement
  process
• part of a faculty-driven, peer review process
• Course does not have to be perfect but better
  than just good enough. (Standards met at about
  85 level or better.)

6
Underlying Principles of QM Continued

• Process designed to ensure all reviewed courses
  will eventually meet expectations
• Collegial review process, not an evaluation
  process
• Review team must include an external peer
  reviewer
• Course faculty or instructor considered part of
  the review team

7
Whats In It For Institutions

• External validation
• Strengthen accreditation package
• Raise QA as a priority activity
• Access to a sustainable, replicable, scalable QA
  process
• Inform online course training practices
• Provide professional development activities

8
Whats In It For Faculty

• Improve online courses
• External quality assurance
• Expand professional community
• Review other courses and gain new ideas for own
  course
• Participation useful for professional development
  plan and portfolio
• Receive 150 for each completed peer course review

9
(No Transcript)
10
Why Adopt
What's Involved
Implementation Strategies
11
Quality MattersPeer Course Review Process
1. Course Selected
2. Trained Review Team Assigned
3. MOUs and Instructor Worksheet
4. Pre-Review Discussion
5. Individuals complete reviews
6. Team Discussion(s)
7. Final Review Report
12
Quality Matters Peer Course ReviewerCertification
Process
Quality Matters Certification
Course Review Experience


Training

• Name on QM website
• Use of QM Logo
• Eligible for Peer Reviewer Pool

Assigned to Peer Review Team
Attend QM Training
Submit Report
Demonstrate Competencies
Trained
Certified
Kane 1/15/05
13
The Rubric

• Eight general standards
• Course Overview and Introduction
• Learning Objectives
• Assessment and Measurement
• Resources and Materials
• Learner Interaction
• Course Technology
• Learner Support
• Accessibility

Key components must align.
14
Key Standards That Must Align

• Objectives
• Standard II.1 Measurable outcomes
• Standard II.2 Content mastery, critical
  thinking, learning skills addressed.
• Assessment and Measurement
• Standard III.1 Measures objectives consistent
  with learning activities
• Learner Interactions and Activities
• Standard V.1 Help students achieve the
  objectives
• Course Materials
• Standard IV.1 Deep and comprehensive enough for
  students to achieve the objective
• Course Technology
• Standard VI.1 Tools and media support the
  objectives

15
General Standard ICourse Overview and
Introduction

• 1.1 Navigational instructions make the
• organization of the course easy to
• understand.
• 1.2 A statement introduces the student to the
• course and to the structure of the student
• learning, and, in the case of a hybrid
• course, clarifies the relationship between
• the face-to-face and online components.

16
Rubric Scoring

• Team of three (3) reviewers
• One score per standard based on team majority
• Assigned point value not sliding scale

17
The Peer Review Team

• 3 Faculty Peer Reviewers
• All must be experienced online instructors
• All must attend QM training
• One MUST be external to the course developers
  institution
• One must be a subject matter expert (SME) The SME
  could also be the external reviewer.
• AND
• Faculty Course Developer
• access to rubric prior to review
• involved in pre-review discussions
• consulted during review

18
Faculty Developer

• Part of the review team.
• Provides access to the course.
• Completes Instructor Worksheet
• Part of the initial team discussion
• Receives compiled report
• Returns Faculty Response Form

19
Peer Reviewers

• Establish Team Calendar
• Review the course individually
• Complete the online web review form
• Discuss review with Team as needed
• Complete an Exit Interview
• Receive
• Recognition as Certified Peer Reviewer.
• Compensation

20
Team Chair

• Reviewer Roles Responsibilities plus.
• Organizes Team calendar
• Confirm Instructor Worksheet is used
• Creates draft report from compiled reviews
• Convenes Team discussions
• Reviews, edits and submits Team Report
• Receives
• Recognition as Certified Peer Reviewer and Chair.
• Compensation

21
Overall Course Review Results

• Upon initial review
• 51 meet expectations
• 19 do not meet expectations - missing at least
  one essential 3-point element(s)
• 30 do not meet expectations - missing at least
  one essential 3 point element(s) and a minimum of
  68 points

22
Why Adopt
What's Involved
Implementation Strategies
23
Range of Implementation Options
Formal QM Peer Review
Internal QM Peer Review
Internal Review
Dean Review
Colleague Review
Cost
Self Administered
24
(No Transcript)
25

• Additional Optional Services and Fees
• Independent (Subscriber-Run) Peer Reviews
• Subscribers may manage their own MOL-recognized
  Peer Reviews after training. Institutions are
  responsible for selecting and managing peer
  reviewers per Quality Matters program guidelines
  and for compensating peer reviewers. Subscribers
  are given access to the Quality Matters Rubric
  for the review team and recognition in the
  Quality Matters Registry of courses that meet
  standards. The annual subscriber fee includes
  registration of up to 30 independent reviewed
  courses for Large Institutions, 20 for Mid-Sized
  Institutions, and 10 for Small Institutions.
  (Subscribers pay a 50 per-course registration
  fee for any additional courses.)
• To be eligible for recognition in the Quality
  Matters Registry, an on-line course must have
  successfully completed a subscriber-run review
  that has met all Quality Matters guidelines
  (which may or may not involve additional costs to
  the Institution).
• Guidelines include
• Review managed by a QM-trained Institutional
  Representative (one seat in training included in
  annual subscriber fee)
• Review completed by a review team of three
  Qualified Quality Matters-Trained Reviewers. Team
  make-up must include a reviewer from outside the
  institution a reviewer trained as a Quality
  Matters Master Reviewer and a reviewer assigned
  as team chair. (A single reviewer may fill one or
  all of these roles.)
• Reviewers used the Quality Matters materials and
  followed the Quality Matters process (provided
  electronically to subscribers as part of annual
  subscriber fee)
• Upon successful completion of review and
  submission of documents to MOL, course
  institution has paid 50 per course for each
  registration over those included in annual
  subscriber fee
• MOL provides access to list of trained Peer
  Reviewers to subscribers. Subscribers must
  negotiate the peer reviewer stipends paid if any.
  (MOL pays 150 per reviewer, 250 to team chair)

26

• MOL-Managed Peer Reviews
• MOL will manage the course review process for
  subscribing and non-subscribing institutions on a
  course-by-course basis. This service includes
  Peer reviewers assigned, managed, and paid by
  MOL compiled final report and recognition of a
  course in the Quality Matters Registry upon its
  meeting Quality Matters program review standards.
• Quality Matters Program Subscribers
  750/course
• Non-Subscribers
  1,000/course

27
Range of Implementation Options
Formal QM Peer Review
Internal QM Peer Review
Internal Review
Dean Review
Colleague Review
Cost
Self Administered
28
Multiple Uses of QM

• Guidelines for initial online course development
• Quality assurance of existing courses
• Ongoing faculty professional development
• Institutional reaccredidation packages
• Formation of distance learning policies
  steering committees