Quality Matters: Peer Review of Online Courses

1
Quality Matters Inter-Institutional Quality
Assurance in Online Learning
Peer Reviewer Training(f2f)
Updated January 2007
Proposed for use by all QM trainers.
2
After this Peer Reviewer Training

• You should be able to
• Describe the critical materials, processes, and
  administrative elements of the Quality Matters
  online course quality assurance program.
• Apply the Peer Course Review rubric to review
  online courses.
• Write useful recommendations for course
  improvement.
• Explain the QM scoring system.
• Describe the Peer Course Review process and your
  role in it.

3
QM Materials Overview

• Todays Agenda
• Binder Overview
• QM Website
• QM Contact Information
• QM Resources

4
Introductions

• Pair up
• Share name, institution, job, and best distance
  learning practice.
• Introduce your partner to the rest of us.
  Briefly (in one sentence) describe your partners
  best practice.

5
About Quality Matters
6
Quality Matters Peer Course Review Process
Faculty Course Developers
Institutions
National Standards Research Literature
Course
Rubric
Faculty Reviewers
Training
Peer Course Review
Feedback
Instructional Designers
7
For Our Purposes, Quality Is

• More than average more than good enough
• An attempt to capture whats expected in an
  effective online course at about an 85 level
• Based on research and widely accepted standards

85
8
What this process is NOT

• Not about an individual instructor
• (its about the course)
• Not about faculty evaluation
• (its about course quality)
• Not about win/lose or pass/fail
• (its about continuous improvement in a
  supportive environment)

9
QM Collegial Review vs. Faculty Evaluation

• A QM Review is
• Ongoing
• Focus course design
• Outcome course improvement
• Non-threatening
• Team approach that includes faculty
• Full disclosure to faculty

• A Faculty Evaluation is
• Single point in time
• Focus delivery
• Outcome decision on performance for
  promotion/tenure
• Win/lose situation
• Confidential/secretive

10
Design vs. Delivery
The faculty member is integral to both design
and delivery.
Course Design is the forethought and planning
that a faculty member puts into the course.
Course Delivery is the actual teaching of the
course, the implementation of the design.
QM is about DESIGN - not delivery or faculty
performance
11
Distinguish Between Design vs. Delivery
Example Discussion Board
Design A discussion board has been planned
into the course students have been told how they
should participate and how they can expect the
faculty to participate.
DeliveryHow often the faculty member actually
participates in the discussion what the faculty
member actually says to students.
12
The Peer Review Team

• 3 faculty peer reviewers
• must be experienced online instructors
• must attend QM training
• one MUST be external to the courses originating
  institution
• there must be a subject matter expert (SME) on
  the team. NOTE The SME could also be the
  external reviewer.
• AND
• Faculty course developer
• access to rubric prior to review
• involved in pre-review discussions
• consulted during review

13
Your Point of View as a QM Peer Course Reviewer

• Take the students point of view
• Advocate for the student
• If you cant find evidence that the standard is
  met, dont assume it is or isnt there.. ask the
  faculty member.

14
Factors Affecting Course Quality

• Course design QM REVIEWS THIS
• Course delivery (i.e. teaching, faculty
  performance)
• Course content
• Course management system
• Institutional infrastructure
• Faculty training and readiness
• Student engagement and readiness

15
Underlying Principles of QM

• The QM toolset and process are
• based in national standards of best practice, the
  research literature and instructional design
  principles
• designed to promote student learning
• integral to a continuous quality improvement
  process
• part of a faculty-driven, peer review process
• Course does not have to be perfect but better
  than just good enough. (Standards met at about
  85 level or better.)

16
Underlying Principles of QM

• Process designed to ensure all reviewed courses
  will eventually meet expectations
• Collegial review process, not an evaluation
  process
• Review team must include an external peer
  reviewer
• Course faculty or instructor considered part of
  the review team

17
Whats In It For Institutions

• External validation
• Strengthen accreditation package
• Raise QA as a priority activity
• Access to a sustainable, replicable, scalable QA
  process
• Inform online course training practices
• Provide professional development activities

18
QM Process Provides

• Institutional toolset and process to meet quality
  expectations
• Online course design
• Student learning
• Improved instruction
• Assessment and feedback loops
• Professional development

19
Whats In It For Faculty

• Improve online courses
• External quality assurance
• Expand professional community
• Review other courses and gain new ideas for own
  course
• Participation useful for professional development
  plan and portfolio
• Receive 150 for each completed peer course
  review

20
QM in Transition

• 2003 August 2006
• QM project funded by FIPSE grant money
• materials and some services freely available
• August 2006 and beyond
• QM project funded by MarylandOnline
• Some limited materials will be freely available
• Other materials available to individuals and
  institutions at nominal fees
• Institutional membership affords full access to
  materials and services

21
The Instructor Worksheet
22
Instructor Worksheet

• Important part of review
• Includes info such as
• Institutionally mandated objectives, materials,
  practices, policies
• Materials outside course pages
• Types of interaction used instructors
  statement on the appropriateness of interaction
  in the course
• Additional items that may require review

23
Instructor Worksheet

• Read it first
• Refer to it during the review
• Use in team discussions

24
The Rubric
25
The Rubric

• Eight standards
• Course Overview and Introduction
• Learning Objectives
• Assessment and Measurement
• Resources and Materials
• Learner Interaction
• Course Technology
• Learner Support
• Accessibility

Key components must align.
26
What is Alignment?

• Critical course elements
• work together to ensure
• that students achieve
• the desired learning outcomes.

27
Key Sections that Must Align
28
Key Standards that Must Align

• Objectives
• Standard II.1 Measurable outcomes
• Standard II.2 The module/unit learning
  objectives describe outcomes that are measurable
  and consistent with the course-level objectives.
• Assessment and Measurement
• Standard III.1 Measures objectives consistent
  with learning activities
• Learner Interactions and Activities
• Standard V.1 Help students achieve the
  objectives
• Course Materials
• Standard IV.1 Deep and comprehensive enough for
  students to achieve the objective
• Course Technology
• Standard VI.1 Tools and media support the
  objectives

29
Rubric Scoring

• Team of three (3) reviewers
• One score per standard based on team majority
• Assigned point value not sliding scale

30
Assigned Point Values
Points are NOT assigned on a sliding scale
31
Awarding Points

• Points are awarded for each standard based on
• the team majority, and
• the pre-assigned weighting of each standard
• If 2-3 Reviewers believe that a standard is
• met, then the full pre-assigned points are
  awarded
• not met, then zero points are awarded

32
How to decide.

• For EACH standard
• Read the specific review standard and the
  annotation review the examples, if needed.
• Look for evidence that the standard is met in
  this course.
• Ask yourself Does this course meet the standard
  at an 85 or better level?
• Decide Yes or No and enter your answer in the
  web-based rubric form.
• Include comments/suggestions as documentation.

33
To Meet Expectations

• A course must achieve
• Yes on all 14 of the 3-point essential
  standards.
• A minimum of 68 out of 80 points

68/80 85
34
Online Hybrid Courses

• Rubric designed for application to fully online
  and hybrid/blended courses
• Same set of standards apply to both
• How we achieve the standards may differ
• For hybrids, focus on pedagogical integration of
  online and F2F components
• Refer to Course Format Definitions document

35
Recommendations
36
Writing a Useful Recommendation
37
When MUST you write a recommendation?

• You MUST write a recommendation if you
  decide that the course does not meet the
  specific standard.

38
Improve these Recommendations

• Assignment instructions werent clear.
• I wasnt sure if assignment 1 required a written
  paper.
• You didn't tell the students how to find the
  additional resources.
• I had trouble locating the resources you may
  want to put a link on the home page for easy
  access.
• The text on the page was too hard to read.
• The green text on a blue background was difficult
  to read. Suggest black on white.

39
Improve these Recommendations

• The learning objectives arent measurable.
• Rephrase the learning objectives to include an
  active verb (explain, distinguish, compare, etc.)
• The assessments are weak.
• The assessments are unrelated to the learning
  objectives. Review your learning objectives and
  make sure that they are reflected in your exams.

40
Hands-On Practice
41
Hands-On Practice

• Work in Pairs
• Review the Anthropology Course
• Go to http//pgcconline.blackboard.com
• Username is fipse
• Password is fipse
• Focus on 14 essential (3 point) standards
• Write 1 recommendation per standard

42
Steps

• Get your training pair assignment
• Find your computer in lab (2 per pair)
• One person logs into the course
• Other person logs into QM rubric
• Read Instructor Worksheet for course
• Then follow facilitators instructions for
  first standard

43
General Standard ICourse Overview and
Introduction

• 1.1 Navigational instructions make the
• organization of the course easy to
• understand.
• 1.2 A statement introduces the student to the
• course and to the structure of the student
• learning, and, in the case of a hybrid
• course, clarifies the relationship between
• the face-to-face and online components.

44
General Standard IILearning Objectives
(Competencies)

• II.I The course learning objectives
• describe outcomes that are
• measurable.
• II.2 The module/unit learning objectives
• describe outcomes that are measurable and
  consistent with the
• course-level objectives.

45
General Standard III Assessment and Measurement

• III.1 The types of assessments selected
• measure the stated learning objectives
• and are consistent with course activities
• and resources.
• III.2 The course grading policy is stated
  clearly.
• III.3 Specific and descriptive criteria are
• provided for the evaluation of students
• work and participation.

46
General Standard IV Resources and Materials

• IV.I The instructional materials support
• the stated learning objectives.
• IV.2 The instructional materials have
• sufficient breadth, depth, and
• currency for the student to learn the
• subject.

47
General Standard VLearner Interaction

• V.1 The learning activities promote the
• achievement of stated learning objectives.
• V.2 Learning activities foster instructor-
• student, content-student, and if appropriate
  to this course, student-student
• interaction.
• V.3 Clear standards are set for instructor
• response and availability (turn-around
• time for email, grade posting, etc.)

48
General Standard VICourse Technology

• VI.I The tools and media support the
• learning objectives and are
• appropriately chosen to deliver the
• content of the course.

49
General Standard VII Learner Support

• No 3-point elementsThis standard has no
  essential 3-point elements because its
  primarily concerned with academic support
  services, student support services and technical
  support services.usually thought to be the
  primary responsibility of the institution and not
  the individual instructor.

50
General Standard VIIIAccessibility

• VIII.1 The course acknowledges the importance of
• ADA requirements.
• To meet this standard, the course must include
  BOTH
• of these elements
• The course must be offered using software that is
  accepted as ADA compliant..
• AND
• The course should include a brief statement that
  clearly tells students how to access ADA services
  at the institution.

51
Showcase Your Courses
52
Course Showcase

• Think about your own course
• What do you do that meets QM expectations?
• What improvements might you make?
• Share your thoughts with your partner
• Summarize for the group

53
Scenarios
54
Evaluating Scenarios

• Divide into groups.
• Discuss the scenarios that focus on the 14
  essential standards.
• Take the quiz with your group. Use the
  material in your books and each other as
  references.
• Jot down your reasoning and be prepared to
  discuss your decision.

55
Peer Course Review Process
56
About the Course

• QM is designed to review mature courses (taught
  at least two semesters)
• QM logo indicates year course met expectations
• Triggers for subsequent reviews
• Faculty request
• More than 3 years since original review
• New textbook or instructor
• Professional or accreditation review pending

57
About the Review

• On average, a course review takes 7-10 hours
• Factors affecting review time include
• Reviewer familiarity with the discipline
• Reviewer familiarity with the CMS
• Reviewer familiarity with the QM review process
• Organization of the course
• Suggested review methodology
• Read Instructor worksheet
• Familiarize yourself with CMS and course
• Pre-review team discussion
• Proceed through standards, but save Standard III
  (Assessment and Measurement) for last
• Post-review team discussion
• Submit your review

58
Your Point of View as a QM Peer Course Reviewer

• Take the students point of view
• Advocate for the student
• If you cant find evidence that the standard is
  met, dont assume it is or isnt there.. ask the
  faculty member.

59
Timeline

• Active review period approximately 3 weeks
• Compiled reports due in approximately 6 weeks
• Teams
• Have pre-review discussion.
• Set Team calendar.
• Commit to 2-3 week review period.
• Have post-review discussion.

60
Post-Review

• Reviewers
• complete exit interview
• receive stipends
• Faculty
• receives Final Review Report
• completes Faculty Response Form

61
Review Outcome

• If meets expectations
• Recognized by Quality Matters
• Notifications distributed
• ID support provided if requested
• If does not yet meet expectations
• ID support provided if requested
• Team Chair and ID approve revisions
• Course meets expectations

62
Roles and Responsibilities
63
Faculty Developer

• Part of the review team.
• Provides access to the course.
• Completes Instructor Worksheet
• Part of the initial team discussion
• Receives compiled report
• Returns Faculty Response Form

64
Peer Reviewers

• Establish Team Calendar
• Review the course individually
• Complete the online web review form
• Discuss review with Team as needed
• Complete an Exit Interview
• Receive
• Recognition as Certified Peer Reviewer.
• Compensation

65
Team Chair

• Reviewer Roles Responsibilities plus.
• Organizes Team calendar
• Confirm Instructor Worksheet is used
• Creates draft report from compiled reviews
• Convenes Team discussions
• Reviews, edits and submits Team Report
• Receives
• Recognition as Certified Peer Reviewer and Chair.
• Compensation

66
QM To Date
67
QM to Date

• Overall Participation
• Individuals programs from 130 institutions
  across 28 states
• Course Reviews
• 103 courses reviewed
• 18 MD schools 10 non-MD schools
• Peer Reviewer Rubric Training
• 600 trained

68
Multiple Uses of QM

• Reported Uses of QM System
• Guidelines for initial online course development
• Quality assurance of existing courses
• Ongoing faculty professional development
• Institutional reaccredidation packages
• Formation of distance learning policies
  steering committees

69
Awards - 2005

• WCET Outstanding Work (WOW) Award, November
  2005.
• USDLA 21st Century Best Practice Award,October
  2005.
• Maryland Distance Learning Association (MDLA)
  Best Program Award,March 2005.

70
QM in Transition

• 2003 August 2006
• QM project funded by FIPSE grant money
• materials and some services freely available
• August 2006 and beyond
• QM project funded by MarylandOnline
• Some limited materials will be freely available
• Other materials available to individuals and
  institutions at nominal fees
• Institutional membership affords full access to
  materials and services

71
Research Findings
72
Overall Course Review Results

• Upon initial review
• 53 meet expectations
• 22 do not meet expectations - missing at least
  one essential 3-point element(s)
• 25 do not meet expectations - missing at least
  one essential 3 point element(s) and a minimum of
  68 points

73
Summary of Course Reviews

• variety of course management systems
• 50 Bb, 35 WebCT, 16 WebTycho,
• 2 other
• 65 of respondent Reviewers felt unfamiliarity
  with CMS did not impede review 34 somewhat
• variety of disciplines
• general studies, information technology, teacher
  education, engineering, allied health
• 75 of respondent Reviewers felt unfamiliarity
  with discipline did not impede review 25
  somewhat

74
Analysis of Infrastructure

• Whether a course meets/does not meet QM
• expectations is NOT correlated with
• Courses institution of origin
• Course management system
• 53 of Bb courses
• 54 of WebCT courses
• 50 of WebTycho courses

75
Analysis of Academic Area
76
Post-Course Review

• Met expectations
• Most faculty made suggested improvements even
  though their course met expectations!
• Did not meet expectations
• Improvements made or in progress
• Most improvements made by faculty
• Some ask for ID support

77
Common Themes

• Course reviews revealed 11 common areas for
  course improvement
• Elements that are missing in 20 or more of the
  courses reviewed
• These are potential targets for
• faculty training
• special attention in the initial course
  development phase

78
Common Areas for Improvement

• Instructor self-introduction (I.4) 22
• Activities that foster interaction (V.2)
  22
• Technology/skills/pre-req knowledge stated (I.6)
  24
• Links to academic support, student services,
  tutorials/resources (VII.2-VII.4)
  24-27
• Learning objectives at module/unit level (II.5)
  27
• Netiquette expectations (I.3) 32
• Self-check/practice with quick feedback (III.5)
  38
• B/W alternatives to color content (VIII.4)
  54
• Alternatives to auditory/visual
  content (VIII.2) 59

79
Serving as a QM Peer Reviewer
80
Quality Matters Peer Course ReviewerCertification
Process
Quality Matters Certification
Course Review Experience


Training

• Name on QM website
• Use of QM Logo
• Eligible for Peer Reviewer Pool

Assigned to Peer Review Team
Attend QM Training
Submit Report
Demonstrate Competencies
Trained
Certified
Kane 1/15/05
81
Quality MattersPeer Course Review Process
1. Course Selected
2. Trained Review Team Assigned
3. MOUs and Instructor Worksheet
4. Pre-Review Discussion
5. Individuals complete reviews
6. Team Discussion(s)
7. Final Review Report
82
Next Steps
83
Next Steps

• Within the next week
• Return to Foundations Blackboard site
• Complete Scenarios Quiz
• Grade of 85 or greater required to pass training
  and be a Peer Reviewer

84
Thanks to YOU Quality Matters!