An Online Peer Review System

1
An Online Peer Review System

• Scott Turner, Aaron Powell, Manas Tungare

2
Outline of Talk

• Why?
• Methods
• Rubric, In-Document Feedback
• Anonymity, Assignment, Evaluation of Reviews
• Implementation
• Workflow
• Design Decisions
• Demo
• Questions

3
Project Summary 4 Main Factors

• Peer review method
• Electronic support for peer review activities
• The Moodle course management system
• Keeping the system flexible to support
• Multiple peer review methods
• Largest possible audience

4
Why Online Peer Review?

• Can be qualitatively equal to face-to-face review
  Cross 1987
• Students can receive quicker and more thorough
  feedback from peers than from instructor
• Higher level learning skills - self-evaluation
• Interdependence, accountability
• More submissions, more reviews ? better grades.

5
Limitations

• Requires student expertise in the field and
  experience with peer review
• Sometimes, unnecessary friction may occur.
• Students evaluation scores can be unreliable for
  actual assessment. Chalk and Adeboyle 2004
• If instructor evaluates reviews, it can be more
  time-consuming
• Many peer review systems seem
• Very context-specific
• Not designed for novice users
• Not designed for flexibility of use

6
Methods

• Walkthroughs
• Inspections
• Online
• Rubric-based and In-document annotation
• Assigning reviewers to work
• by Instructor / Automatic
• Anonymity
• Double-blind, One-way, None.
• Evaluation of reviews

7
Workflow Instructor configures Peer-Review
Assignments

• Selects assignment or resource to review
• Writes instructions using a text editor
• Chooses method In-document, Rubric, or both
• Options
• resubmission of reviews
• Size limit
• Maximum points
• Anonymity
• Email notification
• Deadline

8
Workflow Instructor configures Peer-Review
Assignments

• Assigning Reviewers
• Automatically assign a selected number of reviews
  for each students work.
• Copy reviewer assignments from a previous peer
  review
• Manually

9
Workflow Instructor configures Peer-Review
Assignments

• Rubric Editor
• Create new rubric
• Reuse or copy an existing rubric
• Create criteria
• Reuse existing criteria (from previous rubrics)

10
Workflow Student Provides Feedback

• Receives Email notification that work is ready
  for review
• In-document review annotate in external
  application
• Rubric-based review score and comment the
  criteria on rubric form

11
Workflow Processing of Feedback

• Professor processes feedback
• List of all submissions per assignment, organized
  by reviewee or reviewer
• Provides own feedback on assignment as well as on
  review
• Student processes feedback
• Receives email notification when reviews are
  available
• Can view all feedback in his/her peer review
  assignment view

12
Design Decisions

• Replace Moodle assignment module, or design a
  separate peer-review module?
• All in one place, versus modular design.
• Reviewing other Moodle modules.
• Incorporating peer review into the official
  distribution.
• Upgrades to the assignment module by other devs.

13
Design Decisions

• Question styles, defaults
• Types of Anonymity
• Instructor feedback on peer reviews
• Use in grading

14
Evaluation

• Flexible enough for most peer review needs?
• Usability
• Instructor and student interfaces
• Improvements
• Detailed investigations
• Peer review method, regarding anonymity, rubrics
  and criteria, their affect on peer review quality.

15
Future Work

• More types of reviewable objects
• Enhanced notification
• Importing/exporting of rubrics and criteria
• Statistical analysis of rubrics and reviews
• Publishing best work with ratings, indexing of
  knowledge artifact- education engineering
  Gehringer, 2003

16
Questions, Comments?

• Thank you!