Calibrated Peer Review

1
Calibrated Peer Review (CPR)Using Technology
to Get the Most Out of a Writing Assignment

• Dr. Laurie S. Starkey
• Chemistry Dept., Cal Poly Pomona
• lsstarkey_at_csupomona.edu [phone]
  http//www.csupomona.edu/lsstarkey

2
Calibrated Peer Review What is CPR?

• Web-based tool which enables peer-reviewed
  writing assignments
• Students, faculty log in with password
  athttp//cpr.molsci.ucla.eduto access
  assignments
• Use and access are currently free servers are
  maintained by UCLA

3
CPR Assignment Components Stage 1 Text Entry
Stage

• Essay assignment is given online
• Links to supporting materials, relevant web
  sites, animations, etc., are provided
• Student Instructions, Guiding Questions and
  Writing Prompt are also available
• Essay must be formatted using simple HTML tags
  (can Preview before submitting)
• Student submits essay online before text entry
  deadline (CPR verifies word count)

4
CPR Stage 2 Calibration Stage

• Begins only after text entry deadline
• Student reads, evaluates 3 Calibration Essays
• faculty-generated essays are of varying quality
• Evaluation Student answers 10-12 Calibration
  Questions for each essay and assigns 1-10 score
• Student views Calibration results
• Do Students answers match Instructors answers?
• Feedback is provided for each Calibration Qn
• Student can retake Calibration after failing
• Student becomes Expert Reviewer

5
Sample Calibration Questions

• Content (accurate, complete)
• Does the essay conclude that 150.0 m has four
  significant figures?
• Does the essay identify George Washington as the
  first president of the United States of America?
• Are there scientific errors in the essay, such as
  incorrect definitions or misleading statements?
• Style (coherent, integrated, grammar)
• Does the essay have a descriptive topic sentence?
• Is the authors argument easy to follow? (Y/N)
• Are there spelling errors? (None, Some, Many)

6
CPR Stage 3 Review Stage

• Begins after passing Calibrations Stage
• Review Answer Questions, Score 1-10
• Student reviews three classmates essays
• Anonymous and randomly assigned
• Classmates may include the instructor, TAs,
  grad students, faculty, etc.
• Student performs a self-evaluation (reviews his
  or her own work)

7
CPR Stage 4 Results Stage

• Did Student master review of peers?
• Students 1-10 rating is compared to average
• Instructor sets maximum allowable deviation
• Student views detailed reviews of his/her own
  text
• Four reviews have been performed three peer
  reviews plus Self-Assessment
• Compare answers for each Calibration Question
• View feedback provided by peers

8
Computing CPR Assignment Grade

• Students Text Entry (30 points)
• Calculates weighted average score based on
  reviews
• Score of 10 30 pts Score of 5 15 pts
• Students Calibrations (3 x 10 points each)
• Pass 10 pts Pass after retake 5 pts Fail
  0 pts
• Reviews performed by Student (3 x 10 points ea)
• Within allowable deviation 10 pts exceeds 0
  pts
• Students Self-Assessment (10 points)
• Within allowable deviation 10 pts exceeds 0
  pts

9
Advantages to using CPR

• Getting more out of a writing assignment
• Time on task write, read 6 essays, self-reflect
• Evaluates and promotes reading comprehension and
  critical thinking
• Ideal for content-heavy assignments
• Lots of feedback, little Instructors time
• Peers are trained to be reviewers
• Utilizes time outside of class
• Give Pre-Service teachers grading experience
• Provides experience using HTML

10
Is CPR right for me?

• Ideal for online courses or source materials
• Assignment runs about 2 weeks
• Not suitable for every writing assignment
• Develop Calibration Qns with Y/N answers
• Need significant time for initial investment
  (authoring a CPR assignment)

11
How do I begin using CPR?

• Library contains over 170 CPR assignments
• Chemistry, Biology, English, Theater, etc.
• Contact CPP CPR Administrator (L. Starkey)
• Provide Handouts (Intro to CPR Authoring
  Assignments)
• Create faculty user give authoring privileges
• Create a course
• Create CPR accounts for student users
• Add students to CPR course
• Attend UCLA workshop, July 19-21, 2006

12
Acknowledgments

• UCLA workshop, Summer 2001
• Arlene A. Russell, Krista Motschiedler Brand
• Cal Poly Faculty Computing Support Center
• Cal Poly Pomona Chemistry Department