Calibrated Peer ReviewTM Writing, Collaboration, and Critical Thinking http:cpr'molsci'ucla'edu

1
Calibrated Peer ReviewTMWriting,
Collaboration, and Critical Thinkinghttp//cpr.mo
lsci.ucla.edu

• Arlene A. Russell, UCLA
• University of Maryland
• November 13, 2003

2

• http//cpr.molsci.ucla.edu
• Calibrated Peer Review (CPR)
• was created under the Molecular Science Project,
  an NSF-funded systemic reform project
• is an instructional tool that uses writing and
  peer review to teach higher-order thinking skills
  and peer collaboration
• manages the writing and anonymous review
  processes of instructor-selected assignments

3

• Integrate telecommunications and technology into
  instructional processes so that students learn to
• explore ideas independently
• write about chemistry
• collaborate
• work to a mastery level
• take responsibility for their own learning

4
Tenets of CPR

• Expository writing promotes understanding
• Clear writing demonstrates clear thinking
• Peer review and evaluation require higher-order,
  critical thinking skills
• Writing-to-learn ¹ Learning-to-write
• But, writing-to-learn can (and should) include
  attention to writing skills

5
Writing-to-Learn

• Used extensively in humanities and social
  sciences
• Requires small classes
• Science and engineering have put their resources
  into labs
• Imposes intensive grading workload
• Seldom used in large science classes, because
• Workload too high, not the job of scientists,
  large classes, not objective,.

6
Peer Review

• Accepted process for validating scientific
  research
• Essential skill required of all scientists
• Seldom taught in science classes
• Not used in large lecture courses

7
Peer Review

• Peer Assessment Between Students in Colleges
  and Universities Review of Educational
  Research, Fall 1998, 68, pp 249-276
• Little use of computers except to submit work

8
Peer Review

• Student Peer Assessment in Higher Education A
  Meta-Analysis Comparing Peer and Teacher Marks
  N. Falchikov and J. Goldfinch, Review of
  Educational Research, Fall 2000, 70, pp 287-322
• Peer assessments resembled teacher assessments
  most closely when global judgements based on
  well understood criteria were used.
• Advanced students not better assessors than
  beginner students.

9
Calibrated Peer Review (CPR)

• Both writing and reviewing improve students
  understanding of the topic
• involves writing (reflective thinking)
• involves reviewing (critical thinking)

10
Students Interaction with CPR

• Students
• carry out a guided study of source material
• write an essay (300 words) about what they have
  studied.
• are trained as reviewers of this topic by
  studying a set of Calibration essays and
  evaluating them on specific questions
• anonymously review three of their peers essays
  and provide written feedback to them
• apply their evaluation skills to self-review
  their own essays

11
Instructor Responsibility

• Selects the topics from the library of
  assignments
• Sets grading criteria and due dates for
  submission of essays and reviews
• Monitors student progress and performance
• (Participates in the assignment as a student)

12
(No Transcript)
13
So What? Do CPR Assignments Make a Difference?

• Is content knowledge improved?
• Are evaluation skills developed?
• Do students learn to collaborate?
• What do students think about CPR?

14
Evaluation of CPR Economics - CSUN

• Comparison of intact classes
• Ten case-study assignments
• Three sections (same instructor) of Economics
  200 2 CPR based, 1 traditional with essays but
  no evaluation component
• Repeated over two semesters

15
(No Transcript)
16
Evaluation of CPR Biology - CSUF

• One class, one instructor
• Time-series design
• Ten topics, 5 taught through CPR, alternately
  with 5 taught through lecture and simulation
• Nancy J. Pelaez Problem-based Writing with Peer
  Review Improves Academic Performance in
  Physiology Advan Physiol Educ, Sep 2002 26 174
  - 184.

17
(No Transcript)
18
Evaluation of CPRLife Science 1H - UCLA

• One class, one instructor (26 students)
• Three CPR assignments
• Multiple-choice midterm and final exam questions
  classified as
• relating directly to topics in CPR assignments
• requiring transfer of understanding of CPR topics
• addressing non-CPR topics.
• Mean performance 84.8 on CPR topics, 79.7 on
  non-CPR topics (paired t-test p 0.006)

19
(No Transcript)
20
Evaluation of CPRChemistry - UCLA

• General chemistry
• Midterm question on theory of two-component phase
  diagram
• Two quarters traditional lecture and problems
• One-quarter with an additional CPR assignment

21
Exam Scores

• Quarter CPR Actual Percent
  use grade
• Winter 1998 No 6.2/10 62
• Winter 2000 No 9.2/15 61
• Spring 2000 Yes 15.1/20 71

22
Evaluation of CPR-writing Skills Chemistry- UCLA

• Senior level environmental chemistry all
  evaluation based on writing
• Midterm - 3 essays on issues -topics specified
• Final - 6 essays on issues, two on topics covered
  by CPR assignments
• 3 CPR assignments given between midterm and final

23
(No Transcript)
24
Evaluation of CPR-Reviewing Skill

• Ten courses, taught by 8 instructors, from 8
  different institutions, for 5 different subjects
• Paired t-test comparison of students ability to
  review assignments at beginning of term with
  performance on last assignment in course.

25
(No Transcript)
26
Peer Review Feedback
27
(No Transcript)
28
Peer Review Feedback
29
Introductory Organic Chemistry

• Research University
• lecture three times/week
• no discussion
• separate lab course
• most students premed
• one CPR assignment Study of Cycloalkanes
• uses CHIME interactive tutorial to study ring
  strain
• given as extra credit assignment at end of
  quarter
• feedback e-mail to instructor required for credit

30
Introductory Organic ChemistryStudent Buy-In

• 219 of 372 students did the assignment and
  submitted comments!

31
Student Perceptions
32
Student Comments Technical Issues

• The chime plugin was a little problem because I
  too use netscape 6.1 so I ended up using Internet
  Explorer 6.0. It handled the plugin well and did
  not give me any problems. The web site performed
  decently with a 56K modem connection, except the
  chime illustrations took about 30sec to 45sec to
  load but acceptable. Now that the calibrations
  are available the calibration screens seem very
  user friendly but I would tell everyone that it
  is best to use a screen resolution of at least
  800x600 because there is so much to info to
  display. The questions take half the screen and
  the essay takes up the other half.

33
Source Material/Reviewing

• I thought that the CPR was really interesting.
  I liked the fact that they made us look at 3D
  models before we could answer the questions
  because it gave us a good idea of the concept
  involved in the question. I think that the
  process we had to take before writing the essay
  was great because it took us step by step to the
  answer, allowing us to thoroughly comprehend the
  strain (angle and torsional) in cycloalkanes in
  comparison to their alkane counterparts and to
  other larger or smaller cycloalkanes. I also
  like that we will get to grade other students
  essays and later see if our own essays were of
  good quality. I believe that it will let us know
  if we really understand the part that strain
  plays in cycloalkanes.

34
Workload, Course Value

• This program is similar to something that my
  girlfriend is doing in her Christian theology
  class at BIOLA. She writes a 400-word response
  to a "poll" question based on the 400-500 page
  reading assignment that week, every week. Then
  she reviews the work of three others (but not her
  own), followed by an in class discussion once a
  week. Of course, an in-class discussion is not
  appropriate for a chemistry course and the
  material covered requires more work than a simple
  read and response. This is why I feel that the
  CPR program would be more beneficial if it were
  assigned more than once a quarter. Something
  more like one assignment every 2-3 weeks. I
  really feel that this would improve the overall
  information retention of the class.

35
Writing

• By writing this assignment in a mini-essay form I
  was able to come at my own conclusions and
  express the material in my words. I believe that
  this was very helpful because in order to
  understand the material it is extremely useful to
  be able to describe and put the material one
  learned into his/her own words

36
Workload, Course Value

• I enjoyed the CPR assignment. I liked having
  another format to think about organic chemistry.
  If you do this again, I think optical rotation
  would be a great subject. I thought the writing
  assignment was a good length--not too much to
  read six examples of writing

37
Workload, Value, Reviewing

• I thought the idea of CPR was a very good idea
  because it helped expanding students' knowledge
  and developing insight into the materials we had
  studied. However, I didn't really enjoy this
  project. I didn't like it because there was too
  much research involved and questions were very
  hard to answer. This project itself was time
  consuming as well. It took me a long time to
  finish the practice before I moved onto the
  actual project. Another time-consuming part was
  the peer reviewing. It took me about an average
  of 10 minutes each to review one paper. It
  required a lot of concentration as well

38
Course Value

• I thought CPR to be good learning tool. The
  project made the participants review the course
  material. As a result a better understanding of
  the material is attained. Its like having two
  lectures instead of one. Making an environment
  where students have to think is a good way to
  make the material understandable.

39
Reviewing, Course Value

• Truthfully, I enjoyed CPR. Writing the essay was
  a great way to  review the specifics of
  cyclohxane strain, but when I reviewed the  peer
  essays I think I learned most. I was able to see
  what strong  points my essay had, and more
  importantly where I was lacking. I  think CPR can
  be a very useful tool for students to use if they
  take  it seriously and take the time to explore
  the website. Also, it is  another chance for
  points, which is always good. By the way, thanks
   for the extra credit.

40
Timing, Workload

• this project was sort of like high school work.
  it was only  assigned to keep me busy. first of
  all i don't have a computer so i  had to go out
  of my way to use one. we studied cyclohexanes for
  the  first midterm. it would've been a lot better
  if it was material from  the current chapters.

41
Word Count, Technical Issues

• I just wanted to give you my opinion of cpr. I
  felt like it  was to much of a hassle. It was
  hard to log onto and frustrating to  deal with.
  When I logged on and began writing my text the
  HTML tutor  wasn't accessible so I didn't know
  the editing commands. I know that  we were
  supposed to be concise, but I felt that 230-390
  words just  wasn't enough to truly answer the
  questions and write a good essay.  Anyways, I
  didn't think cpr was really worth it because it
  was to  much of a hassle.

42

• .just submitted my CPR essay. All in all it
  isn't that bad.  It's nice and short so that I
  ended up editting a lot of what I wrote  so that
  I reached the maximum length and that's better
  than trying to  think of crud to insert here and
  there to make a nice long essay.  After the
  hurdle of trying to sit down and writing
  something about  ochem, the experience itself is
  relatively brief. It would have been  nice to
  know who our target audience is. I know that
  other peers are  reading this, but for an
  example, are we supposed to clearly explain  what
  eclipsing bonds means? Initially I did what it
  is, but there  isn't enough word allowance for it
  so I took it out. The bad thing I  don't like is
  that the topic is a bit late. I wouldn't mind
  covering  a subject that is supposed to be
  covered in the next few weeks but I  think
  writing a paper about something we had covered
  for the first  midterm is a waste of time.

43
Foreign Students

• The calibrated peer review helped me to review
  many knowledges about the conformation of
  cyclohexane. I learned a lot from it

44
Technical Difficulties and Attitude to CPR

• Positive Negative
• Comment Comment
• Mentioned problems
• with CHIME 59 41
• (/or UCR and CPR login)
• No technical difficulties 82
  18
• mentioned

45
http//cpr.molsci.ucla.edu

• Integrates telecommunications and technology into
  instructional processes so that students learn to
• explore ideas independently
• write about chemistry
• collaborate
• work to a mastery level
• take responsibility for their own learning

46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
Acknowledgements

• Funding sources
• National Science Foundation
• Howard Hughes Medical Institute
• University of California, Office of the President
• University of California, Los Angeles

50
Acknowledgements

• Colleagues
• Orville L. Chapman
• Tim Su
• Krista Motschiedler Brand
• Stephen Schimpf (Michael Fiore)
• authors
• students

51

• Thank you