Factors Affecting Student Acceptance of Calibrated Peer ReviewTM

1
Factors Affecting Student Acceptance of
Calibrated Peer ReviewTM

• By
• Wendy L. Keeney-Kennicutt, Ph.D.
• Master Administrator, CPR, TAMU
• Associate Director, First Year Chem Program
• k-keeney_at_tamu.edu

2
WHAT IS CPR?

• A Web-based instructional writing and
  peer assessing tool
• Originated in the Molecular Science Project, an
  NSF-sponsored chemistry reform project (DUE
  95-55605) at UCLA
• Enables you to learn by writing about significant
  topics in a course, then going through a
  critiquing process 7 times

FREE
3
HISTORY AT TAMUcpr.tamu.edu

• 2002 - CPRTM was introduced to TAMU by me, Dr.
  Wendy Keeney-Kennicutt after a Chemical
  Education workshop demo.
• 2003 - To avoid FERPA issues because of WALS
  NSF grant thru CTE, CPRTM was housed on a secure
  TAMU server.
• 2003 - I volunteered to be the CPRTM
  administrator because of experience.

WALS Writing for Assessment and Learning in
the Natural and Mathematical Sciences
4
USE OF CPR AT TAMU

• In the last 5 years, CPR at TAMU has been used by
  approximately
• 19,000 undergraduate and graduate students doing
• 492 new assignments by
• 384 instructors in
• 400 courses spread over
• 30 majors in
• 9 colleges. 

5
MAJORS USING CPRTM

• Accounting
• Ag. Economics
• Animal Science
• Anthropology
• Archeology
• Biochemistry
• Biology
• Botany
• Bus. Admin.
• Ctr. Acad. Enh.
• Chemistry

• English
• Ed. Curriculum
• Ed. Psychology
• Engineering
• Film
• French
• German
• Geography
• Kinesiology
• Learn. Comm.
• Math

• Microbiology
• Nutrition
• Physics
• Poultry Science
• Psychology
• Reading
• Secondary Ed.
• Vet Integr. Bio. Sci.
• Wildlife Fish. Sci.
• Zoology

6
What is CPRTM?

• Website cpr.tamu.edu
• Faculty create assignments with
• instructions,
• suggested resources,
• questions to guide student thinking,
• a writing prompt including topic, format,
  audience,
• calibration questions (grading rubric) and
• 3 sample essays (high, average, low quality) with
  feedback for calibration questions

7
What is CPRTM?

• Students work in 3 phases
• Text entry students write/submit essays
• Calibration phase students
• Are presented with 3 calibration essays,
• Answer calibration questions,
• Assign ratings,
• Receive reviewing competency score
• Review phase students
• Are presented with 3 of their peers essays
  (randomly selected and anonymous) and their own
• Review and rate using calibration questions

8
Why do I think CPR is important?(1) Writing is
a Teaching/Learning ToolWriting-Across-the-Disci
pline

• Writing
• Promotes critical thinking skills
• Helps extend knowledge
• Helps to structure rough ideas into coherence
• Helps prepare students for future careers by
  writing in the discipline

9
(2) Peer Review is a Teaching/Learning Tool

• Peer Review
• Has students working at the highest levels of
  Blooms Taxonomy (next slide)
• Gives practice in developing performance criteria
• Encourages self-reflection, responsibility
• Issues
• Students do not like criticizing friends
• Students perceive the grades are arbitrary and
  should be done by instructors

10
Evaluation
Judgment the ability to make decisions and
support views requires understanding of values
Combination of information to form a
unique product requires creativity and
originality
Synthesis
Identification of component parts determination
of arrangement, logic, semantics
Analysis
Use of information to solve problems transfer of
abstract or theoretical ideas to practical
situations.
Application
Identification of connections and relationships
Interpretation
Restatement in your own words paraphrase summary
Translation
Verbatim information memorization with no
evidence of understanding
Recall
Blooms Taxonomy categorizing level of
abstraction of questions
11
Introduction to Study

• This mixed methods study is a continuation of a
  previous study investigating how one instructor
  (me) worked over 7 semesters to try to overcome
  student resistance to a technology-based tool
  Calibrated Peer ReviewTM (CPRTM) . Feedback was
  obtained from SALG Student Assessment of
  Learning Gains.
• The research was published in the International
  Journal of the Scholarship of Teaching and
  Learning (Jan. 2008)
• Overcoming Student Resistance to a Teaching
  Innovation
• http//www.georgiasouthern.edu/ijsotl/issue_v2n1
  .htm
• Here I look in detail at the studys last
  semesters (Spring 2006) results, relating CPR
  grades and student responses on questionnaires on
  CPR to gender, learning styles, and exam grades.

12
Student Assessment of Learning Gainswww.wcer.wisc
.edu/salgains/instructor/

• Students log into the site with their name so
  credit can be given
• SALG dissociates names from responses to ensure
  anonymity

13
Previous Study Quantitative Results
We saw a significant increase in student
acceptance and understanding of CPRTM over time.
14
Previous Study - Qualitative Results Do you
think that future classes should do CPR?
Explain.
15
Methodology for This Study

• Study group
• my Chemistry 101 class in Spring 2006
• Students were asked to complete anonymous SALG
  survey and an identical non-anonymous written
  survey for 5pts each on final exam
• completion rate 98 (N235) for on-line survey
  and 87 (N209) for written survey based on 241
  students who took the final exam.

16
Course Grade Setup

• 4 hr course 3 hr lecture 1 hr lab
• Lecture grade
• 3 exams 100 pts each
• Final exam 170 pts
• In class quizzes 30 pts
• Computerized homework 50 pts
• CPRTM 100 pts total (12 of final class grade)
• Average of best 3 of 4 assignments
• Written homework 0,1,2 pts added onto final
  lecture grade
• of lecture grade that is not exams 30
• of course grade that is not exams 55

17
Methodology Quantitative Results

• Survey Statements
• Q1-4 Likert scale items (1strongly
  disagree5strongly agree)
• Q5 Yes/No with comments (used for qualitative
  study)
• I enjoyed doing the CPRTM assignments.
• CPRTM helped me learn some chemistry.
• CPRTM helped me improve my writing skills.
• CPRTM helped me improve my critiquing skills.
• Do you think that future classes should do CPRTM?
  (Y/N)
• Question data were not significantly different
  (plt0.05) between the surveys except on Q2 CPR
  helped me learn chemistry.
• Students who took written survey gave CPR higher
  marks mean 3.73 vs. 3.42 (plt0.002, Cohens d
  0.30, medium effect)
• Test Reliability (Cronbachs alpha) of Q1-40.771
  (written) and 0.715 (online).

18
What is Cohens d?http//web.uccs.edu/lbecker/Psy
590/es.htm
19
Previous Findings Quantitative Results

• From the odds ratios determined from the
  Chi-square 2x2 contingency tables
• Students who enjoyed CPR were
• 10 times more likely to think they learned more
  chemistry than those who didnt enjoy CPRTM
• 7 times more likely to think CPRTM improved
  writing skills, and
• 12 times more likely to think CPRTM improved
  critiquing skills
• While it is not necessary for students to like
  a particular learning tool in order to benefit
  from it, this analysis demonstrated that students
  who enjoyed CPR reported that they received 7-12
  times more benefit (with regard to their
  learning, writing skills and critiquing skills)
  from it than those who did not.
• This is why it was important for me to look
  further into which students enjoyed CPR.

20
Findings Quantitative Results
Gender Differences How well they did

• Students who took the final exam (N241)
• 53.7 female 46.3 male
• Significant differences (plt0.05) found
• Overall grade 79.74 ? 12.07 (female) Cohens
  d0.37 74.81 ? 14.37 (male)
• Course grade 79.22 ? 13.42 (female) Cohens
  d0.36 75.27 ? 13.57 (male)
• Lab grade 86.02 ? 10.98 (female) Cohens
  d0.52 79.34 ? 14.52 (male)
• Electronic HW 42.29 ? 13.01 (female) Cohens
  d0.40 (max50) 36.41 ? 16.33 (male)
• Written HW 90.41 ? 31.81 (female) Cohens
  d0.45 (max114) 65.34 ? 44.57 (male)
• In Class Quizzes 35.81 ? 7.57 (female)
  Cohens d0.40 (max48) 32.16 ? 10.26 (male)
• No difference (plt0.05) were found for exam score
  average
• 71.6 (female) vs. 69.6 (male)

21
Findings Quantitative Results
Gender Differences How well they did

• What about CPR?
• Significant differences (plt0.05) found
• CPR grade 88.85 ? 9.58 (female) Cohens
  d0.28 85.67 ? 12.55 (male)
• CPR text 7.82 ? 0.90 (female) Cohens
  d0.36 7.42 ? 1.27 (male)
• No differences (plt0.05) were found for
• CPR reviewing,
• critiquing and
• self-assessment scores

22
Findings Quantitative Results
Gender Differences How well they did

• What do the results mean?
• Apparently, female students seem to work
  significantly more diligently than male students,
  since they did better on the assignments that
  required time on task.
• Both achieve similar results on exams.
• In my course where I reward work (55 of grade),
  female students have an edge.
• Female students did significantly better on
  overall CPR grade and writing portion. Why?
• Female students are more willing to put in the
  time?
• Female students have better writing skills?

23
Findings Quantitative Results
Gender Differences What they thought

• No significant differences (plt0.05) between male
  and female students were found in their responses
  to the 5 questions
• I enjoyed doing the CPRTM assignments.
• CPRTM helped me learn some chemistry.
• CPRTM helped me improve my writing skills.
• CPRTM helped me improve my critiquing skills.
• Do you think that future classes should do CPRTM?
  (Y/N)

24
Findings Qualitative Results
Gender Differences What they thought

• No significant differences (plt0.05) between male
  and female students were found in the percentages
  of positive, mixed and negative comments.

25
Learning Style Differences How well they did
and what they thought
Findings Quantitative Results
Students took a free Jung Typology test at
http//www.humanmetrics.com/ as a quiz grade.
NF need acceptance, caring and support, learn
best face to face, enjoy group activities NT
interested in principles and logic, technology,
do not enjoy meaningless assignments SJ value
responsibility, dependability, conformity,
structure, factual information SP thrive on
verbal, visual and immediate approaches,
entertainment
There was no statistical difference (plt0.05)
between their overall CPRTM grades and learning
styles, or in their responses to the five
questions, but something seemed to be going on..
26
Learning Style Differences How well they did
Findings Quantitative Results
NF need acceptance, caring and support, learn
best face to face, enjoy group activities NT
interested in principles and logic, technology,
do not enjoy meaningless assignments SJ value
responsibility, dependability, conformity,
structure, factual information SP thrive on
verbal, visual and immediate approaches,
entertainment
27
Learning Style Differences How well they did
Findings Quantitative Results
NF need acceptance, caring and support, learn
best face to face, enjoy group activities NT
interested in principles and logic, technology,
do not enjoy meaningless assignments SJ value
responsibility, dependability, conformity,
structure, factual information SP thrive on
verbal, visual and immediate approaches,
entertainment
So, how did each group feel about CPR?
28
Learning Style Differences What they thought
Findings Qualitative Results
NF need acceptance, caring and support, learn
best face to face, enjoy group activities NT
interested in principles and logic, technology,
do not enjoy meaningless assignments SJ value
responsibility, dependability, conformity,
structure, factual information SP thrive on
verbal, visual and immediate approaches,
entertainment
N students are more intuitive, skip the
details, seek the big picture and the
identification of patterns, while S students
like the step-by-step approach, are more
utilitarian, use learned skills, care about
details
29
Learning Style Differences How well they did
Findings Quantitative Results
NF need acceptance, caring and support, learn
best face to face, enjoy group activities NT
interested in principles and logic, technology,
do not enjoy meaningless assignments SJ value
responsibility, dependability, conformity,
structure, factual information SP thrive on
verbal, visual and immediate approaches,
entertainment
Pearson Correlation Coefficient, r, between
student exam and CPR averages was 0.355
(plt0.001) accounting for only 13 of the
variability.
30
Questions

• So is CPR a different way of measuring how well
  a student learned chemistry? And not simply
  preparation for an exam? Is it a way that some
  students who arent good test takers can show
  that they have learned a subject?
• These results lead me to directly look at how
  student exam performance affected their
  performance and thoughts on CPR.

31
Findings Quantitative Results
Exam Grade Differences
Obviously there is a large significant difference
in means between the students lecture exam
average and their CPR average (plt0.001, Cohens d
1.26). But the students who did well on exams
are not the same as the students who did well on
CPR.
32
Findings Quantitative Results
Exam Grade Differences
Pearson Correlation Coefficient, r, between
student exam and CPR averages was 0.355
(plt0.001) accounting for only r2x100 13 of the
variability. Even with the extreme outliers
removed, r2x100 19.
33
Findings Quantitative and Qualitative Results
Exam Grade Differences
Q1-4 Likert scale items (1strongly
disagree5strongly agree)
34
Findings Quantitative and Qualitative Results
Exam Grade Differences

• Findings
• My best test-takers were not as thrilled with
  CPR as their less academic peers, except when
  it came to critiquing.
• Number of positive comments almost doubled from
  ?90 students to the lt60 students.

35
Findings - Qualitative
Telling Comments from the ?90 Students
Positive Comments

• CPR taught me more chemistry and it also taught
  me time management because of all the other
  assignments we had in chemistry. It was
  beneficial to me and would be to future students
  as well.
• It gives students an opportunity to experience
  technical writing.
• I think it is a way to try to reach different
  ways that students learn.

36
Findings - Qualitative
Telling Comments from the ?90 Students
Negative Comments

• I don't think CPR helps with chemistry. I have
  gotten low grades not because people grade me
  low, but because I expect a lot in writing out of
  a college student and tend to grade "harder" than
  other students. Therefore, my grade for other
  students is not in the correct range and I get no
  credit for that grading. Also, I don't think
  someone who is not a strong writer should be
  grading the work of others. The grades do not
  adequately represent writing ability.
• The assignments were very time-consuming. People
  were not fair in their grading.
• We didn't have to learn the material for the
  writings. We just had to be able to critique
  well to get a good score. If we disagreed with
  the scorings of our classmates we did poorly. I
  did the best on an assignment for which I
  answered the questions incorrectly.

37
Findings - Qualitative
Telling Comments from the lt60 Students
Positive Comments

• They should do the CPR because it helps you learn
  concepts of chem.
• Because they will be able to write like a true
  science student.
• Because it challenges you and also helps you to
  improve your writing skills.
• CPR is great for students who do bad on the
  tests. It also helped to improve my writing
  skills.
• It helped with understanding the material in a
  different way. It is also a bit of a grade
  boost.
• Science students should know how to frame
  arguments in their respective fields. I think it
  would be silly to cancel this program since so
  few science teachers offer any writing assistance.

38
Findings - Qualitative
Telling Comments from the lt60 Students
Negative Comments

• I felt CPR was very time consuming and feel that
  time could be spent on studying the lecture
  material.
• The class has enough as is, and writing
  assignments just add to the horrible load of the
  class.
• Not an effective use of time.

39
Conclusions

• Gender played a role in how well students did in
  the writing part of CPR, but there was no gender
  difference on what students thought of CPR.
• Females seemed to take assignments overall more
  seriously and do better when time on task was
  important
• Learning styles had a minor effect.
• S students made significantly fewer negative
  comments about CPR than N students
• The students who did best on exams were less
  inclined to appreciate CPRs attributes than
  their peers. The exception was in critiquing.
• Number of positive comments almost doubled from
  ?90 students to the lt60 students

40
Acknowledgements

• Adalet Baris Gunersel
• Nancy J. Simpson, Ph.D.
• My Chem 101 class, Spring 2006