Does peer review improve undergraduates scientific reasoning skills

1
Does peer review improve undergraduates
scientific reasoning skills?

• Briana Timmerman and Denise Strickland
• Department of Biological Sciences
• University of South Carolina

Society Integrative Comparative Biology 2006
Annual Meeting
2
Motivation for the study

• We desire our students to be able to think like
  biologists
• 1100 majors, 31 faculty individual research
  projects are not realistic
• Solution incorporate a spiral of peer review
  and open-ended inquiry projects into lab courses
• Can such curriculum reforms improve students
  scientific reasoning skills?

3
Major Curriculum Reform Efforts

• exclusive use of open-ended labs (no cookbook
  labs with predetermined correct answer) and an
  emphasis on writing as part of the scientific
  process
• students engage in peer review of written work at
  least once per semester in required classes
• spiral of expectations and curriculum goals
• universal criteria for written lab reports used
  to measure student progress over time

4
Universal Criteria for Lab Reports

• Criteria 15 elements that can be expected in
  any good lab report regardless of topic.
• Introduction provides context (primary
  literature), baclground information accurate and
  relevant to question
• Hypotheses testable and consider alternatives,
  scientific merit
• Method controls and replication, experimental
  design
• Results data are comprehensive, relevant,
  accurate and logically presented, statistics
  correctly performed and interpreted
• Discussion conclusions based on data,
  alternative explanations eliminated, limitations
  considered, future implications indicated
• Primary Literature used to set context and
  interpret results
• Writing quality facilitates readers
  understanding
• Constructed as a rubric (expectations for student
  performance are described as performance levels
  from novice to expert).

5
Universal Rubic for Lab Reports

• Instructor selects and incorporates rubric
  elements appropriate for particular assignments.
• Full rubric and curriculum goals available to
  students on departmental website as a learning
  tool (www.biol.sc.edu/undergraduate)

6
Research question and measures

• What is the impact of the curriculum reforms,
  particularly the peer review experiences, on
  students scientific reasoning skills?
• Qualitative direct measure
• Improvement in student scientific thinking as
  evidenced in written lab reports after students
  had engaged in peer review and received peer
  feedback
• Quantitative correlative measure
• Improvement in student scores on a
  multiple-choice scientific reasoning test
  administered in five biology classes
  (introductory to senior level).

7
What was the peer review process?

• Students wrote drafts and uploaded to CPR
  website.
• Each reviewer was asked to provide 10 useful
  feedback items per draft.
• Quality of each feedback item was graded (1pt).
• Useful feedback defined as
• specific and concrete
• focused on the quality of the authors argument
  (not on grammar or writing mechanics)
• likely to result in meaningful change

8
Ability of Freshman as Peer Reviewers

• Freshman are relatively consistent
• Average standard deviation of peers ratings of
  text quality was 16 (n 121 calculations of std
  dev)
• and can provide useful feedback
• Average writer received 10.4 5.0 pieces of
  useful feedback (avg 3.7 per review) (n22
  writers)
• and used approximately 54 of the useful feedback
  they received.

9
Effect of Using Peer Feedback
10
What areas of student papers change as a result
of peer review?

• Biol 102 Draft to Final Lab Report on Evolution
• Areas of weakness in the draft papers ( of
  possible points earned by draft)
• Completeness of data presented (16)
• Clear refutation of alternative explanations
  (38)
• Explanation of evolutionary mechanisms (19)
• Discussion of implications and future directions
  (16)

11
Improvement from draft to final version due to
peer review comments
12
Scores earned by final version of lab reports
(peer reviewed)
(n50 per course)
Areas of weakness that improved from one course
to the next HT Hypothesis Testable and
considers alternatives DC Discussion
Conclusion based on data DA Discussion
elimination of Alternative explanations WQ
Writing Quality
13
Summary of qualitative data

• Freshman can provide effective peer feedback
• Peer review improves the quality of student
  papers.
• In freshman courses major improvement centers on
  concept of eliminating alternative explanations
  with data

14
Stepping back from a single lab report to
spanning the curriculum

• Future work longitudinal qualitative assessment
  of how students scientific abilities develop as
  they progress through the curriculum (portfolios
  of lab reports from freshman to senior year)
• Current work quantitative assessment using
  student cohorts as proxy for longitudinal
  viewpoint.

15
Do students scientific reasoning skills improve
across curriculum?

• Two-tiered scientific reasoning test (AE Lawson,
  2000 Arizona State Univ)
• administered in five biology courses in the same
  semester (cohorts proxy for longitudinal)
• scored on a 12 point scale (24 items, 12 pairs)
• content of scientific reasoning test is mostly
  non-biological

16
What is the correlation between scientific
reasoning and general academic maturity (includes
transfer students)?
Freshman and sophomores differ from upper
classman 4 yr gain 0.58 points (5)
17
What is the correlation between scientific
reasoning and time spent in USC curriculum?
Freshman differ from all other groups. 4 yr gain
0.92 (8)
18
What is the correlation between scientific
reasoning and experience with peer review?
One semester of peer review is insufficient. Two
semesters of experience with peer review is
necessary and produces a larger gain than general
academic maturity or time spent in the USC
curriculum (1.09 pts vs, 0.92 or 0.68) .
19
Summary

• Peer review improves student writing and helps
  students improve scientific skills in specific
  goal areas.
• Peer review improves student scientific reasoning
  skills more than general academic maturity.
• Single exposure to peer review is insufficient
  however multiple exposures needed.

20
Ongoing and Future projects

• Currently completing full-scale study of the
  reliability and validity of the Universal
  Rubric for written lab reports
• Compiling longitudinal portfolios of student lab
  reports from freshman to senior year to assess
  improvement in scientific skills over time.
• Creation and testing of effective teaching
  supports using peer review and formative feedback
  for graduate teaching assistants.
• Assessment of the impact of inquiry-based
  teaching on graduate students research skills.

21
Acknowledgements

• Biology graduate teaching assistants for their
  teaching efforts and administering the pre-post
  tests
• Sue Carstensen and Laurel Hester, (Biol 101 and
  102 lab coordinators)
• Drs. Lincoln (301), Reisman (302) and Sawyer
  (530) for their cooperation
• NSF DUE 0410992 Peer review as a mechanism for
  both curriculum reform and assessment
• Resources
• Universal Lab Rubric www.biol.sc.edu/undergraduat
  e.htm
• Presentation www.biol.sc.edu/timmerman or
  timmerman_at_biol.sc.edu
• Calibrated Peer Review freeware
  http//cpr.molsci.ucla.edu
• SafeAssignment www.safeassignment.com (runs
  through Blackboard)
• Scientific Reasoning Test Revised Edition A.E.
  Lawson, 2000 Arizona State Univ. Based on
  Lawson, A.E. 1978. Development and validation of
  the classroom test of formal reasoning. Journal
  of Research in Science Teaching, 15(1) 11-24.

22
Which lab courses correlate with the greatest
gain in scientific reasoning?
Only freshman level courses produced gains,
though lack of effort likely explanation for
sophomore and upper division courses.
23
Examples of Useful Peer Feedback

• What evidence do you have to support the
  assertion that natural selection did occur? What
  other evidence do you have to support that the
  owls did not kill the finches?
• What data do you have that supports the
  statement that if another drought happen on
  Daphne Major the finches will become extinct?
• You explained that your hypothesis is that the
  finches died due to a lack of food. You should
  include in your hypothesis, why there is a lack
  of food on Daphne Major.
• The amount of Rainfall compared to the number
  of seeds is a good hypothesis, but the idea that
  the hawks killed the finches isn't a good one to
  used since the hawk population was not measured.