What evidence will we accept?

1
Assessing the Impact of Integrating Scientific
Research and Education

• What evidence will we accept?

Diane Ebert-May Michigan State University Celia
Evans Paul Smiths College
2
Anonymous (Change Magazine, 2001)

• I believe we would all agree that the absolute
  best teaching learning-assessment model is the
  one-on-one Socratic apprenticeship model with
  unlimited time with the student. But ever since
  Socrates took on two students rather than only
  one (to double his income), teachers have had to
  make compromises in teaching.

3
Question 1
Please respond on a scale if 0-100 in increments
of 10

• How important is it to use multiple kinds of
  data to assess student learning?

4
How important is it to use multiple forms of data
to assess student learning?

Relative Importance
n127
5
Question 2
Please respond on a scale of 0 - 100 in
increments of 10

• How often do you use data to make instructional
  decisions?

6
How often do you use data to make instructional
decisions?

Frequency
n127
7
True or False?

• Assessing student learning in science parallels
  what scientists do as researchers.

8
Parallel ask questions

• Description
• -What is happening?
• Cause
• -Does x (teaching strategy) affect y
  (understanding)?
• Process or mechanism
• -Why or how does x cause y?

9
Parallel collect data

• We collect data to find out what our students
  know.
• Data helps us understand student thinking about
  concepts and content.
• We use data to guide decisions about
  course/curriculum/innovative instruction

10
Parallel analyze data

• Quantitative data - statistical analysis
• Qualitative data
• break into manageable units and define coding
  categories
• search for patterns, quantify
• interpret and synthesize
• Valid and repeatable measures

11
Parallel peer review

• Ideas and results are peer reviewed - formally
  and/or informally.

12
What is assessment?

• Data collection with the purpose of answering
  questions about
• students understanding
• students attitudes
• students skills
• instructional design and implementation
• curricular reform (at multiple grainsizes)

13
Why do assessment?
Improve student learning and development.
Provides students substantive feedback about
their understanding.
Challenge to use disciplinary research strategies
to assess learning.
14
Research Methods
15
Data collection approaches
16
Theoretical Framework Ausubel 1968 meaningful
learning Novak 1998 visual representations
King and Kitchner 1994 reflective judgment
National Research Council 1999 theoretical
frameworks for assessment
17
Pre-Posttest Analysis
Does active, inquiry-based instructional design
influence students understanding of evolution
and natural selection?
18
Alternative Conceptions Natural Selection
Changes in a population occur through a gradual
change in individual members of a population.
New traits in species are developed in response
to need. All members of a population are
genetically equivalent, variation and fitness are
not considered. Traits acquired during an
individuals lifetime will be inherited by
offspring.
19
Instructional Design
Cooperative groups in class
Guppy Problem sexual vs. natural selection
-PBS film -Simulation -Analyze data -Written
explanation
20
Explain the changes that occurred in the tree and
animal. Use your current understanding of
evolution by natural selection.
(AAAS 1999)
21
Misconception individuals evolve new traits
n80
of Students
22
Misconception evolution is driven by need
n80
of Students
23
In guppy populations, what are the primary
changes that occur gradually over time?

1. The traits of each individual guppy within a
   population gradually change.
2. The proportions of guppies having different
   traits within a population change.
3. Successful behaviors learned by certain guppies
   are passed on to offspring.
4. Mutations occur to meet the needs of the guppies
   as the environment changes.

Anderson et al 2002
24
Posttest Student responses to mc
n171

of Students
25
Animal/Tree Posttest Gain in student
understanding of fitness
n80
of Students
26

• Quantitative Data
• Qualitative Data

Design Experiment
Bioscience 2003
27
Question
How do assessment questions help us determine
students prior understanding and progressive
thinking about the carbon cycle.
28
Instructional Design

• Two class meetings on carbon cycle (160 minutes)
• Active, inquiry-based learning
• Cooperative groups
• Questions, group processing, large lecture
  sections, small discussion sections, multi-week
  laboratory investigation
• Homework problems including web-based modules
• Different faculty for each course
• One graduate/8-10 undergraduate TAs per course

29
Experimental Design

• Two introductory courses for majors
• Bio 1 - organismal/population biology (faculty A)
• Bio 2 - cell and molecular biology (faculty B)
• Three cohorts
• Cohort 1 Bio 1 (n141)
• Cohort 2 Bio1/Bio2 (n63)
• Cohort 3 Other/Bio2 (n40)

30
Assessment Design

• Multiple iterations/versions of the carbon cycle
  problem
• Pretest, midterm, final with additional formative
  assessments during class
• Administered during instruction
• Semester 1 - pretest, midterm, final exam
• Semester 2 - final exam

31
Grandma Johnson Problem

• Hypothetical scenario Grandma Johnson had very
  sentimental feelings toward Johnson Canyon, Utah,
  where she and her late husband had honeymooned
  long ago. Her feelings toward this spot were
  such that upon her death she requested to be
  buried under a creosote bush overlooking the
  canyon. Trace the path of a carbon atom from
  Grandma Johnsons remains to where it could
  become part of a coyote. NOTE the coyote will
  not dig up Grandma Johnson and consume any of her
  remains.

32
Analysis of Responses

• Used same scoring rubric (coding scheme) for all
  three problems - calibrated by adding additional
  criteria when necessary, rescoring
• Examined two major concepts
• Concept 1 Decomposers respire CO2
• Concept 2 Plants uptake of CO2
• Explanations categorized into two groups
• Organisms (trophic levels)
• Processes (metabolic)

33
Coding Scheme
34
Cellular Respiration by Decomposers
Correct Student Responses ()
Bio1/Bio2
Other/Bio2
Freidmans, plt0.01
35
Pathway of Carbon in Photosynthesis
Correct Student Responses ()
Bio1/Bio2
Other/Bio2
Freidmans, plt0.05
36
Is Graduate Education Similar?

• Often excellent at preparing individuals to
  design and carry out disciplinary research.
• Often inadequate and haphazard in preparing
  future faculty/professionals to take on the
  increasingly complex demands of the
  professoriate.
• Teaching is not mentored, peer reviewed, or based
  on accumulated knowledge.

37
Instructional Research and Development teams
(IRDs)

• Who senior faculty, junior faculty, postdoctoral
  and graduate students - intergenerational teams.
• What scholarship of science teaching and
  learning is fully integrated into the
  professional culture along with discipline-based
  activities.
• Assessment is critical to both practices.

38
IRD Team at MSU

• Janet Batzli - Plant Biology U of Wisconsin
• Doug Luckie - Physiology
• Scott Harrison - Microbiology (grad student)
• Tammy Long - Plant Biology
• Jim Smith - Zoology
• Deb Linton - Plant Biology (postdoc)
• Heejun Lim - Chemistry Education
• Duncan Sibley - Geology
• National Science Foundation

39
What evidence will we accept?

• What is the question?
• What research and instructional designs?
• What data collection methods?
• How to analyze and interpret data?
• Are findings valid and generalizable?
• What are the next questions?
• WHO?