What evidence will we accept

1
Assessing the Impact of Integrating Scientific
Research and Education

• What evidence will we accept?

Diane Ebert-May Michigan State University
2
Bottom Left Corner
Please respond on a scale if 0-100 in increments
of 10

• How important is it to use multiple kinds of
  data to assess student learning?

3
How important is it to use multiple forms of data
to assess student learning?

Relative Importance
n127
4
Bottom Right Corner
Please respond on a scale of 0 - 100 in
increments of 10

• How often do you use data to make instructional
  decisions?

5
How often do you use data to make instructional
decisions?

Frequency
n127
6
True or False?

• Assessing student learning in science parallels
  what scientists do as researchers.

7
Parallel ask questions

• Description
• -What is happening?
• Cause
• -Does x (teaching strategy) affect y
  (understanding)?
• Process or mechanism
• -Why or how does x cause y?

8
Parallel collect data

• We collect data to find out what our students
  know.
• Data helps us understand student thinking about
  concepts and content.
• We use data to guide decisions about
  course/curriculum/innovative instruction

9
Parallel analyze data

• Quantitative data - statistical analysis
• Qualitative data
• break into manageable units and define coding
  categories
• search for patterns, quantify
• interpret and synthesize
• Valid and repeatable measures

10
Parallel peer review

• Ideas and results are peer reviewed - formally
  and/or informally.

11
Research Methods
12
Data collection approaches
13
Theoretical Framework Ausubel 1968 meaningful
learning Novak 1998 visual representations
King and Kitchner 1994 reflective judgment
National Research Council 1999 theoretical
frameworks for assessment
14
Goal

• Students will demonstrate their understanding of
  photosynthesis and cellular respiration.

15
Problem

• Experimental setup
• Weighed out 3 batches of radish seeds each
  weighing 1.5 g.
• Experimental treatments
• 1. Seeds placed on moistened paper towels in
  LIGHT
• 2. Seeds placed on moistened paper towels in DARK
• 3. Seeds not moistened (left DRY) placed in light

16
Problem (2)

• After 1 week, all plant material was dried in an
  oven overnight (no water left) and plant biomass
  was measured in grams.
• Predict the biomass of the plant material in the
  various treatments.
• Water, light
• Water, dark
• No water, light

17
Results Mass of Radish Seeds/Seedlings
Write an explanation about the results.
1.46 g 1.63 g
1.20 g
Explain the results. Write individually on
carbonless paper.
18
Assessment

• Exchange papers with another member of your group
  and evaluate their response.
• Use the Quiz/Homework rubric.
• Discuss the problem and revise your explanation
  if necessary.

19
(No Transcript)
20
What is assessment?

• Data collection with the purpose of answering
  questions about
• students understanding
• students attitudes
• students skills
• instructional design and implementation
• curricular reform (at multiple grainsizes)

21
Why do assessment?
Improve student learning and development.
Provides students substantive feedback about
their understanding.
Challenge to use disciplinary research strategies
to assess learning.
22
Pre-Posttest Analysis
Does active, inquiry-based instructional design
influence students understanding of evolution
and natural selection?
23
Data collection approaches
24
Alternative Conceptions Natural Selection
Changes in a population occur through a gradual
change in individual members of a population.
New traits in species are developed in response
to need. All members of a population are
genetically equivalent, variation and fitness are
not considered. Traits acquired during an
individuals lifetime will be inherited by
offspring.
25
Explain the changes that occurred in the tree and
animal. Use your current understanding of
evolution by natural selection.
(AAAS 1999)
26
Misconception individuals evolve new traits
n80
of Students
27
Misconception evolution is driven by need
n80
of Students
28
In guppy populations, what are the primary
changes that occur gradually over time?

• a. The traits of each individual guppy within a
  population gradually change. b. The proportions
  of guppies having different traits within a
  population change. c. Successful behaviors
  learned by certain guppies are passed on to
  offspring. d. Mutations occur to meet the needs
  of the guppies as the environment changes.

Anderson et al 2002
29
Posttest Student responses to mc
n171

of Students
30
Animal/Tree Posttest Gain in student
understanding of fitness
n80
of Students
31

• Quantitative Data
• Qualitative Data

Design Experiment
Bioscience 2003
32
Question
How do assessment questions help us determine
students prior understanding and progressive
thinking about the carbon cycle.
33
Instructional Design

• Two class meetings on carbon cycle (160 minutes)
• Active, inquiry-based learning
• Cooperative groups
• Questions, group processing, large lecture
  sections, small discussion sections, multi-week
  laboratory investigation
• Homework problems including web-based modules
• Different faculty for each course
• One graduate/8-10 undergraduate TAs per course

34
Experimental Design

• Two introductory courses for majors
• Bio 1 - organismal/population biology (faculty A)
• Bio 2 - cell and molecular biology (faculty B)
• Three cohorts
• Cohort 1 Bio 1 (n141)
• Cohort 2 Bio1/Bio2 (n63)
• Cohort 3 Other/Bio2 (n40)

35
Assessment Design

• Multiple iterations/versions of the carbon cycle
  problem
• Pretest, midterm, final with additional formative
  assessments during class
• Administered during instruction
• Semester 1 - pretest, midterm, final exam
• Semester 2 - final exam

36
Multiple choice question Pre-post

• The majority of actual weight (dry biomass)
  gained by plants as they progress from seed to
  adult plant comes from which one of the following
  substances?
• a. Particle substances in soil that are take up
  by plant roots. (15).
• b. Molecules in the air that enter through holes
  in the plant leaves (4).
• c. Substances dissolved in water taken up
  directly by plant root (28).
• d. Energy from the sun (29).
• N138

37
Grandma Johnson Problem

• Hypothetical scenario Grandma Johnson had very
  sentimental feelings toward Johnson Canyon, Utah,
  where she and her late husband had honeymooned
  long ago. Her feelings toward this spot were
  such that upon her death she requested to be
  buried under a creosote bush overlooking the
  canyon. Trace the path of a carbon atom from
  Grandma Johnsons remains to where it could
  become part of a coyote. NOTE the coyote will
  not dig up Grandma Johnson and consume any of her
  remains.

38
Analysis of Responses

• Used same scoring rubric (coding scheme) for all
  three problems - calibrated by adding additional
  criteria when necessary, rescoring
• Examined two major concepts
• Concept 1 Decomposers respire CO2
• Concept 2 Plants uptake of CO2
• Explanations categorized into two groups
• Organisms (trophic levels)
• Processes (metabolic)

39
Coding Scheme
40
Cellular Respiration by Decomposers
Correct Student Responses ()
Bio1/Bio2
Other/Bio2
Freidmans, plt0.01
41
Pathway of Carbon in Photosynthesis
Correct Student Responses ()
Bio1/Bio2
Other/Bio2
Freidmans, plt0.05
42
Is Graduate Education Similar?

• Often excellent at preparing individuals to
  design and carry out disciplinary research.
• Often inadequate and haphazard in preparing
  future faculty/professionals to take on the
  increasingly complex demands of the
  professoriate.
• Teaching is not mentored, peer reviewed, or based
  on accumulated knowledge.

43
Instructional Research and Development teams
(IRDs)

• Who senior faculty, junior faculty, postdoctoral
  and graduate students - intergenerational teams.
• What scholarship of science teaching and
  learning is fully integrated into the
  professional culture along with discipline-based
  activities.
• Assessment is critical to both practices.

44
IRD Team at MSU

• Janet Batzli - Plant Biology U of Wisconsin
• Doug Luckie - Physiology
• Scott Harrison - Microbiology (grad student)
• Tammy Long - Plant Biology
• Jim Smith - Zoology
• Deb Linton - Plant Biology (postdoc)
• Heejun Lim - Chemistry Education
• Duncan Sibley - Geology
• National Science Foundation-DUE

45
What evidence will we accept?

• What is the question?
• What research and instructional design?
• What data collection methods?
• How to analyze and interpret data?
• Are findings generalizable?
• What are the next questions?
• Who?