Teaching, Learning,

1
Teaching, Learning, Transfer of Experimental
Procedures in Elementary School Science

• David Klahr
• Department of Psychology
• Pittsburgh Science of Learning Center (PSLC)
• Program in Interdisciplinary Education Research
  (PIER)
• Carnegie Mellon University

Society for Research on Educational
Effectiveness First Annual Conference Dec 10 -
12, 2006
2
Topic Assessing different methods for teaching
experimental procedures to middle school
children

• In the lab
• In both easy challenging classrooms
• To students of widely varying abilities

More specifically Teaching CVS
3
What is (CVS)?

• CVS Control of Variables Strategy
• A simple procedure for designing unconfounded
  experiments
• - Vary one thing at a time (VOTAT).
• The conceptual basis for making valid inferences
  from data
• - isolation of causal path.

4
Why study CVS?
Practical importance
Topic Core topic in early science
instruction
Assessment State standards High stakes
assessments NCLB to start testing science
Best Instructional approach for teaching
CVS? Heated controversy in profession Legislativ
e battles (e.g., CA and hands on science)
Theoretical issues Surface vs deep mapping during
transfer of procedures and concepts at different
transfer distances.
5
Goal Compare different types of instruction for
teaching CVS.
Chen Klahr (1999), Child Dev.

• Participants 60 2nd - 4th graders
• Assessment
• Measure learning transfer at different
  distances from initial instruction.
• Materials 3 different physcial domains
• Springs
• Ramps
• Sinking objects.

Between subjects design
6
Springs domain
Which attributes determine how far a spring will
stretch?
Materials 8 springs 2 lengths x 2 widths x
2 wire sizes 2 pair of weights
Execution

• Select two springs
• Select two weights

• Hang springs on rack hooks
• Hang weights on springs.
• Compare amount of stretching.

7
Question does the length of a spring make a
difference in how far it stretches?
8
Two types of instruction (between subjects)

• Exploratory
• Explicit Exploratory plus

• Hands on work with physical materials
• Goal provided find out if x makes a
  difference

• Training Explicit, good and bad examples
• Training Reasons why, focus on deep structure
• Probe questions Can you tell for sure? Why?

9
Different transfer distances

• Near transfer (within domain)
• CVS tests in same domain as training, but on a
  different dimension.
• Time minutes after training
• Location, context, etc. same as training

• Far transfer (between domain)
• CVS tests in different domain from training.
• Time few days after training
• Location, context, etc., same as training

• Remote transfer (more later)

10
Study Phases
Day 1 Day 2
70
60
50
40
30
20
10
0
(Pre-test)
Training Manipulation
11
Explicit immediately better than Exploration and
remains so (4 experiments per child in each
phase)
70
60
50
of unconfounded experiments
40
30
20
10
0
12
CVS mastery by individual children
(at least 3 out of 4 unconfounded experiments)
of children becoming Masters
13
Extensions

• Initial transfer measures are very close to
  training objectives.
• Need a more distant ( authentic?) assessment
  of childrens understanding.
• 3. Will training effects remain with such
  extended assessments?

• Procedure
• Create a more authentic assessment
• Ask children to judge science fair posters.
• Score their comments and suggestions.

14
CVS Training and Science Fair Assessments (Klahr
Nigam, 2004)

• Participants 112 3rd 4th graders
• Train on CVS via Explicit or Exploration method.
• Assess effectiveness of CVS skill.
• Present poster evaluation task.
• Look at how CVS skill, training condition, affect
  poster evaluation performance.

15
Study Design
Day 1 1 week
70
60
50
40
30
20
10
0
Exploration
Near transfer
Far transfer
Training Manipulation
16
(No Transcript)
17
Scoring Rubric for Childrens Poster Critiques
1. Adequacy of research design
2. Theoretical explanation
3. Controlling for confounds in
Subjects/Materials, Treatment, Experimenter bias,
etc.
4. Measurement
Reliability/Variability, Error, Data
Representation
5. Statistical Inferences
Sample size/population, effect size
6. Completeness of conclusion
Supported by data, Relate to hypothesis
??
?
all valid, non-redundant, critiques about a
poster
Poster Score
Grand Poster Score (Pingpong Poster) (Memory
Poster)
18
Possible subtle effects of type of instruction

• Do the few kids who master CVS in the Exploratory
  condition do better on poster evaluation than the
  many who master CVS in the Explicit Instruction
  condition?

19
Possible subtle effects of type of instruction
Do the few kids who master CVS in the Exploratory
condition do better on poster evaluation than the
many who master CVS in the Explicit Instruction
condition?

• More specifically
• What is the relation between Poster Scores and
  Path to CVS mastery?
• Method
• Secondary analysis based on learning paths

20
Different paths to mastery or non-mastery of
CVS How do these children following these
different paths perform on poster evaluations?
Note following based on combining results from
two studies original KN plus a replication
21
n.s.
Poster Assessment Score (standardized)
n.s.

• CVS mastery is associated with high poster scores
• Non-mastery with low poster scores
• Path to mastery, or non-mastery is irrelevant

p lt .001
Explicit Masters
Experts
Exploratory Masters
Explicit non-Masters
Exploratory non- Masters
n 59
n 25
n 15
n 66
n 19
22
Question for cognitive researchWhy does
training on CVS (narrow) lead to better poster
evaluations (broad)?
Focused search for causal paths
Decomposition (attention to detail) Nature of
science Rhetorical stance Science as argument
Stay tuned .
23
Question for applied researchCan CVS be taught
in a normal classroom setting?
Procedure (in a nutshell)
Translate experiment script into teacher lesson
plan.
Teach in normal science classes (in high SES
schools). (Toth, Klahr, Chen, 2000)
24

• Participants in Classroom Study
• 77 4th graders from 4 classrooms in two different
  private schools
• 2 different science teachers
• Neither school had participated in lab studies

25
What to hold and what to fold?
These are issues of engineering design.
Keep Change adjust

• Pedagogy
• Goal teach CVS
• Type of teaching Explicit instruction
• Assessment
• Same as laboratory
• Plus, some new assessments in classroom

• Context
• Lesson plan, not script
• Teacher, not researcher
• Scheduling
• Student/teacher ratio
• Group work
• Record keeping
• Error and multiple trials

26
Results of Classroom Implementation
Individual students classified as Experts (8
of 9 correct)

• unconfounded designs

Posttest 91
Pretest 5
27
What about more challenging classrooms?(Lesson
Planning Project, w/Junlei Li, Stephanie Siler,
Mandy Jabbour)

• One facet of the Lesson Planning Project
• Two classrooms (5th and 6th graders) in urban
  school
• 90 eligible for free lunch.
• Teacher is researcher (Junlei Li)

28
Teaching Assessment of CVS with Urban 5th and
6th Graders (n 42) (Klahr Li, 2005)
Standardized Test Items
Our CVS Tests
100
2-Day Classroom Replication of CVS
Training Domain Ramps
2-Day CVS Transfer Retraining Domain Pendulum
2-Week Delay Transfer to real world,
high-stakes items
Dyads Student Design Mastery-based Formative
Assessment
80
Local
(CTBS)
60
Correct
National
(NAEP)
International
40
Dyads Focused Analogical Mapping
(TIMSS)
20
0
29
correct for various groups on a TIMMS CVS item
30
Typical TIMMS CVS item
He wants to test this idea The heavier a cart
is, the greater its speed at the bottom of a
ramp. Which three trials should he compare?
31
Significance

• Brief, theoretically grounded, focused
  instruction
• Is highly effective for middle class students
• In the sort run over longer durations
• On far transfer assessments
• Path independence
• What matters more than how.
• BIG differences in effectiveness with different
  student population. Thus, current approach
  requires
• Adaptation, Modification, Individualization

32
Questions to pursue(Next steps)

• NCLB in the small
• Goal No child who cant understand execute CVS
• Method Develop an intelligent tutor that can
  adapt to wide variability in childrens learning

33
Wide variety of individual learning
patterns(From Chen Klahr, 1999)
TYPE FAST GAIN UP DOWN
UP GRADUAL GAIN HIGH CONSTANT Explicit 31
10 7 7 Socratic 0
0 14 0
TYPE UP DOWN STEADY DECLINE
LOW CONSTANT Explicit 7
0 37 Socratic
18 7
60
34
Design a Tutor for Experimental Designw/ Mari
Strand Cary, Stephanie Siler, Junlei Li
35
Thanks to
Funding ources McDonnell Foundation, NICHD,
NSF, IES
Recent Current collaborators
Zhe Chen, Eva Toth, Junlei Li, Mari Strand Cary,
Stephanie Siler, Milena Nigam, Amy Masnick, Lara
Triona
36
END
37
Extras
38
Remote transfer items
A page from the 15-item test booklet
Why remote?
Does the amount of water affect plant growth?

• Temporal
• Training - test interval 7 months
• Domain
• Physical - biological, et al
• Format
• Physical materials vs. paper and pencil test
  booklet

• Context
• - One on one with Experimenter vs whole class
  test taking

39
Remote Transfer Results
Trained
100
Untrained
75
Does the amount of water affect plant growth?

• Mean correct on 15-item far transfer test

50
Good Test
25
Bad Test
0
3rd
4th
40
Ramps Domain Question Does the surface of a Ramp
make a difference in how far a ball rolls?