An iterative approach to designing scalable and adaptive computer-based science instruction

1
An iterative approach to designing scalable and
adaptive computer-based science instruction

• Mari Strand Cary, David Klahr,
• Stephanie Siler, Cressida Magaro,
• Kevin Willows, Junlei Li
• Carnegie Mellon University University of
  Pittsburgh

2
Goal of this EdBag?

• Discuss this iterative research process and its
  role/prominence in curriculum design, HCI, etc.
• Usefulness to researchers and designers (and,
  more importantly, end-users)
• How different is it from design research
• Difficulties weve encountered
• Other?

3
Overview of the TED project

• Curriculum Experimental design, evaluation, and
  interpretation
• Age 5th-8th grade students
• Schools 6 inner city
• 4 low SES challenging classroom environments
• 2 mid-high SES
• End goal Computer-based adaptive tutor
• 1 student 1 computer in classroom environment
• Provides individualized, adaptive instruction
• Supplements (does not replace!) teacher

4
Experimental design CVS (Control of Variables
Strategy)

• Simple procedure for designing unconfounded
  experiments
• (Vary one thing at a time)
• Conceptual basis for making valid inferences from
  data
• (Isolating the causal path)

5
Does SURFACE affect how far balls roll?
Variable Ramp 1 Ramp 2
Surface Smooth Rough
Track length Short Long
Height High Low
Ball Golf Rubber
Ramp 1 Ramp 2
Smooth Rough
Short Short
High High
Golf Golf
Confounded
Unconfounded
6
Whats the best way to teach CVS?

• As a society (educators, researchers, and
  legislators), we dont know
• Our research team knows of one effective way

7
Our basic CVS instruction (used in various forms
in past studies)

• Students design experiments
• Students answer questions
• Instructor provides explicit instruction about
  CVS
• One domain
• Short instructional period

8
But

• Our brief, focused CVS instruction is
  differentially efficient and effective for
  different student populations, settings, and
  transfer tasks.
• We want to reach ALL students!
• To improve our instruction for the entire student
  population, we must engage in modification
  individualization

9
A computer tutor could facilitate differentiated
instruction

• Instruction that suited for full-class setting
  could be provided by teacher
• Remaining instruction could be provided by
  computer tutor
• Individualized self-paced
• Provides instruction, practice, and feedback
• Teacher freed to provide coaching as needed

10
How are we building our tutor?

• 4 development phases
• Iterative design process

11
4 development phases

• Information gathering What are the novice models
  students hold and how can we address those?
• Refining the basic instruction and going
  virtual
• Building a computer tutor with a few paths
• Building an adaptive computer tutor with a web
  of paths

12
Our iterative design process
13

• An evolving CVS computer tutor

2b) Small groups (us!)
14

• What are we learning from each version that will
  help us design the final, adaptive tutor?

• VERSION 1 (Completed)
• Initial list of student biases, misconceptions,
  errors areas of difficulty
• Inventory of successful tutoring approaches
• familiar domains
• instruction in prerequisite skills
• step-by-step approach
• Student-friendly terminology, definitions, and
  phrasing
• Requiring explicit articulation by student

15
A sampling of what students do wrong

• Common errors
• Vary everything
• Hold target variable constant and vary other
  variables
• Partially confounded
• Nothing varied (identical)

• Common justifications
• I dont know
• You told me to test x!
• Describe their set-up
• Want to see if x happens
• Want to see if this setup is better than that
  setup

16
Why?

• By accident
• misread question
• working carelessly
• Are led astray
• by saliency of physical apparatus (e.g., ramps)
• dont understand written representations (e.g.,
  tables)
• On purpose
• different goals (e.g., engineering)
• misconception of experimental logic
• think other variable(s) dont matter
• Just guessing

17

• VERSION 2a (Complete) 2b (Fall 2007)
• Information regarding
• Addressing most common problems in full-class
  instruction using successful tutoring approaches
• Instructional effectiveness of switching domains
• effect of emphasizing domain-generality
• interface usability
• worksheet usability
• 2b Implementation of successful tutoring
  approaches with small groups (groups will differ
  in the paths they take)

18

• VERSION 3 (being developed)
• Information regarding
• division of instruction between teacher and
  tutor
• individual tutor usability and pitfalls
• comparative efficacy of set learning paths
• efficacy of immediate computer feedback

19
The adaptive tutor will include

• Pre-testing and ongoing monitoring of student
  knowledge
• Self-paced instruction
• Diverse topics matching students interests
• An interactive and engaging interface
• Teacher-controlled and/or computer-controlled
  levels of difficulty
• Level of scaffolding, feedback, and help aligned
  with students needs
• Computerized assessments
• Logging capability (level of output TBD)

20
Discuss!

• Discuss this iterative research process and its
  role/prominence in curriculum design, HCI, etc.
• Usefulness to researchers and designers (and,
  more importantly, end-users)
• How different is it from design research (and
  would it have gotten funded if we had labeled it
  that?)
• Difficulties weve encountered
• Other?

21
Questions? Comments?
Funding provided by Institute of Education
Sciences (IES _____)

• MariStrandCary_at_cmu.edu
• Klahr_at_cmu.edu

22
(No Transcript)
23

• V1 learning examples

• VERSION 1
• Database of student biases, misconceptions
  areas of difficulty
• Inventory of successful tutoring approaches
• familiar domains
• instruction in prerequisite skills
• step-by-step approach
• Student-friendly terminology, definitions, and
  phrasing
• Requiring explicit articulation of understanding
  and reasoning

Create best outcome or Most dramatic difference
Ignore the data or Biased by expectations
Pets, Sports drinks, Cars, Study habits, Running
races
Learn about all variables at once
Variable vs. Value Experiment Result vs.
Conclusion
Table format Remembering the target
variable Drawing conclusions based on the
experiment
Read carefully, Identify question, Identify
variables
Good vs. Fair vs. Informative vs. True Variable
something that can change
24
Every version
Stand-alone, detailed lesson plan with visual aids
Feedback
Assessments (formative and summative)
Asks students to explain, justify, and infer
Examples of exp. designs (good and bad)
Students designing experiments
25
Increasing complexity and adaptiveness

• Physical apparatus ? Virtual simulations
• Full class ? Full class individual computer
  use
• Inflexible ? Individually-adaptive self-paced
• One domain ? Multiple domains

26
What if later versions are less effective than
earlier versions?

• Stop the presses!
• Look for obvious reasons
• Examine lesson components individually
• Consider what is missing

27
Procedures

• Test one variable at a time
• Make the values for the variable youre testing
  be DIFFERENT across groups.
• Make the values for the variables youre not
  testing be the SAME across groups.

28
Concepts

• You need to use different values for the variable
  youre testing in order to know what effect those
  different values have.
• You need to use the same value for all the other
  variables (hold all the other variables constant
  control the other variables) so that they cant
  cause difference in the outcome.
• If you use CVS, you can know that only the
  variable youre testing is causing the
  outcome/result/effect.

29
Beyond our classroom instruction

• Where on the contextual / abstract continuum
  should this type of instruction be focused? When?
• Single vs. multiple domains?
• Static pictures vs. simulations vs. tabular
  representations
• Best mix of explicit instruction, exploration,
  help, feedback, etc.